Know What You’re Modeling

First you need to understand the shape of the data. I don’t have access to a large repository of Bible annotations, but the Twitter and Facebook Bible citations from the Realtime Bible Search section of this website provide a good approximation of how people cite the Bible. (Quite a few Facebook posts appear to involve people responding to their daily devotions.) These tweets and posts are public, and private annotations may take on a slightly different form, but the general shape of the data should be similar: nearly all (99%) refer to a chapter or less.

Compare Bible Gateway reading habits, which are much heavier on chapter-level usage, but 98% of accesses still involve a chapter or less.

The Numbers

The data consists of about 35 million total references.

About 92.9% of posts or tweets cited only one verse or verse range; 7.1% mentioned more than one verse range. Of the latter, 77% cited exactly two verse ranges; the highest had 323 independent verse ranges. Of Facebook posts, 9.1% contained multiple verse ranges, compared to 4.2% of tweets. When there were multiple ranges, 43% of the time they referred to verses in different books from the other ranges; 39% referred to verses in the same book (but not in the same chapter); and 18% referred to verses in the same chapter. (This distribution is a unusual—normally close verses stick together.)

The data, oddly, doesn’t contain any references that span multiple books. Less than 0.01% of passage accesses span multiple books on Bible Gateway, which is probably a useful upper bound for this type of data.

Key Points

1. Nearly all citations involve verses in the same chapter; only 1% involve verses in multiple chapters.
2. Of the 1% spanning two or more chapters, most refer to exact chapter boundaries.
3. Multiple-book references are even more unusual (under 0.01%) but have outsize effects: an annotation that references Genesis 1 to Revelation 22 would be relevant for every verse in the Bible.
4. Around 7% of notes contained multiple independent ranges of verses—the more text you allow for an annotation, the more likely someone is to mention multiple verses.

Download

Download the raw social data (1.4 MB zip) under the usual CC-Attribution license.

Data Modeling

A Bible annotation consists of arbitrary content (a highlight might have one kind of content, while a proper note might have a title, body, attachments, etc., but modeling the content itself isn’t the point of this piece) tied to one or more Bible references:

1. A single verse (John 3:16).
2. A single range (John 3:16-17).
3. Multiple verses or ranges (John 3:16, John 3:18-19)

The Relational Model

One user can have many rows of annotations, and one annotation can have many rows of verses that it refers to. To model a Bible annotation relationally, we set up three tables that look something like this:

users

annotations

annotation_verses

The verse references here are integers to allow for easy range searches: 43 = John (the 43rd book in the typical Protestant Bible); 003 = the third chapter; the last three digits = the verse number.

I like using this approach over others (sequential integer or separate columns for book, chapter, and verse) because it limits the need for a lookup table. (You just need to know that 43 = John, and then you can find any verse or range of verses in that book.) It also lets you find all the annotations for a particular chapter without having to know how many verses are in the chapter. (The longest chapter in the Bible has 176 verses, so you know that all the verses in John 3, for example, fall between 43003001 and 43003176.) This main disadvantage is that you don’t necessarily know how many verses you’re selecting until after you’ve selected them. And using individual columns, unlike here, does allow you to run group by queries to get easy counts.

Querying

In a Bible application, the usual mode of accessing annotations is by passage: if you’re looking at John 3:16-18, you want to see all your annotations that apply to that passage.

Querying MySQL

In SQL terms:

select distinct(annotations.annotation_id)
from annotations, annotation_verses
where annotation_verses.start_verse <= 43003018 and
annotation_verses.end_verse >= 43003016 and
annotations.user_id = 1 and
annotations.annotation_id = annotation_verses.annotation_id
order by annotation_verses.start_verse asc, annotation_verses.end_verse desc

The quirkiest part of the SQL is the first part of the “where” clause, which at first glance looks backward: why is the last verse in the start_verse field and the first verse in the end_verse field? Because the start_verse and end_verse can span any range of verses, you need to make sure that you get any range that overlaps the verses you’re looking for: in other words, the start_verse is before the end of the range, and the end_verse is after the start.

Visually, you can think of each start_verse and end_verse pair as a line: if the line overlaps the shaded area you’re looking for, then it’s a relevant annotation. If not, it’s not relevant. There are six cases:

The other trick in the SQL is the sort order: you generally want to see annotations in canonical order, starting with the longest range first. In other words, you start with an annotation about John 3, then to a section inside John 3, then to individual verses. In this way, you move from the broadest annotations to the narrowest annotations. You may want to switch up this order, but it makes a good default.

The relational approach works pretty well. If you worry about the performance implications of the SQL join, you can always put the user_id in annotation_verses or use a view or something.

Querying CouchDB

CouchDB is one of the oldest entrants in the NoSQL space and distinguishes itself by being both a key-value store and queryable using map-reduce: the usual way to access more than one document in a single query is to write Javascript to output the data you want. It lets you create complex keys to query by, so you might think that you can generate a key like [start_verse,end_verse] and query it like this: ?startkey=[0,43003016]&endkey=[43003018,99999999]

But no. Views are one-dimensional, meaning that CouchDB doesn’t even look at the second element in the key if the first one matches the query. For example, an annotation with both a start and end verse of 19001001 matches the above query, which isn’t useful for this purpose.

I can think of two ways to get around this limitation, both of which have drawbacks.

GeoCouch

CouchDB has a plugin called GeoCouch that lets you query geographic data, which actually maps well to this data model. (I didn’t come up with this approach on my own: see Efficient Time-based Range Queries in CouchDB using GeoCouch for the background.)

The basic idea is to treat each start_verse,end_verse pair as a point on a two-dimensional grid. Here’s the above social data plotted this way:

The line bisects the grid diagonally since an end_verse never precedes a start_verse: the diagonal line where start_verse = end_verse indicates the lower bound of any reference. Here are some points indicating where ranges fall on the plot:

To find all the annotations relevant to John 3:16-18, we draw a region starting in the upper left and continuing to the point 43003018,43003016:

GeoCouch allows exactly this kind of bounding-box query: ?bbox=0,43003016,43003018,99999999

You can even support multiple users in this scheme: just give everyone their own, independent box. I might occupy 1×1 (with an annotation at 1.43003016,1.43003016), while you might occupy 2×2 (with an annotation at 2.43003016,2.43003016); queries for our annotations would never overlap. Each whole number to the left of the decimal acts as a namespace.

The drawbacks:

1. The results aren’t sorted in a useful way. You’ll need to do sorting on the client side or in a show function.
2. You don’t get pagination.

Repetition at Intervals

Given the shape of the data, which is overwhelmingly chapter-bound (and lookups, which at least on Bible Gateway are chapter-based), you could simply repeat chapter-spanning annotations at the beginning of every chapter. In the worst case annotation (Genesis 1-Revelation 22), you end up with about 1200 repetitions.

For example, in the Genesis-Revelation case, for John 3 you might create a key like [43000000.01001001,66022021] so that it sorts at the beginning of the chapter—and if you have multiple annotations with different start verses, they stay sorted properly.

To get annotations for John 3:16-18, you’d query for ?startkey=[43003000]&endkey=[43003018,{}]

The drawbacks:

1. You have to filter out all the irrelevant annotations: if you have a lot of annotations about John 3:14, you have to skip through them all before you get to the ones about John 3:16.
2. You have to filter out duplicates when the range you’re querying for spans multiple chapters.
3. You’re repeating yourself, though given how rarely a multi-chapter span (let alone a multi-book span) happens in the wild, it might not matter that much.

Other CouchDB Approaches

Both these approaches assume that you want to make only one query to retrieve the data. If you’re willing to make multiple queries, you could create different list functions and query them in parallel: for example, you could have one for single-chapter annotations and one for multi-chapter annotations. See interval trees and geohashes for additional ideas. You could also introduce a separate query layer, such as elasticsearch, to sit on top of CouchDB.

Congratulations, I guess, go this year to Charlie Sheen, who came in at both #23 and, with “tiger blood,” at #90. Justin Bieber is up several spots this year, so he hasn’t quite crested yet. The next-highest celebrity, who didn’t make the top 100, is British boy band One Direction.

“Trophies,” at #69, refers to the English soccer club Arsenal‘s recent defeat, or something.

The later start to Lent this year means that “snow” doesn’t appear on the list–last year, it was #48. Myspace hangs on at #99, dropping 48 places.

This list draws from 85,000 tweets from March 7-10, 2011, and excludes retweets.

Image created using Wordle.

Also: PowerPoint, PDF.

I distributed the following handout at the presentation, showing the popularity of Bible chapters and verses cited on Twitter. It displays a lot of data: darker chapters are more popular, the number in the middle of each box is the most popular verse in the chapter, and sparklines in each box show the distribution of the popularity in each chapter. (Genesis 1:1 is by far the most popular verse in Genesis 1, while Genesis 3:15 is only a little more popular than other verses in the chapter.)

Categories of Things Given up by Location

As I only track in English what people are giving up, there are concentrations in English-speaking countries.

Size indicates the relative number of Twitterers in each country giving up something for Lent.

Categories of Things Given up by State

These visualizations show the differences (or lack thereof) in what people are giving up among U.S. states.

Size indicates the relative number of Twitterers in each state giving up something for Lent. Sorry, Alaska and Hawaii.

The composition of each state’s categories of tweets shows mostly minor variations among states. Some states (like Wyoming on the far right) have small numbers of tweets. I would have liked to use opacity or width to indicate this disparity but couldn’t figure out how to do it.

Comparison between 2009 and 2010

This treemap shows how the data changed between 2009 and 2010. The size of the box shows the number of people giving up each category and thing, while color indicates the percentage change from last year: dark blue indicates the steepest drop; dark orange indicates the steepest rise. The second chart shows the same data more conventionally expressed.

About the Visualizations

I created these charts mostly to explore how the new data-analysis software Tableau Public works. One of its claims to fame is that you can publish interactive visualizations to the web, a feature I didn’t take advantage of here. Tableau doesn’t do treemaps, so I used Many Eyes to create the treemap; the closest Tableau equivalent appears below the treemap.

Snow makes the list this year, understandable given the Snowpocalypse and Snowmageddon that gripped much of the Eastern U.S. in the weeks preceding Ash Wednesday. IPods also made the list after the Bishop of Liverpool asked people to consider praying instead of listening to them. This year a celebrity, Justin Bieber, cracks the top 100. He beat out the Jonas Brothers, 64 votes to 11; draw your own conclusions.

The list largely tracks last year’s list. It draws from 40,000 tweets retrieved February 14-20, 2010.

Complete List of the Top 100

Image created using Wordle.

This project uses several APIs from Twitter and is still in a beta stage. It could evolve in several directions, but I want to see how people use it before developing it further.

It’s not quite realtime, but the most recent tweet is rarely more than a few minutes old.

Behind the scenes, it processes tweets to try to ensure their relevance; it has about a 92% accuracy rate based on a training corpus of around 45,000 tweets. Use the “relevant” and “not relevant” buttons in the interface if you see a tweet that you think should or shouldn’t belong. (I’m mostly interested in the latter, but it seems weird not to have both–like Facebook’s lack of an unlike button.)

It currently uses Logos RefTagger to link the Bible references in the tweets.

Feel free to leave a comment here if you have a feature idea or want to make any suggestions.

1. bible
2. lord
3. christ
4. gospel
5. psalm
6. god
7. psalms
8. corinthians
9. preach
10. shall
11. heaven
12. readings
13. church
14. spirit
15. righteous
16. verse
17. lectionary
18. verses
19. spiritual
20. ministry
21. pray
22. enemies
23. thou
24. tongue
25. creation
26. wisdom
27. deuteronomy
28. testament
29. strength
30. refuge
31. therefore
32. kingdom
33. romans
34. holy
35. thankful
36. thy
37. reading
38. rejoice
39. understanding
40. faithful
41. message
42. earth
43. blessed
44. exodus
45. deut
46. faith
47. wise
48. beginning
49. pastor
50. chapel
51. chapter
52. survey
53. anger
54. resurrection
55. risen
56. read
57. hearts
58. chronicles
59. salvation
60. flesh
61. servant
62. glory
63. praying
64. kings
65. sheep
66. praise
67. trust
68. prosperity
69. bless
70. heavens
71. deeds
72. toward
73. discussion
74. whoever
75. speaks
76. ye
77. hath
78. amen
79. teaching
80. thess
81. apostles
82. preparing
83. eph
84. eccl
85. path
86. fear
87. upon
88. presence
89. inspire
90. search
91. zechariah
92. seek
93. teach
94. wrath
95. commandments
96. believers
97. humility
98. spoke
99. thee
100. devo

Background

Extracting Bible references from text means identifying whether a given piece of text is referring to a Bible verse or something else. For example, the meaning of Acts 2 depends on context:

• Referring to Bible passage: Acts 2 recounts the early church.
• Not referring to Bible passage: She’s 5 years old but acts 2.

When you encounter a phrase that could be a Bible reference, you have to look at context to determine whether the phrase is a Bible reference. Humans can make this leap pretty easily, but computers need rigorous models and lots of training data to guess whether an ambiguous phrase is a Bible reference. In the above example, the phrase “early church” is a strong indicator that the phrase “Acts 2″ is a Bible reference, while the phrase “years old” is an indicator the other way.

Twitter, with its high volume of content and decent search engine, provides lots of training data.

Methodology

Using the Twitter Search API, I downloaded 30,000 tweets possibly containing Bible references (e.g., [john 3], [jeremiah 29]) and then categorized them by hand as referring to a Bible verse or not.

I then ran a Naive Bayes algorithm on the resulting tweets to produce the above list, which contains the words that most strongly indicate the presence of a Bible reference.

This list suffers from sample bias, of course: a different set of tweets would produce a different list. In addition, the list is Twitter-centric; the results may not carry over into blogs or other media. (People substitute the number “2” for the word “to” and “4” for “for” on Twitter more frequently than they do elsewhere, for example, which oversamples content like “I’m meeting Matthew 4 dinner.”)

See It in Action

Search for Bible references on Twitter. Use the relevant and not relevant buttons to improve the filtering. I haven’t formally announced this new feature of OpenBible.info yet; consider the link a preview.

Some you’d expect (alcohol, chocolate), some are ironic (giving up Lent for Lent, giving up giving up things), some are odd (pants, lint), some are anti-religious (religion, Catholicism), and some are tech-related (Facebook, Twitter—even “Facebook and Twitter” makes the list).

Complete List

1. Facebook (654)
2. Twitter (317)
3. Chocolate (272)
4. Lent (216)
5. Alcohol (187)
6. Soda (139)
7. Coffee (129)
8. Meat (126)
9. Religion (102)
10. Swearing (94)
11. Sweets (92)
12. Catholicism (90)
13. Giving up things (80)
14. Work (70)
15. Beer (60)
16. Sex (59)
17. Fast food (57)
18. Facebook and twitter (57)
19. Sugar (45)
20. Stuff (43)
21. Booze (41)
22. Smoking (39)
23. Food (39)
24. Procrastination (38)
25. Internet (37)
26. Cursing (36)
27. Caffeine (35)
28. TV (33)
29. Pancakes (33)
30. Social networking (33)
31. Sleep (32)
32. Candy (32)
33. Diet Coke (29)
34. Giving up (29)
35. You (28)
36. Wine (28)
37. Lint (28)
38. Cheese (28)
39. Bread (26)
40. Shopping (26)
41. Sobriety (26)
42. Abstinence (24)
43. Cussing (24)
44. Red meat (24)
45. Chips (23)
46. Internet porn (22)
47. Christianity (22)
48. Nothing (21)
49. French fries (21)
50. Jesus (21)
51. Sarcasm (19)
52. Junk food (19)
53. Starbucks (18)
54. Ice cream (18)
55. MySpace (18)
56. Cookies (18)
57. Fried food (17)
58. Complaining (17)
59. God (16)
60. New years resolutions (15)
61. Social media (15)
62. Pizza (14)
63. Tweeting (14)
64. Carbs (13)
65. MySpace and Facebook (13)
66. Carbon (13)
67. Eating out (13)
68. Stress (13)
69. Flaky guys (12)
70. Laziness (12)
71. Texting (12)
72. Me (11)
73. Some of your money (11)
74. Annoying me (11)
75. Sacrifice (11)
76. School (11)
77. Hope (10)
78. Rice (10)
79. Coke (10)
80. Porn (10)
81. The snooze button (10)
82. Guilt (10)
83. Men (9)
84. Obama (9)
85. Church (9)
86. My job (9)
87. Homework (9)
88. Self denial (9)
89. Moderation (9)
90. Exercise (8)
91. Bacon (8)
92. Dieting (8)
93. Paying taxes (8)
94. Dr Pepper (8)
95. Gossip (8)
96. Beef (8)
97. Pants (7)
98. My sanity (7)
99. Celibacy (7)
100. Shaving (7)

About

Created using the Twitter Search API and Wordle. Data based on analysis of 15,000 tweets from February 22-26, 2009.