In February 2012, I attended a session at the TOC Conference given in part by Kristian Hammond, the CTO and co-founder of Narrative Science. During the session, Hammond mentioned that sports stories have a limited number of angles (e.g., a “blowout win” or a “come-from-behind victory”)—you can probably sit down and think up a fairly comprehensive list in short order. Even in fictional sports stories, writers only use around sixty common tropes as part of the narrative. Once you have an angle (or your algorithm has decided on one), you just slot in the relevant data, add a little color commentary, and you have your story.

At the time, I was struggling to understand how automated content could apply to Bible study; Levy’s article leads me to think that robosermons, or sermons automatically generated by a computer program, are the way of the future.

Parts of a Robosermon

After all, from a data perspective, sermons don’t differ much from sports stories. In particular, they have three components:

First, as with sports stories, sermons follow predictable structures and patterns. David Schmitt of Concordia Theological Seminary suggests a taxonomy of around thirty sermon structures. Even if this list isn’t comprehensive, it would probably take, at most, 100 to 200 structures to categorize nearly all sermons.

Second, sermons deal with predictable content: whereas sports have box scores, sermons have Bible texts and topics. A sermon will probably deal with a passage from the Bible in some way—the 31,000 verses in the Bible comprise a large but manageable set of source material (especially since most sermons involve a passage, not a single verse; you can probably cut this list down to around 2,000 sections). Topically, SermonCentral.com lists only 500 sermon topics in their database of 120,000 sermons. The power-law popularity distribution (i.e., the 80/20 rule) of verses preached on (on SermonCentral.com are 1,200 sermons on John 1 compared to seven on Numbers 35) and topics (1,400 sermons on “Jesus’ teachings” vs. four on “morning”) means that you can categorize most sermons using a small portion of the available possibilities.

Third, sermons generally involve illustrations or stories, much like the color commentary of sports stories. Finding raw material for illustrations shouldn’t present a problem to a computer program; a quick search on Amazon turns up 1,700 books on sermon illustrations and an additional 10,000 or so on general anecdotes. You can probably extract hundreds of thousands of illustrations from just these sources. Alternately, if a recent news story relates to your topic, the system can add the relevant parts to your sermon with little trouble (especially if a computer wrote the news story to begin with).

Application

You end up being able to say, “I want to preach a sermon on Philippians 2 that emphasizes Christ’s humility as a model for us.” Then—and here’s the part that doesn’t exist yet but that technology like Narrative Science’s will provide—an algorithm suggests, say, an amusing but poignant anecdote to start with, followed by three points of exegesis, exhortation, and application, and finishing with a trenchant conclusion. You tweak the content a bit, throwing in a shout-out to a behind-the-scenes parishioner who does a lot of work but rarely receives recognition, and call it done.

Why limit sermons to pastors, though? Why shouldn’t churchgoers be able to ask for custom sermons that fit exactly their circumstances? “I’d like a ten-part audio sermon series on Revelation from a dispensational perspective where each sermon exactly fits the length of my commute.” “Give me six weeks of premarital devotions for my boyfriend and me. I’ve always been a fan of Charles Spurgeon, so make it sound like he wrote them.”

Levy opens his Wired article with an anecdote about how grandparents would find articles about their grandchildren’s Little League games just as interesting as “anything on the sports pages.” He doesn’t mention that what they really want is a recap with their grandchild as the star (or at least as a strong supporting character—it’s like one of those children’s books where you customize the main character’s name and appearance). Robosermons let you tailor the sermon’s content so that your specific problems or questions form the central theme.

The logical end of this technology is a sermonbot that develops a following of eager listeners and readers, in the same way that an automated newspaper reporter would create fans on its way to winning a Pulitzer.

You may argue that robosermons diminish the role of the Holy Spirit in preparing sermons, or that they amount to plagiarism. I’m not inclined to disagree with you.

Conclusion

Building a robosermon system involves five components: (1) sermon structures; (2) Bible verses; (3) topics; (4) illustrations; and (5) technology like Narrative Science’s to put everything together coherently. It would also be helpful to have (6) a large set of existing sermons to serve as raw data. It’s a complicated problem but hardly an insurmountable one over the next ten years, should someone want to tackle it.

I’m not sure they should; that way lies robopologetics and robovangelism.

If you’re not an algorithm and you want to know how to prepare and deliver a sermon, I suggest listening to this 29-part course on preaching by Bryan Chapell at Biblical Training. It’s free and full of homiletic goodness.

Also see Jim LePage’s Illustrations of Every Bible Book.

This year, Twitter continues to take top honors. Facebook drops a few places compared to last year–has it become less-central to people’s lives? This year’s hot new site, Pinterest, almost makes the list, showing up at #118. (Incidentally, Pinterest has a number of Lent-related boards.)

Chocolate comes in at #2–however, if you add up all the mentions of chocolate in its various forms (“chocs,” “chocolate chips,” etc.), it totals over 14,000 mentions, enough to put it at #1.

This year’s biggest gainers are “breathing” and “makeup,” both of which jumped up more than 30 places in the list.

No celebrities make the top 100 this year. Boy band One Direction (aka #1D) is at #144, followed by Justin Bieber at #194 and Tim Tebow at #221. Last year’s curiosity, Charlie Sheen, only got two mentions; he dropped to #10,000 or so.

Overall, food was by far the most popular thing given up.

This list draws from about 300,000 tweets from February 19-25, 2012, and excludes retweets.

Categories

The image is a Wordle.

Look for the usual post-mortem later this week.

This visualization is arranged by book, showing cross-reference sources on the y-axis and targets on the x-axis. Within each square, the first verse in the book or section is at the top, and the last verse is at the bottom. Here’s what a detail of a square looks like:

Genesis 1 is at the top left; Genesis 50 is at the bottom left. Daniel 1 is at the top right; Daniel 12 is at the bottom right. The most-striking cross references between these two books, to me, involve Joseph’s interpretation of dreams in Genesis 40-41 and similar stories in Daniel.

Also see a previous cross reference visualization.

The Sabbath

Two parts of the article stand out. First:

For three decades, Talmud students have been able to use a Nexis-like CD search engine, the Responsa Project, created by Bar Ilan University in Israel…. Bar Ilan officials acknowledged that the CD had one major disadvantage: it cannot be accessed on the Sabbath, when much learning takes place. It also costs $790.

Of course it makes sense that you wouldn’t be able to use a digital study tool on the Sabbath; it had just never occurred to me. The evangelical analog might be having to use a print Bible in church instead of a mobile or projected version.

The Living Index

The second highlight from the article is:

Rabbi Benjamin Blech, professor of Talmud at Yeshiva University, said the rabbis believed that study should not be made too easy. “We want people to struggle with the text because by figuring it out you will have a deeper comprehension,” he said. “They wanted a living index, not a printed index.”

Bible software, websites, and apps are all working to create a “living index” (or at least a “responsive index”) of the Bible that lets you find comprehensive answers to every question that pops into your head while studying the Bible. But will this work devalue the actual experts (pastors and Bible teachers) who currently serve as living indexes?

The book The Lean Startup provides a framework for answering this question. It quotes Farbood Nivi, founder of test-preparation website Grockit:

“Whether you’re studying for the SAT or you’re studying for algebra, you study in one of three ways. You spend some time with experts, you spend some time on your own, and you spend some time with your peers. Grockit offers these three same formats of studying. What we do is we apply technology and algorithms to optimize those three forms.”

Or, in evangelical terms:

Bible software has historically augmented “time on your own” by tying together study materials: connecting documents to each other.

Recently, Bible software has expanded into “time with your peers” by mixing in the social layer that is enveloping the wider world of technology—“away from a web that connects documents together to a web that connects people together,” as Paul Adams puts it in his book Grouped. Bible software has three options when embedding social technologies: (1) Inject technology into existing offline practices (e.g., automating the irritating or the expensive); (2) Copy technology from secular sources (e.g., foursquare-style checkins); or (3) Come up with something new. The most likely outcome will involve a combination of these options.

Eventually, Bible software will delve into “time with experts,” as well, whether those experts are your local pastor or nationally recognized figures. Biblical Training is pioneering this approach in the field of Bible studies and theology, while Stanford and MIT are leading the way in other fields.

Will Bible software someday become an “expert” itself, giving you custom answers to your questions and a personal study plan? It’s certainly possible. Khan Academy is already disrupting math education and assessment. Someone will undoubtedly explore whether a similar approach works for Bible studies.

Of course, the open question is just how much we want Bible software to function as a living index; the rabbis who preferred that students “struggle” with text have a point that learning and wisdom come with effort. In a future where Bible software can provide time for you to study on your own, with peers, and with experts, I guess we’ll find out just how “easy” we want Bible study to become.

From the readme:

This is the fourth complete Bible reference parser that I’ve written. It’s how I try out new programming languages: the first one was in PHP (2002), which saw production usage on a Bible search website from 2002-2011; the second in Perl (2007), which saw production usage on a Bible-related site starting in 2007; and the third in Ruby (2009), which never saw production usage because it was way too slow. This Coffeescript parser (at least on V8) is faster than the Perl one and 100 times faster than the Ruby one.

I chose Coffeescript out of curiosity–does it make Javascript that much more pleasant to work with? From a programming perspective, the easy loops and array comprehensions alone practically justify its use. From a readability perspective, the code is easier to follow (and come back to months later) than the equivalent Javascript–the tests, in particular, are much easier to follow without all the Javascript punctuation.

My main interest in open-sourcing and thoroughly documenting this code lies in giving future programmers data and code that they can use to build better parsers. While this code reflects my experience, it’s hardly the last word on the subject.

Jim also runs Gettin’ Biblical, a site that showcases non-schlocky Christian-themed artwork. I particularly enjoyed The Savior collage and the papercut-esque Burning Bush. Good examples of “Christian art” (a difficult term to define if you’ve ever talked to artists who are Christians) are hard to come by, and I appreciate Jim’s efforts to collect them.

Full size download (.png, 4000×4000 pixels).

Things start off well with creation, turn negative with Job and the patriarchs, improve again with Moses, dip with the period of the judges, recover with David, and have a mixed record (especially negative when Samaria is around) during the monarchy. The exilic period isn’t as negative as you might expect, nor the return period as positive. In the New Testament, things start off fine with Jesus, then quickly turn negative as opposition to his message grows. The story of the early church, especially in the epistles, is largely positive.

Methodology

Sentiment analysis involves algorithmically determining if a piece of text is positive (“I like cheese”) or negative (“I hate cheese”). Think of it as Kurt Vonnegut’s story shapes backed by quantitative data.

I ran the Viralheat Sentiment API over several Bible translations to produce a composite sentiment average for each verse. Strictly speaking, the Viralheat API only returns a probability that the given text is positive or negative, not the intensity of the sentiment. For this purpose, however, probability works as a decent proxy for intensity.

The visualization takes a moving average of the data to provide a coherent story; the raw data is more jittery. Download the raw data (400 KB .zip).

Update October 10, 2011

As requested in the comments, here’s the data arranged by book with a moving average of five verses on either side. (By comparison, the above visualization uses a moving average of 150 verses on either side.)

Full size download (.png, 2680×4000 pixels).

Update December 28, 2011: Christianity Today includes this visualization in their December issue (“How the Bible Feels”).

Know What You’re Modeling

First you need to understand the shape of the data. I don’t have access to a large repository of Bible annotations, but the Twitter and Facebook Bible citations from the Realtime Bible Search section of this website provide a good approximation of how people cite the Bible. (Quite a few Facebook posts appear to involve people responding to their daily devotions.) These tweets and posts are public, and private annotations may take on a slightly different form, but the general shape of the data should be similar: nearly all (99%) refer to a chapter or less.

Compare Bible Gateway reading habits, which are much heavier on chapter-level usage, but 98% of accesses still involve a chapter or less.

The Numbers

The data consists of about 35 million total references.

About 92.9% of posts or tweets cited only one verse or verse range; 7.1% mentioned more than one verse range. Of the latter, 77% cited exactly two verse ranges; the highest had 323 independent verse ranges. Of Facebook posts, 9.1% contained multiple verse ranges, compared to 4.2% of tweets. When there were multiple ranges, 43% of the time they referred to verses in different books from the other ranges; 39% referred to verses in the same book (but not in the same chapter); and 18% referred to verses in the same chapter. (This distribution is a unusual—normally close verses stick together.)

The data, oddly, doesn’t contain any references that span multiple books. Less than 0.01% of passage accesses span multiple books on Bible Gateway, which is probably a useful upper bound for this type of data.

Key Points

1. Nearly all citations involve verses in the same chapter; only 1% involve verses in multiple chapters.
2. Of the 1% spanning two or more chapters, most refer to exact chapter boundaries.
3. Multiple-book references are even more unusual (under 0.01%) but have outsize effects: an annotation that references Genesis 1 to Revelation 22 would be relevant for every verse in the Bible.
4. Around 7% of notes contained multiple independent ranges of verses—the more text you allow for an annotation, the more likely someone is to mention multiple verses.

Download

Download the raw social data (1.4 MB zip) under the usual CC-Attribution license.

Data Modeling

A Bible annotation consists of arbitrary content (a highlight might have one kind of content, while a proper note might have a title, body, attachments, etc., but modeling the content itself isn’t the point of this piece) tied to one or more Bible references:

1. A single verse (John 3:16).
2. A single range (John 3:16-17).
3. Multiple verses or ranges (John 3:16, John 3:18-19)

The Relational Model

One user can have many rows of annotations, and one annotation can have many rows of verses that it refers to. To model a Bible annotation relationally, we set up three tables that look something like this:

users

annotations

annotation_verses

The verse references here are integers to allow for easy range searches: 43 = John (the 43rd book in the typical Protestant Bible); 003 = the third chapter; the last three digits = the verse number.

I like using this approach over others (sequential integer or separate columns for book, chapter, and verse) because it limits the need for a lookup table. (You just need to know that 43 = John, and then you can find any verse or range of verses in that book.) It also lets you find all the annotations for a particular chapter without having to know how many verses are in the chapter. (The longest chapter in the Bible has 176 verses, so you know that all the verses in John 3, for example, fall between 43003001 and 43003176.) This main disadvantage is that you don’t necessarily know how many verses you’re selecting until after you’ve selected them. And using individual columns, unlike here, does allow you to run group by queries to get easy counts.

Querying

In a Bible application, the usual mode of accessing annotations is by passage: if you’re looking at John 3:16-18, you want to see all your annotations that apply to that passage.

Querying MySQL

In SQL terms:

select distinct(annotations.annotation_id)
from annotations, annotation_verses
where annotation_verses.start_verse <= 43003018 and
annotation_verses.end_verse >= 43003016 and
annotations.user_id = 1 and
annotations.annotation_id = annotation_verses.annotation_id
order by annotation_verses.start_verse asc, annotation_verses.end_verse desc

The quirkiest part of the SQL is the first part of the “where” clause, which at first glance looks backward: why is the last verse in the start_verse field and the first verse in the end_verse field? Because the start_verse and end_verse can span any range of verses, you need to make sure that you get any range that overlaps the verses you’re looking for: in other words, the start_verse is before the end of the range, and the end_verse is after the start.

Visually, you can think of each start_verse and end_verse pair as a line: if the line overlaps the shaded area you’re looking for, then it’s a relevant annotation. If not, it’s not relevant. There are six cases:

The other trick in the SQL is the sort order: you generally want to see annotations in canonical order, starting with the longest range first. In other words, you start with an annotation about John 3, then to a section inside John 3, then to individual verses. In this way, you move from the broadest annotations to the narrowest annotations. You may want to switch up this order, but it makes a good default.

The relational approach works pretty well. If you worry about the performance implications of the SQL join, you can always put the user_id in annotation_verses or use a view or something.

Querying CouchDB

CouchDB is one of the oldest entrants in the NoSQL space and distinguishes itself by being both a key-value store and queryable using map-reduce: the usual way to access more than one document in a single query is to write Javascript to output the data you want. It lets you create complex keys to query by, so you might think that you can generate a key like [start_verse,end_verse] and query it like this: ?startkey=[0,43003016]&endkey=[43003018,99999999]

But no. Views are one-dimensional, meaning that CouchDB doesn’t even look at the second element in the key if the first one matches the query. For example, an annotation with both a start and end verse of 19001001 matches the above query, which isn’t useful for this purpose.

I can think of two ways to get around this limitation, both of which have drawbacks.

GeoCouch

CouchDB has a plugin called GeoCouch that lets you query geographic data, which actually maps well to this data model. (I didn’t come up with this approach on my own: see Efficient Time-based Range Queries in CouchDB using GeoCouch for the background.)

The basic idea is to treat each start_verse,end_verse pair as a point on a two-dimensional grid. Here’s the above social data plotted this way:

The line bisects the grid diagonally since an end_verse never precedes a start_verse: the diagonal line where start_verse = end_verse indicates the lower bound of any reference. Here are some points indicating where ranges fall on the plot:

To find all the annotations relevant to John 3:16-18, we draw a region starting in the upper left and continuing to the point 43003018,43003016:

GeoCouch allows exactly this kind of bounding-box query: ?bbox=0,43003016,43003018,99999999

You can even support multiple users in this scheme: just give everyone their own, independent box. I might occupy 1×1 (with an annotation at 1.43003016,1.43003016), while you might occupy 2×2 (with an annotation at 2.43003016,2.43003016); queries for our annotations would never overlap. Each whole number to the left of the decimal acts as a namespace.

The drawbacks:

1. The results aren’t sorted in a useful way. You’ll need to do sorting on the client side or in a show function.
2. You don’t get pagination.

Repetition at Intervals

Given the shape of the data, which is overwhelmingly chapter-bound (and lookups, which at least on Bible Gateway are chapter-based), you could simply repeat chapter-spanning annotations at the beginning of every chapter. In the worst case annotation (Genesis 1-Revelation 22), you end up with about 1200 repetitions.

For example, in the Genesis-Revelation case, for John 3 you might create a key like [43000000.01001001,66022021] so that it sorts at the beginning of the chapter—and if you have multiple annotations with different start verses, they stay sorted properly.

To get annotations for John 3:16-18, you’d query for ?startkey=[43003000]&endkey=[43003018,{}]

The drawbacks:

1. You have to filter out all the irrelevant annotations: if you have a lot of annotations about John 3:14, you have to skip through them all before you get to the ones about John 3:16.
2. You have to filter out duplicates when the range you’re querying for spans multiple chapters.
3. You’re repeating yourself, though given how rarely a multi-chapter span (let alone a multi-book span) happens in the wild, it might not matter that much.

Other CouchDB Approaches

Both these approaches assume that you want to make only one query to retrieve the data. If you’re willing to make multiple queries, you could create different list functions and query them in parallel: for example, you could have one for single-chapter annotations and one for multi-chapter annotations. See interval trees and geohashes for additional ideas. You could also introduce a separate query layer, such as elasticsearch, to sit on top of CouchDB.