Ruby Markdown Performance

Problem

My site builds are getting slow again.

I mean, sure, they don’t take over twenty minutes, but faster is faster, however you look at it.

Aim

Making my site builds (via Jekyll) faster by exploring Markdown engines.

Background

Unfortunately, there’s nothing like DTrace’s hotuser, which is incredibly awesome to use, and really does an excellent job of hunting down hot-points in a process.

Ruby has a benchmarking framework, but I don’t feel particularly like integrating it into Jekyll just to get a bunch of (plausibly meaningless) numbers out.

As a side note, it’s a pain to do any form of decent benchmarking on OS X with any degree of accuracy due to bugs like mds kicking in, destroying my test performance, twice. There were other processes that munched on CPU or I/O time, so there may be a few weird spikes on some of the graphs, but for the most part they all performed fairly well.

Method

I’m doing the benchmark without modifying the Jekyll code to allow the insertion of benchmarking hooks. I checked out Jekyll 41adc30667 from Git (master at the time of writing), then wrote a plugin that monkey-patched Jekyll to allow support for Pandoc-Ruby, BlueCloth and RPEG-Markdown.

I modified my LSI test repository for benchmarking thusly:

> mv _posts _posts_964
> mkdir _posts_{,1,2,3,4,5,6,7,8,9}00
> mkdir _posts_{,1,2,3,4,5,6,7,8,9}50
> rmdir _posts_00
> cd _posts_964 && ln ../_posts_964/* . && ls | tail -n 50 | xargs rm
> for nP in 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900; do (cd $nP && ln ../_posts_`expr ${nP} + 50`/* . && ls | tail -n 50 | xargs rm ); done
> for md in maruku kramdown rdiscount redcarpet pandoc bluecloth rpeg; do echo "markdown: ${md}" > _config_${md}.yml; done

As I write this, Jekyll is currently crunching through the dataset in a nice tight loop, testing each of Maruku, Kramdown, RDiscount, RedCarpet, Pandoc-Ruby, BlueCloth and RPEG-Markdown, to see which performs the best against a given set of posts:

> for md in maruku kramdown rdiscount redcarpet pandoc bluecloth rpeg; do for nP in 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 800 850 900 950 964; do mv _posts_${nP} _posts && time jekyll build --config _config_${md}.yml && mv _posts _posts_${nP; done; done

Results

Before I pass judgement, here are some graphs. First, post count against build durations. These are exponentials, although quite flat.

Next, post count against time per post. A very interesting minimum is noticeable around the 150-200 post mark, and times then increase roughly exponentially.

Finally, post count against posts per second. A matching maximum is observable around the 150-200 post mark, with all processors demonstrating a near halving in the posts per second from the maximum to the extreme of the graph.

In order, the best performers were:

BlueCloth, average 60.7 posts/s (stddev 17.860954)
RedCarpet, average 56.1 posts/s (stddev 16.510995)
RDiscount, average 54.9 posts/s (stddev 16.571145)
RPEG-Markdown, average 52.6 posts/s (stddev 11.800097)
Kramdown, average 40.1 posts/s (stddev 8.438867)
Maruku, average 34.1 post/s (stddev 6.584619)
Pandoc-Ruby, average 17.1 post/s (stddev 1.948292)

Analysis

The most obvious, and in my opinion most interesting, observation is that the best case to run Jekyll is against 150-200 Markdown posts; the performance clearly peaks about here, then falls off quite sharply, up to nearly 50% fewer posts per second in most engines at the maximum number of posts.

Strangely, BlueCloth and RDiscount (which both use the same C library, Discount, to do their processing) perform detectably differently. RedCarpet (which has its own C extension based on Sundown), does fairly well, too, as does the Ruby wrapper around John MacFarlane’s PEG-Markdown, another C library.

Of the pure-Ruby parsers, Kramdown performs the best, but it’s noticeably behind the pace. Maruku is even further back, and Pandoc-Ruby, which calls out through the system to John MacFarlane’s Pandoc (which is written in Haskell and can do everything), is about half as slow again.

The three groupings — native-accelerated, pure Ruby and shelling out — are all very obvious.

Conclusion

I don’t really know what to conclude from this. BlueCloth is fast in benchmarking, but in the real world? How does that compare to it’s feature-set?

I think, though, my performance problems are not associated with Markdown processing.

Jashank Jeremy