Technology1 min read

Open source as a research method

By · Published by Everything Blog

In short

Practical performance bottlenecks drove the 2019 creation of the simdjson project, which reached throughput rates of 4.8 GB/s on an Intel Xeon processor. This effort spurred further innovations, including the Eisel-Lemire algorithm adopted across systems like Go and Rust, as well as the simdutf library for web environments. Modern artificial intelligence code generators are actively reinforcing this model by helping contributors explore unfamiliar repositories and deliver fixes.

Key points

  • Direct engagement with open-source communities supplies bug reports and hardware testing…: Direct engagement with open-source communities supplies bug reports and hardware testing that can exceed traditional peer review.
  • Rather than displacing contributors, artificial intelligence models are actively assistin…: Rather than displacing contributors, artificial intelligence models are actively assisting developers in diagnosing issues and submitting pull requests.
  • Multiple academic institutions including McGill University and Polytechnique Montréal are…: Multiple academic institutions including McGill University and Polytechnique Montréal are supporting SERIQ to advance software engineering across Quebec.

Sources checked by our analysis pipeline: 1

4 November 2026 · 4 min

Many people still picture innovation as a line. The professor thinks and publishes. The engineer reads the paper and designs a product. The customer consumes it.

It rarely works that way. The industrial revolution was not set off by professors writing papers. It was driven by demand for cheap and better clothing for the masses. Demand came first.

Michael Stonebraker made a related point in “Where We Have Failed” (2018). He argued that the database research community has lost its customer: “there are no ‘real world’ clients to keep us focused.” Without a customer, research drifts.

In 2019, JSON was everywhere, and parsing it was slow, at a few hundred megabytes per second. So we built simdjson. Today it parses JSON at gigabytes per second; I measured about 4.8 GB/s on twitter.json on a recent Intel Xeon.

We did not write the paper first. We wrote the code, then a blog post. We put it on GitHub and posted about it on social networks. Industry adopted it. Our first paper was published by the…

Technology