A benchmark paper opened before implementation

Turning a paper into code is never just “implement the paper.” At least, it should not be. A paper is full of equations, tables, assumptions, conventions, and shortcuts that made sense to the authors, but not necessarily to the next person trying to reproduce one part of it.

This is one place where AI agents are genuinely useful. Not because they magically understand the science, but because they are patient with the annoying first layer of work: reading PDFs, extracting equations, doing OCR when the text layer is bad, converting sections to Markdown, listing symbols, and comparing a paper against an existing code base.

The example I have in mind is my work on PenguinReactiveTransferClimentBenchmarks, a Julia benchmark package around scalar and coupled reactive transfer in the PenguinxCutCell ecosystem. The goal was not to make an impressive demo. The goal was more modest and more useful: take benchmark material from the literature, translate it into small reproducible cases, and make the result auditable.

The first pass is reading

The first useful task for the agent is not coding. It is reading the paper in a way that prepares coding.

I want it to extract:

  • the equations;
  • the definitions of symbols;
  • the nondimensional numbers;
  • the boundary and interface conditions;
  • the reference tables and figures;
  • the assumptions that are easy to miss;
  • the cases that are actually comparable with my solver.

That last point matters. A table from a finite-Reynolds DNS paper is not automatically a validation target for a low-Re prescribed-velocity solver. It can be useful context, but it should not become an acceptance criterion if the physics solved by the code is different.

This is where PDF handling matters. Sometimes the equation text is selectable. Sometimes it is not. Sometimes a table needs OCR. Sometimes the cleanest path is to ask the agent to convert a few pages into Markdown, then manually check the equations before going further. That manual check is not optional.

Extracting equations and benchmark definitions from the paper

Then comes the implementation map

Once the paper is readable, I want a map from the paper to the repository. In the Climent benchmark package, this means identifying whether a case belongs near an existing scalar spherical benchmark, a two-phase conjugate transfer problem, a prescribed Stokes-flow diagnostic, or a multi-species reaction block.

The repository already has a useful shape:

  • benchmark scripts such as scripts/run_benchmark.jl;
  • source modules in src/;
  • tests in test/;
  • reference data in data/reference and data/references;
  • generated result CSVs in data/results;
  • plotting and documentation scripts.

The agent should read that structure before proposing code. If a new case resembles an existing one, the right answer is usually to reuse the existing post-processing conventions and add a small benchmark path. The wrong answer is to invent a separate framework because the prompt only mentioned the paper.

Ask for comparisons, not just code

The prompts that work best are not “write the implementation.” They are comparative:

Compare the equations in this paper with the existing B1 and B5 benchmark code.
Which terms are already represented?
Which assumptions differ?
Which reference quantities can be tested on the current grid?
Which rows should be marked diagnostic rather than valid?

This forces the agent to reason across three things at once: the paper, the current code, and the validation layer. It also gives me something to review before any file changes happen. If the mapping is wrong, the code will be wrong too.

Build a ladder

A good implementation should climb a ladder:

  1. formula checks;
  2. geometry and post-processing checks;
  3. scalar smoke benchmarks;
  4. local convergence or resolution studies;
  5. coupled or multi-species benchmarks;
  6. paper-level campaigns and plots.

This ladder is important because it prevents overclaiming. A smoke benchmark only says that the case is wired correctly. A local run gives more confidence. A paper-level campaign is slower and should be reserved for results that are meant to support a figure, a table, or a claim.

In the benchmark repository, this shows up through result levels, usable flags, under-resolution flags, geometry checks, and stored reference tables. Those details may look bureaucratic, but they are what make the output honest.

Keep the final result reviewable

For scientific code, “it runs” is not enough. At the end of an agent session, I want a plain report:

  • changed files;
  • commands run;
  • numerical results;
  • uncertainty or limitations.

If the agent added a benchmark, I want to know which CSV row changed, which figure can be regenerated, which test passed, and what still should not be claimed. The final answer should not sound like a press release. It should sound like a lab notebook.

The useful unit

The productive unit is not “paper to code.” It is smaller:

one equation -> one implementation path -> one testable quantity -> one result row

Once that path is clear, adding more benchmarks becomes less mysterious. The paper provides a case. The code provides operators. The tests provide constraints. The CSV files keep the record of what was actually checked.

AI agents are useful here because they can do the repetitive reading and glue work: extract equations, OCR tables, convert fragments to Markdown, inspect nearby code, draft a runner, and summarize discrepancies. They are dangerous when they flatten scientific distinctions.

For this kind of work, I would rather have a cautious agent that says “this reference is out of scope for the current solver” than a confident agent that makes every table look validated.