Reproduce before publishing

10 Sep, 2026

The most exciting academic use case for LLMs I have come across so far this year has been reproducing your own papers before you publish them. I believe:

1. It will become standard practice to have AI agents reproduce papers before publication.

2. Authors are uniquely placed to own this reproduction process.

3. The result serves as a “certificate of trust” even to an unacquainted reader.

The idea to write about this was sparked by a section of Andrew Hall’s “The 100x Research Institution”, a post I feel is still underappreciated.

I. Introduction

Autonomous research work has been one of the dominant themes in AI in 2026:¹ Largely following the Mythos announcement, and then the leaps Fable and GPT-5.6 made in finding cybersecurity vulnerabilities, letting LLMs work autonomously on solving self-contained (if very hard) research tasks has increasingly become a core part of AI discourse. People proved,² disproved,³ and made progress on⁴ long-standing conjectures in mathematics, autonomous experiment tracking libraries like trackio by Hugging Face have gained a lot of traction,⁵ there have been large-scale reproduction efforts with AI agents especially around ICML 2026 main track papers⁶ and there is even a model now specifically post-trained for this purpose.⁷

Interestingly, this development is also causing negative emotions. Earlier this summer, a finance professor at a well-known UK university told me about a conference participant who had given his paper to an AI agent. The participant then approached him to report that the agent had been unable to reproduce the results. The professor found this comment highly unconstructive.

It is understandable that an author would take issue with such vague criticism. But I believe the burden of pre-empting this criticism may soon shift to the author. This is because the author should always exhaust all means available to make the reader’s experience as pleasant as possible, which includes making them trust the research: And one means that has become available fairly recently is reproducing the paper themselves before publication using an AI agent. What makes this workflow particularly attractive is that unlike other uses of AI that substitute for our own thinking, it does not come at the expense of intellectual rigor. I believe this workflow is well-suited to emerge as an academic norm for quantitative papers (or whatever the future publishing format may be)⁸ because not only do both authors and readers benefit from it (III.), but the setup overhead is also minimal (IV.1. - IV.5.), and it is not cost-intensive to run (IV.6.).

II. Definitions

By reproducibility, I mean computational reproducibility, i.e. the ability to obtain consistent results using the same input data and methods. What is distinctive about this approach, however, is that computational reproduction usually also relies on the code provided,⁹ if available. The type of reproduction described in this post is one that does not rely on any code as input, but instead only takes the input data of the experiment and a plain-text, human-readable description of how the experiment was conducted. This will usually be a version of the paper reduced to its methods-related sections.

III. Motivation

In April 2026, Kohler et al. published a remarkable study in which they attempted to reproduce 48 social science papers using a workflow very similar¹⁰ to the one suggested here, but after publication.¹¹ Regarding the failure modes of agent reproductions, they find:

[...] [T]he largest share of divergences stems from original errors, indicating mismatches between the paper and the underlying code. In other words, the methods descriptions in the paper are often insufficiently precise to enable faithful reimplementation. Another substantial share of errors comes from missing data.

Both of these error types are of human origin. More importantly for my argument, these errors can also be resolved by the author, who knows the method in perfect detail and has full access to all data.¹² Furthermore, regarding the second largest group of errors, they say:

[Reproduction a]gent errors constitute the second largest category of discrepancies. For the strongest agents, the share of agent errors declines markedly, becoming a relatively minor component of total discrepancies.

It seems reasonable to expect the share of agent errors to decline further as agents become more capable. This suggests a world in which, in the limit, the only remaining errors are those only the author can resolve. The author is therefore uniquely placed to own this agent-driven reproduction process.

If the agent is able to reproduce the results using only the input data and a plain-text methods description, this provides strong evidence to the reader that the methods description and supplied data suffice to recover the results. Texts are the most practical basis for any kind of critical engagement with the work of others. In particular, they outperform:

1. plain, documented codebases: publishing the codebase is of course still helpful, but readers will have to jump back and forth between files and make sense of how they fit together, which usually takes way more time than reading a well-structured text describing it. Besides, it remains possible to inspect the codebase.

2. talking to the codebase through an LLM: a reader’s conception of how the code works will shape their questions, guiding the LLM to provide “spotlight” insights into the codebase. Something that does not fit that conception might sit just outside the spotlight and escape attention. This could cause the reader to overlook a design decision they might otherwise question.

Furthermore, this kind of “detective work” of having to find the right spots in the codebase or having to ask the right questions should not fall on the reader in the first place.

Finally, it is also in the interest of any author to have evidence that the paper’s methods description and supplied data suffice to recover the results, so that one can point anyone whose own reproduction fails to the agent’s reproduction first -- an option that may have been valuable to the professor mentioned earlier.

IV. How-to

In the following, I would like to illustrate my way of going about this using the example of a legal benchmarking paper I wrote. I will use Harbor, which is a framework for evaluating agents on long-horizon tasks like this one (and is currently emerging as an industry standard, so it is worth having a look at anyway). Running such evaluations can take several hours, so it is convenient to run them remotely. I run mine on Modal, which is quick to set up and has a very generous free tier.

1. Deciding which claims to reproduce

First, it is helpful to identify specific computational claims that the reproduction should establish. While it may sometimes be possible to reproduce a paper in a single pass, decomposing it into smaller claims makes each one more manageable for the reproduction agent. For my benchmarking paper, I focused on two claims:

1. Given the recorded model responses, the scoring procedure described in the paper exactly produces the scores I arrived at in the paper.

2. Given the annotated full-text court decisions, the dataset construction procedure described in the paper exactly produces the benchmark datasets I constructed.

Each successful reproduction provides a “certificate of trust” for that particular claim. The aim is to cover as much of the paper’s computational work as practical,¹³ while making clear which parts remain unchecked.

For each claim, one would then prepare a separate Harbor task using the following steps.

2. Creating a detailed description of your experiment

The simplest way of obtaining this description is to go through your paper and keep only the methodology-related parts relevant to the respective claim.

For my first claim, I created a stripped-down version of my paper only containing sections where I described the scorer I constructed. Anything that hints at the results or frames the effort in a way that might give away the conclusion of the paper has to be removed.

For my second claim, I wanted to show that I constructed my benchmark datasets from my raw annotated data in a way that is faithful to how I described it in the paper. However, in this case, after a number of attempts with slight changes to the methods section, I did not manage to obtain any scoring improvements. It turned out that a lot of small details on the data generation process that are negligible for understanding the overall process, but are vital for the exact reproduction (such as tedious details on RNG configurations) were missing. So instead, I had another agent write a description of my dataset construction process based on my codebase. A point of criticism I have encountered here is that a highly detailed description of the experiment will often exceed the space usually available for the methods section of a paper. I do not see why one could not simply create a “methods appendix” for this: The methodology section gives an overview of the experiment, and an exact, slightly more verbose, agent-reproduced version of the description goes into the appendix. I attached such an appendix to my paper here.

In your Harbor task, this experiment description goes into your instruction.md file.

3. Add the data

Depending on the type of paper, this may be as simple as sharing the datasets you used for your experiments. However, you will have to provide anything non-deterministic in cached form. If you use LLMs in your experiment, this will e.g. (usually) apply to inference run on third-party hardware.¹⁴ In my case, I made the traces of my legal benchmark evaluations available in the environment of the reproduction agent as well.

In your Harbor task, the data you would like to provide to the reproduction agent should go into the environment subdirectory.

4. Come up with a verifier

The verifier is very simple: score the reproduced output against the reference output using exact match. The implementation will differ based on the shape of your experiment. You can view my verifiers for the above tasks here and here¹⁵.

For the purposes I have in mind here, only a full pass should count as a successful reproduction. Some people understandably dislike full-pass-based evaluations;¹⁶ they make even the smallest errors become significant enough to fail a run, which in many real-world applications is not a useful signal. Here, however, that strictness is especially valuable: If a reproduction diverges by even 0.1%, an unacquainted reader would still have to investigate whether that divergence is genuinely minor. At that point, the reproduction can no longer serve as a clean “certificate of trust.”

In your Harbor task, you will have to define a verifier at tests/test.sh (or tests/test.bat if on Windows). Of course, this does not keep you from still writing the verifier in Python and simply invoking it via the bash/batch script, which is usually more convenient.

5. Running the reproduction agent

Then, you let the reproduction agent run on the task you have constructed this way. It is now very simple to let an agent hillclimb the experiment description in your paper to completeness and close-to-zero ambiguity: Give one orchestration agent permissions to trigger new Modal runs and to modify the methods section you handed it as input, and let it iterate until a perfect score is achieved. Even though the author may not need to review every turn, it is still advisable to do so to maintain close control of the changes made to the paper¹⁷ and to steer the agent. For illustration purposes, I have documented an example where an agent capability error turned out to be an underspecification error:

On a personal note, I find this iterative mode of work of manually applying edits and receiving an immediate signal of their impact on the paper very satisfying.

For the first of my two tasks (reproducing the scorer), you can find the task here and the successful reproduction run here. For the second one (reproducing the dataset construction), I have recorded the process more extensively: You can explore the task and the successful run, but also all runs, and the trajectory of the orchestrator agent. In my case, it took just a handful of iterations to achieve a perfect score.

6. Sharing the runs

Harbor provides a free service called the Harbor Hub that integrates the Harbor trajectory viewer, which in my opinion is the best tool for inspecting agent trajectories as of September 2026. You can share tasks and jobs to the Hub directly via the CLI. My two reproduction runs for the scorer and the dataset construction can be found here:

1. reproducing the scorer - cost: $0.0135 (~1.3¢)

2. reproducing the benchmark dataset construction - cost: $0.0516 (~5.2¢)

The two successful reproduction runs for these components of my paper with deepseek-v4-flash at maximum reasoning effort cost me less than seven cents (!) combined.¹⁹

V. Future developments

Some challenges remain: For example, authors could tamper with the agent trajectories that prove reproduction, and they may also not be able to publish all data that went into the reproduction environment for licensing or security reasons, which would make it harder to verify their validity. Both of these concerns could be solved if there was a third party that provided dedicated environments for this reproduction and also issued the “certificates” described above upon a successful run. Considering that such a third party would have to be able to provide a significant amount of server capacity for provisioning sandboxes and potentially compute, it would have to act somewhat efficiently. At the same time, it could grow deeply entrenched in the research ecosystem, which calls into question whether it should operate as a for-profit enterprise. The position of such an actor within the research ecosystem should therefore be carefully considered.

VI. Conclusion

I believe there are good reasons for reproducing one’s own papers before publication: It makes the reader’s experience more pleasant, builds trust in the research, and can be surprisingly cost-effective to run. Beyond these practical benefits, there is also something inherently satisfying about performing this workflow: all the more reason to try it firsthand.

Thank you for reading. Feel free to subscribe here for more.