Solved – How are we defining ‘reproducible research’

"Reproducible research" as reproducible analysis

Reproducible research is a term used in some research domains to refer specifically to conducting analyses such that

• code transforms raw data and meta-data into processed data,
• code runs analyses on the data, and
• code incorporates analyses into a report.

When such data and code are shared, this allows other researchers to:

• perform analyses not reported by the original researchers
• check the correctness of the analyses performed by the original researchers

This usage can be seen in discussions of technologies like Sweave. E.g., Friedrich Leisch writes in the context of Sweave that "the report can be automatically updated if data or analysis change, which allows for truly reproducible research." It can also be seen in the CRAN Task View on Reproducible Research which states that "the goal of reproducible research is to tie specific instructions to data analysis and experimental data so that scholarship can be recreated, better understood and verified."

Broader usage of the term "reproducibility"

Reproducibility is a fundamental aim of science. It's not new. Research reports include method and results sections that should outline how the data was generated, processed, and analysed. A general rule is that the details provided should be sufficient to enable an appropriately competent researcher to take the information provided and replicate the study.

Reproducibility is also closely related to the concepts of replicability and generalisation.

Thus, the term "reproducible research", taken literally, as applied to technologies like Sweave, is a misnomer, given that it suggests a relevance broader than it covers. Also, when presenting technologies like Sweave to researchers who have not used such technologies, such researchers are often surprised when I call the process "reproducible research".

A better term than "reproducible research"

Given that "reproducible research" as used within Sweave-like contexts only pertains to one aspect of reproducible research, perhaps an alternative term should be adopted. Possible alternatives include:

• Reproducible analysis:
  • John D Cook has used this term
  • Jennifer Blackford uses the term "reliable and reproducible analyses"
• Reproducible data analysis
  • Christophe Pouzat uses this term
• Reproducible statistical analysis
  • A Biostats site at Vanderbilt uses the term "reproducible statistical analysis and reporting activities"
• Reproducible reporting

All of the above terms are a more accurate reflection of what Sweave-like analyses entail. Reproducible analysis is short and sweet. Adding "data" or "statistical" further clarifies things, but also makes the term both longer and narrower. Furthermore, "statistical" has a narrow and a broad meaning, and certainly within the narrow meaning, much of data processing is not statistical. Thus, the breadth implied by the term "reproducible analysis" has its advantages.

It's not just about reproducibility

The other additional issue with the term "reproducible research" is the aim of Sweave-like technologies is not just "reproducibility". There are several interrelated aims:

• Reproducibility
  • Can the analyses easily be re-run to transform raw data into final report with the same results?
• Correctness
  • Is the data analysis consistent with the intentions of the researcher?
  • Are the intentions of the researcher correct?
• Openness
  • Transparency, accountability
    • Can others check and verify the accuracy of analyses performed?
  • Extensibility, modfifiability
    • Can others modify, extend, reuse, and mash, the data, analyses, or both to create new research works?

There is an argument that reproducible analysis should promote correct analyses, because there is a written record of analyses that can be checked. Furthermore if data and code is shared, it creates accountability which motivates researchers to check their analyses, and enables other researchers to note corrections.

Reproducible analysis also fits in closely with concepts around open research. Of course, a researcher can use Sweave-like technologies just for themselves. Open research principles encourage sharing the data and analysis code to enable greater reuse and accountability.

This is not really a critique of the use of the word "reproducible". Rather, it just highlights that using Sweave-like technologies is necessary but not sufficient to achieving open scientific research aims.