I just finished reading the Economists article Trouble in the Lab. Nothing in the article was new to me as such but I am pleased to see such concerns being raised in mainstream publication. One reason mmtests exists is because it is important that any results I use to justify a patches inclusion into the Linux kernel can be reproduced.. There is no guarantee the results can be reproduced for a variety of reasons such as being dependant on the machine configuration but the attempt can be made and anomalies, including bugs in the test itself, reported. That aside, what amused me was the link to Nature’s methodology checklist. The intent of the checklist is to ensure that “all technical and statistical information that is crucial to an experiment’s reproducibility” are available. It’s a great idea (even if there is no guarantee data remains available forever) and point 19 covers software availability. What amused me was that reading the checklist requires proprietary software (Adobe Acrobat in this case) to read what could have been a simple PDF. Clearly the intent was to have a standard form when submitting documents for publication but it is ironic that a document about reproducibility is itself inconvenient to reproduce.