Bashing irreproducibility withshournal

Abstract

The Linux shell is arguably one of the most important computational tools across various scientific disciplines. Its high flexibility makes it the platform of choice for many file operations and smaller scripting tasks. Also, many stand-alone programs are called from the Linux shell - typically followed by multiple command-line parameters. However, in larger analysis projects, keeping track of the work quickly becomes challenging, as a typical shell workflow involves the iterative execution of commands with many parameters, modification of scripts, and editing of configuration files. Too often, researchers find themselves in the uncomfortable position that a computational result generated a few weeks ago can no longer be reproduced, despite having taken great care documenting the work manually. On the other hand, there is a lack of tools able to record the researcher’s shell activity automatically with reasonably low runtime- and storage overhead. To close this critical gap, we developedshournal, a program that tightly integrates with the Linux shell and automatically records every shell command along with the files it reads or writes. Besides logging command- and file metadata, such as working directory, file path, and checksums,shournalcan be configured to archive scripts or configuration files that are not regularly under version control viagitorsvn.