Master Thesis Defense: Lalet Scaria

Speaker: Lalet Scaria

Supervisor: Dr. T. Glatard

Examining Committee: Drs. Y.-G. Gueheneuc, M. Kersten-Oertel, T.-H. Chen (Chair)

Title: A Framework to Evaluate Pipeline Reproducibility Across Operating Systems

Date: Thursday, June 7, 2018

Time: 10:00am

Place: EV 2.260

ABSTRACT

The lack of computational reproducibility threatens data science in several domains. In particular, it has been shown that different operating systems can lead to different analysis results. This study aims to identify and quantify the effect of the operating system on neuroimaging analysis pipelines. We developed a framework to evaluate the reproducibility of these neuroimaging pipelines across operating systems. The framework essentially leverages software containerization and system-call interception to record results provenance without having to instrument the pipelines. A tool (Repro-tools) was developed to compare results obtained in different conditions. We used our framework to evaluate the effect of the operating system on results produced by pipelines from the Human Connectome Project (HCP), a large open data initiative to study the human brain. In particular, we focused on pre-processing pipelines for anatomical and functional data, namely PreFreeSurfer, FreeSurfer, Post- FreeSurfer and fMRI-Volume. We used data from 5 subjects released by the HCP. Results highlight substantial differences in the output of the HCP pipelines obtained in two versions of Linux (CentOS6 and CentOS7). Inter-OS differences corresponding to normalized root mean square errors of up to 0.27 were observed, which corresponds to visually important differences. We provide visualizations of the most important differences for various pipeline steps. No meaningful inter-run differences were observed, which shows that the inter-OS differences do not originate from the use of pseudorandom numbers or silent crashes of the pipelines. We hypothesize that the observed inter-OS differences come from numerical instabilities in the pipelines, triggered by rounding and truncation differences that originate in the update of mathematical libraries in different systems. An apparent solution to this issue is to freeze the execution environment using, for instance, software containers. However, this would only mask instabilities while they should ultimately be corrected in the pipelines.