Permute-match tests: Detecting significant correlations between time series despite nonstationarity and limited replicates

Researchers in fields from ecology to neuroscience analyze correlations between pairs of time series, often working with nonstationary data, wherein statistical properties change over time. This commonly involves a statistical test to determine whether an observed correlation is stronger than expected under the null hypothesis of independence. Testing for dependence between nonstationary time series with only one experimental replicate is exceedingly challenging. However, with many replicates, a nonparametric trial-swapping permutation test can be employed, comparing within-replicate correlations to between-replicate correlations. Although largely assumption-free, this test is severely limited by the number of replicates because its minimum achievablep-value is 1/n! wherenis the number of replicates. This curtails its applicability to many biomedical studies, wherenis frequently as low as 3, which would render significance thresholds like 0.05 unattainable. To address this, we propose modified permutation tests that can report lowerp-values of 2/nⁿor 1/nⁿwhen there is strong evidence of dependence. We prove that the tests guarantee a false positive rate at or below the significance level, as long as replicates come from independent and identical experiments. We demonstrate this approach by confirming the observation that groups of zebrafish swim faster when directionally aligned, using an existing dataset with 3 biological replicates.