Temporal Hierarchy in Spiking Neural Networks

Taking inspiration from the brain to perform efficient computation is a fascinating challenge, often hampered by the complexity of the brain itself. This work investigates a key feature observed across the cortices of various mammals: the hierarchy of time scales in cortical areas. Experimental evidence suggests that the intrinsic speed of activity in neuronal populations progressively slows down through the cortical hierarchy, forming what we define as a ``temporal hierarchy''. We explore whether this property provides an advantage in artificial computational systems. To test this hypothesis, we analyze SNNs, neural networks inspired by biological computation and with inherent temporal dynamics, and we endow them with temporal hierarchy. We incorporate the temporal hierarchy in various SNN temporal parameters, including their neuronal dynamics, synaptic delays, and recurrent dynamics. Our study evaluates these temporal hierarchies under two settings, (1) as an inductive bias, where SNNs temporal parameters are initialized with a predefined hierarchy to evaluate performance improvements, and (2) through optimizing the temporal parameters of SNNs. Benchmarking these networks on temporal tasks, the Multi-Timescale-XOR, and keyword spotting, we validate the benefits of hierarchy in various temporal mechanisms under both settings. In the inductive bias setting, we show that under iso-parameters settings, (i) classification accuracy correlates positively with the magnitude of temporal hierarchy on all the benchmarks, and (ii) there is an accuracy gain between 2 to 6% across tasks, when introducing temporal hierarchy, compared to a network without hierarchy. Moreover, under iso-accuracy settings, introducing hierarchy reduces the required number of parameters by up to 5 times. In the emergence from the optimization setting, we show that temporal hierarchy is naturally found as an emergent property through gradient descent. Finally, we conduct a detailed mathematical analysis of the temporal processing capabilities of SNNs, showing that a hierarchical arrangement of time constants enables a logarithmic reduction in the number of layers required to process temporal signals with multiple frequency components.