In short: This repository contains the code for training a spiking neural network (SNN) on event-based data through a spike-timing-dependent plasticity (STDP) learning rule. Although the focus of the project was on the visual modality (with the N-MNIST dataset), auditory and tactile versions of the code are available as well (with the datasets N-TIDIGITS and ST-MNIST, although performance is very low).
Adapted from Diehl, P. U., & Cook, M. (2015).
This project was carried out as the master thesis on "Local Unsupervised Learning with Multimodal Event-Based Sensors and Spiking Neural Networks" authored by Julian Lopez Gordillo and supervised by: Dr. Lyes Khacef, Prof. Dr. Elisabetta Chicca, Prof. Dr. Alejandro Linares-Barranco, Dr. Antonio Rios-Navarro, and Prof. Dr. Niels Taatgen (MSc in Artificial Intelligence 2019-2021). This project builds upon some previous works that are listed in the section References later on.
The project was done in Python 3.8, PyTorch 1.7.1 and BindsNET 0.2.9, although newer versions will probably work just fine.
For the complete list of dependencies check the file requirements.txt.
Bindsnet is directly included as source code and does not need to be installed.
For GPU acceleration, the appropriate graphic drivers need to be installed and configured. If there is conflict
between PyTorch and CUDA, it might be a good idea to check PyTorch's installation page and install it from there.
- Make sure you have a working installation of Python 3.8+ (as well as the pip package manager).
- Clone this repository.
- Execute
pip install -r requirements.txton the root folder of the repo. - (The downloading, unzipping, and pickling of the dataset should be handled automatically the first time you run the code. If there is any issue with the download, you can download it and unzip it manually (the path to the unzipped Train folder should be
.../stdp_nmnist/data/NMNIST/raw/Train, and the equivalent for Test). - Everything should be ready, you can execute
python MODALITY_network.py --helpfor a description of the available options. "MODALITY" should be replaced with "Visual", "Auditory", or "Tactile". - For example, you can train a visual network by issuing a
python Visual_network.py [--options]command with the specified options (adding no options will execute the default parameters).
Note: The auditory and tactile modalities are not as polished and some things might not work as intended.
The following options are available:
-h, --help show this help message and exit
--thresh THRESH Threshold for the membrane voltage.
--tc_decay TC_DECAY Time constant for the membrane voltage decay.
--x_tar X_TAR Target value for the pre-synaptic trace (STDP).
--n_neurons N_NEURONS
Number of neurons in the excitatory layer.
--exc EXC Strength of excitatory synapses.
--inh INH Strength of inhibitory synapses.
--theta_plus THETA_PLUS
Step increase for the adaptive threshold.
--som Enable for topological self-organisation.
--n_test N_TEST Number of samples for the testing set (if None, all are used)
--n_train N_TRAIN Number of samples for the training set (if None, all are used)
--pattern_time PATTERN_TIME
Duration (in milliseconds) of a single pattern.
--filename FILENAME
Name for the experiment (and resulting files).
--dt DT Simulation timestep.
--n_epochs N_EPOCHS Number of training epochs.
--n_workers N_WORKERS
Number of parallel processes to be created.
--seed SEED Seed for the pseudorandom number generation.
--progress_interval PROGRESS_INTERVAL
Frequency of training progress reports.
--plot Enable plotting (considerably slows the simulation).
--gpu Enable GPU acceleration.
Note: Modifying the --pattern_time option requires having serialised the dataset files with the new pattern_time
beforehand.
For example, changing the --pattern_time to a different value than default when running Visual_network.py
requires regenerating the dataset files with that new pattern time (to change in NMNIST.py).
Throughout the code, you will find some # PARAMETER, # TODO and # FUTURE WORK comments:
# PARAMETER, as the name suggests, tags the lines of code where important parameters are selected.# TODOmarks certain improvements that I couldn't finish but would be nice additions to the code.# FUTURE WORKshows the lines of research we would have taken had we had more time for the project.
In the bigger picture, there are three levels where next steps could be taken:
- Synapse level: this entails the exploration of new plasticity rules, something we did not contemplate in this thesis.
- Neuron level: just like homeostasis from the ALIF model, more complex neuron models might provide other useful features that change the learning dynamics.
- Network level: this includes adding recurrence to the network topology, increasing the number of layers or the complexity of the connectivity.
Particularly, the network level is the most simplistic one at the moment, and thus the one that would benefit the most from an update. Apart from some scattered tasks throughout the code that could be picked up, another interesting line of research would be looking into different datasets. For example, LIPSFUS is an interesting event-based audiovisual dataset where both modalities were recorded simultaneously, and therefore, are naturally synchronised. It would fit very well for this project.
The module eventvision is slightly modified from gorchard/event-Python and only necessary for the reading of the data prior to serialisation.
Orchard, G., Jayawant, A., Cohen, G. K., & Thakor, N. (2015). Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades. Frontiers in Neuroscience, 9. https://doi.org/10.3389/fnins.2015.00437
Diehl, P. U., & Cook, M. (2015). Unsupervised learning of digit recognition using spike-timing-dependent plasticity. Frontiers in Computational Neuroscience, 9. https://doi.org/10.3389/fncom.2015.00099
Hazan, H., Saunders, D., Sanghavi, D. T., Siegelmann, H., & Kozma, R. (2018). Unsupervised Learning with Self-Organizing Spiking Neural Networks. 2018 International Joint Conference on Neural Networks (IJCNN), 1–6. https://doi.org/10.1109/IJCNN.2018.8489673
Iyer, L. R., & Basu, A. (2017). Unsupervised learning of event-based image recordings using spike-timing-dependent plasticity. 2017 International Joint Conference on Neural Networks (IJCNN), 1840–1846. https://doi.org/10.1109/IJCNN.2017.7966074
Iyer, L. R., Chua, Y., & Li, H. (2021). Is Neuromorphic MNIST Neuromorphic? Analyzing the Discriminative Power of Neuromorphic Datasets in the Time Domain. Frontiers in Neuroscience, 15. https://doi.org/10.3389/fnins.2021.608567