This repository contains code for the DeepNSD project, an attempt to characterize the representational structure of human visual cortex with the massive NSD fMRI dataset and a bountiful cornucopia of deep neural network models. It also contains all code and access to data for reproducing the associated manuscript: "What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?", currently in-press at Nature Communications.
Our Google Colab tutorial (bit.ly/Deep-NSD-Tutorial) provides a step by step demonstration of the main functions in this pipeline, fitting the representations of a CLIP model to a single subject subset of the fMRI data using the DeepDive package (soon to be re-released as DeepJuice).
You can use this codebase to quickly load (in a unified API) a number of models and their associated transforms. (Please note that -- pending further development -- you will have to install the underlying model packages manually, as they often require machine-specific settings during installation.)
Models we've preprocessed include:
- the PyTorch-Image-Models library
- the Torchvision model zoo
- the Taskonomy project
- the VISSL (SSL) model zoo
- ISL's MiDas models zoo
- FaceBook's DINO models...
To cite this repository, or the associated manuscript, please use the following BibTex:
@article{conwell2023pressures,
title={What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?},
author={Conwell, Colin and Prince, Jacob S and Kay, Kendrick N and Alvarez, George A and Konkle, Talia},
journal={BioRxiv},
year={2023},
url={https://www.biorxiv.org/content/10.1101/2022.03.28.485868v2}
publisher={Cold Spring Harbor Laboratory}
}- Squeezing your deep nets for science!
Recently, our team has been working on a new, highly-accelerated version of this codebase called Deepjuice -- effectively, a bottom-up reimplementation of all DeepDive functionalities that allows for end-to-end benchmarking (feature extraction, SRP, PCA, CKA, RSA, and regression) without ever removing data from the GPU.
DeepJuice is currently in private beta, but if you're interested in trying out, please feel free to contact me (Colin Conwell) by email: conwell[at]g[dot]harvard[dot]edu