Usage on O2 (Gater)

Note: This workflow is currently optimized only for Gater.

O2 is a high performance computing cluster at Harvard Medical School. Minerva Analysis can be run on O2 and datasets can be loaded from the cluster. There are 2 ways that the tool can be run.

From /n/groups/lsp/gater. We maintain the tool and conda environment in the group directory and a data folder is generated in the your home directory.
From a cloned repo located in your home directory. You are responsible for pulling updates and changes to the tool and conda environment.

We assume that you ran MCMICRO on your sample and generated a multichannel image (e.g., *ome.tif), a segmentation mask (e.g., cellMask*.tif), a spatial feature table (SFT, e.g., *.CSV) for the sample. We also assume here that your data are in /files/ or /n/files which a computation node doesn't have access to, and we first transfer the files to a directory under /n/scratch3, but you can transfer them to anywhere as long as the computation node have access to the path.

Option 1 - Run A Pre-Installed Version of Gater on O2

The easiest option to use Gater on O2 is to run a preinstalled version from '/n/groups/lsp/gater/'. The following description will guide you how to run it.

Note: Please replace <USER> with your HMS user ID and <PORT> with the port number you want to use (e.g. 8008). If the port is unavailable, please try using another port number.

Log into the transfer node and transfer the files to your directory under /n/scratch3.

ssh <USER>@transfer.rc.hms.harvard.edu
cp -r path/to/folder/data /n/scratch3/path/to/folder/

Transfer the SFT (CSV) to a local folder using an SFTP client (e.g. Filezilla for Windows, CyberDuck for Mac).
Log into O2 with port forwarding to localhost:.

ssh -L <PORT>:localhost:<PORT> <USER>@o2.hms.harvard.edu

Run an interactive job.

srun --pty -p interactive --mem 10G -t 0-12 --tunnel <PORT>:<PORT> /bin/bash
NOTE: This can take a minute.

Load the Conda environment.

conda activate /n/groups/lsp/gater/env_gating
- If this is your first time activating the environment, you may see the following error: CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
- Please run conda init bash, then exit the interactive job with exit , and start another interactive job with srun --pty -p interactive --mem 10G -t 0-12 --tunnel 8000:8000 /bin/bash
- Then you will need to re-run conda activate /n/groups/lsp/gater/env_gating

Run the Gater tool.

cd /n/groups/lsp/gater/minerva_analysis
python3 run.py <PORT>

Open http://localhost:<PORT>/ on the browser.
Upload the dataset using paths for files on scratch3. The CSV file should be uploaded from a local directory.

The paths for files on scratch3 should start with /n/scratch3/ (e.g. /n/scratch3/path/to/dataset/registration/image.ome.tif)

NOTE: For the pre-installed version on O2, the uploaded files are in /home/<USER>/.local/share/minerva_analysis/data/

Option 2 - Install and Run your own Version of Gater (in your O2 Home Directory)

There are 3 parts to running the Gater tool from a cloned repo.

First-time setup: You need to clone the repo and create the Conda environment.
Pull updates: When there's an update to the tool, you need to pull them to your repo.
Run the tool.

First-time setup of Gater

Note: Please replace <USER> with your HMS user ID and <PORT> with the port number you want to use (e.g. 8008). If the port is unavailable, please try using another port number.

Connect to O2 via SSH and start an interactive job.

ssh <USER>@o2.hms.harvard.edu
srun --pty -p interactive --mem 10G -t 0-12 /bin/bash

Clone the gating branch of Minerva Analysis.

mkdir ~/src if the directory ~/src doesn't exist
cd ~/src
git clone https://github.com/labsyspharm/minerva_analysis.git
cd ~/src/minerva_analysis
git checkout gating
git pull

Go to Minerva Analysis folder.

cd ~/src/minerva_analysis

Create conda environment for Minerva Analysis.

conda env create -f requirements.yml

Pull updates for Gater

Note: Please replace <USER> with your HMS user ID and <PORT> with the port number you want to use (e.g. 8008). If the port is unavailable, please try using another port number.

Connect to O2 via SSH and start an interactive job.

ssh <USER>@o2.hms.harvard.edu
srun --pty -p interactive --mem 10G -t 0-12 /bin/bash

Go to Minerva Analysis folder.

cd ~/src/minerva_analysis

Pull updates from the GitHub repo.

git checkout gating
git pull

Update the conda environment.

module load conda2
conda env update --name minerva_analysis --file requirements.yml --prune

Run Gater on O2 and upload data from scratch3

Note: Please replace <USER> with your HMS user ID and <PORT> with the port number you want to use (e.g. 8008). If the port is unavailable, please try using another port number.

Log into the transfer node and transfer the files to your directory under /n/scratch3.

ssh <USER>@transfer.rc.hms.harvard.edu
cp -r path/to/folder/data /n/scratch3/path/to/folder/

Transfer the SFT (CSV) to a local folder using an SFTP client (e.g. Filezilla for Windows, CyberDuck for Mac).
Log into O2 with port fowarding to localhost:.

ssh -L <PORT>:localhost:<PORT> <USER>@o2.hms.harvard.edu

Run an interactive job.

srun --pty -p interactive --mem 10G -t 0-12 --tunnel <PORT>:<PORT> /bin/bash
NOTE: This can take a minute.

Run the Gater tool.

module load conda2
conda activate minerva_analysis
- If this is your first time activating the environment, you may see the following error: CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
- Please run conda init bash, then exit the interactive job with exit , and start another interactive job with srun --pty -p interactive --mem 10G -t 0-12 --tunnel 8000:8000 /bin/bash
- Then you will need to re-run module load conda2 and conda activate minerva_analysis
cd ~/src/minerva_analysis
python3 run.py <PORT>

Open http://localhost:8000/ on the browser.
Upload the dataset using paths for files on scratch3. The CSV file should be uploaded from a local directory.