Readme#
𧬠omics-utils#
A modern toolkit for handling larger-than-memory omics datasets using hydra-zen and dask on a Slurm cluster with Docker/Apptainer containerization.
π Key Features#
π§ͺ Concatenation of multiple sub-datasets/cohorts into one larger dataset
π Assigning train/val dataset splits and shuffling
πΎ Sharding of the folds across multiple smaller-than-memory files
π Dask + Slurm integration for efficient parallelization
π§© Reproducibility and modular type-checked configs via hydra-zen
π Table of Contents#
π Container Registry Authentication#
Generate Token#
Create a new GitHub token at Settings β Developer settings β Personal access tokens with:
read:packagespermissionwrite:packagespermission
Log In#
With Apptainer:
apptainer remote login --username <your GitHub username> docker://ghcr.io
With Docker:
docker login ghcr.io -u <your GitHub username>
When prompted, enter your token as the password.
π³ Container Setup#
Choose one of the following methods to set up your environment:
Option 1: Apptainer (Slurm Cluster)#
Install VSCode Remote Tunnels Extension
First, install the Remote Tunnels extension in VSCode.
Connect to compute resources
For CPU resources:
srun --partition=cpu-2h --pty bash
For GPU resources:
srun --partition=gpu-2h --gpus-per-task=1 --pty bash
Launch container
To open a tunnel to connect your local VSCode to the container on the cluster:
apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:latest-sif code tunnel
π‘ You can specify a version tag (e.g.,
v0.0.1) instead oflatest. Available versions are listed at GitHub Container Registry.In VSCode press
Shift+Alt+P(Windows/Linux) orShift+Cmd+P(Mac), type βconnect to tunnelβ, select GitHub and select your named node on the cluster. Your IDE is now connected to the cluster.To open a shell in the container on the cluster:
apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:latest-sif /bin/bash
π‘ This may take a few minutes on the first run as the container image is downloaded.
Option 2: Docker (Local Machine)#
Install VSCode Dev Containers Extension
First, install the Dev Containers extension in VSCode.
Open the Repository in the Dev Container
Click the
Reopen in Containerbutton in the pop-up that appears once you open the repository in VSCode.Alternatively, open the command palette in VSCode by pressing
Shift+Alt+P(Windows/Linux) orShift+Cmd+P(Mac), and typeDev Containers: Reopen in Container.
Using Slurm within Apptainer#
In order to access Slurm with submitit from within the container, you first need to set up passwordless SSH to the login node.
On the cluster, create a new SSH key pair in case you donβt have one yet:
ssh-keygen -t ed25519 -C "your_email@example.com"
and add your public key to the authorized_keys:
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
You can verify that this works by running
ssh $USER@$HOST exit
which should return without any prompt.
𧬠Running Preprocessing#
Make sure you followed the previous sections beforehand and run these commands in the container.
The preprocessing pipeline supports multiple omics datasets:
CellXGene Preprocessing#
Before running the CellXGene preprocessing, download the h5ad files from AWS into a folder as described in the documentation and set DATA_DIR to point to this folder.
The current version was tested on the
2024-07-01release of CellXGene.
python omics_utils/preprocess.py pipeline=cellxgene
HLCA Preprocessing#
python omics_utils/preprocess.py pipeline=hlca
π¦ Package Management#
Update dependencies
This project uses uv for Python dependency management.
Inside the container (!):
# Add a specific package uv add <package-name> # Update all dependencies from pyproject.toml uv sync
Commit changes to the repository:
Use tags for versioning:
git add pyproject.toml uv.lock git commit -m "Updated dependencies" git tag v0.0.1 git push && git push --tags
Use the updated image:
The GitHub Actions workflow automatically builds a new image when changes are pushed.
With Apptainer:
apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:v0.0.1-sif /bin/bash
With Docker:
docker run -it --rm --platform=linux/amd64 ghcr.io/marvinsxtr/omics-utils:v0.0.1 /bin/bash
π Documentation#
To build the documentation locally, run the following command in your container:
uv run sphinx-build docs/source docs/build
π₯ Contributions#
Contributions to this project are very welcome! Feel free to open a PR or reach out with suggestions.