Create Profiles
Contents
5. Create Profiles¶
5.1. Confirm Environment Configuration¶
You CAN, if you choose, make your backends with the same machine that you have used to make CSVs and run DCP for CellProfiler. However, we typically do not, since backend creation can take a long time so it is desirable to use 2 machines - a small inexpensive machine with only a few CPUs for all the steps before backends and a larger machine with at least as many CPUs as (the number of plates in your batch +1) for backend creation. Both machines can be turned off when not in use and when off will generate only minimal charges.
To make a new backend machine, follow the identical instructions as in the section below, only with a larger Instance Type (such as an m4.10xlarge).
Since backend creation also typically requires a large amount of hard disk space, which you pay for whether or not the machine is on, we recommend attaching a large EBS volume to your backend creation machine only when needed, then detaching and terminating it when not in use.
5.2. Add a large EBS volume to your machine¶
Follow the AWS instructions for creating and attaching the EBS volume. Two critical factors to note- the volume must be created in the same subnet as your backend creation machine, and should be approximately 2X the size as the analysis files in your batch - this is most easily figured by navigating to that location in the S3 web console and selecting “Actions -> Calculate total size”.
Once the volume is created and attached, ensure the machine is started and SSH connect to it.
Create a temp directory which is required when creating the database backed using cytominer-database (discussed later).
mkdir ~/ebs_tmp
Get the name of the disk and attach it.
# check the name of the disk
lsblk
#> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
#> xvda 202:0 0 8G 0 disk
#> └─xvda1 202:1 0 8G 0 part /
#> xvdba 202:80 0 100G 0 disk
# check if it has a file system
sudo file -s /dev/xvdba
# ...likely not, in which case you get:
#> /dev/xvdf: data
# if no file system, then create it
sudo mkfs -t ext4 /dev/xvdba
# mount it
sudo mount /dev/xvdba /home/ubuntu/ebs_tmp/
# change perm
sudo chmod 777 ~/ebs_tmp/
If you are starting from here, make sure the following steps have been completed on your ec2 instance and/or session before proceeding
5.3. Create Database Backend¶
Run creation of sqlite backend as well as aggregation of measurements into per-well profiles. This process can be very slow since the files are read from s3fs/EFS. We recommend first downloading the CSVs files locally on an EBS volume attached to the ec2 instance you are running on, and then ingesting.
To do so, first recreate the analysis output folder structure on the EBS volume:
mkdir -p ~/ebs_tmp/${PROJECT_NAME}/workspace/software
cd ~/ebs_tmp/${PROJECT_NAME}/workspace/software
if [ -d pycytominer ]; then rm -rf pycytominer; fi
git clone https://github.com/cytomining/pycytominer.git
cd pycytominer
git checkout jump
python3 -m pip install -e .
The command below first calls cytominer-database ingest to create the SQLite backend, and then pycytominer’s aggregate_profiles to create per-well profiles. Once complete, all files are uploaded to S3 and the local cache are deleted. This step takes several hours, but metadata creation and GitHub setup can be done in this time.
collate.py ingests and indexes the database.
pyenv shell 3.8.10
mkdir -p ../../log/${BATCH_ID}/
parallel \
--max-procs ${MAXPROCS} \
--ungroup \
--eta \
--joblog ../../log/${BATCH_ID}/collate.log \
--results ../../log/${BATCH_ID}/collate \
--files \
--keep-order \
python3 pycytominer/cyto_utils/collate.py ${BATCH_ID} pycytominer/cyto_utils/ingest_config.ini {1} \
--temp ~/ebs_tmp \
--remote=s3://${BUCKET}/projects/${PROJECT_NAME}/workspace :::: ${PLATES}
Note
collate.py does not recreate the SQLite backend if it already exists in the local cache. Add --overwrite flag to recreate.
Note
or pipelines that use FlagImage to skip the measurements modules if the image failed QC, the failed images will have Image.csv files with fewer columns that the rest (because columns corresponding to aggregated measurements will be absent). The ingest command will show a warning related to sqlite: expected X columns but found Y - filling the rest with NULL. This is expected behavior.
Note
There is a known issue where if the alphabetically-first CSV failed QC in a pipeline where “Skip image if flagged” is turned on, the databases will not be created. We are working to fix this, but in the meantime we recommend either not skipping processing of your flagged images (and removing them from your data downstream) or deleting the alphabetically-first CSVs until you come to one where the pipeline ran completely.
This is the resulting structure of backend on S3 (one level below workspace) for SQ00015167:
└── backend
└── 2016_04_01_a549_48hr_batch1
└── SQ00015167
├── SQ00015167.csv
└── SQ00015167.sqlite
At this point, the user needs to use the profiling template to use pycytominer to annotate the profiles with metadata, normalize them, and feature select them.
5.4. Create Metadata Files¶
First, get metadata for the plates. This should be created beforehand and uploaded into S3.
This is the structure of the metadata folder (one level below workspace):
└── metadata
└── platemaps
└── 2016_04_01_a549_48hr_batch1
├── barcode_platemap.csv
└── platemap
└── C-7161-01-LM6-006.txt
2016_04_01_a549_48hr_batch1 is the batch name – the plates (and all related data) are arranged under batches, as seen below.
barcode_platemap.csv is structured as shown below. Assay_Plate_Barcode and Plate_Map_Name are currently the only mandatory columns (they are used to join the metadata of the plate map with each assay plate). Each unique entry in the Plate_Map_Name should have a corresponding tab-separated file .txt file under platemap (e.g. C-7161-01-LM6-006.txt).
Assay_Plate_Barcode,Plate_Map_Name
SQ00015167,C-7161-01-LM6-006
The tab-separated files are plate maps and are structured like this: (This is the typical format followed by Broad Chemical Biology Platform)
plate_map_name well_position broad_sample mg_per_ml mmoles_per_liter solvent
C-7161-01-LM6-006 A07 BRD-K18895904-001-16-1 3.12432000000000016 9.99999999999999999 DMSO
C-7161-01-LM6-006 A08 BRD-K18895904-001-16-1 1.04143999999919895 3.33333333333076923 DMSO
C-7161-01-LM6-006 A09 BRD-K18895904-001-16-1 0.347146666668001866 1.11111111111538462 DMSO
Note
plate_map_nameshould be identical to the name of the file (without extension).plate_map_nameandwell_positionare currently the only mandatory columns.
The external metadata is an optional file tab separated .tsv file that contains the mapping between a perturbation identifier to other metadata. The name of the perturbation identifier column (e.g. broad_sample) should be the same as the column in platemap.txt.
The external metadata file should be placed in a folder named external_metadata within the metadata folder. If this file is provided, then the following should be the folder structure
└── metadata
├── external_metadata
│ └── external_metadata.tsv
└── platemaps
└── 2016_04_01_a549_48hr_batch1
├── barcode_platemap.csv
└── platemap
└── C-7161-01-LM6-006.txt
5.5. Set up GitHub¶
Once and only once - fork the profiling recipe to your own user name (Each time you make a new project - you may want to keep your fork up to date
Once per new PROJECT, not new batch - make a copy of the template repository into your preferred organization with a project name that is similar OR identical to its project tag on S3 and elsewhere.
5.6. Make Profiles¶
5.6.1. Optional - set up compute environment¶
These final steps are small and can be done either in your local environment or on your node used to build the backends. Conda is currently required and not currently on the backend creation VMs. Use these commands to install Miniconda for a Linux ecosystem (otherwise, search for your own OS).
# Miniconda installation
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda
export PATH="/home/ubuntu/miniconda/bin:$PATH"
source .bashrc
conda init bash
source .bashrc
We recommend using DVC for large file management and DVC requires the use of AWS CLI. AWS CLI is already on the backend creation VMs. If you are not using the backend creation VM, use these commands to install AWS CLI for a Linux ecosystem (otherwise, search for your own OS).
# AWS CLI installation - if not on your machine already
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
Configure your AWS CLI installation with
aws configure
It will prompt you for:
AWS Access Key ID:YOUR-KEYAWS Secret Access Key:YOUR-SECRET-KEYDefault region name:e.g. us-east-1Default output format:json
If not using the same machine + tmux as for making backends, where your environment variables are already set, set them up
5.6.2. Set new environment variables¶
Specifically, ORG and DATA should be the GitHub organization and repository name used when creating the data repository from the template. USER should be your GitHub username. CONFIG_FILE will be the name of the config file used for this run, so something that makes it distinguishable (ie, batch numbers being run at this time) is helpful.
ORG=broadinstitute
DATA=2015_10_05_DrugRepurposing_AravindSubramanian_GolubLab_Broad
USER=gh_username
CONFIG_FILE=config_batch1
5.6.3. If a first batch in this compute environment, make some directories¶
mkdir -p ~/work/projects/${PROJECT_NAME}/workspace/{backend,software}
5.6.4. Add your backend files¶
aws s3 sync s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/backend/${BATCH_ID} ~/work/projects/${PROJECT_NAME}/workspace/backend/${BATCH_ID} --exclude="*" --include="*.csv"
5.6.5. If a first batch in this compute environment, clone your repository¶
cd ~/work/projects/${PROJECT_NAME}/workspace/software
git clone git@github.com:${ORG}/${DATA}.git
# depending on your repo/machine set up you may need to provide credentials here
cd ${DATA}
5.6.6. If a first batch in this project, weld the recipe into the repository¶
git submodule add https://github.com/${USER}/profiling-recipe.git profiling-recipe
git add profiling-recipe
git add .gitmodules
git commit -m 'finalizing the recipe weld'
git push
# depending on your repo/machine set up you may need to provide credentials here
git submodule update --init --recursive
5.6.7. If a first batch in this compute environment, set up the environment¶
cp profiling-recipe/environment.yml .
conda env create --force --file environment.yml
5.6.8. Activate the environment¶
conda activate profiling
5.6.9. Set up DVC¶
Initialize DVC for this project and set it to store large files in S3. This needs to happen once per project, not per batch. Skip this step if not using DVC.
# Navigate
cd ~/work/projects/${PROJECT_NAME}/workspace/software/${DATA}/profiling-recipe
# Initialize DVC
dvc init
# Set up remote storage
dvc remote add -d S3storage s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/software/${DATA}_DVC
# Commit new files to git
git add .dvc/.gitignore .dvc/config
git commit -m "Setup DVC"
5.6.10. If a first batch in this project, create the necessary directories¶
profiling-recipe/scripts/create_dirs.sh
5.6.11. Download the load_data_CSVs¶
aws s3 sync s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/load_data_csv/${BATCH_ID} load_data_csv/${BATCH_ID}
gzip -r load_data_csv/${BATCH_ID}
5.6.12. Download the metadata files¶
aws s3 sync s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/metadata/${BATCH_ID} metadata/platemaps/${BATCH_ID}
5.6.13. Make the config file¶
cp profiling-recipe/config_template.yml config_files/${CONFIG_FILE}.yml
nano config_files/${CONFIG_FILE}.yml
Note
The changes you will likely need to make for most small use cases following this handbook. For aggregate set perform to false, for annotate sub-setting external set perform to false, in feature_select set gct to true,
and finally at the bottom set the batch(es) and plates names
For large batches with many DMSO wells and external metadata ala the JUMP project - set perform under external to true, set file to the name of the external metadata file and set merge_column to the name of the compound identifier column in platemap.txt and external_metadata.tsv.
5.6.14. Set up the profiles¶
Note that the “find” step can take a few seconds/minutes
mkdir -p profiles/${BATCH_ID}
find ../../backend/${BATCH_ID}/ -type f -name "*.csv" -exec profiling-recipe/scripts/csv2gz.py {} \;
rsync -arzv --include="*/" --include="*.gz" --exclude "*" ../../backend/${BATCH_ID}/ profiles/${BATCH_ID}/
5.6.15. Run the profiling workflow¶
Especially for large number of plates, this will take some time. Output will be logged to the console as different steps proceed.
python profiling-recipe/profiles/profiling_pipeline.py --config config_files/{$CONFIG_FILE}.yml
5.6.16. Push resulting files back up to GitHub¶
If using a data repository, push the newly created profiles to DVC and the .dvc files and other files to GitHub as follows
dvc add profiles/${BATCH} --recursive
dvc push
git add profiles/${BATCH}/*.dvc profiles/*.gitignore
git commit -m 'add profiles'
git add *
git commit -m 'add files made in profiling'
git push
If not using DVC but using a data repository, push all new files to GitHub as follows
git add *
git commit -m 'add profiles for batch _'
git push
5.6.17. Push resulting files up to S3¶
parallel aws s3 sync {1} s3://${BUCKET}/projects/${PROJECT_NAME}/workspace/{1} ::: config_files gct profiles quality_control