cpac (Python package)¶
cpac is available so that you can easily run analyses without needing interact with the container platform that allows you to run C-PAC without installing all of the underlying software.
cpac requires Python 3.6 or greater. To get cpac, simply
pip install cpac
Download / Upgrade C-PAC with cpac¶
To download or upgrade a particular C-PAC image,
cpac pull
or
cpac upgrade
When downloading/upgrading, the --platform, --image, and --tag let you specify platform (Docker or Singularity), image (Docker image name or URL to image in repository), and tag (version tag, currently only for Docker repositories), respectively.
For example, a development Docker image can be downloaded with
cpac --platform docker --tag nightly pull
Or a Singularity image built from that Docker image can be downloaded with
cpac --platform singularity --tag nightly pull
Run C-PAC with cpac¶
To run C-PAC in participant mode for one participant, using a BIDS dataset stored on your machine or server and using the container image’s default pipeline configuration:
cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant
By default, cpac (the wrapper) will try Docker first and fall back to Singularity if Docker fails. If both fail, an exception is raised.
You can specify a platform with the --platform docker or --platform singularity. If you specify a platform without specifying an image, these are the defaults, using the first successfully found image:
Usage¶
$ cpac --help
usage: cpac [-h] [--version] [-o OPT] [-B CUSTOM_BINDING]
[--platform {docker,singularity}] [--image IMAGE] [--tag TAG]
[--working_dir PATH] [-v] [-vv]
{run,group,utils,pull,upgrade,crash} ...
cpac: a Python package that simplifies using C-PAC <http://fcp-indi.github.io> containerized images.
This commandline interface package is designed to minimize repetition.
As such, nearly all arguments are optional.
When launching a container, this package will try to bind any paths mentioned in
• the command
• the data configuration
An example minimal run command:
cpac run /path/to/data /path/for/outputs
An example run command with optional arguments:
cpac -B /path/to/data/configs:/configs \
--image fcpindi/c-pac --tag latest \
run /path/to/data /path/for/outputs \
--data_config_file /configs/data_config.yml \
--save_working_dir
Each command can take "--help" to provide additonal usage information, e.g.,
cpac run --help
positional arguments:
{run,group,utils,pull,upgrade,crash}
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-o OPT, --container_option OPT
parameters and flags to pass through to Docker or Singularity
This flag can take multiple arguments so cannot be
the final argument before the command argument (i.e.,
run or any other command that does not start with - or --)
-B CUSTOM_BINDING, --custom_binding CUSTOM_BINDING
directories to bind with a different path in
the container than the real path of the directory.
One or more pairs in the format:
real_path:container_path
(eg, /home/C-PAC/run5/outputs:/outputs).
Use absolute paths for both paths.
This flag can take multiple arguments so cannot be
the final argument before the command argument (i.e.,
run or any other command that does not start with - or --)
--platform {docker,singularity}
If neither platform nor image is specified,
cpac will try Docker first, then try
Singularity if Docker fails.
--image IMAGE path to Singularity image file OR name of Docker image (eg, "fcpindi/c-pac").
Will attempt to pull from Singularity Hub or Docker Hub if not provided.
If image is specified but platform is not, platform is
assumed to be Singularity if image is a path or
Docker if image is an image name.
--tag TAG tag of the Docker image to use (eg, "latest" or "nightly").
--working_dir PATH working directory
-v, --verbose set loglevel to INFO
-vv, --very-verbose set loglevel to DEBUG
--platform docker¶
Look for
fcpindi/c-pac:latestlocally.Pull
fcpindi/c-pac:latestfrom Docker Hub.
--platform singularity¶
Look in the present working directory for any Singularity images. If more than one is found, use the most recently modified.
Pull
FCP-INDI/C-PACfrom Singularity Hub.Pull
fcpindi/c-pac:latestfrom Docker Hub and convert to a Singularity image.
You can also specify a container image with an --image argument, passing an image name (e.g., fcpindi/c-pac) for a Docker image or a filepath (e.g. ~/singularity_images/C-PAC.sif) for a Singularity image. You can also specify a --tag (e.g., latest or nightly).
You can also provide a link to an AWS S3 bucket containing a BIDS directory as the data source:
cpac run s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS /Users/You/some_folder_for_outputs participant
In addition to the default pipeline, C-PAC comes packaged with a growing library of pre-configured pipelines that are ready to use. To run C-PAC with one of the pre-packaged pre-configured pipelines, simply invoke the --preconfig flag, shown below. See the full selection of pre-configured pipelines here.
cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs --preconfig anat-only
To run C-PAC with a pipeline configuration file other than one of the pre-configured pipelines, assuming the configuration file is in the /Users/You/Documents directory:
cpac run /Users/You/local_bids_data /Users/You/some_folder_for_outputs participant --pipeline_file /Users/You/Documents/pipeline_config.yml
Finally, to run C-PAC with a specific data configuration file (instead of providing a BIDS data directory):
cpac run /Users/You/any_directory /Users/You/some_folder_for_outputs participant --data_config_file /Users/You/Documents/data_config.yml
Note: we are still providing the postionally-required bids_dir input parameter. However C-PAC will not look for data in this directory when you provide a data configuration YAML with the --data_config_file flag. Providing . or $PWD will simply pass the present working directory. In addition, if the dataset in your data configuration file is not in BIDS format, just make sure to add the --skip_bids_validator flag at the end of your command to bypass the BIDS validation process.
The full list of parameters and options that can be passed to C-PAC are shown below:
Usage: cpac run¶
$ cpac run --help
Loading 🐳 Docker
Loading 🐳 fcpindi/c-pac:latest with these directory bindings:
local Docker mode
---------------------------- -------------------- ------
/home/circleci/build /home/circleci/build rw
/home/circleci/build /tmp rw
/home/circleci/build/log /crash rw
/home/circleci/build/outputs /output rw
Logging messages will refer to the Docker paths.
usage: run.py [-h] [--pipeline_file PIPELINE_FILE] [--group_file GROUP_FILE]
[--data_config_file DATA_CONFIG_FILE] [--preconfig PRECONFIG]
[--aws_input_creds AWS_INPUT_CREDS]
[--aws_output_creds AWS_OUTPUT_CREDS] [--n_cpus N_CPUS]
[--mem_mb MEM_MB] [--mem_gb MEM_GB]
[--save_working_dir [SAVE_WORKING_DIR]] [--disable_file_logging]
[--participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]
[--participant_ndx PARTICIPANT_NDX] [-v]
[--bids_validator_config BIDS_VALIDATOR_CONFIG]
[--skip_bids_validator] [--anat_only] [--tracking_opt-out]
[--monitoring]
bids_dir output_dir {participant,group,test_config,cli}
C-PAC Pipeline Runner
positional arguments:
bids_dir The directory with the input dataset formatted
according to the BIDS standard. Use the format
s3://bucket/path/to/bidsdir to read data directly from
an S3 bucket. This may require AWS S3 credentials
specified via the --aws_input_creds option.
output_dir The directory where the output files should be stored.
If you are running group level analysis this folder
should be prepopulated with the results of the
participant level analysis. Use the format
s3://bucket/path/to/bidsdir to write data directly to
an S3 bucket. This may require AWS S3 credentials
specified via the --aws_output_creds option.
{participant,group,test_config,cli}
Level of the analysis that will be performed. Multiple
participant level analyses can be run independently
(in parallel) using the same output_dir. test_config
will run through the entire configuration process but
will not execute the pipeline.
optional arguments:
-h, --help show this help message and exit
--pipeline_file PIPELINE_FILE
Path for the pipeline configuration file to use. Use
the format s3://bucket/path/to/pipeline_file to read
data directly from an S3 bucket. This may require AWS
S3 credentials specified via the --aws_input_creds
option.
--group_file GROUP_FILE
Path for the group analysis configuration file to use.
Use the format s3://bucket/path/to/pipeline_file to
read data directly from an S3 bucket. This may require
AWS S3 credentials specified via the --aws_input_creds
option. The output directory needs to refer to the
output of a preprocessing individual pipeline.
--data_config_file DATA_CONFIG_FILE
Yaml file containing the location of the data that is
to be processed. Can be generated from the CPAC gui.
This file is not necessary if the data in bids_dir is
organized according to the BIDS format. This enables
support for legacy data organization and cloud based
storage. A bids_dir must still be specified when using
this option, but its value will be ignored. Use the
format s3://bucket/path/to/data_config_file to read
data directly from an S3 bucket. This may require AWS
S3 credentials specified via the --aws_input_creds
option.
--preconfig PRECONFIG
Name of the pre-configured pipeline to run.
--aws_input_creds AWS_INPUT_CREDS
Credentials for reading from S3. If not provided and
s3 paths are specified in the data config we will try
to access the bucket anonymously use the string "env"
to indicate that input credentials should read from
the environment. (E.g. when using AWS iam roles).
--aws_output_creds AWS_OUTPUT_CREDS
Credentials for writing to S3. If not provided and s3
paths are specified in the output directory we will
try to access the bucket anonymously use the string
"env" to indicate that output credentials should read
from the environment. (E.g. when using AWS iam roles).
--n_cpus N_CPUS Number of execution resources per participant
available for the pipeline. This flag takes precidence
over max_cores_per_participant in the pipeline
configuration file.
--mem_mb MEM_MB Amount of RAM available per participant in megabytes.
Included for compatibility with BIDS-Apps standard,
but mem_gb is preferred. This flag takes precedence
over maximum_memory_per_participant in the pipeline
configuration file.
--mem_gb MEM_GB Amount of RAM available per participant in gigabytes.
If this is specified along with mem_mb, this flag will
take precedence. This flag also takes precedence over
maximum_memory_per_participant in the pipeline
configuration file.
--save_working_dir [SAVE_WORKING_DIR]
Save the contents of the working directory.
--disable_file_logging
Disable file logging, this is useful for clusters that
have disabled file locking.
--participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]
The label of the participant that should be analyzed.
The label corresponds to sub-<participant_label> from
the BIDS spec (so it does not include "sub-"). If this
parameter is not provided all participants should be
analyzed. Multiple participants can be specified with
a space separated list. To work correctly this should
come at the end of the command line.
--participant_ndx PARTICIPANT_NDX
The index of the participant that should be analyzed.
This corresponds to the index of the participant in
the data config file. This was added to make it easier
to accommodate SGE array jobs. Only a single
participant will be analyzed. Can be used with
participant label, in which case it is the index into
the list that follows the participant_label flag. Use
the value "-1" to indicate that the participant index
should be read from the AWS_BATCH_JOB_ARRAY_INDEX
environment variable.
-v, --version show program's version number and exit
--bids_validator_config BIDS_VALIDATOR_CONFIG
JSON file specifying configuration of bids-validator:
See https://github.com/bids-standard/bids-validator
for more info.
--skip_bids_validator
Skips bids validation.
--anat_only run only the anatomical preprocessing
--tracking_opt-out Disable usage tracking. Only the number of
participants on the analysis is tracked.
--monitoring Enable monitoring server on port 8080. You need to
bind the port using the Docker flag "-p".
Usage: cpac utils¶
$ cpac utils --help
Loading 🐳 Docker
Loading 🐳 fcpindi/c-pac:latest with these directory bindings:
local Docker mode
---------------------------- -------------------- ------
/home/circleci/build /home/circleci/build rw
/home/circleci/build /tmp rw
/home/circleci/build/outputs /output rw
/home/circleci/build/log /logs rw
Logging messages will refer to the Docker paths.
Usage: run.py utils [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
crash
data_config
group_config
pipe_config
repickle
test
tools
workflows
Note that any of the optional arguments above will over-ride any pipeline settings in the default pipeline or in the pipeline configuration file you provide via the --pipeline_file parameter.
Further usage notes:
You can run only anatomical preprocessing easily, without modifying your data or pipeline configuration files, by providing the
--anat_onlyflag.As stated, the default behavior is to read data that is organized in the BIDS format. This includes data that is in Amazon AWS S3 by using the format
s3://<bucket_name>/<bids_dir>for thebids_dircommand line argument. Outputs can be written to S3 using the same format for theoutput_dir. Credentials for accessing these buckets can be specified on the command line (using--aws_input_credsor--aws_output_creds).When the app is run, a data configuration file is written to the working directory. This directory can be specified with
--working_diror the directory from which you runcpacwill be used. This file can be passed into subsequent runs, which avoids the overhead of re-parsing the BIDS input directory on each run (i.e. for cluster or cloud runs). These files can be generated without executing the C-PAC pipeline using thetest_runcommand line argument.The
participant_labelandparticipant_ndxarguments allow the user to specify which of the many datasets should be processed, which is useful when parallelizing the run of multiple participants.If you want to pass runtime options to your container plaform (Docker or Singularity), you can pass them with
-oor--container_options.
Footnotes
- *
The version restrictions for Singularity are specific to cpac the convenience wrapper. C-PAC itself should run on Singularity 2 or 3.
