CoNNECT Support Tools¶
Several custom support tools have been created to aid in the batch processing. These tools support parallel processing via HTCondor and SGE implemented in the custom batch processing described in the previous section.
Note
FIGURE TO BE INSERTED
bids¶
Note
the functions of support_tools.bids do not support command line operation.
Python Implementation¶
- support_tools.bids.get_bids_filename(subject: str | None = None, session: str | None = None, acquisition: str | None = None, task: str | None = None, direction: str | None = None, run: str | None = None, process: str | None = None, resolution: str | None = None, space: str | None = None, description: str | None = None, suffix: str | None = None, extension: str | None = None) str ¶
return bids compliant filename from input labels
get_bids_filename(subject=SUBJECT,session=SESSION,acquisition=ACQUISITION,task=TASK,direction=DIRECTION,run=RUN,process=PROCESS,resolution=RESOLUTION,space=SPACE,description=DESCRIPTION,suffix=SUFFIX,extension=EXTENSION)
- Parameters:
subject (str, optional) – subject identifier, defaults to None
session (str, optional) – session identifier, defaults to None
acquisition (str, optional) – acquisition identifier, defaults to None
task (str, optional) – task identifier, defaults to None
direction (str, optional) – direction identifier, defaults to None
run (str, optional) – run identifier, defaults to None
process (str, optional) – process identifier, defaults to None
resolution (str, optional) – resolution identifier, defaults to None
space (str, optional) – space identifier, defaults to None
description (str, optional) – description identifier, defaults to None
suffix (str, optional) – suffix identifier, defaults to None
extension (str, optional) – file externsion, defaults to None
- Returns:
BIDS-compliant filename
- Return type:
condor¶
Note
the functions of support_tools.condor do not support command line operation.
Python Implementation¶
- support_tools.condor.create_bin_condor_job(jobName: str, exeName: str, machineNames: list, submit: str, error: str, output: str, log: str, dagman: str, docker: bool = False, docker_image: str = 'wsuconnect/neuro', docker_mount_if: str | None = None, request_cpus: int = 1, request_memory: str = '5g')¶
Creates a pycondor job object to implement a binary from /usr/bin.
create_bin_condor_job(jobName,exeName,machineNames,submit,error,output,log,dagman,docker=False,docker_image=’wsuconenct/neuro’,docker_mount_if=’resshare20’,request_cpus=1,request_memory=’5g’)
- Parameters:
jobName (str) – name for the parallel htcondor job
exeName (str) – helper_function executable name
machineNames (list) – list of machine names to to execute jobs on
submit (str) – fullpath to condor submit file
error (str) – fullpath to condor error file
output (str) – fullpath to condor output file
log (str) – fullpath to condor log file
dagman (pycondor.Dagman) – pointer to created pycondor Dagman object
docker (bool, optional) – flag to execute job using a docker container (default False), defaults to False
docker_image (str, optional) – docker image for docker execution, defaults to ‘wsuconnect/neuro’
docker_mount_if (str, optional) – docker share to mount (see credentials.json), defaults to None
request_cpus (int, optional) – number of cpus to allocate to docker container, defaults to 1
request_memory (str, optional) – memory in bytes to allocate to docker container, defaults to ‘5g’
- Returns:
pointer to a configured pycondor Job object
- Return type:
pycondor.Job
- support_tools.condor.create_freesurfer_condor_job(jobName: str, exeName: str, machineNames: list, submit: str, error: str, output: str, log: str, dagman: str, docker: bool = False, docker_image: str = 'wsuconnect/neuro', docker_mount_if: str | None = None, request_cpus: int = 1, request_memory: str = '5g')¶
Creates a pycondor job object to implement FreeSurfer functions.
create_freesurfer_condor_job(jobName,exeName,machineNames,submit,error,output,log,dagman,docker=False,docker_image=’wsuconenct/neuro’,docker_mount_if=’resshare20’,request_cpus=1,request_memory=’5g’)
- Parameters:
jobName (str) – name for the parallel htcondor job
exeName (str) – helper_function executable name
machineNames (list) – list of machine names to to execute jobs on
submit (str) – fullpath to condor submit file
error (str) – fullpath to condor error file
output (str) – fullpath to condor output file
log (str) – fullpath to condor log file
dagman (pycondor.Dagman) – pointer to created pycondor Dagman object
docker (bool, optional) – flag to execute job using a docker container (default False), defaults to False
docker_image (str, optional) – docker image for docker execution, defaults to ‘wsuconnect/neuro’
docker_mount_if (str, optional) – docker share to mount (see credentials.json), defaults to None
request_cpus (int, optional) – number of cpus to allocate to docker container, defaults to 1
request_memory (str, optional) – memory in bytes to allocate to docker container, defaults to ‘5g’
- Returns:
pointer to a configured pycondor Job object
- Return type:
pycondor.Job
- support_tools.condor.create_fsl_condor_job(jobName: str, exeName: str, machineNames: list, submit: str, error: str, output: str, log: str, dagman: str, docker: bool = False, docker_image: str = 'wsuconnect/neuro', docker_mount_if: str | None = None, request_cpus: int = 1, request_memory: str = '5g')¶
Creates a pycondor job object to implement FSL functions.
create_fsl_condor_job(jobName,exeName,machineNames,submit,error,output,log,dagman,docker=False,docker_image=’wsuconenct/neuro’,docker_mount_if=’resshare20’,request_cpus=1,request_memory=’5g’)
- Parameters:
jobName (str) – name for the parallel htcondor job
exeName (str) – helper_function executable name
machineNames (list) – list of machine names to to execute jobs on
submit (str) – fullpath to condor submit file
error (str) – fullpath to condor error file
output (str) – fullpath to condor output file
log (str) – fullpath to condor log file
dagman (pycondor.Dagman) – pointer to created pycondor Dagman object
docker (bool, optional) – flag to execute job using a docker container (default False), defaults to False
docker_image (str, optional) – docker image for docker execution, defaults to ‘wsuconnect/neuro’
docker_mount_if (str, optional) – docker share to mount (see credentials.json), defaults to None
request_cpus (int, optional) – number of cpus to allocate to docker container, defaults to 1
request_memory (str, optional) – memory in bytes to allocate to docker container, defaults to ‘5g’
- Returns:
pointer to a configured pycondor Job object
- Return type:
pycondor.Job
- support_tools.condor.create_python_condor_job(jobName: str, exeName: str, machineNames: list, submit: str, error: str, output: str, log: str, dagman: str, docker: bool = False, docker_image: str = 'wsuconnect/neuro', docker_mount_if: str | None = None, request_cpus: int = 1, request_memory: str = '5g')¶
Creates a pycondor job object to implement python functions from support_tools.
create_python_condor_job(jobName,exeName,machineNames,submit,error,output,log,dagman,docker=False,docker_image=’wsuconenct/neuro’,docker_mount_if=’resshare20’,request_cpus=1,request_memory=’5g’)
- Parameters:
jobName (str) – name for the parallel htcondor job
exeName (str) – helper_function executable name
machineNames (list) – list of machine names to to execute jobs on
submit (str) – fullpath to condor submit file
error (str) – fullpath to condor error file
output (str) – fullpath to condor output file
log (str) – fullpath to condor log file
dagman (pycondor.Dagman) – pointer to created pycondor Dagman object
docker (bool, optional) – flag to execute job using a docker container (default False), defaults to False
docker_image (str, optional) – docker image for docker execution, defaults to ‘wsuconnect/neuro’
docker_mount_if (str, optional) – docker share to mount (see credentials.json), defaults to None
request_cpus (int, optional) – number of cpus to allocate to docker container, defaults to 1
request_memory (str, optional) – memory in bytes to allocate to docker container, defaults to ‘5g’
- Returns:
pointer to a configured pycondor Job object
- Return type:
pycondor.Job
- support_tools.condor.create_python_venv_condor_job(jobName: str, exeName: str, machineNames: list, submit: str, error: str, output: str, log: str, dagman: str, docker: bool = False, docker_image: str = 'wsuconnect/neuro', docker_mount_if: str | None = None, request_cpus: int = 1, request_memory: str = '5g')¶
Creates a pycondor job object to implement python functions from wsuconnect/python3_venv.
create_python_venv_condor_job(jobName,exeName,machineNames,submit,error,output,log,dagman,docker=False,docker_image=’wsuconenct/neuro’,docker_mount_if=’resshare20’,request_cpus=1,request_memory=’5g’)
- Parameters:
jobName (str) – name for the parallel htcondor job
exeName (str) – helper_function executable name
machineNames (list) – list of machine names to to execute jobs on
submit (str) – fullpath to condor submit file
error (str) – fullpath to condor error file
output (str) – fullpath to condor output file
log (str) – fullpath to condor log file
dagman (pycondor.Dagman) – pointer to created pycondor Dagman object
docker (bool, optional) – flag to execute job using a docker container (default False), defaults to False
docker_image (str, optional) – docker image for docker execution, defaults to ‘wsuconnect/neuro’
docker_mount_if (str, optional) – docker share to mount (see credentials.json), defaults to None
request_cpus (int, optional) – number of cpus to allocate to docker container, defaults to 1
request_memory (str, optional) – memory in bytes to allocate to docker container, defaults to ‘5g’
- Returns:
pointer to a configured pycondor Job object
- Return type:
pycondor.Job
convert_dicoms.py¶
convert_dicoms.py¶
usage: convert_dicoms.py [-h] -i INDIR [--progress]
- -h, --help¶
show this help message and exit
- -i <indir>, --in-dir <indir>¶
(str) fullpath to a sourcedata directory containing DICOM images
- --progress¶
(bool) run in verbose mode
Python Implementation¶
- support_tools.convert_dicoms.convert_dicoms(INDIR: str, PROGRESS: bool = False)¶
This function converts sourcedata DICOM images to NIfTI images in the input sourcedata directory. This function requires the installation of dcm2niix. Converted NIfTI images are placed in the directory containing the imput sourcedata directory.
convert_dicoms(INDIR,PROGRESS=False)
copy_dirs.py¶
Note
the function of support_tools.copy_dirs does not support command line operation.
Python Implementation¶
dti_flirt.py¶
dti_flirt.py¶
usage: dti_flirt.py [-h] -i IN_FILE -d DATA_DIR -f FLIRT_PARAMS [--flirt-params FLIRT_PARAMS] [--overwrite] [--progress]
- -h, --help¶
show this help message and exit
- -i <in_file>¶
Input DTI file
- -d <data_dir>¶
Project’s directory on disk (see dataDir key in credentials JSON file)
- -f <flirt_params>¶
Path to the FLIRT parameter JSON control file
- --flirt-params <flirt_params>¶
- --overwrite¶
- --progress¶
Python Implementation¶
- support_tools.dti_flirt.dti_flirt(IN_FILE: str, DATA_DIR: str, FLIRT_PARAMS: str, overwrite: bool = False, progress: bool = False)¶
This function performs FLIRT registration between IN_FILE and structural/standard brain images similar to flirt.py. Registration will be applied to all secondary images in the same directory as IN_FILE.
dti_flirt(IN_FILE,DATA_DIR,FLIRT_PARAMS,overwrite=False,progress=False)
- Parameters:
IN_FILE (str) – fullpath to a NIfTI file
DATA_DIR (str) – fullpath to the project’s data directory (project’s ‘dataDir’ credential)
FLIRT_PARAMS (str) – fullpath to project’s 2D ASL FLIRT parameter file
overwrite (bool) – OPTIONAL flag to overwrite existing files (default=False)
progress (bool) – OPTIONAL flag to display command line output providing additional details on the processing status
- Raises:
FileNotFoundError – FLIRT_PARAMS file not found on disk
Exception – general error encountered during execution
- Returns:
None
dti_preprocess.py¶
Python Implementation¶
evaluate_raw_file_transfer.py¶
Note
the function of support_tools.evaluate_raw_file_transfer does not support command line operation.
Python Implementation¶
- support_tools.evaluate_raw_file_transfer.evaluate_raw_file_transfer(project: str, inDir: str) bool ¶
This program evaluates files appearing in a single subject/session rawdata directory (inDir). Filenames expected to appear in the rawdata directory are determined from the project’s scan_id.json file found in the project’s code directory.
evaluate_raw_file_transfer(project,inDir)
feat_full_firstlevel.py¶
Python Implementation¶
- support_tools.feat_full_firstlevel.feat_full_firstlevel(DATADIR: str, SUBNAME: str, SESNUM: str, FEATDESIGNDIR: str, FEATOUTPUTDIR: str, DESIGNBASENAME: str, reference: str | None = None, step2_design: str | None = None, struc_reg_matrix: str | None = None, progress: bool = False)¶
This function moves or copies input (source) directories to a single output directory.
feat_full_firstlevel(DATADIR,SUBNAME,SESNUM,FEATDESIGNDIR,FEATOUTPUTDIR,DESIGNBASENAME,reference=None,step2_design=None,struc_reg_matrix=None,progress=False)
- Parameters:
DATADIR (str) – fullpath to the project’s data directory (<dataDir> key in credentials.json)
SUBNAME (str) – subject identifier
SESNUM (str) – session identifier
FEATDESIGNDIR (str) – directory containing the template feat designs
FEATOUTPUTDIR (str) – FEAT output directory
DESIGNBASENAME (str) – basename (no extension) for the design file for step 1
reference (str, optional) – fullpath to T1 reference image, defaults to None ##DEPRECATED
step2_design_basename (str, optional) – basename (no extension) for the design file for step 2, defaults to None
struc_reg_matrix (str, optional) – fullpath to the registration matrix between structural (T1) and standard atlas, defaults to None
progress (bool, optional) – flag to display command line output providing additional details on the processing status, defaults to False
- Raises:
Exception – general error encountered during execution
flirt_pngappend.py¶
Note
the function of support_tools.evaluate_raw_file_transfer does not support command line operation.
Python Implementation¶
- support_tools.flirt_pngappend.flirt_pngappend(in_file: str, reference: str, out_file: str)¶
Creates a registration overlay image (PNG).
Top row contains a background image of in_file in reference space overlaid with the edges of reference Bottom row contains a background image of reference overlaid with the edges of in_file in reference space
flirt_pngappend(in_file,reference,out_file)
flirt.py¶
Python Implementation¶
fsreconall_stage1.py¶
Python Implementation¶
- exception support_tools.fsreconall_stage1.InvalidJsonInput¶
Raised when the input JSON control file does not contain the appropriate mandatory definitions
- __weakref__¶
list of weak references to the object (if defined)
- support_tools.fsreconall_stage1.fsreconall_stage1(IN_FILE: str, DATA_DIR: str, RECONALL_PARAMS: str, MAINRECONALLOUTPUTDIR: str, directive: str | None = None, overwrite: bool = False, progress: bool = False)¶
This function performs stage 1 of FreeSurfer reconall freesurfer_recon-all_input.json control file.
fsreconall_stage1(IN_FILE,DATA_DIR,RECONALL_PARAMS,AINRECONALLOUTPUTDIR,directive=None,overwrite=False,progress=False)
- Parameters:
:param MAINRECONALLOUTPUTDIR:fullpath to the desired FreeSurfer SUBJECTS_DIR :type MAINRECONALLOUTPUTDIR: str
- Parameters:
- Raises:
InvalidJsonInput – error encountered when the input JSON control file does not contain the appropriate mandatory definitions
OSError – OS error encountered during execution
Exception – general error encountered during execution
fsreconall_stage2.py¶
Python Implementation¶
- support_tools.fsreconall_stage2.fsreconall_stage2(STAGE1_DIR: str, DATA_DIR: str, RECONALL_PARAMS: str, overwrite: bool = False, progress: bool = False)¶
This function performs stage 2 and 3 of FreeSurfer reconall freesurfer_recon-all_input.json control file.
fsreconall_stage2(IN_FILE,DATA_DIR,RECONALL_PARAMS,AINRECONALLOUTPUTDIR,directive=None,overwrite=False,progress=False)
- Parameters:
STAGE1_DIR (str) – fullpath to a single subject/session reconall Stage 1 directory
DATA_DIR (str) – fullpath to the project’s data directory (project’s ‘dataDir’ credential)
RECONALL_PARAMS (str) – fullpath to project’s FreeSurfer RECON-ALL parameter control file
overwrite (bool, optional) – flag to overwrite existing files, defaults to False
progress (bool, optional) – flag to display command line output providing additional details on the processing status, defaults to False
- Raises:
InvalidJsonInput – error encountered when the input JSON control file does not contain the appropriate mandatory definitions
Exception – general error encountered during execution
get_scan_id.py¶
Note
the function of support_tools.get_scan_id does not support command line operation.
Python Implementation¶
- support_tools.get_scan_id.get_scan_id(inDir: str, basename: str) Tuple[str, str, dict] ¶
Get metadata from a source NIfTI file
This program return associated scan information into the custom
get_scan_id(inDir,basename)
- Parameters:
- Raises:
Exception – general error encountered during execution
- Returns:
scan name as identified in the Project’s scan_id JSON file within the Project’s ‘code’ directory, bids directory as identified in the Project’s scan_id JSON file within the Project’s ‘code’ directory, scan identifier keys from the scan_id.json control file
- Return type:
mysql.py¶
Note
the functions of support_tools.mysql do not support command line operation.
Python Implementation¶
- support_tools.mysql.create_mysql_connection(host_name: str, user_name: str, user_password: str, db_name: str, progress, unix_socket: str = '/var/run/mysqld/mysqld.sock') Connection ¶
This function creates a MySql connection via pymysql.
- Parameters:
host_name (str) – _description_
user_name (str) – _description_
user_password (str) – _description_
db_name (str)
progress (bool) – flag to display command line output providing additional details on the processing status
unix_socket (str, optional) – path to pysqld.sock, defaults to ‘/var/run/mysqld/mysqldsock’
- Raises:
Exception – general error encountered during execution
- Returns:
_description_
- Return type:
pymysql.connecttion.Connection
- support_tools.mysql.query_file(regex: str, returncol: str = 'fullpath', orderby: str = 'fullpath', inclusion: str | list | None = None, exclusion: str | list | None = None, progress: bool = False) str ¶
This function queries the table specified in the support_tools.creds.searchTable object for a single file. The fullpath to the single file is returned.
query_file(regexStr,PROGRESS=False)
- Parameters:
regex (str) – search string for query
returncol (str, optional) – table column to return in query, defaults to ‘fullpath’
orderby (str, optional) – solumn to sort returns by, defaults to ‘fullpath’
inclusion (str | list, optional) – string or list of strings that are additionally required for an item be returned, defaults to None
exclusion (str | list, optional) – string or list of strings that will exclude items from being returned, defaults to None
progress (bool, optional) – flag to display command line output providing additional details on the processing status, defaults to False
- Returns:
output table column from all items matching the specified search criteria
- Return type:
- support_tools.mysql.query_source_file(regex: str, returncol: str = 'fullpath', orderby: str = 'fullpath', inclusion: str | list | None = None, exclusion: str | list | None = None, progress: bool = False) str ¶
This function queries the table specified in the support_tools.creds.searchTable object for a single file. The fullpath to the single file is returned.
query_file(regexStr,PROGRESS=False)
- Parameters:
regex (str) – search string for query
returncol (str, optional) – table column to return in query, defaults to ‘fullpath’
orderby (str, optional) – solumn to sort returns by, defaults to ‘fullpath’
inclusion (str | list, optional) – string or list of strings that are additionally required for an item be returned, defaults to None
exclusion (str | list, optional) – string or list of strings that will exclude items from being returned, defaults to None
progress (bool, optional) – flag to display command line output providing additional details on the processing status, defaults to False
- Returns:
output table column from all items matching the specified search criteria
- Return type:
- support_tools.mysql.sql_check_table_exists(sqlCursor: Cursor, table: str) bool ¶
This function checks the database pointed to by the pymysql.cursors.Cursor object for the requested table.
sql_check_table_exists(sqlCursor, table)
- support_tools.mysql.sql_create_project_tables(progress: bool = False)¶
This function creates the searchTable and searchSourceTable tables in the database for a given project, all specified in support_tools.creds object.
- Parameters:
progress (bool, optional) – flag to display command line output providing additional details on the processing status, defaults to False
- support_tools.mysql.sql_multiple_query(searchtable: str, regex: str, database: str = 'CoNNECT', returncol: str = 'fullpath', searchcol: str = 'filename', orderby: str = 'fullpath', progress: bool = False) str ¶
Find all items in a MySql table that match the specified search string regex.
- Parameters:
searchtable (str) – MySql table inside of the specified database
regex (str) – string to match in query
database (str, optional) – MySql database containing table, defaults to ‘CoNNECT’
returncol (str, optional) – table column to return in query, defaults to ‘fullpath’
searchcol (str, optional) – table column to perform query, defaults to ‘filename’
orderby (str, optional) – solumn to sort returns by, defaults to ‘fullpath’
progress (bool, optional) – flag to display command line output providing additional details on the processing status, defaults to False
- Returns:
output table column from all items matching the specified search criteria
- Return type:
- support_tools.mysql.sql_query(searchtable: str, regex: str, database: str = 'CoNNECT', returncol: str = 'fullpath', searchcol: str = 'filename', orderby: str = 'fullpath', inclusion: str | list | None = None, exclusion: str | list | None = None, orinclusion: str | list | None = None, progress: bool = False) str ¶
Find all items in a MySql table that match the specified search string regex.
- Parameters:
searchtable (str) – MySql table inside of the specified database
regex (str) – string to match in query
database (str, optional) – MySql database containing table, defaults to ‘CoNNECT’
returncol (str, optional) – table column to return in query, defaults to ‘fullpath’
searchcol (str, optional) – table column to perform query, defaults to ‘filename’
orderby (str, optional) – solumn to sort returns by, defaults to ‘fullpath’
inclusion (str | list, optional) – string or list of strings that are additionally required for an item be returned, defaults to None
exclusion (str | list, optional) – string or list of strings that will exclude items from being returned, defaults to None
progress (bool, optional) – flag to display command line output providing additional details on the processing status, defaults to False
- Returns:
output table column from all items matching the specified search criteria
- Return type:
- support_tools.mysql.sql_query_dir_check(regex: str, targetdir: str, progress: bool = False) bool ¶
This function ealuates the presence of files in a specified directory.
- Parameters:
- Returns:
presencee (True) or absence (False) of files matching the search string in the specified directory
- Return type:
- support_tools.mysql.sql_query_dirs(regex: str, source: bool = False, inclusion: str | list | None = None, exclusion: str | list | None = None, progress: bool = False) list ¶
Find all unique directories containing files in a MySql table that match the specified search string regex.
- Parameters:
regex (str) – string to match in query
source (str, optional) – Utilize searchSourceTable instead of searchTable support_tools.creds object, defaults to False
inclusion (str | list, optional) – string or list of strings that are additionally required for an item be returned, defaults to None
exclusion (str | list, optional) – string or list of strings that will exclude items from being returned, defaults to None
progress (bool, optional) – flag to display command line output providing additional details on the processing status, defaults to False
- Returns:
list of fullpath to unique directories containing files
- Return type:
- support_tools.mysql.sql_table_insert(table: str, item: dict, progress: bool = False)¶
This function inserts entry(ies) from item into table from the database specified in support_tools.creds object.
sql_table_remove(table,item,progress=True)
prepare_examcard_html.py¶
Note
the function of support_tools.prepare_examcard_html does not support command line operation.
Python Implementation¶
- support_tools.prepare_examcard_html.prepare_examcard_html()¶
This function .
query_file(INDIR,PROGRESS=False)
- Parameters:
INDIR (str) – fullpath to a sourcedata directory containing DICOM images for conversion
PROGRESS (BOOL) – OPTIONAL flag to display command line output providing additional details on the processing status
- Returns:
None
remove_dirs.py¶
Note
the function of support_tools.remove_dirs does not support command line operation.
Python Implementation¶
RestToolbox.py¶
Note
the functions of support_tools.RestToolbox do not support command line operation.
Python Implementation¶
- support_tools.RestToolbox.DoDelete(uri)¶
Perform an HTTP DELETE on the PACS server
- support_tools.RestToolbox.DoGet(uri, data={}, interpretAsJson=True)¶
Performs an HTTP GET request on the PACS server
- Parameters:
- Raises:
Exception – generic error connecting to PACS server
- Returns:
results of the HTTP GET request on the PACS server
- Return type:
- support_tools.RestToolbox.DoPost(uri, data={}, contentType='')¶
Performs an HTTP POST request on the PACS server
- Parameters:
- Raises:
Exception – generic error connecting to PACS server
- Returns:
results of the HTTP PUT or POST request on the PACS server
- Return type: