CoNNECT Support Tools¶

Several custom support tools have been created to aid in the batch processing. These tools support parallel processing via HTCondor and SGE implemented in the custom batch processing described in the previous section.

Note

FIGURE TO BE INSERTED

bids_commands.py¶

get_bids_filename¶

get_bids_filename(*args, **kwargs)¶

Create a BIDS-compliant filename from the specified options.

get_scan_id(subject=SUBJECT, session=SESSION, task=TASK, acquisition=ACQUISITION, run=RUN, process=PROCESS, resolution=RESOLUTION, space=SPACE, description=DESCRIPTION, suffix=SUFFIX, extension=EXTENSION)

Parameters:

subject (str) – Optional subject identifier
session (str) – Optional session identifier
task (str) – Optional BIDS description (task-ABC)
acquisition (str) – BIDS description (acq-ABC)
run (str) – Optional BIDS description (run-ABC)
process (str) – Optional BIDS description (proc-ABC)
resolution (str) – Optional BIDS description (res-ABC)
space (str) – Optional BIDS description (space-ABC)
description (str) – Optional BIDS description (desc-ABC)
suffix (str) – Optional BIDS suffix
extension (str) – Optional filename extension (do not begin with ‘.’)

Raises:

Error – Any error occurs.

Returns:

BIDS compliant filename string

Return type:

str

get_bids_labels¶

get_bids_labels(IN_FILE)¶

Get BIDS-compliant labels from specified filename

get_bids_labels(IN_FILE)

Parameters:: IN_FILE (str) – Required fullpath or filename
Raises:: Error – Any error occurs.
Returns:: Dictionary containing extracted BIDS labels
Return type:: dict

convert_dicoms.py¶

create_fsl_condor_job.py¶

create_fsl_condor_job(inDir, basename)¶

Creates an HTCondor job for FSL commands.

create_fsl_condor_job(jobName, exeName, machineNames, submit, error, output, log, dagman)

Parameters:

jobName (str) – Required name for the parallel htcondor job
exeName (str) – FSL executable name
machineNames (list[str]) – list of machine names to execute jobs
submit (str) – fullpath to output submit file
error (str) – fullpath to output submit file
output (str) – fullpath to output submit file
log (str) – fullpath to output submit file
dagman (str) – Pycondor Dagman object

Raises:

Error – any encountered error

Returns:

pycondor Job object

Return type:

pycondor.Job

create_python_condor_job.py¶

create_python_condor_job(inDir, basename)¶

Creates an HTCondor job for helper_function commands.

create_python_condor_job(jobName, exeName, machineNames, submit, error, output, log, dagman)

Parameters:

jobName (str) – Required name for the parallel htcondor job
exeName (str) – FSL executable name
machineNames (list[str]) – list of machine names to execute jobs
submit (str) – fullpath to output submit file
error (str) – fullpath to output submit file
output (str) – fullpath to output submit file
log (str) – fullpath to output submit file
dagman (str) – Pycondor Dagman object

Raises:

Error – any encountered error

Returns:

pycondor Job object

Return type:

pycondor.Job

flirt.py¶

This support tool implements volume extraction (for 4D datasets), brain extraction, and/or registration via FSL’s FLIRT. Specifically, this tools supports paralleled computation implemented from connect_flirt. A flowchart showing the order of operations is depicted in Fig. 13.

Note

NEED TO PROVIDE A REFERENCE TO CONNECT_FLIRT.py

flirt_pngappend.py¶

flirt_pngappend(IN_FILE, REF_FILE, OUT_FILE)¶

Creates a registration overlay image (PNG). This tool is utilized by flirt.py.

Top row contains a background image of IN_FILE in REF_FILE space overlaid with the edges of REF_FILE (in red). Bottom row contains a background image of REF_FILE overlaid with the edges of IN_FILE in REF_FILE space (in red).

flirt_pngappend(IN_FILE, IN_FILE, OUT_FILE)

Parameters:

IN_FILE (str or None or Pathlike object) – REQUIRED String or pathlike object to an input NIfTI file transformed to the space of REF_FILE.
REF_FILE (str or None or Pathlike object) – REQUIRED String or pathlike object to a reference NIfTI file.
OUT_FILE (str or None or Pathlike object) – REQUIRED String or pathlike object to the output PNG file that will be created

Raises:

Error – If any error occurs.

Returns:

None

Return type:

None

get_scan_id.py¶

This function is currently used by connect_create_raw_nii.py.

get_scan_id(inDir, basename)¶

Search the metadata accompanying a sourcedata NIfTI file (found in the accompanying JSON sidecar) using the Project’s scan_id JSON file within the Project’s ‘code’ directory (found Scan Identification).

get_scan_id(inDir, basename)

Parameters:

inDir (str) – Required fullpath to sourcedata sub-directory containing NIfTI file
basename (str) – Required basename of the sourcedata NIfTI file

Raises:

Error – Any error occurs.

Returns:

scan name created from get_bids_filename found in bids_commands.py, bids directory as identified in the Project’s scan_id JSON file within the Project’s ‘code’ directory

Return type:

str, str

get_spec_base.py¶

get_spec_base(inDir, basename)¶

Get metadata from a MRS file.

This program returns the Project credentials into the custom specBase class (NEEDS REFERENCE), which should be imported prior to calling get_spec_base().

get_spec_base(specFile)

Parameters:: specFile (str) – Required path to a MRS file
Raises:: Error – Any error occurs
Returns:: None
Return type:: None

mysql_commands.py¶

sql_multiple_query¶

sql_multiple_query(*args, **kwargs)¶

Performs a query of the identified table and returns multiple columns that matched the search criteria.

Parameters:

database (string) – Optional “kind” of ingredients.
searchtable (string) – REQUIRED

Raises:

lumache.InvalidKindError – If the kind is invalid.

Returns:

The ingredients list.

Return type:

list[str]

query_file¶

query_source_file(regexStr, *args, **kwargs)¶

Performs a query into the Project’s searchTable’s ‘fullpath’ column using the search criteria identified for a single file. Returns a single filepath.

query_file(regexStr, database=’CoNNECT’, searchtable=None, returncol=’fullpath’, searchcol=’filename’, progress=False, orderby=’fullpath’, inclusion=None, exclusion=None, orinclusion=None)

Parameters:

regexStr (str) – Required initial search terms
returncol (str) – Optional table column to return (default ‘fullpath’)
progress (bool) – Optional operate in verbose mode
orderby (str) – Optional column to sort results (default ‘fullpath’)
inclusion (str or list[str]) – Optional inclusion criteria to refine results. These must also be contained within searchcol (AND operation)
exclusion (str or list[str]) – Optional exclusion criteria to refine results. These must not be within the searchcol
orinclusion (str or list[str]) – Optional inclusion criteria to refine results. These may also be contained within the search column (OR operation)

Raises:

Error – Any error occurs

Returns:

Fullpath to a single file if only 1 file matches the search criteria, otherwise None.

Return type:

None or str

query_source_files¶

query_source_file(regexStr, *args, **kwargs)¶

Performs a query into the Project’s searchSourceTable’s ‘fullpath’ column using the search criteria identified for a single file. Returns a single filepath.

query_source_file(regexStr, database=’CoNNECT’, searchtable=None, returncol=’fullpath’, searchcol=’filename’, progress=False, orderby=’fullpath’, inclusion=None, exclusion=None, orinclusion=None)

Parameters:

regexStr (str) – Required initial search terms
returncol (str) – Optional table column to return (default ‘fullpath’)
progress (bool) – Optional operate in verbose mode
orderby (str) – Optional column to sort results (default ‘fullpath’)
inclusion (str or list[str]) – Optional inclusion criteria to refine results. These must also be contained within searchcol (AND operation)
exclusion (str or list[str]) – Optional exclusion criteria to refine results. These must not be within the searchcol
orinclusion (str or list[str]) – Optional inclusion criteria to refine results. These may also be contained within the search column (OR operation)

Raises:

Error – Any error occurs

Returns:

Fullpath to a single sourcedata file if only 1 file matches the search criteria, otherwise None.

Return type:

None or str

sql_query_dir_check¶

sql_query_dirs(regexStr, dirToCheck, showProgress)¶

Queries a directory for existing NIfTI images.

sql_query_dirs(regexStr, dirToCheck, showProgress)

Parameters:

regexStr (str) – Required search string
dirToCheck (str) – Required directory to perform a NIfTI image search
showProgress (bool) – Required verbose mode

Raises:

Error – Any error occurs

Returns:

NIfTI files exist in <dirToCheck>

Return type:

bool

sql_query_dirs¶

sql_query_dirs(regexStr, showProgress, rawFlag, *args, **kwargs)¶

Performs a query into the provided table/column using the search criteria identified. Returns unique directories containing the identified files.

sql_query_dirs(regexStr, showProgress, rawFlag, inclusion=None, exclusion=None)

Parameters:

regexStr (str) – Required search string
showProgress (bool) – Required verbose mode
rawFlag (bool) – Required true for project’s searchSourceTable (default false - project’s searchTable)
inclusion (str or list[str]) – Optional inclusion criteria to refine results. These must also be contained within searchcol (AND operation)
exclusion (str or list[str]) – Optional exclusion criteria to refine results. These must not be within the searchcol

Raises:

Error – Any error occurs

Returns:

The list of directories containing files matching search criteria

Return type:

list[str]

sql_query¶

sql_query(*args, **kwargs)¶

Performs a query into the provided table/column using the search criteria identified.

sql_query(database=’CoNNECT’, searchtable=None, returncol=’fullpath’, searchcol=’filename’, regex=None, progress=False, orderby=’fullpath’, inclusion=None, exclusion=None, orinclusion=None)

Parameters:

database (str) – Optional MySQL database containing the project’s searchtable (default ‘CoNNECT’)
searchtable (str) – Required MySQL table to query
returncol (str) – Optional table column to return (default ‘fullpath’)
searchcol (str) – Optional table column to query (default ‘filepath’)
regex (str) – Required initial search terms
progress (bool) – Optional operate in verbose mode
orderby (str) – Optional column to sort results (default ‘fullpath’)
inclusion (str or list[str]) – Optional inclusion criteria to refine results. These must also be contained within searchcol (AND operation)
exclusion (str or list[str]) – Optional exclusion criteria to refine results. These must not be within the searchcol
orinclusion (str or list[str]) – Optional inclusion criteria to refine results. These may also be contained within the search column (OR operation)

Raises:

Error – Any error occurs

Returns:

The list of columns matching search criteria

Return type:

list[str]

sql_multiple_query¶

sql_multiple_query(*args, **kwargs)¶

Performs a query into the provided table/column using the search criteria identified.

sql_multiple_query(database=’CoNNECT’, searchtable=None, returncol=’fullpath’, searchcol=’filename’, regex=None, progress=False, orderby=’fullpath’)

Parameters:

database (str) – Optional MySQL database containing the project’s searchtable (default ‘CoNNECT’)
searchtable (str) – Required MySQL table to query
returncol (str) – Optional table column to return (default ‘fullpath’)
searchcol (str) – Optional table column to query (default ‘filepath’)
regex (str) – Required initial search terms
progress (bool) – Optional operate in verbose mode
orderby (str) – Optional column to sort results (default ‘fullpath’)

Raises:

Error – Any error occurs

Returns:

All the columns elements from each row where the input column matches the search criteria

Return type:

list[list[str]]

sql_create_project_tables¶

sql_create_project_tables(*args, **kwargs)¶

Creates the tables for a project whose credentials have been loaded into custom creds class via read_credentials().

sql_create_project_tables(progress=False)

Parameters:: progress (bool) – Optional operate in verbose mode
Raises:: Error – Any error occurs
Returns:: None
Return type:: None

sql_table_insert¶

sql_table_insert(table, item, *args, **kwargs)¶

Inserts items into a Project’s table. The project’s credentials must have been loaded into customcreds class via read_credentials().

sql_table_insert(table,item,progress=False)

Parameters:

table (str) – Required MySQL table name
item (dict or list) – Required elements to insert into the table
progress (bool) – Optional operate in verbose mode

Raises:

Error – Any error occurs

Returns:

None

Return type:

None

sql_table_remove¶

sql_table_insert(table, item, *args, **kwargs)¶

Removes items into a Project’s table. The project’s credentials must have been loaded into customcreds class via read_credentials().

sql_table_remove(table,item,progress=False)

Parameters:

table (str) – Required MySQL table name
item (dict or list) – Required elements to remove from the table
progress (bool) – Optional operate in verbose mode

Raises:

Error – Any error occurs

Returns:

None

Return type:

None

sql_check_table_exists¶

sql_check_table_exists(sqlCursor, table)¶

Checks if a table exists

sql_table_remove(sqlCursor, table)

Parameters:

sqlCursor (pymysql.connect.cursor) – Required pymysql connect cursor object
table (str) – Required MySQL table name

Raises:

Error – Any error occurs

Returns:

None

Return type:

None

sql_create_mysql_connection¶

sql_create_mysql_connection(host_name, user_name, user_password, db_name, progress)¶

Creates a connection to the MySQL database.

sql_create_mysql_connection(host_name, user_name, user_password, db_name, progress)

Parameters:

host_name (str) – Required MySQL master hostname
user_name (str) – Required MySQL username
user_password (str) – Required MySQL user password
db_name (str) – Required MySQL database
progress (bool) – Required operate in verbose mode

Raises:

Error – Any error occurs

Returns:

Pymysql connect object

Return type:

pymysql.connect

read_credentials.py¶

Data collected and produced for each project will follow BIDS specifications to ensure community standards are upheld, to improve data integrity and conformity, and to improve data consistency and data processing optimization.

A single JSON file describes various parameters for each project/program. This file is ‘credentials.json’ and is located in the main code directory on the mounted centralized storage (/resshare/general_processing_codes). The table below outlines the keys and their associated descriptions for a project in the credentials file. Each project in the credentials file should be defined as their own key titled by their respective protocol number prescribed by their IRB of record. For pilot studies, a short name may be used in place of the IRB number.

Table 4 Key descriptions for the credentials.json file.¶
Key	Data Type	Description
projects	list[string]	List of all project identifiers (these are referred to as <project_identifier>)
<project_identifier>	dictionary	Dictionary identifying key elements of each project described in `credentials_secondary_table`

Table 5 Dictionary keys for the <project_identifier> elements.¶
Key	Data Type	Description
description	string	Text used to give a short description of the project
title	string	Full project title
database	string	MySQL database for the associated main and source tables described in the MySQL section of this document
dataDir	string	Local directory within the mounted centralized storage’s ‘projects’ folder where data shall be located.
dicom_id	string	Unique string within the DICOM filenames to help identify DICOMS within the PACS and souredata directories
searchSourceTable	string	MySQL table identifying files within the project’s sourcedata directory
searchTable	string	MySQL table identifying files within the project’s directory (excluding sourcedata)

read_credentials(inDir, basename)¶

Read the credential file (credentials.json) from the general_processing_codes directory.

This program returns the Project credentials into the custom creds class (NEEDS REFERENCE), which should be imported prior to calling read_credentials()

read_credentials(project)

Parameters:: project (string) – <project_identifier>. This should correspond to the projects IRB protocol number.
Raises:: Error – if credentials.json is not found
Returns:: None
Return type:: None

CoNNECT Support Tools

This Page

CoNNECT Support Tools¶

bids_commands.py¶

get_bids_filename¶

get_bids_labels¶

convert_dicoms.py¶

create_fsl_condor_job.py¶

create_python_condor_job.py¶

flirt.py¶

flirt_pngappend.py¶

get_scan_id.py¶

get_spec_base.py¶

mysql_commands.py¶

sql_multiple_query¶

query_file¶

query_source_files¶

sql_query_dir_check¶

sql_query_dirs¶

sql_query¶

sql_multiple_query¶

sql_create_project_tables¶

sql_table_insert¶

sql_table_remove¶

sql_check_table_exists¶

sql_create_mysql_connection¶

read_credentials.py¶