populse_mia.data_manager.data_loader

Module for importing MRI scan data from MRIFileManager into a Mia project.

This module provides functionality to:

• Parse and process export logs generated by MRIFileManager.

• Import NIfTI scan files and their associated metadata (e.g., bvec/bval files).

• Handle progress tracking and user feedback during the import process.

• Verify the integrity of imported scans via checksum validation.

• Manage database updates for tags, values, and scan metadata.

Functions

read_log(project, main_window)	Display a progress bar for data loading and return loaded file paths.
tags_from_file(file_path, path)	Returns a list of [tag, value] pairs from a JSON file.
verify_scans(project)	Check if the project's scans have been modified.

Classes

ImportProgress(project)	Displays a progress bar for the import process.
ImportWorker(project, progress)	Worker thread for importing scans into the project database.

class populse_mia.data_manager.data_loader.ImportProgress(project)

Bases: QProgressDialog

Displays a progress bar for the import process.

Contains:

Methods:

• onProgress: Updates the progress bar value.

__init__(project)

Initializes the ImportProgress dialog for tracking the import process.

Parameters:

project – (Project) The project object being imported. Passed to the ImportWorker.

Sets up a modal progress dialog with a window title, custom flags, and macOS compatibility. Progress bar range is fixed at 0 to 3 (as per parent class initialization). Dialog is non-cancelable (no cancel button) and appears immediately (minimum duration = 0).

onProgress(value)

Updates the progress bar value.

Parameters:

value – (int) The current progress value.

class populse_mia.data_manager.data_loader.ImportWorker(project, progress)

Bases: QThread

Worker thread for importing scans into the project database.

This class manages the import process, reading from export logs, processing scan files, and updating the database accordingly.

Contains:

Methods:

• run : override the QThread run method.

• scans_added: get a copy of the scans_added list in a thread-safe manner.

• _add_tag_to_database: add a new tag to the database.

• _apply_default_values: apply default values for user-defined tags.

• _convert_datetime_values: convert string values to datetime objects.

• _ensure_associated_file_tag_exists: ensure the associated file tag exists in the database.

• _extract_tag_info: extract tag information from properties.

• _get_export_logs: get the export logs from the raw data folder.

• _process_associated_file: process an associated file.

• _process_associated_files: process associated bvec/bval files.

• _process_datetime_format: process datetime format strings.

• _process_file_tags: process tags from a file.

• _process_fsl_format_files: process FSL format bvec/bval files.

• _process_list_values: process list values.

• _process_log_entries: process each log entry to import scans.

• _process_mrtrix_format_file: process MRtrix format bvec/bval file.

• _process_scan_file: process a single scan file and its associated files.

• _update_database: update the database with the processed documents.

Signals:

• notifyProgress: Signal to update the progress bar. Emits an integer representing the progress percentage.

__init__(project, progress)

Initialize the ImportWorker thread for importing scans into the project database.

Parameters:

• project – (Project) The project object containing the database and configuration for importing scans.

• progress – (ImportProgress) The progress dialog instance to update the import progress via the notifyProgress signal.

run()

Execute the import process.

This method overrides the QThread run method and is executed when the worker is started. It processes the export logs, imports scans, and updates the database.

property scans_added

Get a copy of the scans_added list in a thread-safe manner.

_add_tag_to_database(tag_info, tags_added, tags_names_added)

Add a new tag to the database.

Parameters:

• tag_info – (dict) Tag information dictionary.

• tags_added – (list) List to track added tags.

• tags_names_added – (list) List to track added tag names.

_apply_default_values(documents, values_added)

Apply default values for user-defined tags.

Parameters:

• documents – (dict) Dictionary containing document information.

• values_added – (list) List to track added values.

_convert_datetime_values(tag_info)

Convert string values to datetime objects.

Parameters:

tag_info – (dict) Tag information dictionary.

Returns:

(dict) Updated tag information dictionary.

_ensure_associated_file_tag_exists(database_data, tag_name, tags_added, tags_names_added)

Ensure the associated file tag exists in the database.

Parameters:

• database_data – Context-Managed database.

• tag_name – (str) Name of the tag.

• tags_added – (list) List to track added tags.

• tags_names_added – (list) List to track added tag names.

_extract_tag_info(tag_name, properties)

Extract tag information from properties.

Parameters:

• tag_name – (str) Name of the tag.

• properties – (any) Properties of the tag.

Returns:

(dict) Dictionary containing tag information.

_get_export_logs(raw_data_folder)

Get the export logs from the raw data folder.

Parameters:

raw_data_folder – (str) Path to the raw data folder.

Returns:

(list) Log entries.

_process_associated_file(database_data, file_database_path, checksum, file_type, associated_file_path, tag_name, documents, values_added)

Process an associated file.

Parameters:

• database_data – Context-Managed database.

• file_database_path – (str) Relative path of the file.

• checksum – (str) MD5 checksum of the file.

• file_type – (Type) Type of the file.

• associated_file_path – (str) Path of the associated main file.

• tag_name – (str) Name of the association tag.

• documents – (dict) Dictionary to store document information.

• values_added – (list) List to track added values.

_process_associated_files(database_data, file_name, raw_data_folder, file_database_path, documents, values_added, tags_added, tags_names_added)

Process associated bvec/bval files.

Parameters:

• database_data – Context-Managed database.

• file_name – (str) Base name of the file.

• raw_data_folder – (str) Path to the raw data folder.

• (str) (file_database_path) – Relative path of the main file.

• documents – (dict) Dictionary to store document information.

• values_added – (list) List to track added values.

• tags_added – (list) List to track added tags.

• tags_names_added – (list) List to track added tag names.

_process_datetime_format(tag_info)

Process datetime format strings.

Parameters:

tag_info – (dict) Tag information dictionary.

Returns:

(dict) Updated tag information dictionary.

_process_file_tags(database_data, file_name, path_name, file_database_path, document_not_existing, documents, values_added, tags_added, tags_names_added, tags_to_exclude)

Process tags from a file.

Parameters:

• database_data – Context-Managed database.

• file_name – (str) Base name of the file.

• path_name – (str) Path to the file directory.

• file_database_path – (str) Relative path in the database.

• document_not_existing – (bool) Whether the document is new.

• documents – (dict) Dictionary to store document information.

• values_added – (list) List to track added values.

• tags_added – (list) List to track added tags.

• tags_names_added – (list) List to track added tag names.

• tags_to_exclude – (list) List of tags to exclude.

_process_fsl_format_files(database_data, bvec_path, bval_path, file_database_path, tag_name, documents, values_added)

Process FSL format bvec/bval files.

Parameters:

• database_data – Context-Managed database.

• bvec_path – (str) Path to the bvec file.

• bval_path – (str) Path to the bval file.

• file_database_path – (str) Relative path of the main file.

• tag_name – (str) Name of the association tag.

• documents – (dict) Dictionary to store document information.

• values_added – (list) List to track added values.

_process_list_values(tag_info)

Process list values.

Parameters:

tag_info – (dict) Tag information dictionary.

Returns:

(dict) Updated tag information dictionary.

_process_log_entries(list_dict_log, raw_data_folder, documents, values_added, tags_added, tags_names_added)

Process each log entry to import scans.

Parameters:

• list_dict_log – (list) Log entries.

• raw_data_folder – (str) Path to the raw data folder.

• documents – (dict) Dictionary to store document information.

• values_added – (list) List to track added values.

• tags_added – (list) List to track added tags.

• tags_names_added – (list) List to track added tag names.

_process_mrtrix_format_file(database_data, bvec_bval_path, file_database_path, tag_name, documents, values_added)

Process MRtrix format bvec/bval file.

Parameters:

• database_data – Context-Managed database.

• bvec_bval_path – (str) Path to the MRtrix format file.

• file_database_path – (str) Relative path of the main file.

• tag_name – (str) Name of the association tag.

• documents – (dict) Dictionary to store document information.

• values_added – (list) List to track added values.

_process_scan_file(database_data, file_name, raw_data_folder, dict_log, documents, values_added, tags_added, tags_names_added, tags_to_exclude)

Process a single scan file and its associated files.

Parameters:

• database_data – Context-Managed database.

• file_name – (str) Base name of the scan file.

• raw_data_folder – (str) Path to the raw data folder.

• dict_log – (dict) Log entry for this scan.

• documents – (dict) Dictionary to store document information.

• values_added – (list) List to track added values.

• tags_added – (list) List to track added tags.

• (list) (tags_to_exclude) – List to track added tag names.

• (list) – List of tags to exclude.

_update_database(documents, tags_added, values_added, historyMaker)

Update the database with the processed documents.

Parameters:

• documents – (dict) Dictionary containing document information.

• tags_added – (list) List of tags to add.

• values_added – (list) List of values to add.

• historyMaker – (list) List for history tracking.

populse_mia.data_manager.data_loader.read_log(project, main_window)

Display a progress bar for data loading and return loaded file paths.

This function shows the evolution of the progress bar while loading data files and returns a list of paths to each data file that was successfully loaded.

Parameters:

• project – (Project) The current project instance in the software.

• main_window – (MainWindow) The software’s main window instance used to display the progress bar.

Returns:

(list) A list of paths to the data files (scans) that were successfully added.

populse_mia.data_manager.data_loader.tags_from_file(file_path, path)

Returns a list of [tag, value] pairs from a JSON file.

Parameters:

• file_path – (str) File path of the Json file (without the extension).

• path – (str) Project path.

Returns:

(List[List[Union[str, dict]]] A list of the Json tags of the file.

populse_mia.data_manager.data_loader.verify_scans(project)

Check if the project’s scans have been modified.

Parameters:

project – (Project) Current project in the software.

Returns:

(List[str]) The list of scans that have been modifiedor are missing.