Log Module

The log module provides logging utilities for data analysis workflows with two main components: the @log_analysis decorator for automatic function logging and the QuickLogger class for manual step-by-step logging.

ipa.data_loader.log_analysis(log_dir='logs')[source]

Decorator: Automatically log function execution process.

This decorator provides automatic logging for data analysis functions. It logs function start/completion, automatically detects and logs file paths from function parameters, handles exceptions, and logs output files.

Parameters:

log_dir (str or Path) – Directory to store log files. Defaults to “logs”

Returns:

Decorated function with automatic logging capabilities

Return type:

function

class ipa.data_loader.QuickLogger(name='analysis', log_dir='logs', console_output=True)[source]

Bases: object

Simple interface for step-by-step logging in data analysis workflows.

Provides predefined methods for common logging scenarios including analysis steps, input/output files, and error handling. Creates timestamped log files automatically.

Parameters:

  • name (str) – Logger name, used in log filename. Defaults to “analysis”

  • log_dir (str or Path) – Directory to store log files. Defaults to “logs”

  • console_output (bool) – Whether to print logs to console. Defaults to True.

logger

Internal logger instance

Type:

logging.Logger

error(message)[source]

Log error message.

Parameters:

message (str) – Error description

Example

>>> logger.error("Failed to load input file")
>>> logger.error(f"Processing failed: {str(e)}")
file_in(file_path)[source]

Log input file path.

Parameters:

file_path (str or Path) – Path to input file

Example

>>> logger.file_in("/data/images/sample.tif")
>>> logger.file_in(mask_pm_path)
file_out(file_path)[source]

Log output file path.

Parameters:

file_path (str or Path) – Path to generated output file

Example

>>> logger.file_out("/results/processed.tif")
>>> logger.file_out(xvg_path)
step(message)[source]

Log analysis step or general information.

Parameters:

message (str) – Step description or information message

Example

>>> logger.step("Loading mask data...")
>>> logger.step(f"Processing dataset: {dataid}")

Overview

The log module offers two complementary approaches for tracking analysis workflows:

  1. @log_analysis decorator - Automatic logging of function execution, parameters, and outputs

  2. QuickLogger class - Manual logging for detailed step-by-step tracking

Key Features:

  • Dual Output: Logs are saved to timestamped files AND optionally printed to console

  • UTF-8 Encoding: Full support for international characters in log messages

  • Automatic Directory Creation: Log directory is created automatically if it doesn’t exist

  • Docker-Friendly: Explicit stdout configuration ensures logs display correctly in containers

  • Duplicate Prevention: Handler clearing prevents duplicate log entries on repeated calls

  • Smart Format: Console uses simple format (INFO: message), files use detailed timestamps

QuickLogger - Manual Step Logging

The QuickLogger class provides a simple interface for logging analysis steps, input/output files, and errors. It has four main methods:

  • step(message) - Log general analysis steps and information

  • file_in(file_path) - Log input file paths

  • file_out(file_path) - Log output file paths

  • error(message) - Log error messages

Basic Usage

from ipa.data_loader import QuickLogger

# Initialize logger with console output enabled (default)
logger = QuickLogger("sxt_partitioning", log_dir="logs", console_output=True)

# Log analysis steps
logger.step("Starting SXT Partitioning Demo")
logger.step(f"Main path: {mainpath}")
logger.step(f"Processing dataset: {dataid}")

# Log input files
logger.file_in(mask_pm_path)
logger.file_in(mask_ne_path)

# Log processing steps with details
logger.step("Loading mask data...")
logger.step(f"PM mask shape: {mask_data_pm.shape}, dtype: {mask_data_pm.dtype}")
logger.step("Initializing partitioning processor...")

# Log output files
logger.file_out(xvg_path)
logger.file_out(comparison_path)

Console Output Control

You can control whether logs are printed to the console:

from ipa.data_loader import QuickLogger

# Enable console output (default) - good for interactive use
logger_verbose = QuickLogger("analysis", console_output=True)
logger_verbose.step("This will print to terminal AND save to file")

# Disable console output - good for background tasks or batch processing
logger_silent = QuickLogger("batch_job", console_output=False)
logger_silent.step("This only saves to file, no terminal output")

When to use each mode:

  • console_output=True: Interactive demos, debugging, real-time monitoring

  • console_output=False: Batch processing, production pipelines, Docker containers

Error Handling

from ipa.data_loader import QuickLogger

logger = QuickLogger("data_analysis")

try:
    logger.step("Loading data from file")
    data = load_data(input_file)
    logger.step(f"Successfully loaded {data.shape} data")

except Exception as e:
    logger.error(f"Failed to load data: {str(e)}")
    raise

@log_analysis - Automatic Function Logging

The @log_analysis decorator automatically logs function execution, parameters, and results.

Function Decoration

from ipa.data_loader import log_analysis

@log_analysis(log_dir="analysis_logs")
def process_tomography_data(input_path, output_dir, threshold=0.5):
    """Process tomography data with automatic logging"""
    # Function logic here
    processed_files = []
    # ... processing ...
    return processed_files

# Usage - logging happens automatically
results = process_tomography_data("data/tomo/", "results/", threshold=0.7)

Parameter Detection

The decorator automatically detects and logs:

  • File paths: Parameters containing ‘path’ or ‘file’ in the name

  • Output directories: Parameters named ‘output_dir’, ‘save_dir’, ‘result_path’

  • Return values: File paths returned as strings, Paths, or lists

@log_analysis(log_dir="batch_logs")
def batch_process_images(image_dir, mask_file, output_dir, n_slices=8):
    """Batch process images with comprehensive logging"""
    # Automatically logs:
    # - image_dir (contains 'dir')
    # - mask_file (contains 'file')
    # - output_dir (recognized output parameter)
    # - Any returned file paths

    result_files = []
    # ... processing logic ...
    return result_files

Example from SXT Demo

This example shows how logging is used in the SXT partitioning demo:

from ipa.data_loader import QuickLogger
import tifffile
import os

def main():
    # Initialize logger with descriptive name
    log_dir = f'{mainpath}/logs'
    logger = QuickLogger("sxt_partitioning", log_dir=log_dir)
    logger.step("Starting SXT Partitioning Demo")

    # Log configuration and paths
    logger.step(f"Main path: {mainpath}")
    dataid = '784_5'
    logger.step(f"Processing dataset: {dataid}")

    # Log directory creation
    results_dir = f'{root_dir}/results/'
    os.makedirs(results_dir, exist_ok=True)
    logger.step(f"Results directory: {results_dir}")

    # Log input files
    mask_pm_path = f'{mainpath}/data/sxt_images/{dataid}_wholecell_label.tiff'
    mask_ne_path = f'{mainpath}/data/sxt_images/{dataid}_NC_label.tiff'
    logger.file_in(mask_pm_path)
    logger.file_in(mask_ne_path)

    # Log data loading and properties
    logger.step("Loading mask data...")
    mask_data_pm = tifffile.imread(mask_pm_path).astype(int)
    mask_data_ne = tifffile.imread(mask_ne_path).astype(int)
    logger.step(f"PM mask shape: {mask_data_pm.shape}, dtype: {mask_data_pm.dtype}")
    logger.step(f"NE mask shape: {mask_data_ne.shape}, dtype: {mask_data_ne.dtype}")

    # Log processing steps
    logger.step("Initializing partitioning processor...")
    logger.step("Extracting boundaries for analysis...")
    logger.step("Creating continuous radial partitions using distance transform...")

    # Log results and outputs
    xvg_path = partitioner.save_partition_coords_to_xvg(partition_coords, dataid, results_dir)
    logger.file_out(xvg_path)

    comparison_path = f"{results_dir}{dataid}_partition_comparison.png"
    logger.file_out(comparison_path)

    logger.step(f"Processing completed for dataset: {dataid}")

Log File Output

The logging system generates timestamped files with structured output. Log files are automatically created in the specified directory.

File Naming Convention:

{name}_{timestamp}.log

Example: sxt_partitioning_20260429_143025.log

Console Output Format (when console_output=True):

INFO: Starting SXT Partitioning Demo
INFO: Main path: /data/sxt_analysis
INFO: Processing dataset: 784_5
INFO: Input file: /data/sxt_images/784_5_wholecell_label.tiff
INFO: Loading mask data...
INFO: PM mask shape: (512, 1024, 1024), dtype: int32
INFO: Output file: /data/results/784_5_partition_coords.xvg
INFO: Processing completed for dataset: 784_5

File Log Format (detailed with timestamps):

2026-04-29 14:30:25,123 - INFO - Starting SXT Partitioning Demo
2026-04-29 14:30:25,124 - INFO - Main path: /data/sxt_analysis
2026-04-29 14:30:25,125 - INFO - Processing dataset: 784_5
2026-04-29 14:30:25,126 - INFO - Results directory: /data/sxt_analysis/results/
2026-04-29 14:30:25,127 - INFO - Input file: /data/sxt_images/784_5_wholecell_label.tiff
2026-04-29 14:30:25,128 - INFO - Input file: /data/sxt_images/784_5_NC_label.tiff
2026-04-29 14:30:25,129 - INFO - Loading mask data...
2026-04-29 14:30:26,456 - INFO - PM mask shape: (512, 1024, 1024), dtype: int32
2026-04-29 14:30:26,457 - INFO - NE mask shape: (512, 1024, 1024), dtype: int32
2026-04-29 14:30:26,458 - INFO - Initializing partitioning processor...
2026-04-29 14:30:27,123 - INFO - Output file: /data/results/784_5_partition_coords.xvg
2026-04-29 14:30:27,456 - INFO - Processing completed for dataset: 784_5

Key Differences:

  • Console: Simple format (LEVEL: message) for easy reading during interactive use

  • File: Detailed format (timestamp - LEVEL - message) for complete audit trail

  • UTF-8 Encoding: Both outputs support international characters