Skip to contents

Project configuration

Source: vignettes/project-structure.Rmd
project-structure.Rmd

Project Configuration Overview

The project configuration is the central piece of any esqlabsR project. It defines the structure of your project, specifies where all necessary files are stored, and provides the foundation for all workflows in the package. The project configuration consists of:

  • Project structure: A standardized folder hierarchy that organizes your models, data, parameters, scenarios, and results
  • ProjectConfiguration object: An R object that stores paths to all project files and directories
  • Configuration files: Excel files (or JSON snapshots) that define the project setup

Initialize the Project Structure

Project structure can be initialized by calling the function initProject().

This will create a set of directories and files that match the expected project’s structure.

#> /home/runner/work/_temp/Library/esqlabsR/extdata/examples/TestProject
#> ├── Configurations
#> │   ├── Applications.xlsx
#> │   ├── Individuals.xlsx
#> │   ├── ModelParameters.xlsx
#> │   ├── Plots.xlsx
#> │   ├── Populations.xlsx
#> │   ├── PopulationsCSV
#> │   │   └── TestPopulation.csv
#> │   └── Scenarios.xlsx
#> ├── Data
#> │   ├── TestProject_TimeValuesData.xlsx
#> │   └── esqlabs_dataImporter_configuration.xml
#> ├── Models
#> │   └── Simulations
#> │       └── Aciclovir.pkml
#> ├── ProjectConfiguration.json
#> ├── ProjectConfiguration.xlsx
#> └── Results
#>     ├── All plots.png
#>     ├── Figures
#>     └── SimulationResults

Create a ProjectConfiguration

The ProjectConfiguration stores the paths to the model-, data-, and other files and directories. It will be used at several steps in esqlabsR workflows so it needs to be created before performing any simulations.

By printing the ProjectConfiguration, we can see the locations of all files used in the workflows:

my_project_configuration <- createProjectConfiguration(path = exampleProjectConfigurationPath())

Now that the project structure is initialized and the ProjectConfiguration is created, read vignette("design-scenarios") to continue the process. To learn more about ProjectConfiguration, read the following sections.

Validating Project Configuration

Before running simulations, it is recommended to validate your project configuration files to catch potential issues early. The validateAllConfigurations() function performs comprehensive validation across all Excel configuration files and returns structured results.

Validation Process

The validation system checks your configuration files for:

  • Structural issues: Missing required sheets or columns
  • Data integrity: Empty sheets, invalid data types, required fields
  • Uniqueness constraints: Duplicate IDs or names that must be unique
  • Cross-references: Invalid references between configuration files (e.g., scenarios referencing non-existent individuals)

Three-Tier Validation Results

Validation results are categorized into three levels:

  1. Critical Errors: Blocking issues that prevent data import and must be fixed (e.g., missing required sheets, missing required columns)
  2. Warnings: Non-blocking issues that allow import but indicate potential problems (e.g., empty optional sheets, data type mismatches)
  3. Valid Data: Successfully validated data ready for use

Usage Example 

# Validate all configuration files in a project
validation_results <- validateAllConfigurations(my_project_configuration)

# Or validate directly from the Excel file path
validation_results <- validateAllConfigurations(exampleProjectConfigurationPath())

# Check if there are any critical errors
if (isAnyCriticalErrors(validation_results)) {
  # Get detailed summary
  summary <- validationSummary(validation_results)
  print(paste("Critical errors found in:", paste(summary$files_with_errors, collapse = ", ")))
  print(paste("Total critical errors:", summary$total_critical_errors))
} else {
  print("All validations passed!")
}

Interpreting Validation Results

The validateAllConfigurations() function returns a named list of validationResult objects, one for each configuration file validated. Each result contains:

  • critical_errors: List of blocking issues
  • warnings: List of non-blocking issues
  • data: The validated data (if validation succeeded)

You can access individual file results:

# Check scenarios validation
scenarios_result <- validation_results$scenarios

if (scenarios_result$has_critical_errors()) {
  print("Scenarios file has critical errors:")
  print(scenarios_result$critical_errors)
}

if (length(scenarios_result$warnings) > 0) {
  print("Scenarios file has warnings:")
  print(scenarios_result$warnings)
}

The validation system ensures data quality before simulation, helping you identify and fix configuration issues early in your workflow.

Details

Change ProjectConfiguration from R

If required, you can change the location of one of the files or folders using relative or absolute paths

# change the location of the output folder
projectConfiguration$outputFolder <- "../anotherOutputFolder"

# change the location of the model parameters file
projectConfiguration$modelParametersFile <- "absolute/path/to/anotherModelParameters.xlsx"

About ProjectConfiguration.xlsx

The ProjectConfiguration.xlsx file will be used by esqlabsR to generate a ProjectConfiguration object which is the central piece of a project. It should be located in the root folder of the project.

This file defines where all the necessary files are stored in the project folder. All the path specified in the Value column should be relative to the ProjectConfiguration.xlsx location.

All these directories and files have a specific purpose and template and are describe in next sections.

Version Control and Project Sharing

While Excel provides an excellent interface for quickly editing and creating new entries in your project configuration, it has limitations when it comes to version control and team collaboration - Excel files are binary and don’t work well with Git, making it cumbersome to share changes between team members.

esqlabsR addresses these limitations by providing powerful version control and project sharing capabilities through its snapshot and restore functionality. This allows you to:

  • Track changes to your project configuration in version control systems like Git
  • Share projects with team members easily
  • Create backups before making significant changes
  • Ensure reproducibility across different environments

Creating Project Snapshots

The snapshotProjectConfiguration() function exports all Excel configuration files in your project to a single JSON file. This JSON format is ideal for version control as it’s:

  • Text-based: Easy to track changes in Git
  • Human-readable: Can be reviewed and compared
  • Complete: Contains all project configuration data
# Create a snapshot from a ProjectConfiguration object
my_project_configuration <- createProjectConfiguration("ProjectConfiguration.xlsx")
snapshotProjectConfiguration(my_project_configuration)

# Or create a snapshot directly from the Excel file path
snapshotProjectConfiguration("ProjectConfiguration.xlsx")

By default, the JSON file will be created in the same directory as your ProjectConfiguration.xlsx file with the name ProjectConfiguration.json.

Restoring from Snapshots

The restoreProjectConfiguration() function recreates your entire project configuration from a JSON snapshot:

# Restore a project from a JSON snapshot
restored_config <- restoreProjectConfiguration("ProjectConfiguration.json")

This function will: 1. Create the necessary directory structure 2. Regenerate all Excel configuration files 3. Return a new ProjectConfiguration object

Checking Project Status

Use projectConfigurationStatus() to check if your Excel files are synchronized with your JSON snapshot:

# Check if files are in sync
status <- projectConfigurationStatus("ProjectConfiguration.xlsx", "ProjectConfiguration.json")
print(status$in_sync) # TRUE if synchronized, FALSE otherwise

Workflow Examples

Team Collaboration: When working with a team:

  1. Initial Setup: Create a snapshot of your project configuration

    snapshotProjectConfiguration("ProjectConfiguration.xlsx")

  2. Version Control: Commit the JSON file to your Git repository

    git add ProjectConfiguration.json
git commit -m "Add project configuration snapshot"

  3. Team Sharing: Team members can restore the project

    restored_config <- restoreProjectConfiguration("ProjectConfiguration.json")

Backup Strategy: Before making significant changes to your project:

# Create a backup snapshot
snapshotProjectConfiguration("ProjectConfiguration.xlsx")

# Make your changes...

# If something goes wrong, restore from the snapshot
restored_config <- restoreProjectConfiguration("ProjectConfiguration.json")

The version control features in esqlabsR provide a robust foundation for team collaboration and project management. By using snapshots, you can maintain project history in version control systems, share configurations easily with team members, create reliable backups before making changes, and ensure reproducibility across different environments.