Run parallel simulations

Now that we have a clear understanding of the SAInt simulation workflow and the case study that will be used in this tutorial, it is time to delve into the steps necessary to run parallel simulations using the SAInt API and Python.

The data for the tutorial are available from the section "TUTORIAL DOWNLOADS" of the category "Model Ready Datasets" of the community Forum. Please, download the latest copy and check the content of the folder .\Scripting\API Advanced.

Please take a moment to read the ReadME.md file in the Advanced API folder for more details on the network and installed generation capacities.

1. Initial setup

To begin, open your preferred Python IDE and create a new Python script called parallel-simulation.py. Save the empty script in a working folder named Parallel. Then, copy the folder Networks available in the tutorial’s files to this new working directory. The Networks directory contains subfolders for the five cases used in the example (i.e., the base case and the four alternative scenarios with a different mix of wind a solar generation). Next, copy the code below to your Python script.

If the multiprocessing Python module is not installed, please install it before moving forward.

 
# Import packages
from ctypes import *
import multiprocessing as mp
import os
import glob

# Path to the SAInt-API.dll file (located in the SAInt installation folder)
DLL_PATH = r"C:\Program Files\encoord\SAInt-v3\SAInt-API.dll"

# Simulation function loading network, scenario, and importing profiles and events
def run_simulation(network, main_dir, source_dir, core):

   ...

# Main function for calculations
if __name__ == "__main__":

   ...

    print("Simulations finished.")

The initial script imports the necessary Python packages, declares the path to our SAInt-API.dll (found in the installation folder), and introduces two main parts: a function called run_simulation that will perform a set of actions ending with running the simulation and writing the results, and another section in the main body of the script where the parallelization is managed.

2. The run_simulation function

This function is executed for each single simulation and needs to perform a specific set of operations, which are in sequence: load an isolated DLL instance of SAInt, open a specific network model from a folder, check the presence and load the scenario file available in the network folder, import profiles (no check on the presence) and import events (no check on the presence) for the scenario, run the optimization problem, and, finally, save the results based on a user specified list of derived properties. Once a simulation finishes, the release method of the Semaphore object core releases a CPU core, signaling the completion of a simulation and freeing up the core for another parallel simulation.

The function takes as arguments:

  • network: the name of the network folder containing the model and scenario to use;

  • main_dir: the path to the folder containing the directories where the networks and scenarios are saved. This is the full path to the folder Networks;

  • source_dir: the path to the folder where the script is (i.e., the full path to the folder Parallel) and which acts as root folder;

  • core: the multiprocessing semaphore used to define the parallel instance of SAInt (see the documentation of the multiprocessing package for details).

The function run_simulation uses SAInt writeESOL function to collect relevant simulation results, as we learned from the "Export operations" tutorial. The writeESOL function needs a result description file as input. Create a new Excel file. Copy the following object identifiers and property extensions into it:

ENET.PFGEN.[MW]
ENET.PXGEN.[MW]
ENET.PHGEN.[MW]
ENET.PWIND.[MW]
ENET.PPV.[MW]
ENET.PNSPV.[MW]
ENET.PNSWIND.[MW]
ENET.PNSDEM.[MW]
ENET.TOTCOSTRATE.[$/h]
ENET.CO2RATE.[t/h]

Next, save the Excel file as Results_Description.xlsx in the Parallel folder.

# Simulation function loading network, scenario, and importing profiles and events
def run_simulation(network, main_dir, source_dir, core):
    try:
        # Each spawned process loads its own DLL instance.
        saint_dll = cdll.LoadLibrary(DLL_PATH)

        # Network directory
        net_dir = os.path.join(main_dir, network)

        # Load the electric network
        saint_dll.openENET(os.path.join(net_dir, "ENET09_13.enet"))

        # Load the electric scenario
        scenario_files = glob.glob(os.path.join(net_dir, "*.esce"))

        if not scenario_files:
            raise FileNotFoundError(f"No .esce file found in {net_dir}")

        saint_dll.openESCE(scenario_files[0])

        # Include profiles
        saint_dll.includeEPRF(os.path.join(net_dir, "Profiles.prfl"))

        # Import events
        saint_dll.importESCE(os.path.join(net_dir, "Events.xlsx"))

        # Hide simulation logs
        saint_dll.showSIMLOG(False)

        # Execute simulation
        saint_dll.runESIM()

        # Extract and write results
        saint_dll.writeESOL(
            os.path.join(source_dir, "Results_Description.xlsx"),
            os.path.join(net_dir, "Results.xlsx"),
        )

    finally:
        # Release the core even if the simulation fails.
        core.release()

3. The main part of the script

This section of the script takes care of executing, only one time, a set of tasks and to set up and launch the parallelization of the simulations by managing a pool of instances.

It starts by defining the path to the folder where the script is (i.e., the folder Parallel) the path to where the folders containing networks and scenarios are (i.e., the folder Networks). The value is printed for the user in the standard output.

After the script identifies the maximum number of possible instances to run in parallel by interrogating the CPU and returning the number of logical CPUs available to Python, and save the result in the variable cores.

Then, we create a list containing the path to all folders with network models and scenarios available inside Networks. The list is printed for the user in the standard output.

Finally, we have a set of commands to define the maximum number of processes to run simultaneously by using the mp.Semaphore() command along with a list to collect the outcome of each parallel instance.

Two sequential for loops are used. The first depends on the number of folders containing networks and scenarios (in our case is five folders) and initializes a core, calls the function run_simulation with the necessary arguments, updates the list of running processes, and launches the simulation. The second loop uses the list to wait for every process to finish.

The code of the main section is:

# Main function for calculations
if __name__ == "__main__":

    # Identify path and subfolder with networks
    source_dir = os.path.dirname(os.path.abspath(__file__))
    main_dir = os.path.join(source_dir, "Networks")

    # Define number of available cores and list of subfolders with networks
    cores = mp.cpu_count()
    networks = os.listdir(main_dir)

    # Print number of cores and list of folders with networks.
    print("Number of cores:", cores)
    print("Networks to simulate:", networks)

    core = mp.Semaphore(cores)
    all_processes = []

    # Loop for parallel execution
    for network in networks:
        core.acquire()

        process = mp.Process(
            target=run_simulation,
            args=(network, main_dir, source_dir, core),
        )
        all_processes.append(process)
        process.start()

    for process in all_processes:
        process.join()

    print("Simulations finished.")

4. Complete version of the script

The complete version of the script should look like the following:

# Import packages
from ctypes import cdll
import glob
import multiprocessing as mp
import os

# Path to the SAInt-API.dll file (located in the SAInt installation folder)
DLL_PATH = r"C:\Program Files\encoord\SAInt-v3\SAInt-API.dll"

# Simulation function loading network, scenario, and importing profiles and events
def run_simulation(network, main_dir, source_dir, core):
    try:
        # Each spawned process loads its own DLL instance.
        saint_dll = cdll.LoadLibrary(DLL_PATH)

        # Network directory
        net_dir = os.path.join(main_dir, network)

        # Load the electric network
        saint_dll.openENET(os.path.join(net_dir, "ENET09_13.enet"))

        # Load the electric scenario
        scenario_files = glob.glob(os.path.join(net_dir, "*.esce"))

        if not scenario_files:
            raise FileNotFoundError(f"No .esce file found in {net_dir}")

        saint_dll.openESCE(scenario_files[0])

        # Include profiles
        saint_dll.includeEPRF(os.path.join(net_dir, "Profiles.prfl"))

        # Import events
        saint_dll.importESCE(os.path.join(net_dir, "Events.xlsx"))

        # Hide simulation logs
        saint_dll.showSIMLOG(False)

        # Execute simulation
        saint_dll.runESIM()

        # Extract and write results
        saint_dll.writeESOL(
            os.path.join(source_dir, "Results_Description.xlsx"),
            os.path.join(net_dir, "Results.xlsx"),
        )

    finally:
        # Release the core even if the simulation fails.
        core.release()

# Main function for calculations
if __name__ == "__main__":

    # Identify path and subfolder with networks
    source_dir = os.path.dirname(os.path.abspath(__file__))
    main_dir = os.path.join(source_dir, "Networks")

    # Define number of available cores and list of subfolders with networks
    cores = mp.cpu_count()
    networks = os.listdir(main_dir)

    # Print number of cores and list of folders with networks.
    print("Number of cores:", cores)
    print("Networks to simulate:", networks)

    core = mp.Semaphore(cores)
    all_processes = []

    # Loop for parallel execution
    for network in networks:
        core.acquire()

        process = mp.Process(
            target=run_simulation,
            args=(network, main_dir, source_dir, core),
        )
        all_processes.append(process)
        process.start()

    for process in all_processes:
        process.join()

    print("Simulations finished.")

Now, we can launch our parallel simulations, for example by calling python.exe parallel-simulation.py from the Parallel folder using the command prompt of Windows.

Check the log for problems or to monitor the execution.