Run parallel simulations

Now that we have a clear understanding of the SAInt simulation workflow and the case study that will be used in this tutorial, it is time to delve into the steps necessary to run parallel simulations using the SAInt API and Python.

Copy the network and scenario files for the case study from the sub-folder .\Scripting\API Advanced of the folder Tutorials in the directory (C:\Users\...\Documents\encoord\SAInt-v3\Projects).

Please take a moment to read the ReadME.md file in the attached folder for more details on the network demand and installed generation capacities.

1. Initial setup

To begin, open your preferred Python IDE and create a new Python script called parallel-simulation.py. Then, copy the tutorial’s files from the tutorial’s directory to your working directory, along with your Python script. Close and re-open your Python script from the new location to avoid directory problems. Next, copy the code below to your Python script and execute.

If the multiprocessing Python module is not installed, please install it before moving forward.

# Import packages
from ctypes import *
import multiprocessing as mp
import os
import glob

# Path to the SAInt-API.dll file (located in the SAInt installation folder)
path = "C:\\Program Files\\encoord\\SAInt-v3\\SAInt-API.dll"

# Create a SAInt API DLL object
saint_dll = cdll.LoadLibrary(path)

2. Check the number of cores in the CPU

It is important to check the number of cores in a CPU before running parallel simulations because the number of cores determines the possible number of simultaneous parallel simulations. Let’s check the number of cores in your CPU using the multiprocessing module’s cpu_count() function. Add the code below to your new Python script and execute.

# Number of CPU cores
cores = mp.cpu_count()

3. Define network directories

When running parallel simulations, unique network directories are important to ensure each simulation process has its network, scenario, events, and profile files. Otherwise, multiple simulation processes may try to access and modify the same data concurrently, leading to inconsistencies and errors in the results. By defining unique network directories for each simulation process, you can ensure that each process operates on its independent data set and that there are no conflicts between processes. Add the code below to your new Python script and execute.

# Source directory
def get_source_directory():
    return os.path.dirname(os.path.abspath(__file__))
source_dir = get_source_directory()

# Main folder
main_dir = source_dir + "\\Networks\\"

# Number of networks to run in parallel
networks = os.listdir(main_dir)

4. Define parallel processing variables

Next, we need to initialize a Semaphore object with the value of the cores available. The Semaphore object controls access to shared resources and limits the number of parallel simulations that run at any given time.

We will also need an array named all_processes to store and track the processes in the simulation queue. Add the code below to your new Python script and execute.

# Variables
core = mp.Semaphore(cores)
all_processes = []

5. Define results description sheet

We will use the writeESOL function to collect relevant simulation results, as we learned from the Export operations tutorial, the writeESOL function needs a result description file as input. Create a new Excel file and copy the following object identifiers and property extensions into it:

ENET.PFGEN.[MW]
ENET.PXGEN.[MW]
ENET.PHGEN.[MW]
ENET.PWIND.[MW]
ENET.PPV.[MW]
ENET.PNSPV.[MW]
ENET.PNSWIND.[MW]
ENET.PNSDEM.[MW]
ENET.TOTCOSTRATE.[$/h]
ENET.CO2RATE.[t/h]

Next, save the Excel file as Results_Description.xlsx in the Parallel folder.

6. Define simulation function

The function RunSimulation is a SAInt simulation workflow for a single simulation in a parallel simulation setup. The function takes two arguments, n, the index of the simulated network, and core, which is the Semaphore object used to manage the number of parallel simulations. Add the code below to your new Python script and execute.

The wind and solar capacities for the reference and high renewable scenarios are in the Events.xlsx sheet found in each network’s folder. Feel free to look at the different renewable capacities to understand the data better.

# Simulation function
def RunSimulation(n, core):

        # Network directory
        net_dir = main_dir + networks[n] + "\\"

        # Load the electric network
        saint_dll.openENET(net_dir + "ENET09_12.enet")

        # Load the electric scenario
        scenario = glob.glob1(net_dir, "*.esce")[0]
        saint_dll.openESCE(net_dir + scenario)

        # Include profiles
        saint_dll.includeEPRF(net_dir + "Profiles.prfl")

        # Import events
        saint_dll.importESCE(net_dir + "Events.xlsx")

        # Hide simulation logs
        saint_dll.showSIMLOG(False)

        # Execute simulation
        saint_dll.runESIM()

        # Extract results
        saint_dll.writeESOL(
                source_dir + "\\Results_Description.xlsx",
                net_dir + "Results.xlsx"
                )

        # Release a CPU core
        core.release()

Once a simulation finishes, the release method of the Semaphore object core releases a CPU core, signaling the completion of a simulation and freeing up the core for another parallel simulation.

7. Define loop to run parallel simulations

We need a loop to run the simulations in parallel. For each network in the list of networks, the code will obtain a core for a process using the acquire method of the Semaphore object. This method ensures the simulation uses all available cores.

The following part of the script creates an object p for each simulation with the target function RunSimulation and the arguments (net, core), where net is the index of the network to simulate and core is the Semaphore object. This Process object will then be added to the all_processes list and started using the start method. Add the code below to your new Python script.

Do not execute the Python script in this step, as we need one final step to complete the code for parallel simulations.

if __name__ == '__main__':
    #Loop to run scenarios in parallel
    for net in range(len(networks)):
            core.acquire()
            p = mp.Process(target= RunSimulation, args=(net, core))
            all_processes.append(p)
            p.start()

8. Define loop to join all processes

Finally, a second loop joins all processes in the all_processes list. This loop will block the main program until completing all processes, ensuring the execution of all simulations before further processing in the code. The p.join() method ensures that the program waits for all processes to finish executing before continuing in the code. Add the code below to your new Python script (nested under the previous if name == 'main' statement) and execute.

    # Loop to join processes
    for p in all_processes:
        p.join()

As a reference, your final script should look like Python code.

Python code

# Import packages
from ctypes import *
import multiprocessing as mp
import os
import glob

# Path to the SAInt-API.dll file (located in the SAInt installation folder)
path = "C:\\Program Files\\encoord\\SAInt-v3\\SAInt-API.dll"

# Create a SAInt API DLL object
saint_dll = cdll.LoadLibrary(path)

# Number of CPU cores
cores = mp.cpu_count()

# Source directory
def get_source_directory():
    return os.path.dirname(os.path.abspath(__file__))
source_dir = get_source_directory()

# Main folder
main_dir = source_dir + "\\Networks\\"

# Number of networks to run in parallel
networks = os.listdir(main_dir)

# Variables
core = mp.Semaphore(cores)
all_processes = []

# Simulation function
def RunSimulation(n, core):

        # Network directory
        net_dir = main_dir + networks[n] + "\\"

        # Load the electric network
        saint_dll.openENET(net_dir + "ENET09_12.enet")

        # Load the electric scenario
        scenario = glob.glob1(net_dir, "*.esce")[0]
        saint_dll.openESCE(net_dir + scenario)

        # Include profiles
        saint_dll.includeEPRF(net_dir + "Profiles.prfl")

        # Import events
        saint_dll.importESCE(net_dir + "Events.xlsx")

        # Hide simulation logs
        saint_dll.showSIMLOG(False)

        # Execute simulation
        saint_dll.runESIM()

        # Extract results
        saint_dll.writeESOL(
                source_dir + "Results_Description.xlsx",
                net_dir + "Results.xlsx"
                )

        # Release a CPU core
        core.release()

if __name__ == '__main__':
    #Loop to run scenarios in parallel
    for net in range(len(networks)):
            core.acquire()
            p = mp.Process(target= RunSimulation, args=(net, core))
            all_processes.append(p)
            p.start()
    # Loop to join processes
    for p in all_processes:
        p.join()