Azzera filtri
Azzera filtri

Clarification regarding 2GB transfer limit between MATLAB and Python

3 visualizzazioni (ultimi 30 giorni)
Could someone please clarify what exactly is meant by 'The size of variables passed between Python and MATLAB is limited to 2 GB when you call a Python function out-of-process. This limit applies to the data plus supporting information passed between the processes.' on https://uk.mathworks.com/help/matlab/matlab_external/limitations-to-python-support.html?
I want to run a sensitivity study using the MATLAB-based UQLab library (https://www.uqlab.com/), which comprises 12 evaluations of a custom Python model spanning many files. The Python model (Python 3.9) is treated as a black box function, which is executed from MATLAB as follows (NOTE: pyenv(ExecutionMode = "OutOfProcess")):
pyrunfile(strcat("python_file.py ", input_variables), output_variable);
where
"python_file.py " is the master script managing my Python model
input_variables is a JSON string of model inputs
output_variable is a single float constituting the model output
Halfway through the 11th model evaluation, MATLAB returns a Python package-specific error ("Python Error: TypeError: Numerical inputs to HAPropsSI must be ints, floats, lists, or 1D numpy arrays."). Now, when I ONLY run the 11th model evaluation I don't get an error, and the results of the individual evaluation are identical to those from the full set of evaluations up to the point where the latter failed. Moreover, when I run all 12 evaluations on a subset of my model (less computationally/memory intensive), I don't get an error either. After repeating the aforementioned steps multiple times, the error message always occurred in the exact same location, halfway through the 11th model evaluation.
All this suggests to me that I am exceeding the 2GB transfer limit between MATLAB and Python, but I don't understand what is meant by 'size of variables passed between Python and MATLAB'? The inputs and outputs described above are only a few byte, but I wouldn't know what falls under 'supporting information'.

Risposte (1)

Sarthak
Sarthak il 30 Apr 2024
Hi Nils,
I think you are facing this error because you are transferring full simulation data for each run to MATLAB which is exceeding the transfer limit. Since you want to run a sensitivity analysis, methods such as Sobol’ indices are usually applied to a limited set of relevant quantities of interest post-processed from your simulation. I believe only this information should be transferred and not the full simulation data for each run which should resolve this error.
I hope this helps!
  1 Commento
Nils Barner
Nils Barner il 2 Mag 2024
Dear Sarthak,
Thank you for your answer. I am pretty certain that I am only transferring the relevant quantities required to calculate Sobol' indices, which should amount to a few kilobytes at most.
Best wishes,
Nils

Accedi per commentare.

Prodotti


Release

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by