Changing Python Environment used by MATLAB

I have mutiple python enviroments on a windows server. I need to change the Python enviroment that MATLAB is using. MATLAB version that I am using is 2017b and I've used the command pyversion('/path/to/python/venv') to change python environment being used by MATLAB. This command gives an error

Python is loaded. The version cannot be changed in this MATLAB session. To change the version, restart MATLAB, and then call 'pyversion'.

To deal with the above error I've made changes to startup.m file by changing two variables

pyPath = '/path/to/python/venv';

pyLibPath = [pyPath 'Lib\site-packages\projects'];

After making changes and restarting MATLAB I get an error

Unable to set Python environment to /path/to/python/venv in this session.
Please restart MATLAB!

I suspect that if results from command py.sys.path contains any path with directory that refers to old python environment I won't be able to set new Python environment. E.g. if the result from command py.sys.path is

['/path/to/python/venv', '','/path/to/python/old_venv']

I need to remove '/path/to/python/old_venv' using remove(py.sys.path, '/path/to/python/old_venv'). I did that but when I restart MATLAB after doing that I see that output from py.sys.path does not change as it still contains '/path/to/python/old_venv'. To be sure that older python paths are removed from py.sys.path I've ran following commands in new Python environment

import sys
sys.path 

Output from above doesnot contain any paths from older Python environment.

Can someone provide a solution on how can I change the Python environment being used by MATLAb?

I've mentioned all the things that I've tried.

[Arturo Pino]

I'm dealing with this exact same issue. After initially setting a python environment, I can't reset even after restarting matlab because it seems to default to the environment I gave it several sessions ago. No matter what I do, it always will load the previously loaded environment and tells me to restart matlab because the environment is already set. So frustrating!

[Jason Kulpe]

I have never been so frustrated, in 15 years of MATLAB usage and wanted to share my experience as its very similar. . I have unit tests running, via Jenkins, in a batch startup mode. My tests were failing only on Mac; windows OK. Running the exact same test commands on my local Mac worked fine so test failing only on a remote system. Per finding this article I have realized the issue and fixed everything. Thanks for the comments and answers!

Original way (tests worked)

% Code in file1.m
pyenv('Version', my_python_dist)

% Execute file2 from inside file1
file2

% This code is in file2.m
% Insert the paths outlined above into the python path
for idx = 1:length(modpaths)
    P = py.sys.path;
    modpath = modpaths(idx);
    if count(P, modpath) == 0
        insert(P, int32(0), modpath);
    end
end

New way - (test failure). I wanted to simplify the commands and reorganize the code. The py.sys.path command, after the pyenv setup, turned out to be the culprit. Perhaps the MATLAB interpreter notice the py.sys usage and defaulted (and loaded) to the system python.

% Code in file1.m
pyenv('Version', my_python_dist)

% This code is in file1.m
% Insert the paths outlined above into the python path
for idx = 1:length(modpaths)
    P = py.sys.path;
    modpath = modpaths(idx);
    if count(P, modpath) == 0
        insert(P, int32(0), modpath);
    end
end

Answer 1

[Yongjian Feng]

virtualenv is challenging. Is it possible to avoid virtualenv?

[Arturo Pino]

I'm not using a virtual environment, I'm just trying to point the App .exe to the correct python distribution to use. It's maintaining a persistent python path, which I can change after restarting matlab, but when I initialize the app window through app designer, it immediately fails to set the environment, even after restarting matlab. Seems like somehow when it compiled/creates the app window it also sets and loads the python persistent path from a previous 'pyenv' call in a previous session, and prevents me from running an 'initializePython' function in the app's startupFcn().

I tried moving all python references (i.e. py.(python reference)) outside of the app itself, only calling python scripts in a .m file, but I don't think that had the intended effect to prevent App Designer from loading a previous python path when it launches the window...

[Gabriel Bergqvist]

Throwing in my 2 cents for this part: "[...] to the correct python distribution to use [...]":

Try exiting + starting Matlab fresh, (i.e. not starting App-Designer, be sure to disable any start-up scripts, avoid auto-loading of previous environment, etc), then setting pyenv manually before pyenv goes to "Status: Loaded". As soon as anything in the startup of Matlab (or related apps) triggers the load of the pyenv, it is too late. The trick - in my case - was to get a chance to change pyenv before anything else "loaded" it. Maybe applicable, maybe not, but kind-of related and worked in my case. The pyenv-handling is quite irritating, albeit powerful, in Matlab. We are migrating towards a separate Python-program and communication using JSON between Matlab and Python worlds, for that reason.

(And if you are on Linux, you may want to throw in a complete computer reboot as part of the "starting Matlab fresh" thing - or maybe that's only relevant for PyCharm, which usually is part of the picture in your typical Matlab-vs-Python bridging development environment.)

Caveat: The above applies to InProcess loading of the pyenv. As for the other variant, maybe terminate() is what you want to google for? Personally, I have found InProcess to work best, at least after you set "pause on".

(Applies to R2021a; I have no idea what's different compared to a 4-year-old version of Matlab)

As for the venv-part: You can achieve a lot using the bash utility "pyenv" (assuming you are on Linux or MacOS - it's slightly less intuitive on Windows, but works there too.) Just be sure to build with --enable-shared, like so:

% BASH for Linux/MacOS:
% ---------------------

% curl https://pyenv.run | bash
% pyenv update
% ... (see current pyenv instructions for details!)

% env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -s 3.9.1  -v 

% pyenv global 3.9.1
% python -V ; pip -V 

After setting up the bash-tool pyenv, then load python from within Matlab with the Matlab-function pyenv, like so:

py_exe_filepath = "<my home folder>/.pyenv/shims/python"
% It also works well with venv, actually:
%      py_exe_filepath = "<my home folder>/virtualenvs/some_venv_3.9.1/bin/python"
pyenv("Version", py_exe_filepath, "ExecutionMode", "InProcess");

[Gabriel Bergqvist]

Completing the above answer, here's some related advice for what usually comes next:

function safe_add_py_path(new_path)
   if sum(cellfun(@string, cell(py.sys.path)) == new_path) == 0
        insert(py.sys.path, int64(0), new_path);
    end
end

safe_add_py_path("/path/to/my/python/source_root";
module_hello = py.importlib.import_module("hello");
my_hello_object = module_hello.HelloClass(params_to_the_init_function_here);
python_data_type_result = my_hello_object.some_hello_method(python_data_type_params)

After that, you probably want to start passing increasingly complex data between Matlab and Python, in which case the difference between the two worlds' data types is going to be a major source of trouble, unless properly handled. If that is the case, I recommend using string-formatted JSON objects between the two worlds. Both Matlab and Python happens to be rather good at reading/writing JSON-strings, so make yourself some nice little wrappers for whatever datatypes you find useful, and/or "stringify" it before sending it to Python (the same goes for the other way as well). Note: This is for advanced use-cases with complex data structures. No need to "stringify" anything for basic use-cases. Example of how to stringify a struct s with JSON-encodable Matlab data on the Matlab side, typically right before sending to it Python:

function stringify_struct = stringify_struct(s)
    stringify_struct = string(jsonencode(s));
end

(On the Python-side, use json.dumps and json.loads)

[Gabriel Bergqvist]

One more thing, seemingly unrelated actions may load the pyenv for you. For example, setting a breakpoint may or may not load the pyenv, depending on exactly in which file you set it (typically setting it in Matlab's internal read-only files may trig this, e.g. getPropertyGroups.m). This makes trying to understand exactly when the pyenv is loaded quite difficult, since the action of setting breakpoints itself may load pyenv. In short, assume pyenv will be loaded at any point during startup of AppDesigner, your App, or the debugger for that matter.

Hence, you may need to explicitly load your preferred pyenv (with whatever config you need) before starting things like AppDesigner, your App, or similar.

From a developer's perspective, I think this should be considered a bug.

I also suspect that certain ways of calling Matlab code may trig the pyenv loading as well. For example, running some commands as a .m function file may load pyenv prematurely (i.e. break the execution) while the same code executed as a .m script file always works (script file = without a function declaration). There is determinism to when Matlab choses to freeze your pyenv for you, albeit absolutely zero transparency or apparent logic. This makes it very hard to use the pyenv functionality in Matlab for more advanced use-cases, unfortunately. It is possible, but bordering reverse-engineering.