Importing custom python module fails

49 views (last 30 days)
I am using Matlab2014b and try to import python packages. I am on a linux computer and executing the command py.numpy.* works and py.importlib.import_module('numpy') gives me the correct output. However, numpy lives in my /usr/lib/pymodules/python2.7/ directory, which according to the output of py.sys.path is in the search directory, so the import should work.
However, I have another package that I would like to include, which is in /usr/lib/python2.7/dist-packages/. Again, according to py.sys.path, this directory is included in the search path, however, importing the package fails ("Import argument 'py.fabio' cannot be found or cannot be imported."). I hope someone has an advice how to solve this problem.
PS: pyversion returns my current installation (2.7) and shows isloaded: 1
PPS: Adding the directory to the search path, as outlined in http://www.mathworks.com/help/matlab/matlab_external/call-python-from-matlab.html has failed.
PPPS: When I try to get feedback from py.importlib.import_module, Matlab crashes.
  4 Comments
Fernando Bechtelar
Fernando Bechtelar on 27 Jan 2016
This is the stack trace when I try to include the fabio package. I tried to skim it a bit to make to post not too cluttered.
Stack Trace (from fault):
[ 0] /lib/x86_64-linux-gnu/libc.so.6+00205157 gsignal+00000053
[ 1] /lib/x86_64-linux-gnu/libc.so.6+00218080 abort+00000384
[ 2] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libhdf5.so.8+00224629 H5check_version+00000325
[ 3] /usr/lib/python2.7/dist-packages/h5py/h5f.so+00042095 inith5f+00006975
[ 4] /usr/lib/libpython2.7.so+01579985 _PyImport_LoadDynamicModule+00000193
[ 5] /usr/lib/libpython2.7.so+01109372
[ 6] /usr/lib/libpython2.7.so+01110002
[ 7] /usr/lib/libpython2.7.so+01110925 PyImport_ImportModuleLevel+00000605
[ 8] /usr/lib/libpython2.7.so+00666527
[ 9] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 10] /usr/lib/libpython2.7.so+01271431 PyEval_CallObjectWithKeywords+00000071
[ 11] /usr/lib/libpython2.7.so+00306307 PyEval_EvalFrameEx+00004915
[ 12] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 13] /usr/lib/libpython2.7.so+00328738 PyEval_EvalCode+00000050
[ 14] /usr/lib/libpython2.7.so+00685773 PyImport_ExecCodeModuleEx+00000173
...
[ 19] /usr/lib/libpython2.7.so+01110685 PyImport_ImportModuleLevel+00000365
[ 20] /usr/lib/libpython2.7.so+00666527
[ 21] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 22] /usr/lib/libpython2.7.so+01271431 PyEval_CallObjectWithKeywords+00000071
[ 23] /usr/lib/libpython2.7.so+00306307 PyEval_EvalFrameEx+00004915
[ 24] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 25] /usr/lib/libpython2.7.so+00328738 PyEval_EvalCode+00000050
[ 26] /usr/lib/libpython2.7.so+00685773 PyImport_ExecCodeModuleEx+00000173
...
[ 30] /usr/lib/libpython2.7.so+01110925 PyImport_ImportModuleLevel+00000605
[ 31] /usr/lib/libpython2.7.so+00666527
[ 32] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 33] /usr/lib/libpython2.7.so+01271431 PyEval_CallObjectWithKeywords+00000071
[ 34] /usr/lib/libpython2.7.so+00306307 PyEval_EvalFrameEx+00004915
[ 35] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 36] /usr/lib/libpython2.7.so+00328738 PyEval_EvalCode+00000050
[ 37] /usr/lib/libpython2.7.so+00685773 PyImport_ExecCodeModuleEx+00000173
...
[ 41] /usr/lib/libpython2.7.so+01110925 PyImport_ImportModuleLevel+00000605
[ 42] /usr/lib/libpython2.7.so+00666527
[ 43] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 44] /usr/lib/libpython2.7.so+01271431 PyEval_CallObjectWithKeywords+00000071
[ 45] /usr/lib/libpython2.7.so+00306307 PyEval_EvalFrameEx+00004915
[ 46] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 47] /usr/lib/libpython2.7.so+00328738 PyEval_EvalCode+00000050
[ 48] /usr/lib/libpython2.7.so+00685773 PyImport_ExecCodeModuleEx+00000173
...
[ 53] /usr/lib/libpython2.7.so+01110685 PyImport_ImportModuleLevel+00000365
[ 54] /usr/lib/libpython2.7.so+00666527
[ 55] /usr/lib/libpython2.7.so+00324642 PyEval_EvalFrameEx+00023250
[ 56] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 57] /usr/lib/libpython2.7.so+00682263
[ 58] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 59] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwpycli.so+00299020
[ 60] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwpycli.so+00289121
[ 61] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwpycli.so+00425954
[ 62] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcos_impl.so+01540515
...
[ 66] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcos_impl.so+02046678
[ 67] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwm_lxe.so+09159856
...
[ 81] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwbridge.so+00168636
[ 82] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwbridge.so+00171767 _Z8mnParserv+00000823
[ 83] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00686927 _ZN11mcrInstance30mnParser_on_interpreter_threadEv+00000031
[ 84] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00607299
[ 85] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00608825 _ZN5boost6detail11task_objectIvNS_3_bi6bind_tIvPFvRKNS_8functionIFvvEEEENS2_5list1INS2_5valueIS6_EEEEEEE6do_runEv+00000025
[ 86] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00614215 _ZN5boost6detail9task_baseIvE3runEv+00000071
[ 87] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00614311
[ 88] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00594682
[ 89] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwuix.so+00313014
[ 90] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwuix.so+00206946
[ 91] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwservices.so+03002591
...
[ 94] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwservices.so+03012684 _Z25svWS_ProcessPendingEventsiib+00000092
...
[ 97] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00516077
[ 98] /lib/x86_64-linux-gnu/libpthread.so.0+00027472
[ 99] /lib/x86_64-linux-gnu/libc.so.6+00899421 clone+00000109

Sign in to comment.

Accepted Answer

Robert Snoeberger
Robert Snoeberger on 28 Jan 2016
Edited: Robert Snoeberger on 28 Jan 2016
Based on the stack trace, your issue looks like a library conflict with libhdf5.so. It seems like the libhdf5 that is in the stack trace is a library that is installed with MATLAB; I am assuming this based on the '/matlab_2015b/bin/glnxa64/' in the path to the library. I would guess that the appropriate version of libhdf5.so for Python should be located at /usr/lib/libhdf5.so. Try the following command to look for the correct libhdf5.so.
ldd /usr/lib/python2.7/dist-packages/h5py/h5f.so | grep libhdf5
My hunch about the library conflict is further confirmed by the fact that the libhdf5.so function that is executing is called H5check_version (line [2] of the stack trace). The documentation I found for that function says, "If this check fails, H5check_version causes the application to abort (by means of a standard C abort() call)..." [1].
Assuming you aren't using MATLAB's interface to HDF5, you should be able to use LD_PRELOAD to load the correct version of libhdf5.so [2,3].
References
  1. https://www.hdfgroup.org/HDF5/doc1.6/RM_H5.html#Library-VersCheck
  2. http://man7.org/linux/man-pages/man8/ld.so.8.html -- Search for "LD_PRELOAD". It should say, "A list of additional, user-specified, ELF shared objects to be loaded before all others..."
  3. http://stackoverflow.com/questions/426230/what-is-the-ld-preload-trick
  6 Comments
Michael Thieme
Michael Thieme on 18 Oct 2018
Edited: Michael Thieme on 18 Oct 2018
All,
I found a solution to this problem. My problem was related to tensorflow/keras rather than fabio. This thread (https://github.com/NeurodataWithoutBorders/api-python/issues/28) led me to the solution. Basically my order of operations was:
  • conda uninstall hdf5
  • conda uninstall h5py
  • pip uninstall hdf5
  • pip uninstall h5py
  • conda install hdf5=1.8.12 h5py=2.7.0 -c jonc
  • pip install keras
  • pip install tensorflow-gpu

Sign in to comment.

More Answers (1)

Rafael Rebouças
Rafael Rebouças on 19 Jan 2021
Edited: Rafael Rebouças on 19 Jan 2021
My solution:
% Simple way to load your library
copyfile(my_library_file_path, pwd, 'f');
% It's necessary to load library
py.my_library.any_method_or_function();
delete 'my_library.py';
Now, your Python library is loaded to use in any directory.
  1 Comment
Ronan Fleming
Ronan Fleming on 12 Apr 2021
https://nl.mathworks.com/help/releases/R2021a/matlab/matlab_external/out-of-process-execution-of-python-functionality.html

Sign in to comment.

Tags

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by