mxSetData to subset of mxArray

Is it possible, perhaps with a certain set of restrictions on what the user can do, to assign data to an mxArray which is a subset of some memory that has already been allocated?

[Jim Hokanson]

A comment on the use case: I have a long string and I'd like to take parts of that long string and dole it out to substrings for struct entries.

Answer 1

[James Tursa]

Not without a hack. The reason is as follows:

The API functions mxSetPr, mxSetPi, mxSetData, mxSetImagData, mxSetIr, and mxSetJc etc all check to see if the pointer passed in for the data memory is from the MATLAB Memory Manger allocation list. If it isn't, then the mex routine will crash MATLAB with an assertion fault. In practical terms, this means that you need to individually allocate each memory block using MATLAB API functions ... you can't allocate one big block and then piece it out to multiple mxArray's, and you can't use C native allocated memory either (e.g., can't use malloc and friends).(*)

So, the only way to attach such memory to the data area of an mxArray is to hack into the mxArray structure itself and set the desired pointer manually. But this leaves you in a precarious state, because if MATLAB ever tried to destroy this mxArray (e.g., maybe there is an error return from your mex routine and garbage collection takes place) then MATLAB would crash when it tried to free this memory. So you could do some things with it (e.g., as an input to mexCallMATLAB where you knew the function you were calling would not make a shared data copy of the input). But to be safe you would probably need to protect this mxArray from being garbage collected (i.e., make it persistent and set up a mexAtExit function to clean things up properly), and you would need a way for the mex routine to safely detach the data pointers before this mxArray gets destroyed. Bottom line is that doing this is possible, but very tricky to get things set up so that MATLAB does not crash ... and it requires a lot of hacks.

(*) Note: Earlier versions of MATLAB did not make these checks, so you could get away with using the API functions to set the data pointers with non-MATLAB allocated memory. But you would still be faced with a potential crash if MATLAB ever tried to destroy this mxArray while it was in the invalid state.

[Jim Hokanson]

It seems like ideally Matlab would have some algorithm like the following.

a = 1:4

b = a(2:3)
%or a better syntax in terms of an explicit range
b = view(a,2,3);
%1) Create a header entry with a data pointer starting at the appropriate point based on the indexing
%2) Set a flag in 'b' that this is a reference to some other data
%3) Update in 'a' a variable that keeps track of the # of arrays that are a subset of 'a'

%Let's say we rewrote 'b'
b(1) = 5;
%1) 'b' is linked, so do a memory copy of the relevant parts of 'a'
%2) decrement in 'a' the subset count

%Let's say we delete 'b'
%1) decrement in 'a' the subset count

%Let's say we change 'a'
%Then we take a big hit, and we need to go through our variables until we find everything pointing to it, and delink it. We could potentially be smart about how we do this in terms of covered ranges and not delinking everything ...

%Let's say we delete 'a'
%Perhaps we don't actually delete it until the subset count is 0

From what you have said, it sounds like an approach that might work is the following: 1) create new headers 2) manually set the pointer instead of using mxSetData 3) update the reference count somewhere

This is where I get confused. With this approach I want something like the following:

a = 1:10;
b = a(2:3); %Increment ref count of a
c = a(5:6); %Increment ref count of a
d = a(9:10); %Increment ref count of a

So now I have 4 'a's, and it isn't until all of them are deleted that 'a' actually get's freed. The problem then, is that 'a' needs to be the last 'a' freed, as it is the only valid 'a' (i.e. we can't try and free the other data pointers since they don't point to the start of the data). I think this is where your persistence approach would come in so that we guarantee that 'a' is the last 'a'?

Unfortunately, I don't see any way to know when 'a' finally becomes the last 'a', without doing some sort of periodic check myself ...

Am I understanding the situation correctly? I wonder if TMW would be sympathetic to a feature request ...?

[James Tursa]

"... I don't see any way to know when 'a' finally becomes the last 'a' ..."

You could create a shared data copy of "a", make it persistent, and keep it inside the mex routine with the mex routine locked with mexLock(). MATLAB would not be able to destroy this mxArray because it isn't on the garbage collection list and is not in any workspace either. Destroying it would be part of a mexAtExit procedure, but it would only be destroyed if there were no other shared data copies (detectable from the CrossLink field of the mxArray structure), no other reference copies (detectable from the refCount field of the mxArray structure), and no other subset copies (detectable some TBD way) in existence. That last part would be very tricky ... I don't have any suggestion for that yet (other than being part of some OOP class).