(matlab coder)How should the coder::array member function set convert opencv's mat type to its type in-place?

When using matlab coder to generate C++ code, the generator automatically generates a C++ class for coder::array, which is defined in the `coder_array.h` file (located in fullfile(matlabroot,'extern','include','coder','coder _array','coder_array_rtw_cpp11.h')), which has this usage note.

// set(T const *data, SizeType sz1, SizeType sz2, ...)
//                : Set data with dimensions.
//                : (Data is not copied, data is not deleted)

In addition, I have read in an image from an external Opencv and would like to convert it directly in situ using this set() member

cv::Mat srcImg = cv::imread("C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");

// test cv::Mat to coder::array type directly
coder::array<unsigned char, 3U> result;
result.set((unsigned char*)srcImg.data, srcImg.rows, srcImg.cols,srcImg.channels());

But unfortunately the above conversion did not work and the result debugging found empty, how do I use it correctly?

Run in R2022b

Answer 1

[xingxingcui]

After repeated investigations and trials over this period of time, in relation to my current problem, I have found out what the problem is and am sharing the exact answer in full.

Run R2022b, win10, test in mingw64 and vc complier.

Conclusion：

• the member function set in the generated coder::array class is not able to convert opencv's Mat type to coder::array<typename T, int N> type in situ, unless you rewrite the inherited class yourself.
• The entry-point function in Matlab, whether declared as `coder.rowMajor;` or specified directly on the command line using the `-codegen` argument `-rowmajor`, does not change the fact that the matlab array is still in coloum-major form (bug).

What can be done to solve this at the moment is to assign values element by element or to use memcpy to copy the data block form to do it.

-------

With test cases：

function outImg = mytest(inImg)%#codegen
% in = coder.typeof(uint8(0),[1000,1000,3],[1,1,0]);% :1000x:1000x3
% codegen -config:lib -c mytest -args in -lang:c++ -report

arguments
    inImg uint8
end
outImg = imresize(inImg,0.5);
end

use above comment in command line, generated C++ code,test set() method :

// omit some code

// my custom function, use to convert opencv's mat to matlab matrix
void convertCVToMatrix(cv::Mat &srcImg, int rows, int cols, int channels,
                       unsigned char dst[])
{
  CV_Assert(srcImg.type() == CV_8UC1 || srcImg.type() == CV_8UC3);
  size_t elems = rows * cols;
  if (channels == 3) {
    cv::Mat channels[3];
    cv::split(srcImg.t(), channels);
    memcpy(dst, channels[2].data,
           elems * sizeof(unsigned char)); // copy channel[2] to the red channel
    memcpy(dst + elems, channels[1].data,
           elems * sizeof(unsigned char)); // green
    memcpy(dst + 2 * elems, channels[0].data,
           elems * sizeof(unsigned char)); // blue
  } else {
    srcImg = srcImg.t();
    memcpy(dst, srcImg.data, elems * sizeof(unsigned char));
  }
}

int main(int, char **)
{
  // The initialize function is being called automatically from your entry-point
  // function. So, a call to initialize is not included here. Invoke the
  // entry-point functions.
  // You can call entry-point functions multiple times.
  {
    coder::array<unsigned char, 3U> inImg;
    coder::array<unsigned char, 3U> outImg;

    cv::Mat srcImg = cv::imread(
        "C:/Program Files/MATLAB/R2022b/toolbox/matlab/imagesci/peppers.png"); // 384*512*3 size

    // Since set_size() was set beforehand, then after set srcImg retains the array size of the set method, i.e. the prior set_size method is invalid! Also, if only set is used for the conversion, there is a dimension mismatch problem, i.e. it leads directly to the later calculation in the wrong dimension!
    // inImg.set_size(384, 512, 3);
    // inImg.set((unsigned char *)dst, srcImg.channels(), srcImg.cols,srcImg.rows); 
    

    // The following 2 sentences are the effective and correct way
     inImg.set_size(384, 512, 3);
     convertCVToMatrix(srcImg, srcImg.rows, srcImg.cols, srcImg.channels(),inImg.data());
     
     
    // Call the entry-point 'mytest'.
    mytest(inImg, outImg);
 

    // output matlab matrix `outImg` to file,so i can read this matrix into matlab to show for debug.
    std::cout << " outImg.size:" << outImg.size(0) << "," << outImg.size(1)
              << "," << outImg.size(2) << std::endl;
    std::ofstream fid("outImg.txt", std::ios::out);
    for (size_t i = 0; i < outImg.size(0); i++) {
      if (i != 0) {
        fid << std::endl;
      }

      for (size_t j = 0; j < outImg.size(1); j++) {
        fid << (int)outImg.at(i, j, 0) << " "; // only see R channel array.
      }
    }
    fid.close();
  }
    // Terminate the application.
  // You do not need to do this more than one time.
  mytest_terminate();
  return 0;
}

// End of code generation (main.cpp)

Read the text file output from the above C++ code in matlab：

aa = uint8(readmatrix("outImg.txt"));figure;imshow(aa)

debug found that the only way to display the image correctly was to assign it element by element or my custom conversion function convertCVToMatrix() above.

In addition, due to space limitations above, I have tried specifying `coder.rowMajor` in the matlab entry function or `-rowmajor` on the command line, but this does not work, i.e. the resulting C++ code is still in coloum-major

However, thanks very much to @Alexander Bottema for his positive answer, even if it didn't solve the problem directly.

Answer 2

[Alexander Bottema]

The issue here is that MATLAB (and MATLAB Coder) uses column-major format for matrices. This means the first dimension is the one that traverses memory locations consecutively, whereas C usually uses row-major memory layout; the last dimension traverses consecutive memory locations. You can toggle between the two memory layouts by reversing the size vector. You can either do this manually, or automatically. Manually:

codegen -args { coder.typeof(uint8(0), [inf inf inf] } mytest

function im = mytest(im)
% You have size(im,1) == channels, size(im,2) == columns, size(im,3) == rows
...
    
and:

coder::array<unsigned char, 3U> result;
result.set((unsigned char*)srcImg.data, srcImg.channels(), srcImg.cols, srcImg.rows);

mytest(arr);

If the top-level input is in row-major format you can also annotate your function to use row-major, and thus you don't need to reverse the size vector in your head:

function im = mytest(im)
coder.rowMajor;
% You have size(im,1) == rows, size(im,2) == cols, size(im,3) == channels
...
    
and:

coder::array<unsinged char, 3U> result;
result.set((unsigned char*)srcImg.data, srcImg.rows, srcImg.cols, srcImg.channels());

mytest(arr);

[xingxingcui]

@Alexander Bottema，

Hi, at the moment these method does not work, the elements of the matrix do change after set(), but this is only valid in the scope of the function, however the array as a return parameter does not return its data, after debugging, its internal member variable `owner_` has changed to false, is this the cause? Then I tried to set_owner to true and it threw an exception straight away.

--------------------

Strangely enough, the following program always works using the traditional element-by-element assignment, and `owner_` is always true, and the array returns no change to the data within the other functions!

  cv::Mat srcImg = cv::imread("C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");
  result.set_size(srcImg.rows, srcImg.cols, srcImg.channels());
  // Loop over the array to initialize each element.
 for (int idx0{0}; idx0 < result.size(0); idx0++) {
   for (int idx1{0}; idx1 < result.size(1); idx1++) {
     for (int idx2{0}; idx2 < result.size(2); idx2++) {
       // Set the value of the array element.
       // Change this value to the value that the application requires.
       result[(idx0 + result.size(0) * idx1) +
               result.size(0) * result.size(1) * idx2] =
           (unsigned char)srcImg.at<cv::Vec3b>(idx0, idx1)[idx2];
    }
 }
}

---------------------

I then subsequently tried the set_size() or reshape() member functions to change to the dimension size I initially wanted, but passing the `result` variable into the other functions (the return result method) the data was always 205, which was very weird and awkward to use.

result.set_size(srcImg.rows, srcImg.cols, srcImg.channels());// or use bellow line
result = result.reshape(srcImg.rows, srcImg.cols, srcImg.channels());
return result;

Apparently, I use MATLAB's built-in "peppers.png" image, type uint8, and the pixel value 205 does not appear when I look at the original pixels

[xingxingcui]

for reference only:

static coder::array<unsigned char, 3U> argInit_d1000xd1000xd3_uint8_T()
{
  coder::array<unsigned char, 3U> result;

  cv::Mat srcImg = cv::imread(
      "C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");
  result.set_size(srcImg.rows, srcImg.cols, srcImg.channels());
  // Loop over the array to initialize each element.
  // for (int idx0{0}; idx0 < result.size(0); idx0++) {
  //  for (int idx1{0}; idx1 < result.size(1); idx1++) {
  //    for (int idx2{0}; idx2 < result.size(2); idx2++) {
  //      // Set the value of the array element.
  //      // Change this value to the value that the application requires.
  //      result[(idx0 + result.size(0) * idx1) +
  //             result.size(0) * result.size(1) * idx2] =
  //          (unsigned char)srcImg.at<cv::Vec3b>(idx0, idx1)[idx2];
  //    }
  //  }
  //}

  result.set((unsigned char*)srcImg.data, srcImg.channels(), srcImg.cols,
             srcImg.rows);
 // result.set_owner(true);
  //result.set_size(srcImg.rows, srcImg.cols, srcImg.channels());
  result = result.reshape(srcImg.rows, srcImg.cols, srcImg.channels());

  return result;
}

[Alexander Bottema]

The issue here is that coder::array and OpenCV use different memory managers. coder::array has support to not bother about memory managers when the owner is set to false (set_owner(false), which also happens implicitly if you use set(....)). You should never set_owner(true) for a coder::array object that hasn't allocated the data. If you use "set_size", then there's a chance that coder::array will reallocate the data, and when that happens, it owns the data; i.e. the memory manager of coder::array owns it and also knows how to destroy it.

The similar is true for OpenCV, and there's no guarantee that both memory managers are compatible with each other. Furthermore, when coder::array has (re)allocated the data, then the data() pointer will be something different and does becomes detached from cv::Mat's representation.

So to transfer the data from coder::array back to a cv::Mat would require reading the documentation of OpenCV to see if there's a way to create a similar object as you can with coder::array so that it doesn't own the data. I don't know whether this is possible to do so. Skimming through the documentation of OpenCV indicates that this is a reference counted system and uses a flexible system where memory can optionally be on a CUDA host, etc. There's something called a MatAllocator as well, but it requires a context to be used. In general, the memory management for OpenCV is quite different from coder::array.

Furthermore, even if you manage to transfer the data pointer to the cv::Mat object you would also need to make sure that coder::array uses the same memory manager as OpenCV. For this there are macro hooks for coder::array that you can use (CODER_ARRAY_ALLOC, CODER_ALLOC, CODER_DEALLOC). Once the memory is transferred you need to set_owner(false) on the coder::array object to indicate that it shouldn't own the memory (and will not get freed when its destructor is called.)

Memory interoperability between OpenCV and MATLAB Coder is no easy task, but I think it is possible. I'm not sure how though.

The safest option is to copy the data between OpenCV and MATLAB Coder. Then each respective memory manager knows how to deal with it.

[Alexander Bottema]

Another idea is to write your own custom coder::array class that wraps a cv::Mat. coder::array is a boilerplate template class (and is not auto generated.) It is by design so that you can replace it with your own custom coder::array version. There aren't that many methods you need to implement.

[Alexander Bottema]

Just to clarify:

1) coder::array has two modes: either it owns the data (owner = true) which means that it is supposed to be in control over the data. It knows how to destroy it, etc. This is likely incompatible with cv::Mat.

2) it does not own the data. This happens when you use the ".set" method. Then you need to make sure that coder::array doesn't reallocate the memory (don't call set_size, or anything else that may result in memory reallocation)

In the first version you posted you issued "set_size" and then iterated over the pixels to transfer the data. This means that coder::array owns the data. You need to do a similar operation to transfer it back to an cv::Mat. You cannot mix (1) and (2). So you need to choose between set_size() + writing pixels or only use set() (and then make sure coder::array doesn't reallocate it = don't use set_size on it)

If coder::array allocates its data, then you need to create a new cv::Mat and copy over the data. Again, this is because the two representations use different memory managers. The only way of making sure that there are no copying between coder::array and cv::Mat is to make sure they are fully compatible. This would require you to write your own custom coder::array that uses cv::Mat.

[Alexander Bottema]

I just saw in the documentation of OpenCV that there's a similar way of creating an non-owned cv::Mat (that you can do with coder::array's set method):

cv::Mat(int rows, int cols, int type, char* preAllocatedPointerToData);

So there's a constructor which enables you to pass a pre-allocated piece of data.

This means that whenever coder::array owns the data, you can do:

coder::array<unsigned char, 3U> arr;
arr.set_size( ... );
...

cv::Mat im(arr.size(0), arr.size(1), CV_8UC3, arr.data());
.... to be used in OpenCV ...
    
coder::~array() is called and frees the data of 'arr'.

[xingxingcui]

@Alexander Bottema

Thank you so much for your professional and detailed elaboration and clarification! It made me understand the details of the memory management aspect of them, thanks again sincerely!

------------

However, I solved my actual problem in the following way, but instead of an in-place operation(I.e. no deep copy of memory occurs),it is essentially still went through the deep copy method.

static coder::array<unsigned char, 3U>& argInit_d1000xd1000xd3_uint8_T()
{
  coder::array<unsigned char, 3U> result,temp;
  cv::Mat dst;
  cv::Mat srcImg = cv::imread(
      "C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");

 unsigned char matlab_data[589824]; // numbers of pixels, peppers.png,384*521*3=589824
 convertCVToMatrix(srcImg, srcImg.rows, srcImg.cols, srcImg.channels(),
                   matlab_data);

result.set((unsigned char *)matlab_data,srcImg.rows, srcImg.cols,srcImg.channels());

  return result; // This return value has the correct dimension and matlab type memory layout！
}

note: owner_ of "result" variable becomes true when the function returns a value to an external scope.

Where above function convertCVToMatrix is a memory transformation from opencv row-dominated to MATLAB column-dominated.

static void convertCVToMatrix(cv::Mat &srcImg, int rows, int cols, int channels,
                       unsigned char dst[])
{
  CV_Assert(srcImg.type() == CV_8UC1 || srcImg.type() == CV_8UC3);
  size_t elems = rows * cols;
  if (channels == 3) {
    cv::Mat channels[3];
    cv::split(srcImg.t(), channels);

    memcpy(dst, channels[2].data,
           elems * sizeof(unsigned char)); // copy channel[2] to the red channel
    memcpy(dst + elems, channels[1].data,
           elems * sizeof(unsigned char)); // green
    memcpy(dst + 2 * elems, channels[0].data,
           elems * sizeof(unsigned char)); // blue
  } else {
    srcImg = srcImg.t();
    memcpy(dst, srcImg.data, elems * sizeof(unsigned char));
  }
}

---------------

BTW, the original Manually method given by @Alexander Bottema solves the problem of memory layout arrangement, but not solve dimensionality of the "result" variable, and I can't get the correct dimension unless I rewrite the subclass implementation of the coder::array class((similar matlab build-in function permute)), which is more problematic.

The second way, by asserting coder.rowMajor, formally solves the dimensionality problem but not the OpenCV channel order problem, i.e. the BGR to RGB order conversion, which seems to require a little manual processing.

[xingxingcui]

In addition, regarding the pixel value of 205 that kept appearing yesterday, switching to any other image would have returned a pixel value of 205, which is indeed a problem that most likely occurs in the set() function of coder::array!

my env:

Microsoft Visual Studio Enterprise 2019 version 16.11.5

Matlab R2022b

windows10

[Alexander Bottema]

The argInit version you wrote above is incorrect. You return a reference to a coder::array that has been allocated in the local stack frame. This is an invalid operation in C++. In general, I discourage you to return coder::array from functions. Pass them around by reference instead.

[Alexander Bottema]

To clarify, I recommend something like this:

void argInit( coder::array<unsigned char> &result ) {

...

}

Furthermore, you should be very careful when using .set(...) which does not copy tbe data. This means that the data pointer that is being wrapped by the coder::array is owned elsewhere and whenever that object is freed, then the data will be freed as well.

[xingxingcui]

You're right, but that's not the main problem. Of course I could have returned coder::array without the reference form, by copy assignment, like the following, and not achieved the solution to my original problem.

static coder::array<unsigned char, 3U> argInit_d1000xd1000xd3_uint8_T()
{
  coder::array<unsigned char, 3U> result,temp;
  cv::Mat dst;
  cv::Mat srcImg = cv::imread(
      "C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");
 unsigned char matlab_data[589824]; // numbers of pixels, peppers.png,384*521*3=589824
 convertCVToMatrix(srcImg, srcImg.rows, srcImg.cols, srcImg.channels(),
                   matlab_data);
result.set((unsigned char *)matlab_data,srcImg.rows, srcImg.cols,srcImg.channels());
temp = result;
  return temp; // This return value has the correct dimension and matlab type memory layout！
}

[Alexander Bottema]

Your argInit_.... function have are more problems. You allocate a variable in the function's local stack frame (matlab_data), and then wrap that data inside a coder::array (that you attempt to return). Once the function returns, the stack allocated matlab_data is deallocated and you can no longer trust the data inside coder::array. You need to be very careful on who owns the memory and how. C/C++ in general can be complicated when it comes to memory management.

[xingxingcui]

I understand what you're saying, so change the interface to the way it was originally generated by the MATLAB coder, like this

static coder::array<unsigned char, 3U> argInit_d1000xd1000xd3_uint8_T();

Again, this way the "result" variable is automatically changed to true in the external scope owner_, and the way it was yesterday, the pixel values inside become 205, which is very unusual

[xingxingcui]

I initially just didn't return by reference, and only after discovering many of the above problems did I return by reference for debugging purposes, but the problems persisted.

[Alexander Bottema]

Again, I'd recommend that you don't return coder::array and instead use the pattern as shown above; i.e. pass the variables as a function argument by reference. I know that the example main code generation does return coder::array, but it is of historical reasons what we did for C code generation and that idiom got transferred to the C++ version. Note that all other code generation logic doesn't do this. It's only the example function that returns coder::array.

The reason why you see changes in pixel values has nothing to do with coder::array, set() or anything else. It's just the fact that the pixel data gets deallocated (somewhere) and the coder::array uses a dangling pointer.

If you use .set() to reuse an existing pointer, and the original data gets deallocated (either on stack, or by some other memory manager), then the coder:array data will continue reference unallocated data.

Changing a function to return a coder::array by reference isn't suitable for debugging purpose either; it is undefined behavior in C++. You cannot trust the code, not even during debugging.

[Alexander Bottema]

Instead of doing this:

static coder::array<unsigned char, 3U> argInit_d1000xd1000xd3_uint8_T()
{
    coder::array<unsigned char, 3U> result,temp;
    cv::Mat dst;
    cv::Mat srcImg = cv::imread(
"C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");
    unsigned char matlab_data[589824]; // numbers of pixels, peppers.png,384*521*3=589824
    convertCVToMatrix(srcImg, srcImg.rows, srcImg.cols, srcImg.channels(),
        matlab_data);
    result.set((unsigned char *)matlab_data,srcImg.rows, srcImg.cols,srcImg.channels());
    temp = result;
    return temp; // This return value has the correct dimension and matlab type memory layout！
}

Do this instead:

static void argInit_d1000xd1000xd3_uint8_T(coder::array<unsigned char, 3U> &result)
{
    cv::Mat dst;
    cv::Mat srcImg = cv::imread(
"C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");
    unsigned char matlab_data[589824]; // numbers of pixels, peppers.png,384*521*3=589824
    result.set_size(srcImg.rows, srcImg.cols, 3);
    convertCVToMatrix(srcImg, srcImg.rows, srcImg.cols, srcImg.channels(),
        result.data());
}

Don't rely on coder::array move semantics, especially when you wrap an existing data pointer.

[xingxingcui]

@Alexander Bottema

Again, I'd recommend that you don't return coder::array and instead use the pattern as shown above; i.e. pass the variables as a function argument by reference. I know that the example main code generation does return coder::array, but it is of historical reasons what we did for C code generation and that idiom got transferred to the C++ version. Note that all other code generation logic doesn't do this. It's only the example function that returns coder::array.

This is not a major issue, whether it is placed on a per-copy value return value or as a function reference to an output parameter, it has no bearing on the above issue.

---------

The reason why you see changes in pixel values has nothing to do with coder::array, set() or anything else. It's just the fact that the pixel data gets deallocated (somewhere) and the coder::array uses a dangling pointer.

I don't think the lack of change in image pixel values is irrelevant to coder::array for the following reasons：

use original set() method

 static void argInit_d1000xd1000xd3_uint8_T(coder::array<unsigned char, 3U> &result)
{
  cv::Mat dst;
  cv::Mat srcImg = cv::imread(
      "C:\\Program "
      "Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");
  // or add result.set_size(srcImg.rows, srcImg.cols, 3); // Debugging up to this point is the constant 205 pixel value, a dangling pointer caused by coder::array ???
  result.set((unsigned char *)srcImg.data,
             srcImg.channels(),srcImg.cols,srcImg.rows);
}

The "result" variable is indeed correctly assigned a pixel value within the scope of the function, but once out of scope to an external function, the pixel value is unchanged (this time it's a 215 constant or some other value, i.e. a dangling pointer reference as you say)

Again, the downside of this approach is that you can't directly get the "result" variable correct dimension to pass to the external function.( for example, mytest(result) )

---------

Instead of doing this:

static coder::array<unsigned char, 3U> argInit_d1000xd1000xd3_uint8_T()
{
    coder::array<unsigned char, 3U> result,temp;
    cv::Mat dst;
    cv::Mat srcImg = cv::imread(
"C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");
unsigned char matlab_data[589824]; // numbers of pixels, peppers.png,384*521*3=589824
convertCVToMatrix(srcImg, srcImg.rows, srcImg.cols, srcImg.channels(),
matlab_data);
result.set((unsigned char *)matlab_data,srcImg.rows, srcImg.cols,srcImg.channels());
temp = result;
return temp; // This return value has the correct dimension and matlab type memory layout！
}

Do this instead:

static void argInit_d1000xd1000xd3_uint8_T(coder::array<unsigned char, 3U> &result)
{
    cv::Mat dst;
    cv::Mat srcImg = cv::imread(
"C:\\Program Files\\MATLAB\\R2022b\\toolbox\\matlab\\imagesci\\peppers.png");
unsigned char matlab_data[589824]; // numbers of pixels, peppers.png,384*521*3=589824
result.set_size(srcImg.rows, srcImg.cols, 3);
convertCVToMatrix(srcImg, srcImg.rows, srcImg.cols, srcImg.channels(),
result.data());
}

Don't rely on coder::array move semantics, especially when you wrap an existing data pointer.

Both of these methods work, except for a little difference in the form of the return argument, as both use my own converted memory method function convertCVToMatrix()，but it use deep-copy array,not in-place by set()

[xingxingcui]

Therefore, it would be better to give direct examples of actual proofs of the set() method, in-place methods, in code that actually works! However, after trying hard above, I can only get it to work by copying element by element or by copying the whole memory block separately.