Matlab crashes while running a mex function with data.

I have written a mex function(in C) which takes 2 arrays and a scalar as inputs and after doing some mathematical calculations, it returns a scalar quantity as output. I can compile the corresponding mex function successfully on the MATLAB platform but as soon as I am gonna run it with some input data, its causing a crash to MATLAB. The error log has a heading "Segmentation Violation detected at Mon Apr 25 ..:..:.. 2016". I also tried to debug it in Linux platform using the GNU debugger 'gdb'. It is showing a problem with all the if statements that I have used to verify the number and type of input/output arguments using nrhs,prhs[],nlhs,plhs[]. For example my first statement for checking the number of input arguments is,

 if(nrhs!=3) 

 mexErrMsgTxt("Error..Three inputs required.");

and likewise others for nlhs. The GNU debugger is placing its first breakpoint at the above if statement and if I am commenting it out it is causing a problem with the second if statement and likewise. When I am commenting out all the if statements the mex function is running successfully and also giving me the desired output.

It's been long since I am trying to remove this bug but I am not unable to do so. Can you please help me with the above problem. For your assistance I am also attaching the mexFunction content which is causing the above problem.

Thanks in advance.

Answer 1

[James Tursa]

This is causing your problem:

  /* Make sure that the output is of type double and is a scalar */
        if(!mxIsDouble(plhs[0]) || mxIsComplex(plhs[0]) || mxGetNumberOfElements(plhs[0])!=1) 
    mexErrMsgTxt("Error..Image Euclidean Distance must be a scalar.");

plhs[0] is something that you, as the programmer, need to create. It is not something that is pre-existing. So when this if-test is encountered at runtime, plhs[0] is invalid (not yet created) and using it in your test causes access violations and a crash.

Get rid of this entire block of code. You already have code downstream for creating plhs[0].

Also, another check you should do with the input is for sparse. If the input is sparse your code could crash since sparse storage differs from full storage. So add mxIsSparse(prhs[1]) and mxIsSparse(prhs[2]) to your input checking.

Finally, as a suggestion, I would change your nlhs check to allow for the result to be placed directly in ans. E.g., I would change this block of code:

  if(nlhs!=1) 
    mexErrMsgTxt("Error..Only one output required.");

to this:

  if(nlhs>1) 
    mexErrMsgTxt("Error..Only one output required.");

The reason for this is to allow the user to call your function without assigning the result explicitly to a variable (in which case it will go into ans). This will work even if nlhs=0. I.e., MATLAB always allocates space for one output variable in plhs[0], even if nlhs=0. This change will make your code more user friendly.

[Niraj Agarwal]

Alright..very nicely expressed but it's too late to get the answer for my question. I already figured out this problem. Anyways thanks again for your efforts.