Actually this error was caused by a previous line specifying a value for 'PlotFcns'. If this is nonempty, even with 'Display' set to 'off' (or equivalently, 'none'), Matlab will make a call to Java graphics. If fminsearch options' 'PlotFcns' is [] (the default) and 'Display' is 'off', it is indeed possible to run fminsearch with -nojvm.
This leaves open the question how you can debug Matlab on MacOS with Java activated. At this hour I haven't found a way.