First you should test
That will eliminate everything except function handles.
To specifically narrow it down to anonymous functions... I am not sure what the best way is at the moment. I do note, though, that you could use
This would eliminate the case of function handles that are not anonymous functions.
What is your use case for wanting to restrict to anonymous functions? For example why would you want to prevent
[x,n_iter,ERR] = newton_fun(@cos,@(x)-sin(x),2,0.004,100)
and force the user to use
[x,n_iter,ERR] = newton_fun(@(x)cos(x),@(x)-sin(x),2,0.004,100)