Frame matching in a video

I have an image and should check whether it is present in a video. Image will not be same in the video little rotated and scaled so tried surf matching technique but if the video recorded is slightly rotated no of matched features are less, so cannot check image is present in the video based on count of the matched features. Will histogram matching work as it does not change even when image is slightly rotated please give me a solution for this problem.

[Walter Roberson]

I thought SURF was independent of rotation ?

[Akhilkumar1307]

This happens when images are taken from different cameras with slightly varied calibration and lighting condition

Any way now am taking the reference image and videos to be tested from same camera and error is almost negligible even if it is rotated.