It's not clear what formatting you're trying to set. Also, you don't state how you used the Modify() method to generate the error. The following worked for me. Hopefully it's helpful or at least instructive enough to get you started.
Excel = actxserver('Excel.Application');
Excel.Visible = true
xlCellValue = 1;
xlGreater = 5;
Range = Excel.Worksheets.Item(1).Range('A1:A4');
Range.FormatConditions.Add(xlCellValue, xlGreater, '=2.5')
Range.FormatConditions.Item(1).Font.Color = -16383844
Range.FormatConditions.Item(1).Font.TintAndShade = 0
1. I recommend against using Excel.Selection. This can cause problems if more than one instance of Excel is running and will cause problems if the user does anything to change the selection. Your code will be much more robust if you define and use Range objects separately.
2. I've done a lot of interacting with Excel from Matlab and have never found it useful to use Excel's built-in conditional formating. I do all of the conditional operations in Matlab and format cells appropriately from Matlab. You might try that instead. That being said, I can see how using the conditional formatting in Excel might be useful at times.