Does fitcsvm downweight ||w||^2 when sample size is increased? (is margin size not scaled by 1/n?)

Hi,

I was looking into the specific svm plane fit for an SVM model as I was getting confusing results. I was manually comparing the fit parameters to the minimization task below (from Wikipedia) and getting confusing results. For example, total loss was >1 when in the equation below, loss could be set to 1 via w=0 and b=0.

After investigation, I realized that it seems like fitcsvm does not do the scaling by 1/n present in the minimization equation above.

Two questions:

1. Is my diagnosis of this problem correct? If not, I will post some example code to show what I believe to be erroneous optimization of the SVM.
2. Is there an easy way to turn this setting off, or must I modify BoxConstraint manually?

Thank you!

Answer 1

[Avadhoot]

Hi @Elliot Chin,

From the description you have provided, I understand that you are concerned whether "fitcsvm" uses the scaling of 1/n as mentioned in your formula. You are getting different results when you compare your manual calculations according to the given formula to the support vector produced by the "fitcsvm" function. Let us get into both of your questions one by one.

1) The scaling factor issue: The role of the parameter "C" in the problem is to control the trade-off between maximizing the margin and minimizing the classification error. It's not directly about scaling by (1/n), but rather about how much penalty is applied to errors. The formula you mentioned is a scaled version for regularization purposes. This scaling by (1/n) is done to normalize the objective by the number of samples to make the calculations independent of the sample size. But the core SVM optimization problem does not scale by 1/n.

2) Modifying the box constraints: In "fitcsvm" the "BoxConstraint" parameter corresponds to the "C" parameter in the optimization problem. It controls the maximum penalty imposed on margin-violating observations, which helps to prevent overfitting. There is no direct option to turn the setting off because "fitcsvm" does not operate like that. Instead you can adjust the "BoxConstraint" to change the regularization strength so that you can achieve your desired effect. You will need to experiment with the values of "BoxConstraint" to do this.

For more information on the following topics refer to the adjacent documentation links:

1. "fitcsvm" function: https://www.mathworks.com/help/stats/fitcsvm.html
2. "BoxConstraints" parameter: https://www.mathworks.com/help/stats/fitcsvm.html?s_tid=doc_ta#d126e437749
3. SVM and box constraints: https://www.mathworks.com/help/stats/fitcsvm.html#bt8v_z4-1

I hope this helps.