symsum(), like the rest of MATLAB, evaluates the symbolic expression that is passed in as its first argument, before the function itself gets control. This is a problem with your formula because mod() with symbolic inputs is.. weird...
Instead, construct vectors of powers:
NL = 1:n;
Nm3 = mod(NL-3,n);
Nm1 = mod(NL-1,n);
NNm3 = n - Nm3;
NNm1 = n - Nm1;
Nm1n = mod(NL-1-n, n);
and so on. Then
sum( (a.^Nm3 + a.^NNm3 + a.^Nm1n + a.^NNm1n) .* second_term )
I suggest that you only use symsum() when you are trying to find formula, such as symsum(1/2^x, 1, N) and expecting back the formula 1 - (1/2)^N .
When you have finite limits, it is almost always better to create a vector containing the individual terms, and sum() the vector.
If you were hoping that you could use symsum() to create an expression with generalized n so that you could reason with the formula, or fill in particular n values later, then you should probably give up all hope of that with symsum() or with symfun().