Setting transparency of markers with gscatter is not possible as of r2020a.
Workarounds
You could use scatter() which does support transparency. Here's one way to translate a gscatter syntax to scatter syntax using arrayfun.
gscatter(x,y,g)
cla()
hold on
uniqueGroups = unique(group);
h = arrayfun(@(g)scatter(x(group==g), y(group==g), 'filled'), uniqueGroups);
set(h, 'MarkerFaceAlpha', 0.6)
colors = 'rb';
markers = '^v';
uniqueGroups = unique(group);
cla()
hold on
h = arrayfun(@(g)scatter(x(group==g), y(group==g), ...
100, colors(g), 'filled', 'Marker', markers(g)), uniqueGroups);
set(h, 'MarkerFaceAlpha', 0.4, 'MarkerEdgeColor', 'k', 'MarkerEdgeAlpha', .2)
Alternatively, you could use gscatter without filled markers.
h = gscatter(x,y,g,colors,'os',10);
set(h,'LineWidth',1)
Why transparency in gscatter doesn't work
The scatter function creates scatter objects. Scatter objects have MarkerFaceAlpha and MarkerEdgeAlpha properties that allow you to set transparency levels.
The gscatter function creates line objects. Line objects do not have these properties. Furthermore, an undocumented method of adding transparancy to some graphics objects by adding a 4th element (0:1) to the RGB color definition does not work with gscatter.