I found a solution.
 clear; clc;
 J = 152e-6; % moment of inertia
 W = 200*pi; % desired crossover freq
 K = 0.1;    % torque constant
 syms Ki Kp real
 assumeAlso(Ki>0);
 assumeAlso(Kp>0);
 G = Ki/(i*W)*(1 + i*W/(Ki/Kp))*K/(i*W*J); % transfer-function
 F = [abs(G), angle(G)]; % symbolic output 
 H = [rewrite(F(1),'sqrt') == 1, simplify(F(2)) == -120*pi/180]; % 2 equations
 [Ki,Kp] = solve(H,Ki,Kp);
So 'rewrite' and 'simplify' made the equations more managable for 'solve'

