Some teacher assigned a problem like this one? It's more maths than C I think.
If you want my advice replace c by the value the other variables produce (((int)(sqrt(a*a+b*b)+0.5)) and run a double loop for the other variables (a,b) until your condition is met (a+b+c=1000).
In other words compare a+b+ (int)(sqrt(a*a+b*b)+0.5) with 1000 until you find the numbers.
Limits for your loops up to 333 is satisfactory I guess (since a+b+c=1000).