Please:
- put your code sample within code tags, so that we can see the structure more easily.
- tell us what the input should be;
- tell us why you think the output is "weird".
This line: tou = pow(a, 3/2);
doesn't do what you think it does, because of integer division. However, I doubt that has much to do with your problem.
0.01 is presumably your timestep - don't hard-code it. Put it in a variable so that it only needs changing once, not a myriad of times.
Don't write lines like F[3] = AU * r_y[j]/pow(rad, 3.0) * -1;
The minus sign can just go in front; it doesn't require a multiplication by -1.
And here ... I'll hazard a guess that those j ... should be i. Occurs several times.