There are two things you have to deal with here: the overflow and the division by 4 at the end. Consider the largest case: N=1,000,000,006. You can't multiply N*N in a 32 bit number, so you'll need a 64 bit value for that (probably an unsigned long long). But even there, if you multiply N*N*(N+1)*(N+1), that result won't fit in 64 bits either, so you need to use the property
(a*b) mod c == (a mod c)*(b mod c) mod c |
. In other words, take the modulo after each multiplication.
That handles the multiplication but what about the division by 4? I don't think that you can simply take the result and divide by 4. Instead, consider the factors: N*N*(N+1)*(N+1). Either N is even or N+1 is even. So N*(N+1) is guaranteed to be even and that means you can divide it by 2. So lets rearrange the terms to be (N*(N+1)/2) * (N*(N+1)/2) = (N*(N+1)/2)^2.
Now it's real easy, but I'll let you do the final bit of code. Just remember:
- use unsigned long long for the multiplication.
- Take the modulo after each term, and again with the result.