The C++ language does not guarantee what the result of that line will be. It's simply undefined. You should avoid writing code like this that modifies and then reads the variable in the same expression.
The result is well-defined if you write it as two statements.
1 2
x -= x*x; // 6 - (6 * 6) = -30
x += x; // (-30) + (-30) = -60