method 1) Point is in a triangle if it is on the correct side of each edge. You have a triangle ABC. Find vectors AB, BC and CA. Then find vectors perpendicular to them pAB, pBC and pCA. Check the sign of dot product of pAB and AC. Then check the dot product of AM (where M is the mouse ) and pAB, pBC, pCA. If all signs are the same as the first one, M is in ABC.
method 2) If M is in ABC, area of AMB + BMC + CMA = area of ABC. IIRC area of a triangle is half of the cross product of two vectors coming from one point (use absolute value). Though checking for equality will often fail due to floating point number imprecision. I think you should check that the sum of any two smaller striangles is lower than the area of ABC.
method 3) Something with matrices. Don't ask..
Take a matrix
and multiply it's inverse with vector AM. Now if M was in ABC, AM' will be in a triangle with corners at coordinates 0;0, 1;0 and 0;1 and it's trivial to check that.
I'm sure there are more methods but I can't think of any.
I hope I didn't mess up these 3..