Given a vector of interest and two "down" and "forward" reference vectors, is there some way to rotate them so the "down" vector points down and the "forward" vector is on the vertical forward plane, without decomposing the vectors and using atan2()?
Are the forward and down vectors perpendicular? Some test data with expected results might help clarify what you want.
You could convert the forward and down vectors into a 3x3 rotation matrix, normalizing the vectors, taking the cross product to get the third "side" component. If forward and down aren't perpendicular, you could use the "side" result of the cross product to then get a proper perpendicular "forward" vector by taking the cross product of down with side. This should result in forward being on the forward vertical plane and down pointing directly down. You can then use this rotation matrix to rotate the vector of interest, assuming that's what you wanted.
Well the you can do it any which way so long as you follow the convention. If you fill matrix in row major you just will be doing the multiplication of the vector the other way around.
For column major, should look like this:
1 2 3 4 5
// these being the columns
// forward ~ (1, 0, 0)
// side ~ (0, 1, 0)
// up ~ (0, 0, 1)
output = Mat3( forward, side, up ) * Vec3();
Basically, you are building the rotation matrix to transform down from (0, 0, -1) into the position it is actually at. Then you take the transpose of that matrix, which is the same as the inverse (rotation matrix specific property). So it ends up being a matrix to rotate your vectors, with down rotating into the desired (0, 0, -1) position.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
// untested
// i believe you want something like this, ultimately:
up = -down;
side = cross(up, forward); // might be able to do cross(forward, down), but just for examples sake/simplicity
perpendicularForward = cross(side, up);
// normalize perpendicularForward, side, up...
Mat3 rotation = transpose( Mat3(perpendicularForward, side, up) );
forward = rotation * forward;
up = rotation * up;
interest = rotation * interest;
Here's what I'm using it for: http://i59.tinypic.com/11radee.png
(S up, E right, center is ascension +/-90°, circumference is ascension 0°, red is path below the horizon)
That's supposed to be yesterday's ecliptic above my city , although it doesn't look quite right yet. The sun appears to be changing direction for no reason, and the South Pole is calculated to currently be under a midnight sun. I might be applying the axial tilt in the wrong order.