I was trying to find to find the intersection (the area that they have in common even if their perimeters don't intersect) between two quadrilaterals and I came up with this.
1 2 3 4 5 6 7 8 9 10 11 12
intersection(A,B):
C=B
for each pair of adjacent vertices (a,b) in A
L=the line that contains a and b
for each pair of adjacent vertices (c,d) in C
S=the segment that goes from c to d
if S crosses L
add the intersection of S and L as a new vertex of C between c and d
for each vertex c in C:
if c is on a different side of L than A's geometric center
remove c from C
return C
Note that it applies for any convex polygon A and any convex or concave polygon B.
I do not understand: are you trying to find the area, or the polygon that is the intersection of the two?
[Edit:] Second remark: if A is a very small quadrilateral lying entirely inside a very large quadrilateral B, then the intersection in question should be equal to A.
I tried to run your algorithm through the so described setup and I don't see how I will get the right answer.
The intersection, but I read that in mathematics the intersection of two polygons is actually the intersection of their perimeters, so I wanted to make it clear that what I want to intersect is the surfaces inside of them. E.g. if you intersect a square by itself rotated 45°, you'd get an octagon, and if a polygon completely encloses another, their intersection is the smaller one.
EDIT: I've already tested the algorithm with hand-drawn polygons, so I know it works, but I might have made a small mistake when writing it here. The basic procedure is:
1. You have two pieces of paper A and B shaped like convex polygons.
2. Put one on top of the other.
3. Take a ruler and place it aligned to each side of one paper (A), running a box cutter over its length at each step, cutting the other piece of paper (B).
4. The resulting piece of paper (B) is the intersection.
Ok, I think I see what is the idea of your algorithm. I would explain its idea like this.
The polygon A is convex, that means it is the intersection of the half-planes given by the lines passing through its sides.
Intersecting B with A is the same as intersecting B with any single one of those half-planes, and then repeating the same with the result of this operation and the remaining half-planes. This corresponds to your out-most cycle.
The inner cycle takes care of intersecting your polygon B with a single half-plane. Indeed, the only new candidates for vertices are the ones obtained by intersecting the sides of B with the line determining the half-plane. You throw those into B. However, some pieces of B will be chopped off by the line. To find the points that are chopped away from B, you simply need to look whether they lie on the "wrong half" of the line determining the half-plane. This is the precisely the half that does not contain the geometric center of A.
The last thing you need to take care of is how to link the newly obtained vertices (the ones that lie on the line) with segments. This of course you do in the same order in which those were linked in the original B.