I was following this article (http://developer.download.nvidia.com/whitepapers/2010/PN-AEN-Triangles-Whitepaper.pdf), and there is an detail explanation how to build index buffer suitable for this kind of mesh tessellation.
Since i dont get correct results i need someone to tell me if my algorithm is right (as described in article) or i need to debug some things elsewhere.
Implementation details:
1. Create an output IB that is 3 times the size of input IB.
2. For each input Triangle in IB, with indices i0, i1 and i2:
a. Write out an initial output entry of: i0, i1, i2, i0, i1, i1, i2, i2, i0, which sets edges to
initially be neighbors of themselves. This would produce identical results to PN Triangles.
b. Lookup the positions p0, p1, and p2, using i0, i1 and i2 to perform a lookup for
position of the associated vertex in VB.
c. Define 3 Edges, which consist of the two indices and two positions that make up
the corresponding Edge. An Edge should consist of the origin index, the
destination index, the origin position and the destination position.
d. For each edge, store the reverse of that edge in an easily searchable data structure
for the next step. The reference implementation uses an stdext::hash_map<Edge, Edge>
for this purpose. Reverse simply flips the sense of the edge (originating at
the destination position and index and heading to the origin position and index).
3. Walk the output index buffer (OB) constructed in step 2. For each patch of 9 indices:
a. For each Edge in the current Patch, perform a lookup into Edge->Edge mapping
created in step 2d.
b. If found, replace the current indices with the indices found in the map. Note that
two edges should be considered matching if their "from" and "to" indices match,
OR if their "from" and "to" positions match.
c. If not, continue to use the existing indices.