Why is this segfaulting?

[cheapie]

// This is part of a library called OgreBullet. The program segfaults when this function is called.
void TriangleMeshCollisionShape::AddTriangle(
Vector3 &vertex1,
Vector3 &vertex2,
Vector3 &vertex3)
{
btVector3 vertexPos[3];

// GDB identifies this next line as the problem.
vertexPos[0][0] = vertex1.x;
vertexPos[0][1] = vertex1.y;
vertexPos[0][2] = vertex1.z;
vertexPos[1][0] = vertex2.x;
vertexPos[1][1] = vertex2.y;
vertexPos[1][2] = vertex2.z;
vertexPos[2][0] = vertex3.x;
vertexPos[2][1] = vertex3.y;
vertexPos[2][2] = vertex3.z;

mTriMesh->addTriangle(vertexPos[0], vertexPos[1], vertexPos[2]);
}

[kbw]

That looks like an infinitly recursive function to me. I take it you've blown the stack.

[eypros]

I don't know what type btVector3 is but you are declaring an array of 3 elements.

Then of course you are trying to use a 2-D array instead.
e.g. vertexPos[0][0]
You have never declared this.

Maybe you should declare it like this:
btVector3 vertexPos[3][3];

Almost always segfault has to do with boundary overpassing

[ne555]

Probably a vector has overloaded the operator[]
But, ¿can't you avoid the Vector3 to btVector3 conversion?

mTriMesh->addTriangle(vertexPos[0], vertexPos[1], vertexPos[2]); ¿why a pointer? ¿is it valid?

[cheapie]

kbw: I tried compiling it with -fstack-protector, and it still crashes exactly the same way.

eypros: I tried that and got:
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp: In member function ‘void OgreBulletCollisions::TriangleMeshCollisionShape::AddTriangle(Ogre::Vector3&, Ogre::Vector3&, Ogre::Vector3&)’:
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:116:29: error: no match for ‘operator=’ in ‘vertexPos[0][0] = vertex1.Ogre::Vector3::x’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:116:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:117:29: error: no match for ‘operator=’ in ‘vertexPos[0][1] = vertex1.Ogre::Vector3::y’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:117:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:118:29: error: no match for ‘operator=’ in ‘vertexPos[0][2] = vertex1.Ogre::Vector3::z’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:118:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:119:29: error: no match for ‘operator=’ in ‘vertexPos[1][0] = vertex2.Ogre::Vector3::x’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:119:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:120:29: error: no match for ‘operator=’ in ‘vertexPos[1][1] = vertex2.Ogre::Vector3::y’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:120:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:121:29: error: no match for ‘operator=’ in ‘vertexPos[1][2] = vertex2.Ogre::Vector3::z’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:121:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:122:29: error: no match for ‘operator=’ in ‘vertexPos[2][0] = vertex3.Ogre::Vector3::x’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:122:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:123:29: error: no match for ‘operator=’ in ‘vertexPos[2][1] = vertex3.Ogre::Vector3::y’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:123:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:124:29: error: no match for ‘operator=’ in ‘vertexPos[2][2] = vertex3.Ogre::Vector3::z’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:124:29: note: candidate is:
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: btVector3& btVector3::operator=(const btVector3&)
/usr/local/include/bullet/LinearMath/btVector3.h:39:28: note: no known conversion for argument 1 from ‘Ogre::Real {aka float}’ to ‘const btVector3&’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:126:71: error: no matching function for call to ‘btTriangleMesh::addTriangle(btVector3 [3], btVector3 [3], btVector3 [3])’
/home/cheapie/skyscraper/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp:126:71: note: candidate is:
/usr/local/include/bullet/BulletCollision/CollisionShapes/btTriangleMesh.h:54:8: note: void btTriangleMesh::addTriangle(const btVector3&, const btVector3&, const btVector3&, bool)
/usr/local/include/bullet/BulletCollision/CollisionShapes/btTriangleMesh.h:54:8: note: no known conversion for argument 1 from ‘btVector3 [3]’ to ‘const btVector3&’
make[2]: *** [CMakeFiles/OgreBulletCol.dir/ogrebullet/Collisions/src/Shapes/OgreBulletCollisionsTrimeshShape.cpp.o] Error 1
make[1]: *** [CMakeFiles/OgreBulletCol.dir/all] Error 2
make: *** [all] Error 2

ne555: Overloaded the what?

[kbw]

It would help if you posted the code for btVector3 and Vector3.

[roberts]

It would help if you posted the code for btVector3 and Vector3.

Likely, both are defined as typedef float btVector[3], Vector3[3];

I used to do this type of redefining all the time. It made typing easier.

[ne555]

Probably they are these (nothing interesting to see)
http://www.ogre3d.org/docs/api/html/classOgre_1_1Vector3.html
http://bulletphysics.com/Bullet/BulletFull/classbtVector3.html

operator btScalar*() replaces operator[], using implicit conversion. We added operator != and operator == to avoid pointer comparisons.

So

btVector3 v;
v[0]; //perfectly legal, access first component 

But you should know that...

void TriangleMeshCollisionShape::AddTriangle(
  Vector3 &vertex1, //these will not be modified by the function. They should be constant
  Vector3 &vertex2,
  Vector3 &vertex3)
{
/* Vector3 to btVector3 conversion*/

  mTriMesh->addTriangle(vertexPos[0], vertexPos[1], vertexPos[2]);
}

AFAICS there is no recursive calling. ¿what is mTriMesh? ¿why is it a pointer? ¿are you sure it is valid?
¿what does addTriangle() of mTriMesh do?

[cheapie]

ne555: Yes, they are those. On the other hand, I can't seem to figure out where mTriMesh comes from. The code I'm having issues with can be found at http://svn.tliquest.net/viewvc/skyscraper/trunk/ogrebullet/ if that helps at all.

Also, I still can't seem to get any response from the guy that wrote the code.

[LB]

Just FYI there is no recursion here, as I am sure "AddTriangle" (the function in the OP) is not the same as "addTriangle" (the function being called from "AddTriangle")