OpenGL Color Keying

[N36fSL3A]

I've been messing around with pixel data lately, and this is really bugging me. How would I be able to add color keying to my game. I have a function already, however it's mucking up, adding red pixels, making it like a checkerboard.

The function is supposed to return a new pointer, which is essentially just the same thing with a full alpha for the color I'd like to key.

This function also assumes the image is 24 bit.

Here's my code:

Uint8* ColorKey(void* pixels, int w, int h, Uint16 pitch, const TexColor key, GLint &mode)
{
	bool hasKey = false; // If the image actually has the color we'd like to key

	mode = GL_RGB;       // Set the mode

	// Cast original pixel data to a Uint8 pointer if it already wasn't
	Uint8* originalPixels = (Uint8*)(pixels);

	Uint8* keyedIMG       = new Uint8[w * h * 4]; // Allocate pixel data for the keyed image
	
	// Go through the image, extract each pixel value, then if it's equal
	// to the color key, change it to alpha value

	int p2 = 0;
	for(int p = 0; p < (w * h * 4); p += 3)
	{
		keyedIMG[p2]     = originalPixels[p];
		keyedIMG[p2 + 1] = originalPixels[p + 1];
		keyedIMG[p2 + 2] = originalPixels[p + 2];

		Uint8 l[3] = {keyedIMG[p2], keyedIMG[p2 + 1], keyedIMG[p2 + 2]};

		if(originalPixels[p] == key.r && originalPixels[p + 1] == key.g && originalPixels[p + 2] == key.b)
		{
			keyedIMG[p + 3] = 0;
			std::cout << "Match.\n";
			hasKey = true;
		}

		else {keyedIMG[p + 3] = 1;}
		
		p2 += 4;
	}

	// If there was a color value, change the mode
	if(hasKey)
	{
		mode = GL_RGBA;
	}

	return keyedIMG;
}

[cire]

It would appear that pixels is supposed to point to an array of rgb values, which would make each pixel 3 bytes in size. You're using p to index the pixels array. Why, then, is the loop condition on line 16 p < ( w * h * 4 ) ?

What do you hope to accomplish with line 22?

It escapes me why you're modifying the mode value. The array you return will have alpha information in it, regardless of whether there were pixels that should be completely transparent in the original array.

[Edit: And why don't you just store the alpha information with the image/texture rather than jumping through hoops?]

[helios]

Are you totally sure about the channel ordering for the GL_RGBA format? I can never remember this, but I think "RGBA" refers to the order when you read the pixel as a number (alpha on the LSB, red on the MSB), so if you're using an x86, the memory layout would be ABGR.

Another thing you're doing wrong is that you're ignoring the pitch. Most 24-bit bitmap formats add 4 - (width*3)%4 bytes of padding at the end of each scanline. A proper pixel iteration would be something like

for (int y = 0; y < h; y++){
    byte_t *p = origin + y * pitch;
    for (int x = 0; x < w; x++, p += 3){
        //...
    }
}

[N36fSL3A]

I'd usually just do the alpha channel, but I'm using SDL as my .png loader (I wrote my own .bmp loader to avoid the headache)

Whenever I load a png with alpha I get 1 bit per pixel or whatever.