Increasing Size of Vector

[Duthomhas]

Part of game design, even today, is to minimize resource usage. You’re throwing a swimming pool’s worth of water at a bucket. I can’t imagine having three thousand sprites in a single scene.

Color tricks are not new. IDK if hardware support still exists for playing with an image’s palette or not (I haven’t played with OGL for a while now), but I’d honestly be surprised if it didn’t. You’ll likely have to write your own shader fragment for it.

Ultimately, though, even if you have to do it by hand before the graphics card gets it, exactly how many palette variations do you need at one time?

Reduce, reduce, reduce. If it means having to pause between spaces, etc just to load up the next batch of assets, do it.

[Cyclone]

I think I got the logic now. It's almost working. I was able to use the .clear() once I finished with the first pass of the function. Just two issues remain. First its very slow once it reaches the second vector and I'm not sure how to make it global enough so that I can write to the correct directory and file name. As if I make it global there will be the memory error



Edit: Sorry for the multiple posts. It was too big. And I couldn't delete one.

Just saw your post now. Idiot I am... I have to think about it some. I just spent so much time on it already I would hate to have to redo it all. The idea is to extract images to use in a tool. Think of it as an install for the tool. I know that might not be the best approach.

[Cyclone]

#include "C:\Users\Chris\source\repos\PNG\lodepng.h"
#include "C:\Users\Chris\source\repos\PNG\lodepng.cpp"
#include <fstream>
#include <sstream>
#include <string>
#include <atlstr.h>
#include <iterator>
#include <iostream>
#include <bitset>
#include <Windows.h>
#include <vector> 

[Cyclone]

using namespace std;

//Encode from raw pixels to disk with a single function call
//The image argument has width * height RGBA pixels or width * height * 4 bytes
void encodeOneStep(const char* filename, std::vector<unsigned char>& image, unsigned width, unsigned height) {
    //Encode the image
    unsigned error = lodepng::encode(filename, image, width, height);

    //if there's an error, display it
    if (error) std::cout << "encoder error " << error << ": " << lodepng_error_text(error) << std::endl;
}

int main()
{
    const size_t width = 256, height = 256;
    size_t no_elems = width * height * 4;
    using Image = std::vector<unsigned char>;
    std::vector<Image> Image_PNG(7000, Image(no_elems));  // Increasing this to more then ~7000 the program will crash

    char sztmp[1024];
    char sztmp_ROM[1024];
    const char* filepath = " ";
    const char* filepath_ROM = " ";

    cin.clear();
    cout << "\nEnter Directory\n " << endl;
    cin.getline(sztmp_ROM, sizeof(sztmp_ROM), '\n');
    std::string directory(sztmp_ROM);
    directory.erase(remove(directory.begin(), directory.end(), '\"'), directory.end());

    string path;
    const size_t last_slash_idx = directory.rfind('\\');
    if (std::string::npos != last_slash_idx)
    {
        path = directory.substr(0, last_slash_idx);
    }

    // Create Sub folders
    string combined_file_path(string(path) + "\\Diddy");
    CreateDirectory(combined_file_path.c_str(), NULL);
    cout << "\nChose a Palette";
    cin.get();

    cout << "1 Diddy\n\n";

    int Palette_Number = 0;
    std::cin >> Palette_Number;

    std::cout << "Your slection was "; std::cout << Palette_Number;

    // Boombbox and Cannonball Pallette  ** Checked
    static const unsigned char coloursV2[16][4] = { // An array of colour values. 
       //RRR  GGG  BBB
       { 0, 0, 0, 0 }, /* White    */
       {   16,  16,  16, 255 }, /* Index  1 */
       {   40,  40, 40, 255 }, /* Index  2 */
       {  56, 56, 56, 255 }, /* Index  3 */
       {   72, 72, 72, 255 }, /* Index  4 */
       {   88, 88, 88, 255 }, /* Index  5 */
       { 112, 112, 112, 255 }, /* Index  6 */
       { 128, 128, 128, 255 }, /* Index  7 */
       { 144, 144, 144, 255 }, /* Index  8 */
       { 160,   160,  160, 255 }, /* Index  9 */
       {  184,   184, 184, 255 }, /* Index 10 */
       {   200,  200, 200, 255 }, /* Index 11 */
       {  216,  216, 216, 255 }, /* Index 12 */
       {   232, 232, 232, 255 }, /* Index 13 */
       {  248, 248,   248, 255 }, /* Index 14 */
       {   8,   0,   0, 255 }  /* Index 15 */
    };

    // Create the Bitplanes
    int* Bitplane0__ROW = new int[20000];
    int* Bitplane1__ROW = new int[20000];
    int* Bitplane2__ROW = new int[20000];
    int* Bitplane3__ROW = new int[20000];

    long long int address = 0x1D70C6;  // add 0xC00000 to get the SNES address.
    int Data_Size = 0;
    int Next_sprite = 0;
    int coord_X_Location = 0;
    int coord_Y_Location = 0;

    int tile_count = 0;


    for (int X = 0; X <= 4000; X++)
    {

        int bit_offset;
        // Go to the next area in the ROM that contains sprites. Add 0XC00000 to get the snes ADDRESS
        if (X == 22) { address = 0x6DA77; Data_Size = 0; } // Diddy Part 1

        char oDataK[0x40000]; // Sprite buffer
        ifstream inFileK;
        inFileK.open(directory, ios::binary); // Note. this opens a binary stream. This is needed. cuzz the program was using 0x1a values as a Substitute AscII charter which halts the program/Using it as End Of File.
        inFileK.seekg(address + Data_Size, 1); // Seek to the current sprite header location.
        inFileK.read(oDataK, 0x40000);
        inFileK.close();

        int Header = 8;
        int number_of_large_tiles = oDataK[0]; // The number of 2x2 chars (16 x 16 pixel tiles)
        int size_of_large_tiles = oDataK[0] * 32;
        int tile_offset = 0;
        int Offset_Group_Two = oDataK[4];
        int DMA_Group_ONE = oDataK[5];
        int Size_of_coordenants = (oDataK[0] + oDataK[1]) * 2;   // 0 contains the value of the number of 2x2 chars (16 x 16 pixel tiles). 1 contains the value of the number of 1x1 chars (8x8 pixel tiles). These two values are multiplied by 2 (two bytes for each set of coordenants).

        //int Tile_Count_All = (tile_count_small * 8 + number_of_large_tiles * 32);
        int Start_Of_First_Tile = Header + Size_of_coordenants;  // 8 is the size of the header. 6 is the total size of coordinates(3 chars (tiles) * 2 bytes for earch char) (these are also part of the header...) 
        int p = 0;
        int chars_small = oDataK[1];  // The number of 1x1 chars (8 x 8 pixel tiles)
        int chars_small_Group2 = oDataK[3]; // The number of 1x1 chars Group 2 (8 x 8 pixel tiles)




        /////////////////////////////////////////////////////////////////////////////////////////////////

        Start_Of_First_Tile = Start_Of_First_Tile + Data_Size; // the location of the curent sprite.
        Data_Size = Data_Size + Header + Size_of_coordenants + (number_of_large_tiles * 4 * 32) + (chars_small * 1 * 32);
        int c = 0;
        int x_coord_Large;
        int y_coord_Large;
        coord_X_Location = tile_count;
        coord_Y_Location = tile_count + 1;
        int offset = 32;

        unsigned char oDataL[0x80000]; // Increase this value to prevent crashing?

        // Sprite buffer
        ifstream inFileL;
        inFileL.open(directory, ios::binary); // Note. this opens a binary streem. This is needed. cuzz the program was using 0x1a values as a Substitute AscII charter which halts the program/Using it as End Of File.
        inFileL.seekg(address, 1); // Seek to the current sprite header location.
        inFileL.read((char*)oDataL, 0x40000);
        inFileL.close();
        int Pixel_count = ((number_of_large_tiles * 32) + (chars_small * 8) + (chars_small_Group2 * 8));

        for (int p = 0, t = 0; p < Pixel_count; p++, t++)
        {
            // Write the PNG Images
            if (p <= Pixel_count && X <= 4000 && Palette_Number == 1)  // BoomBox and Cannon Ball
            {
                for (int j = 0, N = 7; j < 8; j++, N--) // Check which bits are set for each pixel of the tiles. Then assign the index colour to the image Array location.
                {
                    std::bitset<4> bs;
                    bs.set(0, (Bitplane0__ROW[p] >> N) & 1);
                    bs.set(1, (Bitplane1__ROW[p] >> N) & 1);
                    bs.set(2, (Bitplane2__ROW[p] >> N) & 1);
                    bs.set(3, (Bitplane3__ROW[p] >> N) & 1);
                    unsigned long Index = bs.to_ulong();

                    if (X <= 2855)
                    {
                        Image_PNG[X][c++] = coloursV2[Index][0]; // Red
                        Image_PNG[X][c++] = coloursV2[Index][1]; // Green
                        Image_PNG[X][c++] = coloursV2[Index][2]; // Blue 
                        Image_PNG[X][c++] = coloursV2[Index][3]; // Alpha
                    }

                    if (X > 2855)
                    {
                        Image_PNG.clear();
                        std::vector<Image> Image_PNGB(7000, Image(no_elems));
                        Image_PNGB[X][c++] = coloursV2[Index][0]; // Red
                        Image_PNGB[X][c++] = coloursV2[Index][1]; // Green
                        Image_PNGB[X][c++] = coloursV2[Index][2]; // Blue 
                        Image_PNGB[X][c++] = coloursV2[Index][3]; // Alpha
                         std::cout << "test";
                    }
                }   
            }   
        }

        if (X <= 2855)
        {
            string combined_file_path(string(path) + "\\Diddy\\" + to_string(X) + ".png");
            encodeOneStep(combined_file_path.c_str(), Image_PNG[X], width, height);  // Write the .png image file.
        }    
    }
    cout << "Extraction Complete! ";
    system("pause");
    return 0;
}

[Duthomhas]

So it isn’t the game itself?

Do all the images need to be extracted at the same time? Would it be worth building a dependency graph so you only need to load images as needed?

(Sorry, I haven’t looked at your code. My brain doesn’t function like it used to and can’t handle that right now.)

[seeplus]

1) What is created with NEW should be deleted with DELETE or DELETE [] for arrays (as here)

2) odatak and odatal are still being created on the stack. Irrespective of the amount of memory available, there is only a fixed amount of memory available for stack-based variables. Suggest you make them into dynamic arrays, make them static or move their definition to be before main().

3) With 7000 elements, Image_PNG has a memory size of about 1.8Gb! Are you compiling as 32bit or 64bit? I suspect 32 bit. Change it to compile for 64bit. You're running out of memory for 32bit.

[Cyclone]

1) Hmmm. I haven't tried that.

2) "make them static or move their definition to be before main()."

Didn't seem to make a difference.

3) Changed it to 64bit. Seems to work ok now! At least for the time being lol.

How much more memory is available , and what is a safe max ram usage for a console program?

Also I had an idea of running the program more then once and passing a single variable to it. Is this possible or bad practice?


Thanks so much for your help!

[Ganado]

seeplus -- good thinking with the 32-bit addressing thought.

Cyclone --
> How much more memory is available, and what is a safe max ram usage for a console program?
In 64-bit addressing, the only thing limiting you on the "heap" is the physical RAM on your computer. [ technically it's like...16 exabytes, so give it a few years ;) ]

But I wouldn't put much more than 10s of megabytes on the same contiguous block of memory, just because that can make it harder for the memory allocator to find usable memory. It's probably not an issue though.

The std::vector<std::vector<byte>> creates a bunch of discontiguous memory allocations, so you're actually fine.

> Also I had an idea of running the program more then once and passing a single variable to it.
Like, a command-line argument? There's nothing wrong with command-line arguments. Go for it!

[Duthomhas]

Cyclone wrote:

Also I had an idea of running the program more then once and passing a single variable to it. Is this possible or bad practice?

It is not bad practice, but for a program that reads/uses that much memory every time it is invoked, it is very unwieldly. At best your memory allocator will handle it just fine. At worst it reads a bazillion files and becomes a disk thrasher (which is an issue for solid state drives too). Not to mention issues that arise if the user manages to invoke multiple copies of the program at once.