C++

Forum

 
LZO1x-999 Decompression

MexUK (10)
Hi there

I am trying to decompress a file, as it is compressed with LZO 1x-999 compression.

Here's my code:

1
2
3
4
5
6
7
8
9
10
11
12
13
 
...
			size_t uiCompressedSize = pEntry->m_uiFileSize;
			size_t uiUncompressedSize = uiCompressedSize + uiCompressedSize / 16 + 64 + 3;
			CFileParser::getInstance()->seek(pEntry->m_uiFileOffset * 2048);
			string strEntryData = CFileParser::getInstance()->readString(uiCompressedSize);
			//strEntryData.append(1, '\0');
			lzo_bytep pOutData = (lzo_bytep) new char[uiUncompressedSize];
			lzo_uint *length = new lzo_uint(uiUncompressedSize);
			lzo_bytep pEntryData = (lzo_bytep)strEntryData.c_str();
			lzo1x_decompress(pEntryData, uiCompressedSize, pOutData, length, NULL);
			delete length;
			delete [] pOutData;
...


But this code causes a crash.
MiiNiPaa (8886)
If you allocated char array, you need to delete char array, and not lzo_byte array
MexUK (10)
okay thanks, I have changed the variable type to char*

1
2
3
4
5
6
7
8
9
10
11
 
			size_t uiCompressedSize = pEntry->m_uiFileSize;
			size_t uiUncompressedSize = uiCompressedSize + uiCompressedSize / 16 + 64 + 3;
			CFileParser::getInstance()->seek(pEntry->m_uiFileOffset * 2048);
			string strEntryData = CFileParser::getInstance()->readString(uiCompressedSize);
			//strEntryData.append(1, '\0');
			char *pOutData = new char[uiUncompressedSize];
			lzo_uint *length = new lzo_uint(uiUncompressedSize);
			lzo_bytep pEntryData = (lzo_bytep)strEntryData.c_str();
			lzo1x_decompress(pEntryData, uiCompressedSize, (lzo_bytep) pOutData, length, NULL);
			delete length;
			delete [] pOutData;


but it still crashes.
Last edited on
MiiNiPaa (8886)
(lzo_bytep)strEntryData.c_str();
What is lzo_bytep?
If it is not a const pointer, and lzo1x_decompress changes value pEntryData points to, it will yield undefined behavior.

Do not use c-casts on pointers (actually do not use c-casts at all), using static_cast instead will eliminate possibility of accidentally casting away constness.

Where does it crash? What is in stack trace? Why exactly it crashes? What does debugger says?
MexUK (10)
lzo_bytep is a unsigned char *

I don't know if lzo1x_decompress() changes the pEntryData variable value, the output from this function is stored in pOutData.

I have tried to use static_cast instead of C casts but it's giving me an error
"Invalid type conversion"
With this code:
1
2
3
4
5
6
7
8
9
10
11
 
			size_t uiCompressedSize = pEntry->m_uiFileSize;
			size_t uiUncompressedSize = uiCompressedSize + uiCompressedSize / 16 + 64 + 3;
			CFileParser::getInstance()->seek(pEntry->m_uiFileOffset * 2048);
			string strEntryData = CFileParser::getInstance()->readString(uiCompressedSize);
			//strEntryData.append(1, '\0');
			char *pOutData = new char[uiUncompressedSize];
			lzo_uint *length = new lzo_uint(uiUncompressedSize);
			lzo_bytep pEntryData = static_cast<lzo_bytep>(strEntryData.c_str());
			lzo1x_decompress(pEntryData, uiCompressedSize, static_cast<lzo_bytep>(pOutData), length, NULL);
			delete length;
			delete [] pOutData;


When debugging in Visual Studio in Release Mode it says it crashes at this line:
 
 
			delete [] pOutData;


Here is the CallStack, is this the same as stack trace?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
 
 	ntdll.dll!_RtlpBreakPointHeap@4()	Unknown
 	ntdll.dll!_RtlpCheckBusyBlockTail@8()	Unknown
 	ntdll.dll!_RtlpValidateHeapEntry@12()	Unknown
 	ntdll.dll!_RtlDebugFreeHeap@12()	Unknown
 	ntdll.dll!@RtlpFreeHeap@16()	Unknown
 	ntdll.dll!_RtlFreeHeap@12()	Unknown
 	kernel32.dll!759714ad()	Unknown
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	
 	[External Code]	
>	IMG Factory - Alpha 0.26.exe!CIMGManager::parseIMG_Version1(std::basic_string<char,std::char_traits<char>,std::allocator<char> > strIMGPath) Line 117	C++
 	IMG Factory - Alpha 0.26.exe!CIMGManager::parseFile(std::basic_string<char,std::char_traits<char>,std::allocator<char> > strIMGPath, eIMGVersion eIMGVersion) Line 32	C++
 	IMG Factory - Alpha 0.26.exe!CIMGFactory::addViewInstance(std::basic_string<char,std::char_traits<char>,std::allocator<char> > strIMGPath, eIMGVersion eIMGVersion, bool bNew) Line 3708	C++
 	IMG Factory - Alpha 0.26.exe!CIMGFactory::onRequestOpen2(std::basic_string<char,std::char_traits<char>,std::allocator<char> > strPath) Line 210	C++
 	IMG Factory - Alpha 0.26.exe!CIMGFactoryDlg::WindowProc(unsigned int msg, unsigned int wp, long lp) Line 961	C++
 	[External Code]	
 	IMG Factory - Alpha 0.26.exe!CIMGFactoryDlg::CIMGFactoryDlg(CWnd * pParent) Line 72	C++
 	IMG Factory - Alpha 0.26.exe!CIMGFactoryApp::InitInstance() Line 93	C++
 	[External Code]


I'm not sure exactly as to why it crashes, however it does not crash when I comment out this line:
 
 
			//lzo1x_decompress(pEntryData, uiCompressedSize, (lzo_bytep)(pOutData), length, NULL);


In terms of what the debugger says, well it has a list of locals.
Last edited on
MiiNiPaa (8886)
1) replace your char array with library native type:
lzo_bytep pOutData = new lzo_byte[uiUncompressedSize];

2) Which casts are giving you the error?

3) Are you sure that your buffer is of sufficient size?
MexUK (10)
I have now changed to native type.

The C++ cast error is with static_cast in these 2 lines:

1
2
 
			lzo_bytep pEntryData = static_cast<lzo_bytep>(strEntryData.c_str());
			lzo1x_decompress(pEntryData, uiCompressedSize, static_cast<lzo_bytep>(pOutData), length, NULL);


To ensure buffer is big enough I have added:
 
 
uiUncompressedSize *= 200;


But it still crashes.
MiiNiPaa (8886)
was insput actually encoded with lzo1x_compress? Are sizes correct? Is input is in correct format? Is everything correct according to documentation?
MexUK (10)
The data would have been compressed with lzo1x_999_compress().
There is no 999 for decompress, just lzo1x_decompress().

Sizes are:
uiCompressedSize: 280576
uiUncompressedSize: 298179

The formula I use to calculate the uncompressed entry size is taken from documentation.

I'm not sure if input is in correct format, I take a const char * from std::string.c_str() (pEntryData) and I cast it to a unsigned char *, (lzo_bytep).

Everything appears to be correct according to documentation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
			size_t uiCompressedSize = pEntry->m_uiFileSize;
			size_t uiUncompressedSize = uiCompressedSize + uiCompressedSize / 16 + 64 + 3;
			CDebugger::log("uiCompressedSize: " + CStringUtility::toString(uiCompressedSize));
			CDebugger::log("uiUncompressedSize: " + CStringUtility::toString(uiUncompressedSize));
			//uiUncompressedSize *= 200;
			CFileParser::getInstance()->seek(pEntry->m_uiFileOffset * 2048);
			string strEntryData = CFileParser::getInstance()->readString(uiCompressedSize);
			//strEntryData.append(1, '\0');
			lzo_bytep pOutData = new lzo_byte[uiUncompressedSize];
			lzo_uint *length = new lzo_uint;
			*length = (lzo_uint) uiUncompressedSize;
			lzo_bytep pEntryData = (lzo_bytep)strEntryData.c_str();
			lzo1x_decompress(pEntryData, uiCompressedSize, pOutData, length, NULL);
			delete length;
			delete [] pOutData;


I'm using LZO compression code from here:
https://github.com/joyent/syslinux/blob/master/lzo/include/lzo/lzo1x.h

MiiNiPaa (8886)
Do you read it from file? If so, was file written in text or binary mode?
MexUK (10)
I read the entry data from file, one entry at a time.
The file is opened in binary mode, as the uncompressed data utilizes full 8 bits.

In fact here is my entire function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
CIMGFile*		CIMGManager::parseIMG_Version1(string strIMGPath)
{
	string strDIRPath = CPathUtility::replaceExtension(strIMGPath, "dir");

	CIMGFile *pIMGFile = new CIMGFile;
	pIMGFile->m_eVersion = IMG_1;
	pIMGFile->m_strPath = strIMGPath;

	CFileParser::getInstance()->setReadAllAtOnce(true);
	CFileParser::getInstance()->setEndian(LITTLE_ENDIAN);
	pIMGFile->m_bFileFound = CFileParser::getInstance()->open(strDIRPath, true);

	if (!pIMGFile->m_bFileFound)
	{
		return pIMGFile;
	}

	do
	{
		CIMGEntry *pEntry = new CIMGEntry;
		pIMGFile->m_vecEntries.push_back(pEntry);

		pEntry->m_uiFileOffset					= CFileParser::getInstance()->readULong();
		pEntry->m_uiFileSize					= CFileParser::getInstance()->readULong() * 2048;
		pEntry->m_strFileName					= CStringUtility::rtrimFromLeft(CFileParser::getInstance()->readString(24));
	}
	while (!CFileParser::getInstance()->isEOF());

	CFileParser::getInstance()->close();

	// fetch RW version for entries in IMG file
	CFileParser::getInstance()->setEndian(LITTLE_ENDIAN);
	CFileParser::getInstance()->setReadAllAtOnce(false);
	pIMGFile->m_bFileFound = CFileParser::getInstance()->open(strIMGPath, true);
	//pIMGFile->m_strIMGContent = CFileParser::getInstance()->getFileContent();
	//pIMGFile->m_pIMGContent = CFileParser::getInstance()->getFileContent();

	for (auto pEntry : pIMGFile->m_vecEntries)
	{
		CFileParser::getInstance()->seek(pEntry->m_uiFileOffset * 2048);
		string strData = CFileParser::getInstance()->readString(2, true);
		//CDebugger::log("b0: " + CStringUtility::toString((unsigned char)strData.c_str()[0]));
		//CDebugger::log("b1: " + CStringUtility::toString((unsigned char)strData.c_str()[1]));
		if (((unsigned char)strData.c_str()[0] & 0xFF) == 206 && ((unsigned char)strData.c_str()[1] & 0xFF) == 161)
		{
			///*
			// the entry data is compressed with LZO1X-999 compression
			size_t uiCompressedSize = pEntry->m_uiFileSize;
			size_t uiUncompressedSize = uiCompressedSize + uiCompressedSize / 16 + 64 + 3;
			CDebugger::log("uiCompressedSize: " + CStringUtility::toString(uiCompressedSize));
			CDebugger::log("uiUncompressedSize: " + CStringUtility::toString(uiUncompressedSize));
			//uiUncompressedSize *= 200;
			CFileParser::getInstance()->seek(pEntry->m_uiFileOffset * 2048);
			string strEntryData = CFileParser::getInstance()->readString(uiCompressedSize);
			//strEntryData.append(1, '\0');
			lzo_bytep pOutData = new lzo_byte[uiUncompressedSize];
			lzo_uint *length = new lzo_uint;
			*length = (lzo_uint) uiUncompressedSize;
			lzo_bytep pEntryData = (lzo_bytep)strEntryData.c_str();
			lzo1x_decompress(pEntryData, uiCompressedSize, pOutData, length, NULL);
			delete length;
			delete [] pOutData;
			//*/

			/*
			unsigned char pOutData2[2048];
			unsigned long uiLength = 2048;
			unsigned char ucChars[1024];
			for (int i = 0; i < 1024; i++)
			{
				ucChars[i] = 'a';
			}
			lzo1x_decompress(ucChars, 1024, pOutData2, &uiLength, NULL);
			*/

			//delete length;

			//string strDecompressedEntryData((char*)pOutData, strEntryData.length());
			//pEntry->m_uiRWVersion = CStringUtility::unpackULong(strDecompressedEntryData.substr(8, 4), false);
			//delete pOutData;
		}
		else
		{
			CFileParser::getInstance()->seek((pEntry->m_uiFileOffset * 2048) + 8);
			pEntry->m_uiRWVersion = CFileParser::getInstance()->readULong();
		}
	}

	CFileParser::getInstance()->close();

	return pIMGFile;
}
Last edited on
Topic archived. No new replies allowed.