I was assigned a problem involving using buffer overflow to access a different function than I was supposed to. I was able to figure out how to modify the point in stack that I need to change using a printf statement, however what doesn't make sense is that when I use the input "AABBCCDDEEFFGG\x86\x64\x00\x00" the stack changes to 78363878 I looked up the ascii codes and I assume that it's not converting \x86, but using the input x86 instead. I don't see why this is wrong, any help I could get would be appreciated.
If you are just typing in the string as written, then the backslash is treated as a literal backslash. You will want to input whatever character corresponds to the hex values you want (perhaps by using some file as input).
Does the file contain the backslash character? Or the actual character with code 0x86? You may also want to note that having null characters (code 0) in the string terminates it.
The command I use to edit the file is printf ''AABBCCDDEEFFGG\x86\x64\x00\x00" < input, then I set args < input using gdb while running the program. Would excluding the "\x00\x00" help me?