I license a game engine so have to use a compiled exe to run my app.
I know about a line of code in the source that is causing a lot of unnecessary compute overhead (I have the source but cannot compile myself due to other libraries included in my current licensed exe).
How can modify the .exe to comment out one particular line. Or unset a particular #define?
You can't. For a native C++ application, when a executable is created, all that's left are the assembly instructions. The best you can do is some reverse-engineering with the assembly to pinpoint which instructions you want to remove or modify (and I don't know the specifics of low-level assembly). Probably easier to re-compile.
Unfortunately I cannot recompile because (although I do have the source), I don't have the source for some other libraries that are included in the licensed .exe.
Given that i do have the source code though, how could I use that to detect which instructions to set to NOP?
Trying to unset a #define in an executable is likely to be a lot of hack work. Do you have a clue how many places in the source code that #defined value is used?
A #define is a pre-processor command that does text replacement before the compiler is given the source code to process. That replaced value could be in hundreds of locations.
there really isnt a sane answer to this. Either you need the code, or you need them to give you a special version of it, or you need to throw it away and write it yourself, or mix and match, write the one bad routine yourself and use their other stuff maybe. The price of using someone else's code is it will frequently do more than you need it to, or do it slightly differently, or do something you don't want or like. If you only have the compiled version, you live with it, code around it, or get another library, or whatever.