Edit: Err, just read the difference between your title and your question in your post. They generally imply quite different things.
To Mod: Just start with something simple like the map editor, or editing the tech and unit stats from Sheets.vol.
To Hack: (If this is REALLY what you want to do, even though it's unlikely)
Grab a debugger/disassembler, and go. You will need to know assembly language to really understand much, but you can sort of pick that up as you go along. Just google for OllyDbg, it's a free download. Also nice to keep an x86 instruction reference available, such as Intels "IA32 - Vol 2" document. You can search Intel's website for a free download of that too.
Granted, it really helps to be at least resonably familiar with programming, and maybe do a bit of experimenting with a C++ compiler and the debugger/disassembler so you can see what the code gets translated into.
Don't bother reading the whole doc on the Intel instruction set. It's got hundred of instructions, and hundreds of pages. Most of the instructions won't be used. Just look things up as you need them.
