Forced Exports

[Hooman]

I've got this ongoing project, which more or less tries to force export symbols from Outpost2.exe. Using this project I've been able to access code and data from Outpost2.exe much like it was written in a seperate .cpp file, or declared as an extern variable. There are some down sides to certain aspects of this project, but so far they've proved minor, and don't tend to occur in practice.


Previously, this sort of thing was typically done by hardcoding pointer values, or storing the pointers in int or DWORD variable, and used lots of ugly type casting (or inline assembly that isn't type checked). The pointer casting was especially noticable for function pointers, which usually had rather long notations. Notations that you quickly forget after not using them for a while. Accessing variables this way also needed to make use of dereferencing, so it was obvious that a pointer was being used. The code ended up looking like this:

Code: [Select]

int* tickPtr = (int*)0x56EB1C;  	// Data pointer cast
int (*funcPtr)(int value1, int value2) = (int (*)(int, int))0x400120;	// Function pointer cast

void SomeFunction()
{
	int value = *tickPtr;	// Access data through pointer
	(*funcPtr)(3,4);  // Call through function pointer
	funcPtr(3, 4);  	// Call through function pointer (short form)
}

This method was also somewhat limited in what kinds of functions you could setup pointers to. Declaring a pointer to a member function of some class is quite a pain to do. This may be partly because the internal representation of pointers to members is sometimes larger than 4 bytes. In any case, the compiler gives an error message if you try to type cast an int into one of those pointers. Knowing the internal representation, you can always get around this with other hacks, such as using a union, but this makes already ugly notation much uglier.

Here is normal member function pointer access:

Code: [Select]

class SomeClass
{
public:
	void ClassMethod();
};

void (SomeClass::*methodPtr)() = &SomeClass::ClassMethod;  // Setting up a function pointer

void SomeFunction2()
{
	SomeClass someClassInstance;
	SomeClass* someClassPtr = &someClassInstance;	// Initialize a pointer to the class

	(someClassInstance.*methodPtr)();	// Call class method through method function pointer
	(someClassPtr->*methodPtr)();  // Call class method through method function pointer, using class pointer
}

Keep in mind where the "*" goes.

Now, if you try to type cast an int to a pointer of that form, such as like this:

Code: [Select]

int (SomeClass::*methodPtr2)() = (int (SomeClass::*)())0x400140;// Attempted function pointer cast

You'll end up getting the following compiler error message:

Code: [Select]

Main.cpp(35) : error C2440: 'type cast' : cannot convert from 'const int' to 'int (__thiscall SomeClass::*)(void)'
        There are no conversions from integral values to pointer-to-member values

So you'd need to set the pointer using something beyond a simple type cast.


As a side note, I found through experimentation that this:

Code: [Select]

class SomeClass2;
int (*SomeClass2::methodPtr3)() = (int (*)())0x400140;

Produces the following compiler error message:

Code: [Select]

Main.cpp(41) : fatal error C1001: INTERNAL COMPILER ERROR
        (compiler file 'msc1.cpp', line 1794) 
         Please choose the Technical Support command on the Visual C++ 
         Help menu, or open the Technical Support help file for more information

Notice the misplaced "*". Also, I used a forward declaration for a class, whose definition was never supplied. Please note that "msc1.cpp" is not one of my files. Probably an indication that we're getting into some fringe area of the language/compiler, where not too many people have tread before.

As another side note, I once got a core dump from a compiler on unix. Might have been with g++. (Not exactly the kind of encouragement I needed during the wee hours of the morning, while trying to get an assignment done by 9:00 AM).



Anyways, back on topic. We'd like some way to call class member functions inside Outpost2.exe. Assume we know the address of the function, it's calling convention, number of parameters, and return type. If this were code in a seperate .cpp file, we'd just have included it's header file to make use of it. Now remember that C++ compilers compile each .cpp file seperately. By simply writing a header file for the class, and including it, we can get something to compile into a .obj file. It's not until the link stage when it goes looking for the code that it will give an error about an undefined symbol. For example:

Code: [Select]

class A
{
public:
	void F1();
};


void G(A* a)
{
	a->F1();
}

Will compile, and then fail to link, since we haven't defined the class member function F1(). Compiling this produces the following compiler output:

Code: [Select]

Compiling...
Main.cpp
Linking...
Main.obj : error LNK2001: unresolved external symbol "public: void __thiscall A::F1(void)" (?F1@A@@QAEXXZ)

Notice the seperate "Compiling..." and "Linking..." steps. You can also verify that the compiler produced a Main.obj in the intermediate files folder. If you were to turn on assembly listing, you can even see that it produced code to call that function F1, but the assembly listing is really messy and hard to read, so we won't do that here. Also, note that the decorated symbol name is given in the error message. We'll make use of this later.


Now what is a symbol? It's essentially a named memory address. Since we already know what address we want, we just need a way of defining that symbol to have that address. Once we do that, we can get the linker to stop complaining, and produce an exe or dll. Granted, the C++ compiler doesn't seem to have a way to assign those symbols a value, but your typical assembler will.

What you can do, is create an assembly file (NASM format used here), and define that symbol to have the address of the code using something like:

Code: [Select]

symbolName EQU numericValue

We also need the symbol to be accessible from outside the .obj file the assembler will produce, otherwise the linker won't see it. We do this by defining the symbol to be global. We end up with something like this:

Code: [Select]

global ?F1@A@@QAEXXZ
?F1@A@@QAEXXZ EQU 0x400140

Here I actually filled in the symbol name, and some (random) numeric value.

Assemble the .asm file using NASM, and then link it all together to produce an exe or dll. It's a free download from Source Forge, and pretty easy to setup and use. If you include the assembly file into your project, it can be built along with the rest of your project, but you'll probably have to setup a custom build rule to tell your IDE how to build it. In MSVC this can be done by right-clicking on the file, and choosing settings. From there, you can enter a command line under the Custom Build tab to compile the file. I use the following settings.
Description: Performing Custom Build Step on $(InputPath)
Commands: NASMw -f win32 -o "$(InputDir)\Build\$(InputName).obj" "$(InputPath)"
Outputs: $(InputDir)\Build\$(InputName).obj
The -f sets the output format to win32 (COFF), and the -o sets the output file name. The rest is just where the input and output files are located (and I used macros provided by MSVC). I put the output file in a "Build" subfolder from where the source file is. This is not the usual place for output files, but my project was setup a bit differently. I'll probably get to that in a later post.

There is probably a similar setup for CodeBlocks, although, I don't know it. I did notice that the default assembler used by CodeBlock is MASM, not NASM. The syntax is not compatible, so you'd either have to convert the assembly file, or setup the IDE to use a new compiler.

If all that fails, you can always assemble the file yourself from the command line using Nasmw (the Windows version of NASM), and if you can't get your IDE to include the extra .obj file in the link step, you can also call Link yourself from the command line. It's usually a good idea to place Nasmw.exe somewhere in your path, or a bin folder where your IDE will look for it, so you don't always have to specify it's location.


So what does all this accomplish? Well, you can build header files in C++ describing all the code and data in Outpost2.exe, and then just include them into your source files enabling you to make direct calls to the internal functions of Outpost2.exe, and access all the variables, just as if it was code from a seperate .cpp file. Some code to dump a list of all tech names to a file at tick 3 might look like this (untested):

Code: [Select]

	if (tethysGame.tick == 3)
	{
  int i;

  ofstream techList("TechList.txt");
  for (i = 0; i < research.numTechs; i++)
  {
  	techList << i << ") " << research.techInfo[i]->techName << endl;
  }
}

Note the casing of tethysGame, and the use of .tick, instead of .Tick(). Here, tethysGame is a global class instance, and we are reading a member variable. This was done by declaring a TethysGame class (in a namespace, to prevent redefinitions with the class of the same name which is exported by Outpost2.exe) in a header file, and then an extern variable of that type using "extern TethysGame tethysGame;". The address of this variable was then placed in the assembly file, along with the decorated name of this global variable, enabling access in this manner. Similarly, a header file was created for the Research class, and a global extern variable named "research", as well as a TechInfo struct, whose fields are being accessed to find the tech name. (Code very similar to this was used to produce the tech list I posted a few days ago). Compare that to the older method of writing (untested):

Code: [Select]

	if (*(int*)(0x56EB1C) == 3)
	{
  int i;

  ofstream techList("TechList.txt");
  for (i = 0; i < *(int*)0x56C230; i++)
  {
  	techList << i << ") " << *(char**)((*(int**)0x56C234)[i] + 0x28) << endl;
  }
}

It took me a few minutes to get the casting to work so it'd compile.


The previous example was simple data access. What about calling class member functions? Well, not only can you call class member functions, but once you have all it's member variables declared so it's size is correctly determined by the compiler, you can also declare local variables of existing class types. The compiler will even call a constructor function located inside Outpost2.exe to initialize your object too. This lets you code something like:

Code: [Select]

  MultiplayerPreGameSetupWnd preGameSetupWnd;

  preGameSetupWnd.ShowHostGame(hostGameParameters);

Complete with IntelliSense, giving you the list of member functions as you type that ".". (Yes, that's pretty much what's in the NetPatch I've been working on).


This can be taken even further too. You can also declare your own classes derived from classes that exist in Outpost2.exe. Here's a somewhat larger example doing exactly that:

Code: [Select]

class CheatView : public CommandPaneView
{
public:
	// Member variables
	// ----------------
	// vtbl
	// ----------------
	UICommandButton disasterButton[4];
	CreateUnitCommand spawnDisasterCommand[4];

public:
	// Virtual member functions (inherited)
	virtual void UpdateView() {};
	virtual void OnAddView()
	{
  // Setup action parameters
  spawnDisasterCommand[0].unitType = mapMeteor;
  spawnDisasterCommand[0].cargo = (map_id)1;
  spawnDisasterCommand[1].unitType = mapLightning;
  spawnDisasterCommand[1].cargo = (map_id)1;
  spawnDisasterCommand[2].unitType = mapVortex;
  spawnDisasterCommand[2].cargo = (map_id)1;
  spawnDisasterCommand[3].unitType = mapEarthquake;
  spawnDisasterCommand[3].cargo = (map_id)1;

  // Set the button actions
  disasterButton[0].command = &spawnDisasterCommand[0];
  disasterButton[1].command = &spawnDisasterCommand[1];
  disasterButton[2].command = &spawnDisasterCommand[2];
  disasterButton[3].command = &spawnDisasterCommand[3];

  // Add the buttons to the display
  AddButtons(this, 4, 
  	&disasterButton[0], "&Meteor", "Create &Meteor",
  	&disasterButton[1], "&Storm", "Create &Storm",
  	&disasterButton[2], "&Vortex", "Create &Vortex",
  	&disasterButton[3], "&Earthquake", "Create &Earthquake");
	};
	virtual void OnRemoveView() {};
	virtual bool IsNewView() { return 1; };
	virtual void Draw(Rect* drawRect, GFXClippedSurface* surface) {};
	virtual void SetReportPageIndex() {};
	virtual bool DoesUnitSelectionChangeCauseUpdate() { return 0; };
	virtual void OnAction() {};
	virtual int GetSelectedReportButtonIndex() { return 0; };
};

This is a section of code I used for the DLL I posted a while back with the custom user interface that let you create disasters, and target them with the mouse. I did something similar for a custom mouse control class, as well as for a custom filter class that checked for the hotkey to display my hidden user interface. (And no, the code doesn't work in multiplayer).



Limitations
------------
There are a few limitations with this approach. Symbols used for relative jumps and calls are treated a bit differently than symbols used to access absolute data, such as variable access, and virtual function table entries. With the relative jumps and calls, you need to adjust the address of the symbol by the load address of the DLL, or it will link to the wrong address. For absolute data addresses, there is no adjustment. This means the same symbol can not be accessed in two different ways. If you want a virtual function to be inheritable by derived class, you'll need the absolute address associated with the symbol. That way, the correct absolute address will be placed in the virtual function table of the derived class. If this is done, then you can no longer call the original function in a relative way. This can occur is you prefix the method by the class name using the scope resolution operator ( :: ), such as if you were chaining to a base class function from an overridden virtual function. In practice, I've only run into this problem once in a very minor way, and there was an easy work around.

The other potential limitation I noticed is with virtual destructors. The virtual function table will point to either a Scalar Deleting Destructor, or to a Vector Deleting Destructor, which are both different from the normal destructor. These all have seperate decorated symbol names too, so the correct function address needs to be associated with the correct symbol. The limitation here comes from a possible mismatch between the type of destructor you need to call virtually, from the actual implemented function. The Scalar and Vector deleting destructors take two bit flags packed in a 4 byte int. The lowest bit, if set, means the object should free itself after running any destructor code. The next bit is used by the Vector deleting destructor only, and is used to check if an array of objects needs to be destructed, rather than just one object. The potential problem then, is that the virtual function table might only have a scalar deleting destructor, which might be called if you try to delete[] and array of that object type. This will likely cause a crash. However, I've had trouble trying to get the compiler to generate dangerous code to demonstrate this case, and I'm not exactly sure what conditions would actually cause this sort of thing to happen. Plus, I don't see much reason to go deleting arrays of the built in object types, or really doing much of anything that would cause issues with the destructors.



I'll begin posting source to parts of this project as I clean it up.

[Sirbomber]

And then you wonder why nobody has any idea what you're talking about...

[Hooman]

In summary, write up header files that look how the source header files for Outpost2 might have looked, and #include them. Then program like you had access to the source code, and this will get the compiler to compile your .cpp files into .obj files. But since you've only defined the interface to the Outpost2 code, and not written the code yourself, the link step will give errors when it tries to link. So you need an extra .obj file that defines those symbols, and points them to the actual code in Outpost2.exe. This is done using an assembler that lets you define such arbitrary mappings without hassle. Once that is done, the link step will succeed. The end result is a DLL that can access all the code and data in Outpost2.exe (that you've written mappings to), just as if you had the source code available.


That, and I was also saying how ugly the other way of doing something similar is, with a few ugly examples. i.e. This way is much nicer.


Or did you just not bother reading it all due to the length?  <_<  

[Sirbomber]

Oh, I read it all, alright.
Your synopsis is much easier to read and I understand it.

[Hooman]

Here's a sample header file from the project. I choose this one because it's probably the cleanest at the moment. I'll attach a copy, since this likely won't paste nicely, but for those who can't be bothered to download something, here it is:

Code: [Select]

#ifndef Font_H
#define Font_H


#include <windef.h>
#include <wingdi.h>


namespace OP2ForcedExport
{
	
	class GFXSurface;
	struct RenderDataBase;
	struct Point;
	struct Size;
	struct Rect;



	// Min Size: 0x4
	class FontBase
	{
	public:
  // Virtual member functions
  virtual ~FontBase();    // 0x416BE0  (scalar-deleting: 0,1)

	public:
  // Member variables
  // ----
  // vtbl        // 0x4CFBD0
  // ----
	};


	// Min Size: 0x1C60
	class Font : public FontBase
	{
	public:
  // Virtual member functions
  virtual ~Font();    	// 0x415CB0  (scalar-deleting: 0,1)
  virtual Size* GetTextSize(Size* size, char* string, int stringLength);	// 0x415F60

  // Constructor/Destructor
  Font();        // 0x415C80
  // ~Font();      	// 0x415CF0  (non-virtual destructor)

  // Member functions
  void CreateFont(LOGFONT* logFont);	// 0x415D10
  void DrawString(char* string, int stringLength, GFXSurface* surface, Point* drawPos, Rect* drawRect, COLORREF color);      	// 0x4162B0
  void DrawString(char* string, RenderDataBase* renderData, GFXSurface* surface, Point* drawPos, Rect* drawRect, COLORREF color, int vSpacing);	// 0x4161F0
  void DrawString(char* string, RenderDataBase* renderData, GFXSurface* surface, Rect* textRect, Rect* drawRect, int vSpacing);    	// 0x416B20
  void ParseString(char* string, int stringLength, int boundingWidth, RenderDataBase* renderData, Size* size);        	// 0x416770
  void ParseString(char* string, int stringLength, int boundingWidth, RenderDataBase* renderData, Size* size, int a1);      	// 0x415FE0 **

	public:
  // Member variables
  // ----
  // vtbl        // 0x4CFBC8
  // ----
  LOGFONT logFont;    	// 0x4  
  int b1;        // 0x40 ** Find out what this field is for
  int tmHeight;      // 0x44 
  int tmAscent;      // 0x48 
  int tmDescent;      // 0x4C 
  int tmInternalLeading;    // 0x50 
  int tmExternalLeading;    // 0x54 
  int tmMaxCharWidth;    	// 0x58 MaxCharWidth+1
  GLYPHMETRICS glyphMetrics[256];  // 0x5C
  char* characterImageBuffer;  	// 0x1C5C
  // ----
	};



	struct RenderChunk
	{
  int xOffset;      // 0x0  Pixel offset of this chunk of text
  int stringStart;    	// 0x4  Index of first strin char in this chunk
  int stringLen;      // 0x8  Length of this chunk in characters
  int bIsEOL;      	// 0xC  Is-End-Of-Line, used for text wrapping
  COLORREF color;      // 0x10 Color used to render this chunk of text
	};

	struct RenderDataBase
	{
  int numChunks;      // 0x0 Total allocated RenderChunk space
  int numLines;      // 0x4 
  int numChunksUsed;    	// 0x8 Used RenderChunk space
  //RenderChunk renderChunk[1];  // 0xC Describes a section of unbroken text in a single color

  RenderDataBase() { numChunks = 0; };// Convenient initalizer for the struct array size
	};
	// For convenience to make a varible of a specific size
	template <int N>
	struct RenderData : public RenderDataBase
	{
  RenderChunk renderChunk[N];  	// 0xC Describes a section of unbroken text in a single color

  RenderData<N>()	{ numChunks = N; };	// Convenient initalizer for the struct array size
	};

	// Globals
	extern Font buttonFont;    	// 0x577EE8
	extern Font aFont;      // 0x579B48  ** Find out what this font is for


}	// End Namespace


#endif

This header is essentially the format I'd like to follow for the other header files in the project.

I've been keeping eveything in namespaces so they won't collide with symbols in your projects or other APIs. I had that problem initially with a few things I named the same as something that was exported. I also found out that the namespace is part of the symbol decoration, and had to go fix up all the symbol definitions in my assembly file. From now on, everything in this API is being placed in a namespace from the start.

I've also been using forward declares of classes, structs, and enums, instead of #including their full definition. This should help reduce compile times a little, particularly when changes are made to the headers. It's a bit annoying to have your entire project recompile after a minor edit to a header file, because every source file either includes that header, either directly, or through inclusion of some other header that includes it.

I've included the addresses of all declared functions. The list of functions might not be complete, but the header file can always be added to later without problems. Virtual functions usually at least get a placeholder. Non virtual functions are just plain absent from the file until they're understood. I also include a comment placeholder for the virtual function table pointer in the class layout, and a comment giving the address of the virtual function table for that class.

I've also started adding comments about what type of destructor is located at what address, and included the flag combinations that it handles properly (0-3). Only one of these can be uncommented. The other is just there as a way to document the address. Generally, the virtual destructor is the one that needs to be declared, and the non virtual address is the one that needs to get placed in the assembly file.

I included offsets of most member variables in a comment. This is particularly useful if you're using the header as a guide while debugging.

Global variables of that class type are declared at the end of the file as extern, and the global address is given in the constants.


Oh, and I included a nice template hack to get a struct with a variable sized array in it, and allow you to specify the array size you want at the point you declare the variable. I'm proud of that one.  
 

[Hidiot]

In response to Sirbomber's comment:
I actually understood some parts, including the general idea, despite having little to no knowledge about this stuff.

[Hooman]

I've got a bit more to add.


TethysGame.h
----------------

Code: [Select]

// Include Guard
#ifndef TethysGame_H
#define TethysGame_H


#include "GameOpt.h"
#include "GameStartInfo.h"
#include "Player.h"


namespace OP2ForcedExport
{

	class GameNetLayer;
	enum GameTermReasons;
	class TFileDialog;


	class TethysGame
	{
	public:
  // Static Constructor/Destructor
  // [Static]Constructor()	// 0x4890C0
  // [Static]Destructor()  // 0x489100

  // Constructor/Destructor
  TethysGame();    // 0x489130

  // Member functions
  bool LoadDebugMap();            // 0x489170
  bool StartGame(GameStartInfo* gameStartInfo);      // 0x4893B0 bSuccess
  void LoadPlayerGameStartInfo(GameStartInfo* gameStartInfo);  	// 0x4899C0
  int F1(MissionResults* missionResults);        // 0x489AB0 **
  MissionResults* GetMissionResults();        // 0x489B00
  void InitPlayers(int numPlayers);        	// 0x489E50
  void ProcessGameCycle();          	// 0x489EA0
  void Deinitialize();            // 0x48A080
  bool Save(StreamIO* savedGame);          // 0x48A150 ** [Dead code?]
  bool Load(StreamIO* savedGame);          // 0x48A170 ** [Dead code?]
  bool VerifySavedGameTileTag(StreamIO* savedGame);    	// 0x48A190 bSuccess
  bool SaveGame(StreamIO* savedGame, TFileDialog* savedGameDialog);	// 0x48A1F0 bSuccess
  bool LoadGame(StreamIO* savedGame);        	// 0x48A3E0 bSuccess
  void LocalPlayerQuit(char a1, char a2);        // 0x48A6A0 **
  void PostQuitMessage(int quitMessageIndex, int quitDelay);  	// 0x48A720
  // ----

  // Static member functions
  static void __fastcall AddMessage(int pixelX, int pixelY, const char* message, int toPlayerNum, int soundIndex);	// 0x478710

	public:
  // Member variables
  GameOpt gameOpt;    // 0x0
  int numPlayers;    	// 0x68 Initialized from level DescBlock
  int numHumanPlayers;  	// 0x6C 
  int cpProcessingInterval;  // 0x70 Note: Must be a power of 2
  int lgCpProcessingInterval;  // 0x74 Cached base 2 log, so a bit shift can be used to multiply
  int networkCpArraySpace;  // 0x78 Space available for Command Packet data in network packets
  int b1;      	// 0x7C **
  int b2;      	// 0x80 **
  int tick;      // 0x84 Note: 100 ticks per mark, most input processing done every 4 ticks
  int tickOfLastSetGameOptCp;  // 0x88 Not set by FreeMoraleLevel, SetGameSpeed, SetDaylightEverywhere
  int b3;      	// 0x8C **
  int localPlayerNum;    // 0x90 
  int b4;      	// 0x94 **
  int startFadeOutTick;  	// 0x98 
  GameTermReasons termReason;  // 0x9C 
  int b5;      	// 0xA0 **
  GameStartInfo gameStartInfo;	// 0xA4  Game starting parameters, used to restart a level, and return MissionResults
  char padding[3];    // 0x481 Padding, due to unaligned size of GameStartInfo
  Player player[7];    // 0x484
  GameNetLayer* gameNetLayer;  // 0x5980
	};


	// Globals
	extern TethysGame tethysGame;  // 0x56EA98

}	// End namespace


#endif	// End Include Guard

I might get rid of the static member here (AddMessage), and add it to a "Log" class.

Unknown functions are listed as F1, F2, F3, .... Unknown parameters are listed as a1, a2, a3, .... Unknown member variables are listed as b1, b2, b3, .... Any lines with an unknown part have a "**" in a comment, so they can be found easily through search, and fixed or finished later.

I was also tempted to start wrapping some of the definitions in #defines. That way you can choose not to define the global variables, or member functions. It might help with debugging, although it's mostly only an issue while this project is only partly released. Any suggestions on this?

Currently, without the assembly file, if you try to use one of these extern globals, you won't get errors until the link step. If the globals weren't defined in the header, then you'd get errors as soon as you tried to use them (in the compile step). If you don't use one of these variables, then of course it doesn't care if it was declared extern (in the header) and then the symbol never defined (in the assembly file). In other words, the suggested option just lets you move the error from the link step to the compile step, so you can actually find the offending line.

What the above means is, as long as you don't try to use the global variable defined in the header, and don't try to use any member functions, you can #include this file into your projects, and it will link correctly. (That is, don't use anything with an address after it). This is useful if you want to use the pointer casting tricks to start making use of these headers without the assembly component. You might have to comment out included header files and variable declarations until those header files become available though. (If you comment out a variable, then every variable declared after it in a struct or class will no longer point to the correct memory though).

Example (after commenting out *some* stuff that gives errors):

Code: [Select]

#include "TethysGame.h"
using namespace OP2ForcedExport;
...

TethysGame* tGame = (TethysGame*)0x56EA98  // Pointer casting trick (use address specified for the extern variable)

tGame->gameOpt.forceMoraleExcellentPlayerBitMask = -1;  // Fix morale  (without "Cheated Game!" message)

To make use of the GameOpt fields, you can't exactly comment out that one header. I'll post that one next.

[Hooman]

I was a bit tempted to gut a bit of this file, just in case. However, I am curious about a couple unknown fields, so if someone wants to play around and figure them out, I want to know. (Check the "**" lines).

Btw, settings these options through the exported functions are what cause the "Cheated Game!" message to appear. There are 3 exclusions to the cheat message that I noticed. They are marked by "(Not Cheat)" in the list. If for some special reason you need to set any of these options in your level, directly writing to them should change them without displaying the message. Bear in mind that direct memory writes don't synch in multiplayer of course. (Otherwise I wouldn't be posting this). Any use of direct writes in a multiplayer level will of course require some sort of synchronization built into the level. (Such as writing the option in a trigger callback that triggers on all computers). In other words, you can only really do this in new levels, and we'd know where to look if you tried any monkey business.


GameOpt.h
-------------

Code: [Select]

#ifndef GameOpt_H
#define GameOpt_H


namespace OP2ForcedExport
{

	const int numOptions = 26;

	enum GameOptIndex
	{
  // Main Debug flags
  GameOptBoolPowerBoost  	= 0,	// 0x0 Adds 0x100000 = 1048576 power, buildings work without CC, and 100000 common ore storage (rare storage?)
  GameOptBoolAllTechsAvailable	= 1,	// 0x1 Research.HasTech always returns true, and all beacons surveyed. Lab research list not updated: prevents research (fails to start as expected)
  GameOptBoolLogMoraleOutput  = 2,	// 0x2 Log Morale status to "MORALE.LOG"
  GameOptBoolDoDamage4X  	= 3,	// 0x3 
  GameOptBoolFastUnits  	= 4,	// 0x4 
  GameOptBoolInstantKitBuild  = 5,	// 0x5 Instant research
  GameOptBoolShowUnitPaths  = 6,	// 0x6 
  GameOptBoolShowMiniMapUnits  = 7,	// 0x7 Does not hide vehicles in the dark when headlights are off
  
  // Placeholder Cheats  [switched through to specified Player]
  GameOptSetCommonOre    = 8,	// 0x8
  GameOptSetRarePre    = 9,	// 0x9
  GameOptSetScientists  	= 10,	// 0xA
  GameOptSetWorkers    = 11,	// 0xB
  GameOptSetFoodStored  	= 12,	// 0xC
  GameOptSetKids    	= 13,	// 0xD

  // RCC Effect Debug flags
  GameOptBoolDisableRccEffect  = 14,	// 0xE
  GameOptBoolEnableRccEffect  = 15,	// 0xF

  // Unknown, possibly removed  [Set by hidden or removed UI]
  // = 16,	// 0x10 **
  // = 17,	// 0x11 **

  // Daylight flags
  GameOptBoolDaylightMoves  = 18,	// 0x12
  GameOptBoolDaylightEverywhere	= 19,	// 0x13 (Not Cheat)

  // Game Speed
  GameOptGameSpeed4X    = 20,	// 0x14 (Not Cheat)

  // Fixed morale
  GameOptForceMoraleExcellentMask	= 21,	// 0x15 
  GameOptForceMoraleGoodMask  = 22,	// 0x16 
  GameOptForceMoraleFairMask  = 23,	// 0x17 
  GameOptForceMoralePoorMask  = 24,	// 0x18 
  GameOptForceMoraleTerribleMask	= 25,	// 0x19 

  // -- End Global options --

  // Player resource cheats
  GameOptPlayerAddResources  = 26,	// 0x1A  Workers + 50, Scientists + 25, Kids + 10, Common Ore + 1000, Rare Ore + 1000, Food Stored + 25000
  GameOptAllPlayerAddResources	= 27,	// 0x1B  Workers + 50, Scientists + 25, Kids + 10, Common Ore + 1000, Rare Ore + 1000, Food Stored + 25000

  // Disaster creation debug options
  GameOptCreateVortex    = 28,	// 0x1C  Vortex (0,   700):8,  RoboDozer (0, 21)
  GameOptCreateLightning  	= 29,	// 0x1D  Storm  (0,   500):12, RoboDozer (0, 15)
  GameOptCreateEarthquake  	= 30,	// 0x1E  Quake  (512, 2048):4
  GameOptCreateMeteor    = 31,	// 0x1F  Meteor (100, 700):4,  RoboDozer (3, 15)

  // Free morale level
  GameOptFreeMoraleLevel  	= 32,	// 0x20 (Not Cheat)
	};

	union GameOpt
	{
  struct
  {
  	// Main Debug flags
  	int bPowerBoost;    // 0x0  Adds 0x100000 = 1048576 power, buildings work without CC, and 100000 common ore storage (rare storage?)
  	int bAllTechsAvailable;  	// 0x4  All bulding options available. Prevents research  (fails to start as expected)
  	int bLogMoraleOutput;  	// 0x8  
  	int bDoDamage4X;    // 0xC  
  	int bFastUnits;    	// 0x10
  	int bInstantKitBuild;  	// 0x14 Instant research
  	int bShowUnitPaths;    // 0x18
  	int bAllMiniMapUnitsVisible;	// 0x1C

  	// [No global references, special cased to map to specified player]
  	int setCommonOrePlaceholder;	// 0x20
  	int setRareOrePlaceholder;  // 0x24
  	int setScientistsPlaceholder;	// 0x28
  	int setWorkersPlaceholder;  // 0x2C
  	int setFoodStoredPlaceholder;	// 0x30
  	int setKidsPlaceholder;  	// 0x34

  	// RCC Effect Debug flags
  	int bForceDisableRCCEffect;  // 0x38 For all Human Players
  	int bForceEnableRCCEffect;  // 0x3C For all Human Players

  	// Unknown, possibly removed
  	int b1;      	// 0x40 ** [Possibly set through hidden or removed UI]
  	int b2;      	// 0x44 ** [Possibly set through hidden or removed UI]

  	// Daylight flags
  	int bDaylightMoves;    // 0x48
  	int bDaylightEverywhere;  // 0x4C (Not Cheat)

  	// Game Speed
  	int gameSpeed4X;    // 0x50 (Not Cheat)

  	// Fixed morale
  	union
  	{
    struct
    {
    	int forceMoraleExcellentPlayerBitMask;	// 0x54
    	int forceMoraleGoodPlayerBitMask;  // 0x58
    	int forceMoraleFairPlayerBitMask;  // 0x5C
    	int forceMoralePoorPlayerBitMask;  // 0x60
    	int forceMoraleTerriblePlayerBitMask;	// 0x64
    };
    int forceMoralePlayerBitMask[5];
  	};
  };
  int option[numOptions];    // 26 Options
	};

}	// End namespace


#endif

I used a few unions to allow multiple ways of accessing certain fields. That way you can easily name them by name, or through the array using a named index (from the enum), or just by iterating through the array to adjust a bunch of options at once.


Btw, a hint for the two unknown options. If you have a resource decompiler, find the game options window (probably), and look for any controls with IDs 0x9CBC = 40124 or 0x9CBD = 40125. Other options from that window are modified in the same section of code, and they also have nearby control IDs.
 

[Hooman]

I've cleaned up most of the User Interface stuff. There's quite a lot in that section, so I'm attaching it all as a zip.

There are a number of unknowns, some of which could probably be easily determined through experimentation. (Anyone care to save me some time?)


This is probably the most useful part of the project. The majority of the work put into this project was to do with the user interface. This is partly because both the net patch, and the disaster targetting test was quite reliant on user interface code.

Using the classes in this zip, you can create custom Command Pane Views, add Buttons to them, display info using the Font class, install global Filters to capture new hot keys, and create new in game commands, including Mouse Commands that require the user to click the detail pane. You can also do a fair bit of low level graphics work, and access properties of the Detail Pane, Command Pane, and Mini Map Pane. You can also make use of Dans_RULE_UIFrame, the MultiplayerPreGameSetupWnd, and create your own custom skinned windows, like the Net Patch does.


I suppose people will need examples of how to use this stuff, but I don't have the time at the moment. If there is a particular class you're interested in, then ask, and I'll post some example code using it.
 

[Mcshay]

I'd like to test this out, but I need a complete copy of your SDK. Could you post any missing headers? I'm sorry if I missed them somewhere but I'm still trying to figure out how all of this works.

[Hooman]

I'm still completing them. I'm posting them as I finish cleaning up that section of the source tree. One of the reasons why I'm posting early is because it gives me a little more incentive to keep working on it.

What I'll eventually do is zip up the whole project and/or put it up on SVN. There are still a few files that I've added to the project as placeholders, but haven't gotten around to putting anything into them yet. I feel the need to fill in those files before posting that section of the source.

As for now, I'm gonna guess the headers you need are the ones defining the really basic types. Other than those, each tree of the project is pretty independent.

PointTypes.h
--------------

Code: [Select]

#ifndef PointTypes_H
#define PointTypes_H


namespace OP2ForcedExport
{
	// Common Types

	struct Point
	{
  int x;
  int y;
	};

	struct Rect
	{
  union
  {
  	struct
  	{
    int x1;
    int y1;
    int x2;
    int y2;
  	};
  	struct
  	{
    Point topLeft;
    Point bottomRight;
  	};
  };
	};

}	// End namespace


#endif

BasicTypes.h
---------------

Code: [Select]

// Win32 data types, compatable with windows.h, but doesn't require windows.h


#ifndef BasicTypes_H
#define BasicTypes_H


typedef long LONG;
typedef unsigned int UINT;
typedef unsigned long ULONG;
typedef __int64 LONGLONG;
typedef UINT WPARAM;
typedef LONG LPARAM;
typedef unsigned long DWORD;

typedef DWORD COLORREF;


#define MakeHandleType(name) typedef struct name##__* name

MakeHandleType(HINSTANCE);
MakeHandleType(HHOOK);
MakeHandleType(HACCEL);
MakeHandleType(HMENU);
MakeHandleType(HWND);
MakeHandleType(HDC);
MakeHandleType(HFONT);
MakeHandleType(HICON);
MakeHandleType(HBITMAP);
MakeHandleType(HPALETTE);

typedef HICON HCURSOR;


#endif

Please note I may rename a few files along the way. In particular, I'm thinking I should rename BasicTypes.h to WinTypes.h. I think I'll keep all the Win32 stuff together in one file, and seperate from the rest.
 

[Hooman]

I've been working on the StreamIO classes. I've got them mostly cleaned up, except for a few that handle Vol files.

Seems I found something new too. From way back we had the Vol IndexEntry listed as follows:

Code: [Select]

	struct VolIndexEntry
	{
  DWORD fileNameOffset;	// 0x0 Offset in string table
  DWORD dataOffset;  // 0x4 Offset to the data in the VOL file
  DWORD dataLength;  // 0x8 Length of the internal file data
  BYTE compressionCode;	// 0xC Compression used on file data
  BYTE bUsed;    // 0xD Indicates index entry is valid  (not quite bool: tested against 1)  [Unused entries must only occur at the end of the table (binary searched)]
	};

It seems that bUsed isn't quite what I expected it to be. There is some special use for it. If it's set to 1, then it's a normal file entry. But if it's set to 0, then it seems to be a special file entry. Instead of doing a lookup based on the file name, it uses an ID instead, which is stored in the fileNameOffset field.

I also noticed some code using the fileNameOffset field like a char* pointer, instead of an offset. Not sure if I had that documented anywhere either. I suspect it's just a difference between the disk format, and the memory format. Most likely translated when saving or loading.
 

[Hooman]

Time for some updated code to be posted. It's been a while, and a fair bit of work was done. Keep in mind, that a fair bit of work also involves reading, figuring out, and commenting lots of assembly. Without that, these files wouldn't be as accurate, or nearly as complete. Granted, there are some things we'll probably never figure out, so this project will necessarily remain and "unfinished" work.

I still haven't gotten around to the Game section of this project, which contains things like Units. Sadly this is probably one of the more useful parts for what some people want done, but it's also the part best covered by the already exported functions. It will probably be next.


This package contains some header files that would be useful for building a new underlying network protocol, as well as resource management classes, and the user interface stuff, which I think I already posted.

There are class descriptions in the resource management part that can compress files, and store them into VOL files. We have standalone decompressors for one of the methods, but nothing to recompress files. If someone wanted to toy around, they could use these classes. There is also one used to parse text streams, that was used to process the sheets files.

This update also includes classes for handling bitmaps, in a number of formats, including Windows bitmaps, and the other custom formats that OP2 supports. It also describes the classes for loading the tile sets, and op2_art.bmp. The Prt processing code hasn't really received any significant updates lately, like I hoped it would. Maybe sometime in the future. Some corrections and updates regarding palettes were made. Also, the Bitmap classes can be passed to certain low level drawing functions on the GFXSurface classes to draw custom images in game, and they can even handle the color fading that happens at night. Although, I haven't gotten around to documenting the needed functions in the GFXSurface header files yet. *notes on to-do list*.

 

[Mcshay]

Thanks for the update, I'm really looking forward to using this later (when I have time).

[Mcshay]

Any recent progress?

[Hooman]

There's been a little bit of unreported progress, but it's stalled a little as of late. Most of the easy stuff is done, and I've gotten kind of busy with work.

I was working on the game engine related classes last. I had lots of details worked out for TethysGame and some minor work on the Unit classes. The Unit class hierarchy will be a huge job to work out though. Mind you, I've got a lot of info to help with that, such as class sizes, but it's still a tree structure with over 100 leaves. I also have to work out the UnitTypeInfo tree structure, which is equally large, but should probably match the Unit tree structure.

 

[Mcshay]

Holy cow, that's big. Good luck with it, I'm really looking forward to using this.

[Arklon]

Last night I figured I'd modify Forced Exports by making it use a helper DLL that exports the functions instead of using the asm file hack. In order for the DLL to export them, it has to define and export dummy versions of every function, and then it patches up its Export Address Table upon loading so it points to addresses in Outpost2.exe.
I've been running into obstacles that have made me use more and more disgusting code, though. For example, the only way to get the true address of a virtual function programatically is to use the inline assembler. To get the address of any other non-static class member function, you have to either use the non-standard (void*)&(void*&)Class::Function cast, or the assembler. And if you need the address of one of x function overloads? It gets even worse.
Another caveat is the DLL must be compiled with debug information, or else the compiler will discard a bunch of the declared functions regardless of optimization settings, which breaks it of course.
Still, after I write up a (sick) placeholder solution for the overloaded functions, I'll put it up on the SVN. Honestly the whole thing got a bit clunky though. It ended up not being the elegant solution I thought it'd be. Still, it has the advantage of not breaking when Outpost2.exe and/or modules using it don't get loaded at their expected addresses, and you don't have to set up your compiler to use NASM.

Edit: It's committed to the SVN under /LevelsAndMods/trunk/API/ForcedExports-ExporterDLLTest. To use it, you'll need to compile it, place the op2exp.dll file into your Outpost2 folder, and place the op2exp.lib file where you would normally put your lib files. Make your DLL include ForcedExports.h (important: this include must come before any Windows header includes), and be dependent on op2exp.lib. Finally, for the DLL to be loaded by Outpost2, in Outpost2.ini, edit the line:

Code: [Select]

LoadAddons = "NetFix"

To be

Code: [Select]

LoadAddons = "NetFix, ForcedExports"

And add this after the [NetFix] section:

Code: [Select]

[ForcedExports]
Dll = "op2exp.dll"

I added macros to help with exporting overloaded functions, but they're unused as of right now; therefore, any functions with overloads aren't having their exports remapped in the current revision, and cannot be used. Also, only functions/data that are usable in the original ForcedExports (except for overloaded ones), as in, they are "resolved" by ForcedExports.asm, are actually usable as of right now.

The whole thing didn't work out quite as well as I had hoped (although it is definitely usable), so I'm probably going to drop the project. Fortunately I didn't spend too much time on it.

[TH300]

Thats the right thinking. Its always good to make something easier usable. Not to mention the benefits of deduplicated code. And the way your dll works/is supposed to work is quite smart, because it eliminates all overhead. A pity that it turned out to be so complicated.

[Arklon]

Actually, compiling it with debug information isn't needed. Just turn "keep unreferenced data" on under project settings->linker->optimization. Still, the whole thing is weird because the things it throws out ARE referenced (and it throws out the reference too!). Of course, the DLL is still relatively big because of this at ~250 kb, but the DLL with debug information was ~350 kb.
Edit: Ok, it wasn't discarding the reference (that conclusion didn't really make sense anyway). It was making merging all similar "dummy" functions into one shared function; this, of course, would make figuring out what entry to replace by the address of the export impossible.

Hmm, another potential problem... with classes with virtual destructors, instead of placing Class::~Class directly into the vftable, it places a function in there ("Class::`vector deleting destructor'") that wraps Class::~Class (and internally the DLL doesn't know about the export table hack, if you called Research::GetTechNum within the DLL for instance it would still call the blank "dummy" function). Unless there's a way to stop this behavior, these classes won't use the destructors in OP2.
Edit: Actually, as the vector deleting destructor isn't being exported, any DLL using it will make its own copy, that will wrap the imported Class::~Class, so everything should still work out as expected.

Damn it, ran into another problem with overloads: when constructors are overloaded, which is actually what most of the overloads are. My sick method for getting the address of a function overload doesn't work with constructors (yay C++ language restrictions). The only way to take the address of a constructor is with the assembler, which is (apparently) forbidden from having anything to do with C++ overloaded functions.

The only other method I can think of is, use the exported function names along with a name unmangler (for simplicity, I'd want to keep most unmangling "simple" as in Variable/Function/Class::Function, although for overloads the full unmangled name would be needed), and just compare strings. Kinda disgusting though, but it would work, whereas the current method isn't any better and has show-stopping pitfalls.

Update: I figured out how to expose the C++ RTL name undecorator (I could've used UnDecorateSymbolName from dbghlp but I didn't want to add a dependency to that), so I'm gonna experiment with the name string comparison method.

Another update: Committed a new version of the project that uses undecorated names to figure out what export needs to be remapped to what.

[Arklon]

Did a little more work on it. Decided that it'd be best to use the assembler method for non-overloaded functions, and the name string comparison method for overloads, as the string manipulation functions are fairly expensive when using them for every export "remap".

A couple alternatives to this method of forcing exports:
a) Instead of patching the export address table, make all the "dummy" functions call the desired function, or make them naked functions that jump to the desired function perhaps. This would be fairly useless if you wanted to use it for API hooking, though; if you just take a pointer to the function, that would of course give you a pointer to the "thunk" function and not the desired target within OP2. You could parse the assembled bytes of the thunk, but that would be sick, and would become more complicated if they weren't simple naked functions with a jmp instruction. Even if they were functions that jmp'd, there would be instructions for adjusting the offset of the jmp with the base address of OP2 that would complicate things; unless, the jmp instruction would be inserted into the functions at runtime rather than compile time, but that would be about zero improvement over the method I've used here (it would also mean making it all a static library would be virtually out, as in order to easily patch the functions, in the event they are overloaded, the only method I've found was to compare the undecorated symbol name of the function, which only works if the function is exported, or if you compile your DLL with debug info (nasty to do that with release versions, plus I think you need to prove the PDB of your DLL as well, which makes it even less attractive)).
b) Manually add entries to the export table within Outpost2.exe. BlackBox and I talked a bit about this the other day. We'd have to add a new data section to the exe (there are programs out there that could do this for us), then relocate the existing export table there so we have room to add new entries. This would be by far the nicest and cleanest solution, but would be a lot of work. We'd likely want to write a program to help us easily add new export table entries. Additionally, it's probable it would get updated relatively often, and we'd need to write some kind of auto-update system for OP2 in order to make distribution painless. This method can also have the problem of creating a sort of "dependency hell": consider a case where we decide the name of a new function entry we added wasn't the most appropriate one upon further inspection of it; if we change it though, it would break compatibility with any module that used it previously. (op2exp.dll can suffer from this same problem.) Could always add duplicate entries with different names for compatibility, but that could clutter up the export table with a bunch of depreciated entries.

[Sirbomber]

B) Manually add entries to the export table within Outpost2.exe.

May wanna turn off smilies Arklon.

[TH300]

Part of this is beyond my scope. I have the impression that replacing the export-table would be the nicer solution, because it doesn't require sick function pointer replacements and leaves everything in a natural state.

An automatic update system is not required. Just finish the thing and THEN include it in the game download. Its not like its absolutely needed now.

[Arklon]

An automatic update system is not required. Just finish the thing and THEN include it in the game download. Its not like its absolutely needed now.

Easier said than done. Even ForcedExports wasn't nearly complete. Plus, there are still many, many functions (for example, much of the pathfinder) we haven't yet looked at a whole lot that we'll need to to properly name them. If we waited until we were totally done categorizing every internal function, at that point, we'd be very close to having reverse engineered enough of the game to start rewriting it anyway.

Edit: Turns out I was wrong about the assumption where creating an instance of a derived class caused the non-forced-exported base class constructors to be called. That really made no sense to me; calling the base class constructor happens via a function call to it within the derived class constructor function, which was being remapped to the constructor within OP2, so it should have for sure been calling the base class constructors within OP2. That's good at least.

Edit 2: Uploaded the latest revision to the SVN. Also, I was thinking that perhaps there could be a way to inject EAT entries programmatically rather than do horrible horrible things to the exe file's actual EAT. You'd have a DLL that allocates some memory for a new EAT, which would then copy the contents of Outpost 2's EAT into itself, add entries for out custom exports, then overwrite the pointer to the old EAT and point it at the new one. The RVAs inside the new EAT would be relative to Outpost2.exe and not the DLL whose memory contains it, I believe. Generating an import library so the new entries can be used wouldn't be very straightforward, though. Additionally finding out what the mangled name for a new entry should look like could be a hassle. Ideally I'd want to use a function that converts a regular name into a decorated symbol name (it would also need to be aware of things like typedefs and macros). Maybe the WINE project has implemented such a thing, I know for sure they wrote a name demangler. If not, their demangler could probably be rewritten to accomplish the reverse effect.  

[Arklon]

I've been testing out the method mentioned in the last edit of the above post. I have it partially working right now. I can import injected C-style imports this way so far. C++-style with all the extra mangling, on the other hand, is refusing to work for some odd reason; C++-style is necessary due to using C++ constructs like classes and whatnot that the C-style symbol names don't support. I'm not gonna commit what I have so far to the SVN because it's very messy right now.

Update: Apparently the "?" at the beginning of the C++ symbol name is causing it... still haven't figured out how to fix it though.

Another update: For some odd reason it wants to only check the entry as described by the "hint" in the import thunk of the exported names table to try and find the export when it is a C++-style name, rather than searching through the whole table. GetProcAddress, on the other hand, decides to only check the first exported name entry. Again, this is all assuming the name is a C++ symbol name; as stated before, C symbol names work fine. This also assumes the OS is or is based on Windows NT. It might not have this goofy behavior (and would therefore work) in Windows 9x for all I know. It may also work under WINE, and possibly ReactOS.
I'm probably gonna need help from someone who knows how the Win NT internals work (specifically the ntdll loader functions) in order to fix it - assuming there is a fix, that doesn't involve having to insert hooks inside system DLLs, which would be a really bad idea and likely different hooks would be needed for different OS versions (9x vs. NT-based, and then some if/when there are future Windows versions that are totally different). This whole problem may possibly be caused by the logic for forwarded exports within the NT loader functions that deal with exports, but that is speculation.

Yet another update: As a test, I hex edited the export data directory and swapped a the RVA pointers to a couple exported symbol names around, and tried loading up a DLL that referenced one of the associated exports. Surprise, surprise, it failed to load with a "procedure entry point could not be found" error (because the export name was no longer at the ordinal the hint says it should be at). I wonder if this could be some kind of bug within ntdll when, if the ordinal the hint says to check first is a mismatch, it's supposed to walk through the exports table to find the entry it needs. Or, if the import libraries created for Outpost2 (op2.lib) and the one I created for the custom exports are, for some odd reason, only wanting to really import by ordinal somehow?

Update: I think I figured it out. The export names table MUST be sorted alphabetically, in this order:
?
0-9
_
A-Z
a-z
That's the sorting criteria I know of, anyway. Well, that isn't a good description at all; I'll attach a dump of OP2's exports with some comments added showing where new exports with different name schemes would likely need to be. That should help give you an idea.
Actually, it's sorted by the hex codes of the ANSI characters that make up the name strings. (A reference of the character hex codes is at http://www.alanwood.net/demos/ansi.html )

Also, there is an upper limit of 65535 things you can export by name (and a maximum of 4294967295 total exports). There's < 4600 functions in Outpost2.exe, and there wouldn't be a whole lot of objects/variables we'd want to export, so that's not gonna be an issue. Even if it was, it could be worked around by making a dummy DLL that would get loaded by Outpost 2 that would have an export table written to it containing the exports that wouldn't fit inside Outpost2.exe's export table, then DLLs needing them would just import them from the dummy DLL. Or you could just import by ordinal, but that would be kinda icky since the ordinals of the new exports would probably constantly be jumbled around most any time new exports were added.