Okay, so I got this far: I can parse a .map file using the instructions at http://forum.outpostuniverse.net/index.php?showtopic=1250 (http://forum.outpostuniverse.net/index.php?showtopic=1250). I can check, for every tile, which image from the sets should be displayed. I can load that image an display it.
Only, the stuff is mangled. The map should be 128x128 if I remember correctly, but I didn't figure out the exact tile ordering yet. It doesn't seem to be by width, and not by height.
What I did find out is that the map seems to be "interleaved". I saw some pieces of a vulcano, which should display something like this:
______V________
_____VVV_______
____VVVVV______
_____VVV_______
______V________
_______________
but it displays like:
______V________ _____VVV_______
____VVVVV______ _____VVV_______
______V________ ________________
(if that makes any sense..)
I'll attach some screenshots if needed... Otherwise, is there anything known about this? Maybe even some code that ultimately decides in what order they should be displayed? I have one central place where a tile is fetched from the array, so I need the algorithm to translate the X and Y parameters to the exact location in the array.
Thanks!
Okay. Now for saved maps loading support, this is weird. This is the documentation in the old topic (http://forum.outpostuniverse.net/index.php?showtopic=1250):
0x0 0x19 "OUTPOST 2.00 SAVED GAME", 0x1A, 0 - String must match exactly or else load error
---------------------------
Note: The following is repeated as an array of 0x74 (116 decimal) elements
0x19+i*0x1E0 4 **TODO** Figure this out
0x1D+i*0x1E0 0x1DC **TODO** Figure this out
---------------------------
Note: From here, the format matches that of .map files
This is what hexdump says about the first part of the file:
00000000 4f 55 54 50 4f 53 54 20 32 2e 30 30 20 53 41 56 |OUTPOST 2.00 SAV|
00000010 45 44 20 47 41 4d 45 1a 00 94 c9 21 8c f8 75 1c |ED GAME....!..u.|
00000020 4a 00 00 00 00 62 6c 61 61 74 73 63 68 61 61 70 |J....blaatschaap|
00000030 20 6e 61 61 6d 20 76 61 6e 20 73 61 76 65 67 61 | naam van savega|
00000040 6d 65 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |me..............|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000070 00 00 00 00 00 94 c9 83 b8 15 8d a0 c8 15 8d a0 |................|
00000080 c8 15 8d a0 c8 15 8d a0 c8 15 8d a0 c8 15 8d a0 |................|
*
("blaatschaap naam van savegame" is some foo long name I came up with in a split second)
- so I can see the "OUTPOST 2.00 SAVED GAME\x1a\x00" but then there's 8 bytes, 4 zero bytes, then 64 bytes containing the name of the savegame and the rest null... and then semi-random data. Those 8 bytes at the top could mean, in numbers:
94 c9 21 8c = 148 201 33 140 / 51604 35873 / 2351024532
f8 75 1c 4a = 248 117 28 74 / 30200 18972 / 1243379192
Now I checked, at least I tried to check, the length of the data behind that header. It was nothing near 51604, 35873, 30200 or 18972, but the data seemed to switch to something else after 0x10880 (= 67712 bytes). The bytes themselves seem to be mostly repeated all the time except for the first three bytes, 94 c9 83, which could be 148 201 131 / 51604 / 33737 / 8636820 - numbers which also make few sense, except that the first two are the same as the two bytes right after "OUTPOST 2.00 SAVED GAME\x1a\x00"... if these are check bytes, that leaves 131 as a mysterious number....
The file size is 467026 bytes, I've found the location where the actual .map format part starts, that's at 0x1e025 = 122917 bytes. 467026-122917=344109 which may come close to 35873 or 33737 or so, but now I'm just doing random guesses.
Is anything other than this already known about this, Hooman (or anybody else)?
As you've probably understood by now, I'm trying to recreate some of Outpost 2's code, in order to have something as a community to build upon and improve. In the spirit of OpenTTD (http://openttd.org/), a remake of Transport Tycoon Deluxe, I've named my project OpenOutpost2. Once I'm happy with what I have, it will be freely available under the GNU General Public License - i.e. freely modifiable by anyone as long as they put a patch available... :) (and some other stuff, read the license if you want to know, it's not that big)
So I have two concrete questions:
1) Could you clear up what's on that topic? So there's an array of 116 times 480 bytes, is there nothing known about the values of that? Also, so the array is 55680 bytes; could you maybe even send me the save file you have done your testing on to see if that 116 is named somewhere? That could be the mysterious 131 number; if mine is 131 times 480 bytes, that leaves 62880 bytes, which comes close to the 67712 number already there. Any ideas?
2) Is there maybe even an existing implementation to read the save files? I found this old topic: http://forum.outpostuniverse.net/index.php?showtopic=882 (http://forum.outpostuniverse.net/index.php?showtopic=882) - is BlackBox still active and around? Does he have any idea how to go about and start parsing this?
Thanks!
Okay, I wrote code to load all data, but I think a lot will still be incorrect for now, because I didn't have the original code for the Load methods.
For example, the Research class is pretty much a mystery for me:
class Research
{
public:
// Member variables
int numTechs;
TechInfo **techInfo; // TechInfo*[]*
int maxTechID; // 12999 [maxTechLevel * 1000 + 999]
// ...
// 0x2C
};
I can't just read TechInfo** from file, and the class is supposedly 0x30 bytes in length (assuming 0x2c is the starting code of the last dword); I don't know how to load that from file, I don't know how long the piece is I have to load, etc. Could you clear up on the length of the research data, and maybe even how to load it? I wrote a lot of code but haven't tested a bit of it yet; but anything will surely fail if I don't know the real size of the Research class. ;)
Another question is regarding the IWnd data - of course, I don't use IWnd. Is there anything important in there? If so, I'll need to take the data I need, and call some Qt functions to get it out the same. But if there isn't anything important, I'll just skip it like I do now, no problem.
In the same way I'm also ignoring the LevelDLL data, since I don't have any DLL's to use them with. Once most of the project is done, we can go reconstruct that data (or we don't, the clone doesn't have to have the same campaigns etc). Also, is there anything important in the LevelDLL data when the game is multiplayer (i.e. has no prebuilt script)?
Edit: It seems to be humming along OKishly. However, it fails even before the research data. It is at position 158791 in a 298601 byte large savegame; the location of 1023 * 120 bytes of unit data. My program currently skips that and then it's at position 281551 (so 17050 bytes left in the file for the rest of the data); it reads the map tag which is correct, and then it goes on loading the message log. After that, the program reads 0x484 bytes of TethysGame data, it skips the LevelDLL data, and then it reads Player objects - but these classes are 3120 bytes of data, so after having read five players from the file, it bumps out at the sixth because there's no more data left. (this is a single-player game with just one player, or maybe there's a CPU player without any units.)
Two (additional) questions:
1) I can't disassemble at all, could you check if TethysGame should really load 0x484 bytes, and how many bytes Player::Load(StreamIO* savedGameFile) reads from the file? I think the fix will be in there.
2) Is it true that Outpost 2 games can have 7 players exactly? :) Seems like a weird number to me...
Thanks!
Edit 2 - little status report:
edit 3 beh, no hotlinking, image (http://imagebin.ca/img/QMaA_fJ.png).
If I hack the saved game file reader to stop right after the map load, this is what we get: the map displays correctly, only unit and building locations aren't known at all yet. The red dot means the mouse was there, the program supports map scrolling already and that's a little leftover from an earlier milestone, I'm leaving it in until it becomes annoying ;)
edit 4: (removed a piece of this comment - I thought something was wrong but it turned out to be completely correct)
Okay, looks like the information on LevelDLL I have isn't enough too...
The class is 92 bytes big when packed; however, I see a *lot* more data in the hexdump:
00045100 00 00 07 00 00 00 00 00 00 00 00 00 00 00 08 00 |................|
00045110 00 00 00 00 00 00 09 00 00 00 0a 00 00 00 00 00 |................|
00045120 00 00 0b 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00045130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00045160 00 00 01 00 00 00 40 00 00 00 42 00 00 00 44 00 |......@...B...D.|
00045170 00 00 48 00 00 00 4a 00 00 00 46 00 00 00 ff ff |..H...J...F.....|
00045180 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 01 00 |................|
00045190 00 00 00 00 00 00 02 00 00 00 00 00 00 00 03 00 |................|
000451a0 00 00 00 00 00 00 04 00 00 00 00 00 00 00 1d 00 |................|
000451b0 00 00 0f 00 00 00 41 74 74 61 63 6b 65 64 54 72 |......AttackedTr|
000451c0 69 67 67 65 72 0a 00 00 00 42 75 69 6c 64 47 72 |igger....BuildGr|
000451d0 6f 75 70 14 00 00 00 42 75 69 6c 64 69 6e 67 43 |oup....BuildingC|
With a class 92 bytes big, 4510b would be the place to stop loading LevelDLL data; however it seems clear that that's not the right place, because a hundred bytes later, there are a lot of triggers, conditions, things that look like mission objectives and responses to triggers etc. To me, the place where player data starts seems only to be at around 46590 or so. Could you tell me what LevelDLL::Load() does? I think, with that info, I'll be able to finish this saved game loader.
Thanks!
Oh, and the image:
(http://files.dazjorz.com/cache/openop2_status_rev21.png)
(I currently throw away most data I read, after checking it a little and printing some statements about it. However, it should be possible to get some information in the sidebars to the right. The reason I'm still splitting between SDL and Qt is because both have their pros and cons - eventually I'll probably revert to using just one of the two, but I haven't decided which one yet; Qt has QPainter and Phonon, so it's probably going to be Qt...)
Nice.
Ok, here's a bit of pseudo code. It starts getting a little vague, but hopefully it will help parse over at least most of the data. If not, scan for the sentinel values (tag words). ;)
LevelDll.Load(StreamIO* file)
{
if (file.Read(4) != 0xAEDE0034) return false; // Error
scriptNameLength = file.Read(4);
scriptName = file.read(scriptNameLength);
LoadScript(scriptName); // Load the level DLL
// Note: Loading the script sets LevelDll fields, like the SaveRegion (offset 0x10)
if (saveRegion.buffer* != Null)
{
bufferSize = file.Read(4);
if (bufferSize != saveRegion.bufferSize) return false; // Error
saveRegion.buffer* = file.Read(bufferSize); // Note: bufferSize = 0 is valid here
}
defaultGroup[] = file.Read(0x30); // See offset 0x1C in LevelDll
???.Load(file);
if (file.Read(4) != 0xAEDE0034) return false; // Error
return true; // Success
}
???.Load(StreamIO* file)
{
numScStubCreators = file.Read(4);
if (numScStubCreators != this?.numScStubCreators) return false; // Error
for (i = 0; i < numScStubCreators; i++)
{
num? = file.Read(4);
file.Read(num?);
}
? = file.Read(4);
do
{
?? = file.Read(4);
if (?? == 0xFF) return 1; // Success
file.Read(4);
?.Load(file); // Virtual function dispatch (could very well load a variable amount of data, depending on the exact object type)
} while (?)
}
I was just thinking as I looked at the images, that'll you'll be needing the link to: http://forum.outpost2.net/index.php?showto...indpost&p=64758 (http://forum.outpost2.net/index.php?showtopic=4285&view=findpost&p=64758).
It should be good for the basics of units and buildings, but is sadly incomplete. The movement images, and many building images were extracted. However, much of the remaining information is hardcoded into the game, and there is no automated way to extract all of it. Things like cargo trucks dumping ore is special cased.
I have no idea how to read that... When I try to find the image ID's there in Cynex's art viewer, I can't find anything. For example:
AnimationList: 25 STORAGE_ORE
Eden:
0: 69 # ?
1: 59 # actually 44
2: 60 # actually 45
3: 193 # ?
4: 1109 # ?
5: 2059 # actually 43
I figured these numbers would be image ID's, but apparently, they aren't... How do I use these? :)
Edit: Oh s***. I just created a full unit/imageid mapping table, but it turns out I've been an idiot. I should use the frames, not the image ID's. I get it now... This is actually a day of mapping all image ID's gone, but luckily, it's not *completely* gone; I just have to adapt a lot of my code to get it right..... :(
Edit 2: Even more s***: It's not the frames, it's the *animations* which are important! Okay, I know how to do this now. Damn, took me a while :P But I'll get it right.
Edit 3: I'm close again. However, I have one question: Cynex' art viewer seemed to know pretty well the width and height of any frame animation. However, I don't think I see that width and height anywhere. Does his art viewer scan through all images of all frames, take their width and height and position, and then calculate the end width and height of the complete frame? Or is there an easier way to simply get the width and height? (Also, are width and height for any animation static, or does it depend on the frame? :) )
Edit 4: And shadows? In these 1 bpp images, when zero the RGBA value is probably {0,0,0,255} (fully transparent); when one what is the RGBA value? {0,0,0,128} or so? I see these shadow images have their palette set to two, yet if I try to simply use them as-is with bpp 1 and palette 2 I get some blue-ish crap under my vehicles, instead of the shadow, so I figure I have to write a palette for it myself.
The shadows are processed high bit first. If the bit is set, it does a double fade on the destination bitmap, adding the intermediates, to get 3/4 of the original intensity for each color component.
if (bSet)
{
// Process 16 bit color words.
// Halve each color, by shifting, with a mask to prevent low order bits
// from being shifted into the higher order bits of the next color component.
// (The mask clears the high bit of each color component, which should be 0
// after the "divide by 2", at least in the absence of bit shift carries).
halfColor = (*(short*)destImage >> 1) & 0x3DEF3DEF;
quaterColor = (halfColor >> 1) & 0x3DEF3DEF;
// Combine into new image color
newColor = halfColor + quarterColor; // "color*(3/4)"
// Write sum to image
*(short*)destImage = newColor;
}
Note there is a 32 bit mask value even though only 16 bits are used. Perhaps they were thinking of processing 32 bits at a time (2 color words). This however complicates the case when you have an odd number of pixels to process in a scanline.
As for processing the entire scanline or image, you may need to do some shifting of bits to account for clipping. That is, if you want to draw the shadow, starting 1 pixel into the shadow (such as when it overlaps the top left corner of the screen, and isn't completely visible), you would need to shift the first byte of each scanline by 1 bit before processing it, and going to the next byte earlier. Remaining bytes should be processed normally.
Note there is a 32 bit mask value even though only 16 bits are used. Perhaps they were thinking of processing 32 bits at a time (2 color words). This however complicates the case when you have an odd number of pixels to process in a scanline.
So, uh, if I understand correctly, in an 8 bits-per-pixel image, there are actually 32 bits per pixel? :/
And then, of those 32 bits per pixel, only 16 are used...?
see edit below
As for processing the entire scanline or image, you may need to do some shifting of bits to account for clipping. That is, if you want to draw the shadow, starting 1 pixel into the shadow (such as when it overlaps the top left corner of the screen, and isn't completely visible), you would need to shift the first byte of each scanline by 1 bit before processing it, and going to the next byte earlier. Remaining bytes should be processed normally.
With clipping, do you mean edges of the screen to what is and what isn't visible? I currently calculate all tiles of which a little bit is visible on-screen; those tiles I retrieve and paint onto a temporary surface, and then I blit the right piece from the temporary surface on to the screen. I don't really have to do any clipping like that and the pixels that are uselessly painted onto the temporary surface don't make it that much of a slowdown (but I'll try to benchmark that).
if (bSet)
{
// Process 16 bit color words.
// Halve each color, by shifting, with a mask to prevent low order bits
// from being shifted into the higher order bits of the next color component.
// (The mask clears the high bit of each color component, which should be 0
// after the "divide by 2", at least in the absence of bit shift carries).
halfColor = (*(short*)destImage >> 1) & 0x3DEF3DEF;
quaterColor = (halfColor >> 1) & 0x3DEF3DEF;
// Combine into new image color
newColor = halfColor + quarterColor; // "color*(3/4)"
// Write sum to image
*(short*)destImage = newColor;
}
This is complicated for me, since I actually work with surfaces everywhere and can't just change the colors of the parent surface in another call. However, I *can* create a plain black surface and make it 75% transparent on the places where the bit is set. Do you think that would have the same effect?
I'll try it, and let you know how it looks.
Edit: The pixel image is indeed 8 bits per pixel, but I doubt your "if the highest bit is set". What I currently do, is create a black surface the same size as the original surface; then I check every byte in the original surface to see if it & 0x80 is true (highest bit), if so I set the alpha value to 100% (opaque) else I set it to 0% (transparent). What I see is that there's a weird pattern of black on about the right spot, but not exactly the right spot as I see it on the op2 art viewer. This is, in more-or-less-pseudocode, the code I have:
SDL_Surface *surface = SDL_CreateRGBSurface( SDL_SWSURFACE | SDL_SRCALPHA, width, height, 32, 0x0000ff00, 0x00ff0000, 0xff000000, 0x000000ff );
for( int y = 0; y < height; ++y )
{
bmpFile_->seek( 0x436 + dataPtr + y * scanLineByteWidth );
QByteArray pixels = bmpFile_->read( scanLineByteWidth );
assert( pixels.size() == scanLineByteWidth );
for( int x = 0; x < width; ++x )
{
Uint32 pixel;
// if this pixel has shadow set...
if( pixels.at( x ) & 0x80 )
// set it as 25% opaque (currently 100% for testing)
pixel = SDL_MapRGBA( surface->format, 0x00, 0x00, 0x00, SDL_ALPHA_OPAQUE );
else
// set it as 0% opaque
pixel = SDL_MapRGBA( surface->format, 0x00, 0x00, 0x00, SDL_ALPHA_TRANSPARENT );
// and copy it to the surface
memcpy( (char*)surface->pixels + y * surface->pitch + x * 4, &pixel, 4 );
}
}
return surface;
The question is: Should I really be looking at 0x80? The patterns I'm seeing may even indicate I'm reading a wrong number of bytes every time, but I just read what the image info header tells me to read. Or maybe 0x80 is incidentally set around the positions I expect shadows. I am noticing the patterns are, namely, on the right places where I'd expect shadows for buildings, but they are all around the vehicles. Do you have any idea?
Err, the shadows are a monochrome 1 bit per pixel bitmap. However, the computer being only byte addressable means you have to process them in 8 bit blocks. You do this by checking the high bit, and shift up, rather than say, checking the low bit and shifting them down. That is, the direction on screen, from left to right, corresponds to the bits in a byte in a high to low order.
// Process the bits (pixels) in an 8 bit block
while (numBits > 0)
{
bSet = (byte & 0x80) != 0; // Check high bit
byte <<= 1; // Shift left
numBits--;
// Process the bit, as previously posted
}
Normally, the initialization before that loop would be:
byte = *sourceImage;
numBits = 8;
However, if it is the first in a scanline, and you want to avoid drawing stuff that will never be seen, you may need to adjust it. Say you want to start drawing xOffset pixels in.
byte = *sourceImage;
byte << xOffset;
numBits = 8 - xOffset;
Also remember that you'll need to clip that 8 near the end of the scanline so you don't overdraw. In the worst case, that could potentially lead to a segfault. Of course, if you're drawing to an enlarged back buffer where clipping is never an issue, at worst it should only cause a slowdown. Mind you, it keeps the code simpler if you don't have to clip.
I'll try to get it done once again. However, I did notice the width of an image was, for example, 80 pixels, and its scanLineByteWidth was also 80, ie 1 byte per pixel for every scanline. Is the scanLineByteWidth actually incorrect for these images, ie should it actually be ceil(width/8), or is there a lot of useless data after the real useful bytes?
Edit: I can't figure out the answer to my own question, at least. Every time I get something *close* to what I want to see, but never really the real deal. This is currently, in unoptimized code, what I do:
bmpFile_->seek( 0x436 + images[ imageId ].dataPtr );
int numBits = 0;
char byte;
for( int y = 0; y < images[ imageId ].height; ++y )
{
for( int x = 0; x < images[ imageId ].width; ++x )
{
if( numBits == 0 )
{
if( bmpFile_->read( &byte, 1 ) != 1 )
{
qWarning() << "Failed to read image data: " << bmpFile_->errorString();
SDL_FreeSurface( surface );
return 0;
}
numBits = 8;
}
bool shadowSet = byte & 0x80;
byte <<= 1;
numBits--;
// if this pixel has shadow set...
if( shadowSet )
{
[...]
}
}
}
Ie, I just completely ignore image scanline and just read the bytes I need. I have also tried reading the bytes for one scanline, and start with a fresh byte the next vertical line; and I've also tried just reading scanLineByteWidth bytes and using these, and assuming the rest of the bytes are useless. This (http://imagebin.ca/view/U0dOl2za.html) is what I currently see. The shadows just seem to appear at the right places for the buildings, but in a weird pattern that just indicates I read too few or too many bytes. Whatever I try, the image constantly looks something like that... Any ideas?
Edit: We talked about it on IRC; I just won't support shadows atm, oh well. Next up: Try to clone some interface stuff, and add support for selecting units and showing data on them in the menu...