Ordering Of The Tiles In An Outpost2 .map File

[dazjorz]

Okay, so I got this far: I can parse a .map file using the instructions at http://forum.outpostuniverse.net/index.php?showtopic=1250. I can check, for every tile, which image from the sets should be displayed. I can load that image an display it.
Only, the stuff is mangled. The map should be 128x128 if I remember correctly, but I didn't figure out the exact tile ordering yet. It doesn't seem to be by width, and not by height.

What I did find out is that the map seems to be "interleaved". I saw some pieces of a vulcano, which should display something like this:

Code: [Select]

______V________
_____VVV_______
____VVVVV______
_____VVV_______
______V________
_______________

but it displays like:

Code: [Select]

______V________ _____VVV_______
____VVVVV______ _____VVV_______
______V________ ________________

(if that makes any sense..)
I'll attach some screenshots if needed... Otherwise, is there anything known about this? Maybe even some code that ultimately decides in what order they should be displayed? I have one central place where a tile is fetched from the array, so I need the algorithm to translate the X and Y parameters to the exact location in the array.

Thanks!

[Hooman]

The tiles are stored in columns. Each column is 32 tiles wide. If you have an (x, y) pair, zero based, you can translate to the correct array offset by splitting the x bits.

xLow = x & 31;  // Grab only low 5 bits
xHigh = x >> 5;  // Get rid of low 5 bits
tile = tileData[xHigh][y][xLow];  // Or was this reversed order?


I believe this was done to make the game more page friendly. To display any section of the map (keeping in mind that the screen dimensions don't let you view more than 33x33 tiles at a time), only requires 4 page accesses to get all the tile data. If instead, it was stored as a regular 2D array, then for wide maps (which are quite common in Outpost 2), it would need to access a new page for each scanline. This means viewing an area of the map, or doing localized calculations, which Outpost 2 seems to have lots of, will make more efficient use of memory. This is particularly important on low memory systems, as having your working set exceed the amount of free memory will really kill performance.


Also, bitmaps are often stored upsidedown. So if the graphics still look funny, that might be the next step.

 

[dazjorz]

Yay!! I got it to work. I can correctly display a map now! Next up: Showing saved games, I guess... that will also contain displaying existing buildings and units, if possible. Once that's done, I'm getting close to being happy with the app I have now

Also, bitmaps are often stored upsidedown. So if the graphics still look funny, that might be the next step.

By the way: Yes, this was necessary. I wrote a little function that turns them around again

[dazjorz]

Okay. Now for saved maps loading support, this is weird. This is the documentation in the old topic:

0x0 0x19 "OUTPOST 2.00 SAVED GAME", 0x1A, 0 - String must match exactly or else load error
---------------------------
Note: The following is repeated as an array of 0x74 (116 decimal) elements
0x19+i*0x1E0 4 **TODO** Figure this out
0x1D+i*0x1E0 0x1DC **TODO** Figure this out
---------------------------
Note: From here, the format matches that of .map files

This is what hexdump says about the first part of the file:

Code: [Select]

00000000  4f 55 54 50 4f 53 54 20  32 2e 30 30 20 53 41 56  |OUTPOST 2.00 SAV|
00000010  45 44 20 47 41 4d 45 1a  00 94 c9 21 8c f8 75 1c  |ED GAME....!..u.|
00000020  4a 00 00 00 00 62 6c 61  61 74 73 63 68 61 61 70  |J....blaatschaap|
00000030  20 6e 61 61 6d 20 76 61  6e 20 73 61 76 65 67 61  | naam van savega|
00000040  6d 65 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |me..............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000070  00 00 00 00 00 94 c9 83  b8 15 8d a0 c8 15 8d a0  |................|
00000080  c8 15 8d a0 c8 15 8d a0  c8 15 8d a0 c8 15 8d a0  |................|
*

("blaatschaap naam van savegame" is some foo long name I came up with in a split second)
- so I can see the "OUTPOST 2.00 SAVED GAME\x1a\x00" but then there's 8 bytes, 4 zero bytes, then 64 bytes containing the name of the savegame and the rest null... and then semi-random data. Those 8 bytes at the top could mean, in numbers:

Code: [Select]

94 c9 21 8c = 148 201 33 140 / 51604 35873 / 2351024532
f8 75 1c 4a = 248 117 28 74  / 30200 18972 / 1243379192

Now I checked, at least I tried to check, the length of the data behind that header. It was nothing near 51604, 35873, 30200 or 18972, but the data seemed to switch to something else after 0x10880 (= 67712 bytes). The bytes themselves seem to be mostly repeated all the time except for the first three bytes, 94 c9 83, which could be 148 201 131 / 51604 / 33737 / 8636820 - numbers which also make few sense, except that the first two are the same as the two bytes right after "OUTPOST 2.00 SAVED GAME\x1a\x00"... if these are check bytes, that leaves 131 as a mysterious number....
The file size is 467026 bytes, I've found the location where the actual .map format part starts, that's at 0x1e025 = 122917 bytes. 467026-122917=344109 which may come close to 35873 or 33737 or so, but now I'm just doing random guesses.

Is anything other than this already known about this, Hooman (or anybody else)?
As you've probably understood by now, I'm trying to recreate some of Outpost 2's code, in order to have something as a community to build upon and improve. In the spirit of OpenTTD, a remake of Transport Tycoon Deluxe, I've named my project OpenOutpost2. Once I'm happy with what I have, it will be freely available under the GNU General Public License - i.e. freely modifiable by anyone as long as they put a patch available... (and some other stuff, read the license if you want to know, it's not that big)

So I have two concrete questions:
1) Could you clear up what's on that topic? So there's an array of 116 times 480 bytes, is there nothing known about the values of that? Also, so the array is 55680 bytes; could you maybe even send me the save file you have done your testing on to see if that 116 is named somewhere? That could be the mysterious 131 number; if mine is 131 times 480 bytes, that leaves 62880 bytes, which comes close to the 67712 number already there. Any ideas?

2) Is there maybe even an existing implementation to read the save files? I found this old topic: http://forum.outpostuniverse.net/index.php?showtopic=882 - is BlackBox still active and around? Does he have any idea how to go about and start parsing this?

Thanks!

[Hooman]

Checking OllyDbg with my comments file loaded, the saved game files are processed at:
0048A3E0 Function: TethysGame.LoadGame(StreamIO* savedGameFile)

Here is an overview of the saved game loading code:
0) Verify string "OUTPOST 2.00 SAVED GAME", 0x1A, 0
1) TFileDialog.LoadAndDiscard([ECX] StreamIO* savedGameFile)
2) Sheet.Load(StreamIO* savedGameFile)
3) Map.Load(StreamIO* savedGameFile, bSavedGame = true)
4) MessageLog.Load(StreamIO* savedGameFile)
5) Load TethysGame member variables
6) LevelDll.Load(StreamIO* savedGameFile)
7) For i in 0..6: Player.Load(StreamIO* savedGameFile)
Research.Load(StreamIO* savedGameFile)
9) DayNight?.Load(StreamIO* savedGameFile)
10) UnitHotKeyGroups.Load(StreamIO* savedGameFile)
11) IWnd:Pane:Detail.Load(StreamIO* savedGameFile)
12) Load Random Number Generator seed (2 dwords, seperate error checking)
13) Lava.Load(StreamIO* savedGameFile)
14) Blight.Load(StreamIO* savedGameFile)


It checks for stream read errors between each of the numbered steps above, and also twice in step 12, after reading each of the 2 dwords. It then finishes initializing the game using data already loaded into memory.

Note that step 3 is just a call to the map loading code used when starting a new game. Hence the notes about how the map and saved game formats are more or less the same.


In step 1, the function call used when loading the game will read exactly 0x1086C (67692) bytes, and then promptly discard them and returning. This might be the image for the load game screen.

In step 2, it basically loads a binary format of what's in sheets.vol. It loops for each unit type, and loads it's sheet data. Note that it uses virtual function dispatch here, and the amount of data loaded may change between unit types. The units are from 0..72 (inclusive).

In step 3, note that it passes a second parameter which is set to true when loading a saved game, and false when loading a new map. This flag controls loading of the unit data, which is mentioned in the middle of the .map format. That data contains the unit records (each is 120 bytes long), as well as booking information on the number of units, which unit records are free, and possibly some waypoint information too.

In step 4, it loads three fields, each of 4 bytes, and then goes into a loop reading 0x4C (76) byte chunks. The number of loop iterations is from the third dword read.

In step 5, it reads 0x484 (1156) bytes into the TethysGame object. Note that the function analysed above is a member of that object. You can check the TethysGame.h file in the ForcedExports project for details on their meaning. You can also then go on to suggest a better name for that project, as I've never much liked the name.

You should also be able to find definitions for most of the remaining objects in the ForcedExports project. Some of it may be a little sketchy though.


In step 9, the DayNight object loads six fields of 4 bytes each.

In step 10, the UnitHotKeyGroups loads 10 groups worth of data. (Endpoint non-inclusive intervals used here):
For i in 0...10:
    numUnits = StreamIO.Read(4)  // 4 bytes
    For j in 0...numUnits:
        unitIndex[j] = StreamIO.Read(2)  // 2 bytes
    Next j
Next i

In step 11, the IWnd:Pane:Detail pane loads it's data. Information on this class can be found in the user interface section of the ForcedExports project.

The Lava and Blight objects share the same loading code. They are both derived from a common parent class. They load four fields, each of 4 bytes. These fields are probably used as random number seeds (at least 8 bytes of it anyways).


Standard Disclaimer: I'm building these notes on the fly by reading off some assembly code. Don't be too surprised by any errors.

[dazjorz]

Thanks so much! That was more detailed than what I expected I'll let you know if I can get it all to load.

[dazjorz]

Okay, I wrote code to load all data, but I think a lot will still be incorrect for now, because I didn't have the original code for the Load methods.
For example, the Research class is pretty much a mystery for me:

Code: [Select]

  class Research
  {
  public:
    // Member variables
    int numTechs;
    TechInfo **techInfo;    // TechInfo*[]*
    int maxTechID;        // 12999  [maxTechLevel * 1000 + 999]
    // ...
    // 0x2C
  };

I can't just read TechInfo** from file, and the class is supposedly 0x30 bytes in length (assuming 0x2c is the starting code of the last dword); I don't know how to load that from file, I don't know how long the piece is I have to load, etc. Could you clear up on the length of the research data, and maybe even how to load it? I wrote a lot of code but haven't tested a bit of it yet; but anything will surely fail if I don't know the real size of the Research class.

Another question is regarding the IWnd data - of course, I don't use IWnd. Is there anything important in there? If so, I'll need to take the data I need, and call some Qt functions to get it out the same. But if there isn't anything important, I'll just skip it like I do now, no problem.

In the same way I'm also ignoring the LevelDLL data, since I don't have any DLL's to use them with. Once most of the project is done, we can go reconstruct that data (or we don't, the clone doesn't have to have the same campaigns etc). Also, is there anything important in the LevelDLL data when the game is multiplayer (i.e. has no prebuilt script)?

Edit: It seems to be humming along OKishly. However, it fails even before the research data. It is at position 158791 in a 298601 byte large savegame; the location of 1023 * 120 bytes of unit data. My program currently skips that and then it's at position 281551 (so 17050 bytes left in the file for the rest of the data); it reads the map tag which is correct, and then it goes on loading the message log. After that, the program reads 0x484 bytes of TethysGame data, it skips the LevelDLL data, and then it reads Player objects - but these classes are 3120 bytes of data, so after having read five players from the file, it bumps out at the sixth because there's no more data left. (this is a single-player game with just one player, or maybe there's a CPU player without any units.)
Two (additional) questions:
1) I can't disassemble at all, could you check if TethysGame should really load 0x484 bytes, and how many bytes Player::Load(StreamIO* savedGameFile) reads from the file? I think the fix will be in there.
2) Is it true that Outpost 2 games can have 7 players exactly? Seems like a weird number to me...

Thanks!

Edit 2 - little status report:
edit 3 beh, no hotlinking, image.
If I hack the saved game file reader to stop right after the map load, this is what we get: the map displays correctly, only unit and building locations aren't known at all yet. The red dot means the mouse was there, the program supports map scrolling already and that's a little leftover from an earlier milestone, I'm leaving it in until it becomes annoying

edit 4: (removed a piece of this comment - I thought something was wrong but it turned out to be completely correct)

[Hooman]

Image gives 403 Forbidden error.

Research Data:
numBytes  Description
-----------   -------------
4  numStoredTechs
4  maxTechId
32  techFileName
  -> Research.Load(techFileName) -> int numTechs
  -> Check that numTechs - numStoredTechs == 1 (tech #0 isn't stored to file, I believe it was a dummy entry)
2*numStoredTechs short[numStoredTechs] playerHasTechBitVector

I'm quite sure TethysGame loads 0x484 (hex) bytes. It also unconditionally loads 7 players worth of player data. (The last player is Gaia basically, so 6 usable players). Note that the in memory format is 0xC24 bytes, but it may very well be more tightly packed when written to a file.

Player Data:
4  tagWord (0xE3D4C5B4)
0x4C8  data (first 4 bytes? => num?)
0xE*num?  moreData (offset 4 => num?2)
num?2  yetMoreData
4  noIdea
4  stillNoIdea
4  iDontKnow
4  tagWord (0xEDA34CEF)

Gee, hope that helps.  


I don't remember anything particularly important in IWnd off the top of my head. (Maybe check the headers?). LevelDll I believe contains stuff like triggers, which are of some importance to game play.


Edit: Ok, a little more seriously now.
Player Data:
4  tagWord  (0xE3D4C5B4)
0x4C8  PlayerData  (first 0x4C8 bytes, as from the forced export Player header)
16 times:
  0xE  commandPacket.header  (see forced export CommandPacket header)
  header.dataLength  commandPacket.data[]
4  buildingUnitListHead  [unitIndex in file, Unit* in memory]
4  vehicleUnitListHead  [unitIndex in file, Unit* in memory]
4  *unknown*UnitListHead  [unitIndex in file, Unit* in memory]
4  tagWord (0xEDA34CEF)
 

[dazjorz]

Okay, looks like the information on LevelDLL I have isn't enough too...

The class is 92 bytes big when packed; however, I see a *lot* more data in the hexdump:

Code: [Select]

00045100  00 00 07 00 00 00 00 00  00 00 00 00 00 00 08 00  |................|
00045110  00 00 00 00 00 00 09 00  00 00 0a 00 00 00 00 00  |................|
00045120  00 00 0b 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00045130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00045160  00 00 01 00 00 00 40 00  00 00 42 00 00 00 44 00  |......@...B...D.|
00045170  00 00 48 00 00 00 4a 00  00 00 46 00 00 00 ff ff  |..H...J...F.....|
00045180  ff ff ff ff ff ff 00 00  00 00 00 00 00 00 01 00  |................|
00045190  00 00 00 00 00 00 02 00  00 00 00 00 00 00 03 00  |................|
000451a0  00 00 00 00 00 00 04 00  00 00 00 00 00 00 1d 00  |................|
000451b0  00 00 0f 00 00 00 41 74  74 61 63 6b 65 64 54 72  |......AttackedTr|
000451c0  69 67 67 65 72 0a 00 00  00 42 75 69 6c 64 47 72  |igger....BuildGr|
000451d0  6f 75 70 14 00 00 00 42  75 69 6c 64 69 6e 67 43  |oup....BuildingC|

With a class 92 bytes big, 4510b would be the place to stop loading LevelDLL data; however it seems clear that that's not the right place, because a hundred bytes later, there are a lot of triggers, conditions, things that look like mission objectives and responses to triggers etc. To me, the place where player data starts seems only to be at around 46590 or so. Could you tell me what LevelDLL::Load() does? I think, with that info, I'll be able to finish this saved game loader.

Thanks!

Oh, and the image:

(I currently throw away most data I read, after checking it a little and printing some statements about it. However, it should be possible to get some information in the sidebars to the right. The reason I'm still splitting between SDL and Qt is because both have their pros and cons - eventually I'll probably revert to using just one of the two, but I haven't decided which one yet; Qt has QPainter and Phonon, so it's probably going to be Qt...)

[Hooman]

Nice.


Ok, here's a bit of pseudo code. It starts getting a little vague, but hopefully it will help parse over at least most of the data. If not, scan for the sentinel values (tag words).  

Code: [Select]

LevelDll.Load(StreamIO* file)
{
  if (file.Read(4) != 0xAEDE0034) return false; // Error
  scriptNameLength = file.Read(4);
  scriptName = file.read(scriptNameLength);
  LoadScript(scriptName); // Load the level DLL
  // Note: Loading the script sets LevelDll fields, like the SaveRegion (offset 0x10)
  if (saveRegion.buffer* != Null)
  {
    bufferSize = file.Read(4);
    if (bufferSize != saveRegion.bufferSize) return false; // Error
    saveRegion.buffer* = file.Read(bufferSize); // Note: bufferSize = 0 is valid here
  }
  defaultGroup[] = file.Read(0x30); // See offset 0x1C in LevelDll
  ???.Load(file);
  if (file.Read(4) != 0xAEDE0034) return false; // Error
  return true; // Success
}

???.Load(StreamIO* file)
{
  numScStubCreators = file.Read(4);
  if (numScStubCreators != this?.numScStubCreators) return false; // Error
  for (i = 0; i < numScStubCreators; i++)
  {
    num? = file.Read(4);
    file.Read(num?);
  }
  ? = file.Read(4);
  do
  {
    ?? = file.Read(4);
    if (?? == 0xFF) return 1;  // Success
    file.Read(4);
    ?.Load(file); // Virtual function dispatch (could very well load a variable amount of data, depending on the exact object type)
  } while (?)
}

[dazjorz]

Okay, I wrote some code to read the filename, and then some code which searches for that end tag - and surprisingly, suddenly when I finished that and re-ran the program, it worked! It happily told me it loaded the map. There were still some quirks in the source, it didn't check the player tags so that was still quite a little off, but I think I'm done now...

...except for one small very unimportant thing, but it could make this just a little more perfect. The loader currently stucks at reading dword 1 of the random number generator, just 10*4 bytes from the end of the file (from the end: blight, lava, dword2, dword1). However, at that point, the loader reached EOF. In other words, something before that reads too much data. I can't modify my saved games right now because of a hard drive crash (so I can't run outpost2 atm, will need to wait a week for that); but I think it's the IWnd:Pane:Detail.Load() function. I wrote a piece of code which skips exactly 0x150 bytes, which is the size listed for that object in the ForcedExports file; when I count back using the size of the file, it works fine when I skip exactly 0x50 bytes. Does this match with the load() routine, or is something in that data variable size too?

When this is done, I'm going to try to get it to parse building and unit details, and see if I can get some images on the screen. I see you have succeeded to extract images of the buildings already; ie http://www.outpost2.net/images/ConsumerFactory.gif as on the front page. Do you already know how to get those pictures? (I guess they are in a special format to get player colors to work, etc)

Anyway, thanks so much for your help. I couldn't ever have got this far without it

[Hooman]

Hmm, it would be a bad assumption to assume that the in-memory-size of the objects in the forced exports project is the same as the on-disk-size. Particularly since the in memory formats use a lot of pointers, which are useless if stored to a file. Most of that memory is dynamically allocated, so there is pretty much no way they would be valid pointers after loading anything. There is pretty much always some conversion, and things usually end up shrinking. When I wrote object.Load(StreamIO* file), I definately did not mean something simple like ReadFile(&object, sizeof(object)). That would be completely wrong in the majority of cases.

Keep in mind that the forced exports project was meant as a way of interfacing with the game while it was running, not as a library for processing files off the disk. It has been used mainly for extending the game's user interface, or to make levels that do non standard things that can't be accomplished with the standard level interface.

However, there is obviously going to be some correlation between the memory format and the disk format. Hence why I referenced that project. That and I'd somewhat forgotten just how different some of the formats are. Usually there is some degree of ReadFile(&struct, sizeof(struct)) done within objects, but almost never to objects as a whole. Basically, if there is a pointer somewhere, then it can't be read from disk, so there will be some format conversion around that bit of data. Sometimes field widths will also change. Usually though, the order things are stored in memory matches up with the order it appears on disk.



To read the images, you'll need to parse OP2_ART.prt and OP2_ART.bmp. The actual bitmap data is in the .bmp file, which is for the most part just some really huge Windows bitmap file (that Paint can't load). All the indexing information to find out what blocks of bytes correspond to what frame of what what animation are found in the .prt file, along with palette information (excluding player colors). I believe there was some VB source posted that parsed the prt and bmp files and displayed things in them.
 

[dazjorz]

For reference: The thread with the information is http://forum.outpost2.net/index.php?showtopic=1469. There is also quite some information at http://forum.outpostuniverse.net/index.php?showtopic=1386.

I didn't find any Visual Basic code nor any other language, yet. I'm going to try to throw some of my own code at it, to see if I can paint anything useful. If there's anything I'm missing, please drop me a link

[dazjorz]

Okay, little update: I wrote some code to read the .prt format using code I found on the boards (see previous reply). However, it fails somewhere in between. For now, I got the Outpost 2 art viewer from Arklon on IRC, I'll use that to extract some stub images and paint them to get at least a viewable game field. That'll probably take long enough, so I'll drop another note here when I'm done with that.

I'm planning to release an alpha version once I get everything painted and shown correctly. Then, I may start trying to continue an existing saved game, feature by feature. I'm not sure how that will end up, but at least there's some hope it will end up in a complete clone, something like OpenTTD.

[Hooman]

VB Art Viewer Project


I think you've found most of the other important sources of info on the topic.
 

[Leviathan]

Nice work dazjorz

Oh and OpenTTD is win! And a great example of a very successful open source project.

[dazjorz]

A day later. The game currently just loads an image file for every unit/building; if it can't find the image, it simply substitutes the unknown.bmp image you see used a few times, for the buildings. For the visible units, the game currently just loads a bmp file. As you can see, this doesn't really come out well... Since the prt/bmp file is *much much* more detailed than a collection of bmp files will be unless it becomes a *huge* collection, I'll be hacking in some code from your Art Viewer into my OP2LegacyImages source file. Hopefully, I can then also get the transparency and palette stuff right - currently, that's done by SDL and obviously it doesn't really come out well.

(Also I think the units may be one place too much to the right. Little bug. )

I'm looking forward to putting this code publicly visible once I'm happy with it

edit: Oh hai leviathan Yeah, OpenTTD is really cool, I'm hoping to come even a little bit close to that with this project...

[dazjorz]

and we're a step further again! And quite a step, this time. I fixed some more bugs in the prt file reader, and Hooman found a bug I couldn't find for a day; and the clone now contains a complete Outpost2 art reader!

To show off, I made these screenshots:





Though, Hooman, I have one question: Currently, I hardcode all the vehicle image ID's. As you can see, I don't check rotation or team or whatever at all yet. So what I wanted to know: Is anything on this stored somewhere, or does Outpost2 really hardcode this and map all unit ID's with their rotation and eden/plymouth to a specific image ID?

Also: All images seem to be by default blue. Do you have a clue as to what palette data to change to get a different team color? That would be amazing

Thanks a lot again!

[Hooman]

I was just thinking as I looked at the images, that'll you'll be needing the link to: http://forum.outpost2.net/index.php?showto...indpost&p=64758.

It should be good for the basics of units and buildings, but is sadly incomplete. The movement images, and many building images were extracted. However, much of the remaining information is hardcoded into the game, and there is no automated way to extract all of it. Things like cargo trucks dumping ore is special cased.


The unit rotation can be read from the unit records at offset 0x1C. It is a single byte, from 0-255, where 0 is facing right/east, and it increases counterclockwise. According to my notes, this is backwards from the order the tables are indexed. However, the data in the tables seems to be backwards again. So I guess, find the animation index for the first image in the table, and just divide the 0-255 range to match the smaller range of animation index values, and you'll probably be good.


Hmm, interesting, I'm noticing the bulldozer has doubled up values. That would seem to imply it has half as many graphics for the rotation angle. I've never really noticed that before. Yet, it feels somehow obvious now that I see it.


Edit: Forgot some info.
To determine whether you need to display an Eden or Plymouth graphic for the unit, you probably need to check the flags. There is a 4 byte flags value at offset 0x44 in the unit record. Within that value, the mask 0x1000 gives the bIsEden bit.


The player color palette is in color.bmp. There are 24 entries per player. I don't remember exactly how the colors got remapped, but it might have been something simple like re-mapping the first 24 colors of the palette using the 24 value blocks from color.bmp, indexed by player color. The player's color number is the 4 byte value at offset 0x30 in the memory image of the player object. I don't know how closely that corresponds to the file format.

[dazjorz]

I was just thinking as I looked at the images, that'll you'll be needing the link to: http://forum.outpost2.net/index.php?showto...indpost&p=64758.

It should be good for the basics of units and buildings, but is sadly incomplete. The movement images, and many building images were extracted. However, much of the remaining information is hardcoded into the game, and there is no automated way to extract all of it. Things like cargo trucks dumping ore is special cased.

I have no idea how to read that... When I try to find the image ID's there in Cynex's art viewer, I can't find anything. For example:

Code: [Select]

AnimationList: 25 STORAGE_ORE
 Eden:
  0: 69 # ?
  1: 59 # actually 44
  2: 60 # actually 45
  3: 193 # ?
  4: 1109 # ?
  5: 2059 # actually 43

I figured these numbers would be image ID's, but apparently, they aren't... How do I use these?

Edit: Oh s***. I just created a full unit/imageid mapping table, but it turns out I've been an idiot. I should use the frames, not the image ID's. I get it now... This is actually a day of mapping all image ID's gone, but luckily, it's not *completely* gone; I just have to adapt a lot of my code to get it right.....

Edit 2: Even more s***: It's not the frames, it's the *animations* which are important! Okay, I know how to do this now. Damn, took me a while But I'll get it right.

Edit 3: I'm close again. However, I have one question: Cynex' art viewer seemed to know pretty well the width and height of any frame animation. However, I don't think I see that width and height anywhere. Does his art viewer scan through all images of all frames, take their width and height and position, and then calculate the end width and height of the complete frame? Or is there an easier way to simply get the width and height? (Also, are width and height for any animation static, or does it depend on the frame? )
Edit 4: And shadows? In these 1 bpp images, when zero the RGBA value is probably {0,0,0,255} (fully transparent); when one what is the RGBA value? {0,0,0,128} or so? I see these shadow images have their palette set to two, yet if I try to simply use them as-is with bpp 1 and palette 2 I get some blue-ish crap under my vehicles, instead of the shadow, so I figure I have to write a palette for it myself.

[Hooman]

I'll try to look into these on the weekend. (I might need a reminder).

I have a feeling the width/height and shadow questions might be a bit involved, so I'll want to take a look back at the code. I know I've gone over both at one point or another.

It might be that each frame has a width/height, and some sort of an offset. Using that offset might be what keeps all the frames lined up, even with different sizes. I suppose that might mean that animations as a whole don't even really have a width/height to speak of. Not that I can really remember though.

I think the shadows use some sort of bit trickery.
 

[dazjorz]

It might be that each frame has a width/height, and some sort of an offset. Using that offset might be what keeps all the frames lined up, even with different sizes. I suppose that might mean that animations as a whole don't even really have a width/height to speak of. Not that I can really remember though.

I got it! Cynex' art viewer helped me a lot in retrieving and using the right values. Little summary: Every image has a width and height, and every component has an offset and an image. Therefore, you can retrieve the exact position of the component in the end image (from offset to offset + size); and by computing the maxima and minima of those values, you know how large the image is.

Next up: The shadows.

[Hooman]

The shadows are processed high bit first. If the bit is set, it does a double fade on the destination bitmap, adding the intermediates, to get 3/4 of the original intensity for each color component.

Code: [Select]

if (bSet)
{
  // Process 16 bit color words.
  // Halve each color, by shifting, with a mask to prevent low order bits
  // from being shifted into the higher order bits of the next color component.
  // (The mask clears the high bit of each color component, which should be 0
  // after the "divide by 2", at least in the absence of bit shift carries).
  halfColor = (*(short*)destImage >> 1) & 0x3DEF3DEF;
  quaterColor = (halfColor >> 1) & 0x3DEF3DEF;
  // Combine into new image color
  newColor = halfColor + quarterColor;  // "color*(3/4)"
  // Write sum to image
  *(short*)destImage = newColor;
}

Note there is a 32 bit mask value even though only 16 bits are used. Perhaps they were thinking of processing 32 bits at a time (2 color words). This however complicates the case when you have an odd number of pixels to process in a scanline.


As for processing the entire scanline or image, you may need to do some shifting of bits to account for clipping. That is, if you want to draw the shadow, starting 1 pixel into the shadow (such as when it overlaps the top left corner of the screen, and isn't completely visible), you would need to shift the first byte of each scanline by 1 bit before processing it, and going to the next byte earlier. Remaining bytes should be processed normally.

 

[dazjorz]

Note there is a 32 bit mask value even though only 16 bits are used. Perhaps they were thinking of processing 32 bits at a time (2 color words). This however complicates the case when you have an odd number of pixels to process in a scanline.

So, uh, if I understand correctly, in an 8 bits-per-pixel image, there are actually 32 bits per pixel? :/

And then, of those 32 bits per pixel, only 16 are used...?

see edit below

As for processing the entire scanline or image, you may need to do some shifting of bits to account for clipping. That is, if you want to draw the shadow, starting 1 pixel into the shadow (such as when it overlaps the top left corner of the screen, and isn't completely visible), you would need to shift the first byte of each scanline by 1 bit before processing it, and going to the next byte earlier. Remaining bytes should be processed normally.

With clipping, do you mean edges of the screen to what is and what isn't visible? I currently calculate all tiles of which a little bit is visible on-screen; those tiles I retrieve and paint onto a temporary surface, and then I blit the right piece from the temporary surface on to the screen. I don't really have to do any clipping like that and the pixels that are uselessly painted onto the temporary surface don't make it that much of a slowdown (but I'll try to benchmark that).

Code: [Select]

if (bSet)
{
  // Process 16 bit color words.
  // Halve each color, by shifting, with a mask to prevent low order bits
  // from being shifted into the higher order bits of the next color component.
  // (The mask clears the high bit of each color component, which should be 0
  // after the "divide by 2", at least in the absence of bit shift carries).
  halfColor = (*(short*)destImage >> 1) & 0x3DEF3DEF;
  quaterColor = (halfColor >> 1) & 0x3DEF3DEF;
  // Combine into new image color
  newColor = halfColor + quarterColor;  // "color*(3/4)"
  // Write sum to image
  *(short*)destImage = newColor;
}

This is complicated for me, since I actually work with surfaces everywhere and can't just change the colors of the parent surface in another call. However, I *can* create a plain black surface and make it 75% transparent on the places where the bit is set. Do you think that would have the same effect?
I'll try it, and let you know how it looks.

Edit: The pixel image is indeed 8 bits per pixel, but I doubt your "if the highest bit is set". What I currently do, is create a black surface the same size as the original surface; then I check every byte in the original surface to see if it & 0x80 is true (highest bit), if so I set the alpha value to 100% (opaque) else I set it to 0% (transparent). What I see is that there's a weird pattern of black on about the right spot, but not exactly the right spot as I see it on the op2 art viewer. This is, in more-or-less-pseudocode, the code I have:

Code: [Select]

 SDL_Surface *surface = SDL_CreateRGBSurface( SDL_SWSURFACE | SDL_SRCALPHA, width, height, 32, 0x0000ff00, 0x00ff0000, 0xff000000, 0x000000ff );

  for( int y = 0; y < height; ++y )
  {
    bmpFile_->seek( 0x436 + dataPtr + y * scanLineByteWidth );

    QByteArray pixels = bmpFile_->read( scanLineByteWidth );
    assert( pixels.size() == scanLineByteWidth );

    for( int x = 0; x < width; ++x )
    {
      Uint32 pixel;                               

      // if this pixel has shadow set...
      if( pixels.at( x ) & 0x80 )
        // set it as 25% opaque (currently 100% for testing)
        pixel = SDL_MapRGBA( surface->format, 0x00, 0x00, 0x00, SDL_ALPHA_OPAQUE );
      else
        // set it as 0% opaque
        pixel = SDL_MapRGBA( surface->format, 0x00, 0x00, 0x00, SDL_ALPHA_TRANSPARENT );

      // and copy it to the surface
      memcpy( (char*)surface->pixels + y * surface->pitch + x * 4, &pixel, 4 );
    }
  }

  return surface;

The question is: Should I really be looking at 0x80? The patterns I'm seeing may even indicate I'm reading a wrong number of bytes every time, but I just read what the image info header tells me to read. Or maybe 0x80 is incidentally set around the positions I expect shadows. I am noticing the patterns are, namely, on the right places where I'd expect shadows for buildings, but they are all around the vehicles. Do you have any idea?

[Hooman]

Err, the shadows are a monochrome 1 bit per pixel bitmap. However, the computer being only byte addressable means you have to process them in 8 bit blocks. You do this by checking the high bit, and shift up, rather than say, checking the low bit and shifting them down. That is, the direction on screen, from left to right, corresponds to the bits in a byte in a high to low order.

Code: [Select]

// Process the bits (pixels) in an 8 bit block
while (numBits > 0)
{
  bSet = (byte & 0x80) != 0;  // Check high bit
  byte <<= 1;  // Shift left
  numBits--;

  // Process the bit, as previously posted
}

Normally, the initialization before that loop would be:

Code: [Select]

byte = *sourceImage;
numBits = 8;

However, if it is the first in a scanline, and you want to avoid drawing stuff that will never be seen, you may need to adjust it. Say you want to start drawing xOffset pixels in.

Code: [Select]

byte = *sourceImage;
byte << xOffset;
numBits = 8 - xOffset;

Also remember that you'll need to clip that 8 near the end of the scanline so you don't overdraw. In the worst case, that could potentially lead to a segfault. Of course, if you're drawing to an enlarged back buffer where clipping is never an issue, at worst it should only cause a slowdown. Mind you, it keeps the code simpler if you don't have to clip.