Cross Platform Outpost 2 Utility Library

[Vagabond]

Hey everyone,

Currently there is a C++ Utility Library for Outpost 2 that lives on GitHub here https://github.com/OutpostUniverse/OP2Utility. It is a library to assit in Outpost 2 related tasks such as opening and saving map files, save files, clm files, and volume (vol) files. OP2Utility is also the backend for OP2Archive and OP2MapImager. Someday, I wouldn't mind seeing it used in a future map editor to facilitate Outpost 2 specific tasks.

There is a fairly robust Stream library that could easily be lifted for use in non Outpost 2 projects, which will likely continue to be improved on in the future.

OP2Utility currently only compiles for Windows x86.

Hooman and I have been putting a lot of time into improving the library and removing the Windows specific code. A couple of open branches will make all the size_t calls Clang compliant once merged. We are probably only a few lines of code from removing the Windows.h header altogether, which means it should compile on Linux and Windows very soon. Nothing should keep the library from compiling on Macintosh if anyone is interested?

Once it is compiling on both systems, I'd like to look at removing the Visual Studio (Windows) and Make (Linux) specific project files. Instead we can use cMake to build the platform specific project files. This would have the benefit of standardizing compiler settings and allowing updating new source code files in only one place, cMake.

When someone wants to compile the code, cMake generates the proper platform specific tooling. This means someone who uses Linux doesn't have to worry about how to add new files and settings to a Visual Studio project/solution that they cannot check on Linux and the same applies to the Windows users figuring out the Make file.

The downsides are you have to download and install cMake on your computer and learn the commands for generating the platform specific tooling. It also means if you want to make major edits to the project structure, you would have to dig into cMake enough to understand how to make those changes.

The end goal will be compiling OP2Archive and OP2MapImager for both Windows and Linux. I'd like to look at compiling them both for x86 and x64 operating systems. And finally keep a small symbols library for both (.PDB files on Windows, and I think usually .debug files for Linux) that match up with each release cycle. So you could drop in the symbols for full debugging no matter which operating system you are using.

This means both OP2Archive and OP2MapImager will have to switch to cMake for their build tool generation to hook into OP2Utility. The good news is both of these projects should already be almost fully Linux compatible already.

I've been happy working on this project, as it is a great opportunity to continue learning and practicing C++ in a project. I recognize there are probably limited gains to the community (unless you are a Linux fan!).

Anyways, I don't know how others feel about how this is all moving forward. We are happy to have help from others if anyone else is interested in checking it out. Open to suggestions as well if people love/hate cMake or are generally ambivalent to this entire discussion. I'd like to discuss a standardized file structure for the projects in the near future as well.

-Brett

[leeor_net]

You and Hooman have done absolutely amazing work! I've been following development on GitHub and an pleased to see how this progressed.

When I eventually pick up on my map editor I'll pull the NAS2D filesystem and use your stream library instead. I think it's quite well built and I look forward to being able to drop the dependency (though having ZIP and 7z support is still attractive).

Anyway, this should make it a lot easier to build high quality, robust tools.

Again, awesome work!

[Hooman]

Thank you Leeor. And a very big thank you to Brett for all the work he's put into it.

As of today, the project now compiles on Linux.

There might still be a few rough edges, as some of the changes still need to settle into the master branch, and a few warnings may need to be addressed.

I'm particularly excited about the Streams portion of it, as I've been thinking about that part for quite some time. Though I admit there is so much more that can be done there. There may be some API changes coming to address a few shortcomings recently encountered, but likely mostly in the form of additions or restrictions on bad usage to prevent accidental bugs. I doubt changes will break existing good code from a user perspective. No promises yet for people implementing derived classes though.

Using CMake could be cool. We've talked about using that here for a while now. This just might be the project that forces us to do it. I'm still a little apprehensive about having to learn it, though now is probably a good time.

Overall, this project has been a lot of fun. It's also been a great learning opportunity. Both about tooling, such as using GitHub, and about the additions from the newer C++ standards. The C++ language has really changed in recent years, and it's received a lot of much needed improvement.

[Vagabond]

I wanted to post an update on the progress OP2Utility, OP2Archive, and OP2MapImager have made. We have both OP2Archive and OP2MapImager compiling and running on Linux now.

Also, we have successfully compiled OP2Archive for x64. It appears capable of creating and extracting from Clm and VOL archive files equally whether compiled in x86 or x64. This was very encouraging. Although there were quite a few compiler warnings, especially on MSVC that should be addressed.

Most of these warnings deal with casting integer values that are represented differently based on which architecture they are compiled for. Addressing these warnings will mostly protect from edge case scenarios such as attempting to pack extremely large numbers of files into the archive. In the process, other minor improvements are being made to the code base in general.

These changes are mostly complete for OP2Utility. Next will be making the changes in OP2Archive. After that, I would like to attempt compiling OP2MapImager in X64 and see how it goes.

---

The stream library continues to improve. It has (or will will soon have):

 * expanded helper functions allowing auto reading/writing of vectors and strings
 * Allows StreamWriter to consume the contents of a StreamReader automatically
 * The ability to slice existing MemoryStreams and FileStreams into an independent substreams.

We are looking at adding options to FileStream on construction that allow for things like truncating existing files, not overwriting existing files, etc.

---

OP2Archive and OP2Utility now contain CMake project generation files. CMake can be used to quickly auto-generate valid project files for both MSVC and Linux. Adapting it for use on Macintosh may not be difficult, although no one has explored the topic. The original Visual Studio solution and Linux makefile still exist in the repositories, so using CMake to generate the project files is currently optional.

In general, I have been disappointed with the CMake results. The resulting Visual Studio solution works, but does not really proximate what I come to expect as standard in a Visual Studio project that would be created without CMake.

The CMake documentation is very dry, no outstanding books on the subject seem to exist, and web tutorials are a mixed bag of old or not recommended techniques.

We are approaching a crossroads where we need to decide what to do with CMake. If we want to continue supporting it, we need to sink a lot more time and effort into it and probably delete the other project files so to force its use. Currently I just use the built in Visual Studio project files since they are easier to use and do not contain all the CMake added oddities. Deleting them would force everyone to generate a CMake project before coding instead of relying on the existing makefile and Visual Studio project.

At this point, I would be okay deleting the CMake projects from the repository and just using a native makefile and Visual Studio solution. It would allow others to use the projects without having to download and use CMake for project generation.

I would be curious to hear how others feel about CMake being present or being removed from the projects. Both from the standpoint of people who have experience with it and from people who do not have experience but would be forced to use it.

-Brett

[Hooman]

Indeed, the library now works for both Windows and Linux, and compiles for both 32-bit and 64-bit.

Initial attempts at a 64-bit build on Windows produced a number of compiler warnings. Recent work has cleaned that up, and looks ready to be merged into the master branch. On Linux, the library builds cleanly for either 32-bit or 64-bit.


The stream library has a few new additions that make processing files much easier. The new helper methods shorten and simplify code needed to parse or write generic binary files, while also providing a number of safety checks by default.


I think you kind of hit the nail on the head about the CMake experience. So far it hasn't been very compelling. It seems like for everything we can make it do, there's another equivalent problem that crops up earlier in the CMake system. It's just moving the problems, rather than solving them.

I could go either way on removing the CMake stuff. I think maybe I'm leaning slightly towards removing it, though right now, I want to leave it in for a little while longer, just to make sure we're not simply running from the discomfort of using a new system.

Perhaps we should set a deadline for a decision so we don't leave this hanging open.

[Vagabond]

I wanted to tag onto this that I'm glad that I now know how to generate platform specific build systems using CMake. I'm also glad that we tried it out. Just not so sure about its future in the project.

---

Attached are some partially mangled decompressed contents from Sheets.vol. It was produced by updates to the Archive code in OP2Utility in case anyone wants to look at it for troubleshooting what we changed that affected it so negatively.

-Brett

[Vagabond]

Another update on OP2Utility / OP2Archive / OP2MapImager as we have made good progress.

We are routinely building code on x86, x64, Windows, and Linux that operates in the same capacity. There are no warnings in the code base for standard MSVC compilations, and only a few for Clang/g++. I think the code base is now fairly efficiently cross platform, especially for a newbie like me working on the project.

Hooman added automated builds to OP2Utility. This means when someone opens a new pull request, the code automatically compiles for both Linux and Windows in x86 and x64 mode. So even though I do not develop in Linux, I would see quickly (well sort of, the free packages build pretty slowly) that it doesn't work on Linux and vice versa for Hooman working on Linux. We are using AppVeyor for the Windows builds and TravisCI and CircleCI for the Linux builds. I'd like the Windows version to also compile in Debug and Release configuration, but this is really extending the time it requires to perform the automated build, so we will see.

For unit testing, we have chosen Google Test since it is popular, well documented, and has been around for a long time. We have a smattering of tests written and are working towards getting all the tests to run as part of the automated build process. With this in place, you would know that pushed code compiled properly and passed all unit tests on both Windows and Linux, which should add peace of mind when modifying the source code.

There have been several changes proposed to improve the Stream library, but these changes would not be testable using the current OP2Utility code as they are not required in archives or OP2 maps code. Having unit tests would add a way to test them.

I think the near future probably holds continuing to write unit tests and some minor improvements to the stream library.

I am curious, is there anything anyone in the community would like to see incorporated into these projects?

Thanks for reading,
Brett

[leeor_net]

Just wanted to put out there that your continued involvement in the project is very much appreciated. You're also far from a newbie now, you've made great strides in your use of C++. Major pat on the back!

[Hooman]

One slight correction: It's only the g++ builds that show warnings, and that's due to an acknowledged g++ compiler bug concerning the warning it outputs. The Clang builds are clean, with no warnings or errors.

We could probably do MacOS builds as well if we could find a compiler and standard library that supports the new <filesystem> extension. From my initial attempt, that was the only sticking point.

----

Leeor is right, you've advanced quite quickly in your C++ coding abilities.

----

In terms of what I want to see incorporated into the OP2Utility project, that would be sprite support. The library currently handles the tilesets. I want it to also handle the unit sprites. The current tools for extracting and manipulating sprites are a bit hacky to use, so it'd be nice to have a good tested library with an easy to use API to support working with sprites.

In particular, I want a utility that can extract sprite sheets from the game. The basic sprite extraction and handling could go into OP2Utility. I think the actual sprite sheet extraction would be a new utility program that uses the OP2Utility library.

Having a decent set of unit images, and animated sheets, could really help spruce up the website and wiki. Plus, they would be available to anyone working on Outpost themed games.

[Vagabond]

Well, we currently have Cynex's designed extractor which has aged well I think. No source code to play with though. There is also this GitHub project written in Python that allows repacking: https://github.com/phossy/op2art.

Do you think these applications do not meet the need?

Another project I was thinking of was trying to integrate OP2Utility into NetFix to allow playing with network streams if you felt there would be room in NetFix for this to be helpful? It would allow working on both the stream library and Outpost 2 simultaneously.

Brett

[Hooman]

Perhaps we could use those tools, though I feel like things will be a bit hacked together, and mixing languages. What I want is not simply raw data extraction, though that is the part I was thinking of for OP2Utility. Rather, I wanted that functionally to be wrapped up in a utility that re-organizes the data into sprite sheets, and organizes them by unit type and purpose for the animation.

Currently we don't have all the data available to organize by unit type and purpose. We might need to extract some of the data from the exe, or perhaps manually sort through it.

I think such a task would be easier if all tools were integrated. But then, I may have an aversion to using existing external libraries. 


I'm having a bit of trouble envisioning how OP2Utility would relate to NetFix. As for the Streams concept, NetFix uses UDP packets, not TCP streams, so the stream interface might not be a good fit.

With that said, I was looking to enlist your help on a few NetFix upgrades.

[Vagabond]

Even with the Python code available to extract, I think there is some value in getting it in C++ form within OP2Utility. There are some interesting but not too difficult challenges to organizing the sprite sheet. How to maximize space on a canvas with different sized sprites, picking a script such as XML to store the position of each sprite and animation, etc.

I was hoping there would be a way to work the Streams constructively into NetFix, but I didn't actually look. Oh well.

I think from a community perspective, improving NetFix is probably more important than easing modding of the game art. Although this would mean a lot less work on OP2Utility while the NetFix was going on.

-Brett

[Hooman]

Indeed. The original game graphics have animations, which consists of frames, which consist of multiple layers. I don't want to be storing a lot of metadata. I think it might be better to produce a slightly less optimized but more regular graphical format, rather than overload the consumer with loads of metadata to parse and use. In particular, I want a bounding box to be calculated for each animation, based on the offsets and sizes of all layers within all frames, and then build a regular sized sprite sheet for that one animation. For now I want to focus on individual animations. Perhaps at some point they can be merged into unit specific sprite sheets, depending on what works best. In terms of web graphics, I think individual animations would be most convenient to consume. As for meta data, the number of frames would be sufficient. Frame width is bitmap width, and frame height is image height / number of frames.


You're right though, improving NetFix might be a more important community priority.

[Vagabond]

Wow, it has already been a month since this post was active. Guess it is time for another update.

Hooman and I have developed an ArtFile class that representing the Outpost 2 .prt file in memory. It also allows for reading and writing of the .prt file. The PRT file contains all the metadata involved in Outpost 2 sprites and animations.

In the process, we discovered some minor new aspects of the .prt file that I think still need officially documented in the SVN repository. Additionally, there is a chunk of data at the end of the .prt file that is not documented at all. We haven't teased out the real purpose or structure of this data yet, so the prt Writer may not properly form data for the game yet. (This data may not be needed, but I haven't tested yet).

I also wrote an IndexedBmpWriter. It is a C++ platform agnostic way to write a bitmap file from in data memory. It cannot handle bitmaps with a bit count over 8 (non-indexed). I've been working on a BitmapReader as well, although it hasn't been committed yet.

Hooman has been providing a lot of code review for me and technical help. Especially with bitwise operations and improving naming of functions and tips on improving unit test code.

An OP2BitmapLoader allows loading the Outpost 2 specific art assets and then transfers them into a generic bitmap for saving to disk. It works on the images right now, but does not yet form actual frames from the data. All platform agnostic.

Hooman also submitted a decent number of improvements to Outpost2Utility in general for the Archive code and Stream library.

The next steps are probably commiting the BitmapReader to the library, teasing out the purpose of the data at the end of the .prt file, drawing frames from multiple images, etc.

-Brett

[Vagabond]

Another update on OP2Utility and its relation with OP2Archive & OP2MapImager.

OP2Utility now contains a basic Bitmap file reader. It only works with indexed bitmaps which serves Outpost 2's needs. After writing some unit tests for the bitmap reader I moved on to working on new releases for OP2Archive and OP2MapImager.

Releasing OP2Archive went smoothly and I think with the updates it is a fairly robust tool. OP2MapImager is proving significantly more troublesome.

We updated OP2MapImager so it can read tilesets (wells) without extracting them from volume files. A nice feature but nothing groundbreaking.

Hooman started pulling the string on some bugs related to locating files in OP2MapImager from different directories or from within archives. Turns out there were bugs in both OP2MapImager and the underlying file management code in OP2Utility. We have fixed the OP2MapImager bugs (at least mostly I think).

OP2Utility has proven more difficult. Hooman was writing unit tests for the XFile namespace in OP2Utility and we learned that Google Mock is not installed via Microsoft's official Gtest release. We have sort of circled up and tried to remedy the situation, but a good option hasn't presented itself yet.

progress has been slower since I was trying to focus on some code for the next Outpost 2 release.

-Brett

[Hooman]

Yes, the library is quickly becoming capable of processing game sprites, used for unit graphics and such. Previously it could only handle tilesets used for map imaging.

The OP2MapImager updates make it more convenient to use. Previously when I tried it on Linux, I got confusing error messages, and needed to manually unpack archives with the OP2Archive tool. Now it mostly just works, though there are some bugs when running it from outside the game folder.

The current difficulties with the test project setup is prompting us to address some things we'd previously left as good enough. In particular, there's a bit of duplication of effort with the Continuous Integration builds, and a bit of platform inconsistency with the test project setup, namely how the Google Test library is found, and which version is used.

[Vagabond]

Just a quick update.

We put some time into improving the map parsing code. It has a better understanding of how a saved game file works and naming conventions were improved.

The stream library now contains a BidirectionalSeekableReader/Writer and ForwardSeekableReader/Writer. This should give some more flexibility in implementing different types of streams.

I spent a lot of time trying to clear out the issues queue, but progress was painfully slow. A lot of our issues are written in sweeping statements that are difficult to implement. Also the XFile codebase continues to contain bugs and subtle platform differences that has stunted development.

We have teased out a lot of bugs in OP2MapImager. It needs a little more attention before being ready for its next release. This release will allow rendering maps without extracting the tilesets from the volumes. Otherwise it is just a bugfix release to fix the large number of bugs associated with searching for files in different directories.

-Brett

[Hooman]

You're right about the sweeping statements. We probably put more into the Issue tracker than we should. In particular, there is a Projects section we could use to track feature requests. That might be better than mixing them in with Issues/bugs.

[Vagabond]

It has been a long time, but wanted to mention that development on OP2Utility has continued, although at a much slower pace.

In particular:
 * More unit tests
 * Improvements to the Stream classes (Slice, FileWriter)
 * Improved ResourceManager
 * A new DynamicMemoryWriter for test applications
 * Fixes and new functionality for XFile, esp directory functions

We are slowly increasing the unit test coverage on the codebase, although it is still spotty or nonexistant in many areas.

My contributions were geared towards improving OP2Archive and OP2MapImager directory traversal issues. If you have used these programs, you have probably noticed a large number of bugs associated with packaging files from different directories. I think it is time to publish new releases of both OP2Archive and OP2MapImager to fix these issues.

-Brett

[TechCor]

Hey, this looks awesome.

I looked at this on GitHub a few weeks ago, as making a full-featured scenario editor is on my to-do list, but I'm having trouble figuring out what I can do with it.

Can you update the GitHub readme with an overview of the important features and what they are used for? And maybe include what file/folder to look for.

Outpost2.net projects generally have lackluster documentation, and there isn't always someone knowledgeable around to help.

[Vagabond]

Thanks TechCor,

I just checked the readme and it is laughably out of date and small.

I'll add expanding the readme to my TODO list, although it may fall after the release of OP2Archive and OP2MapImager unless you are looking to use it very soon.

-Brett

[Hooman]

Huh, interesting to see people are checking that stuff out. I guess we should probably pay more attention to documentation.

I just submitted an updated ReadMe.md for the project. Hopefully we'll have better descriptions merged into the repository soon.

[Vagabond]

@Techcor,

Hooman and I spent quite a bit of time on OP2Utility's readme to document the purpose of different library sections and basic calling instructions. I'd be curious to get some feedback on if it is useful or not.

The stream code hasn't been documented yet...

Also we added the following updates:

 - Stream::Reader can now read a null terminated stream out of the box. This is useful if reading portions of a PE file in particular, although I'm sure there are other file formats it would come in handy for.
 - Windows gmock support was added. Previously, we were just testing gmock on Linux builds since we only had access to gtest on Windows. I finally found and added the official gmock nuget package to the project, which for some reason was a real challenge for me to understand and find.
 - I fixed a hang in the volume creation subroutine. If you tried to create a volume ontop of an existing volume and then added that existing volume to itself, the program would hang and add a 4 gigabyte file to your hard drive. Sort of expected that you shouldn't be able to add an archive to itself, but the consequences seemed bad enough they were worth preventing.
 - A bug in error handling display for CLM file creation was fixed.

OP2Utility development is a lot slower than it used to be. Many of the core functions have matured and are serving their purpose. The Bitmap/sprite code in particular could use a lot of help if anyone wants to work on it. The Map code is also not well encapsulated yet, although complete enough to work with.

I'll be putting out a new OP2Archive version soon since it will benefit from the updates in OP2Utility.

MissionScanner relies on OP2Utility for stream support. This drove the update to reading null terminated strings. There could be room to move the PE export load code out of MissionScanner and into OP2Utility if this was desired. Not sure it is a good fit or not for OP2Utility or if other programs would benefit from it.

-Brett

[TechCor]

@Techcor,

Hooman and I spent quite a bit of time on OP2Utility's readme to document the purpose of different library sections and basic calling instructions. I'd be curious to get some feedback on if it is useful or not.

The stream code hasn't been documented yet...

Very nice. A lot more thorough than I was expecting.

I will need to look into creating a spec for a comprehensive map/scenario editor.

[TechCor]

I don't know if this is a mistake, but the ReadMe says:

#include "OP2Utility.h"

OP2BmpLoader bmpLoader(bmpFilename, artFilename);
bmpLoader.ExtractImage(0, "extractedFilename.bmp");

However, OP2BmpLoader is not defined when I use it with just that header. I need to include:

#include "OP2Utility/src/Sprite/OP2BmpLoader.h"