Author Topic: Cross Platform Outpost 2 Utility Library  (Read 719 times)

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 724
Cross Platform Outpost 2 Utility Library
« on: June 21, 2018, 11:22:13 PM »
Hey everyone,

Currently there is a C++ Utility Library for Outpost 2 that lives on GitHub here https://github.com/OutpostUniverse/OP2Utility. It is a library to assit in Outpost 2 related tasks such as opening and saving map files, save files, clm files, and volume (vol) files. OP2Utility is also the backend for OP2Archive and OP2MapImager. Someday, I wouldn't mind seeing it used in a future map editor to facilitate Outpost 2 specific tasks.

There is a fairly robust Stream library that could easily be lifted for use in non Outpost 2 projects, which will likely continue to be improved on in the future.

OP2Utility currently only compiles for Windows x86.

Hooman and I have been putting a lot of time into improving the library and removing the Windows specific code. A couple of open branches will make all the size_t calls Clang compliant once merged. We are probably only a few lines of code from removing the Windows.h header altogether, which means it should compile on Linux and Windows very soon. Nothing should keep the library from compiling on Macintosh if anyone is interested?

Once it is compiling on both systems, I'd like to look at removing the Visual Studio (Windows) and Make (Linux) specific project files. Instead we can use cMake to build the platform specific project files. This would have the benefit of standardizing compiler settings and allowing updating new source code files in only one place, cMake.

When someone wants to compile the code, cMake generates the proper platform specific tooling. This means someone who uses Linux doesn't have to worry about how to add new files and settings to a Visual Studio project/solution that they cannot check on Linux and the same applies to the Windows users figuring out the Make file.

The downsides are you have to download and install cMake on your computer and learn the commands for generating the platform specific tooling. It also means if you want to make major edits to the project structure, you would have to dig into cMake enough to understand how to make those changes.

The end goal will be compiling OP2Archive and OP2MapImager for both Windows and Linux. I'd like to look at compiling them both for x86 and x64 operating systems. And finally keep a small symbols library for both (.PDB files on Windows, and I think usually .debug files for Linux) that match up with each release cycle. So you could drop in the symbols for full debugging no matter which operating system you are using.

This means both OP2Archive and OP2MapImager will have to switch to cMake for their build tool generation to hook into OP2Utility. The good news is both of these projects should already be almost fully Linux compatible already.

I've been happy working on this project, as it is a great opportunity to continue learning and practicing C++ in a project. I recognize there are probably limited gains to the community (unless you are a Linux fan!).

Anyways, I don't know how others feel about how this is all moving forward. We are happy to have help from others if anyone else is interested in checking it out. Open to suggestions as well if people love/hate cMake or are generally ambivalent to this entire discussion. I'd like to discuss a standardized file structure for the projects in the near future as well.

-Brett

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 2021
    • LairWorks Entertainment
Re: Cross Platform Outpost 2 Utility Library
« Reply #1 on: June 22, 2018, 09:54:42 AM »
You and Hooman have done absolutely amazing work! I've been following development on GitHub and an pleased to see how this progressed.

When I eventually pick up on my map editor I'll pull the NAS2D filesystem and use your stream library instead. I think it's quite well built and I look forward to being able to drop the dependency (though having ZIP and 7z support is still attractive).

Anyway, this should make it a lot easier to build high quality, robust tools.

Again, awesome work!
- Leeor
LairWorks Entertainment

Titanum UFO's

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4536
Re: Cross Platform Outpost 2 Utility Library
« Reply #2 on: June 24, 2018, 07:09:17 AM »
Thank you Leeor. And a very big thank you to Brett for all the work he's put into it.

As of today, the project now compiles on Linux.

There might still be a few rough edges, as some of the changes still need to settle into the master branch, and a few warnings may need to be addressed.

I'm particularly excited about the Streams portion of it, as I've been thinking about that part for quite some time. Though I admit there is so much more that can be done there. There may be some API changes coming to address a few shortcomings recently encountered, but likely mostly in the form of additions or restrictions on bad usage to prevent accidental bugs. I doubt changes will break existing good code from a user perspective. No promises yet for people implementing derived classes though.

Using CMake could be cool. We've talked about using that here for a while now. This just might be the project that forces us to do it. I'm still a little apprehensive about having to learn it, though now is probably a good time.

Overall, this project has been a lot of fun. It's also been a great learning opportunity. Both about tooling, such as using GitHub, and about the additions from the newer C++ standards. The C++ language has really changed in recent years, and it's received a lot of much needed improvement.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 724
Re: Cross Platform Outpost 2 Utility Library
« Reply #3 on: August 07, 2018, 09:06:40 PM »
I wanted to post an update on the progress OP2Utility, OP2Archive, and OP2MapImager have made. We have both OP2Archive and OP2MapImager compiling and running on Linux now.

Also, we have successfully compiled OP2Archive for x64. It appears capable of creating and extracting from Clm and VOL archive files equally whether compiled in x86 or x64. This was very encouraging. Although there were quite a few compiler warnings, especially on MSVC that should be addressed.

Most of these warnings deal with casting integer values that are represented differently based on which architecture they are compiled for. Addressing these warnings will mostly protect from edge case scenarios such as attempting to pack extremely large numbers of files into the archive. In the process, other minor improvements are being made to the code base in general.

These changes are mostly complete for OP2Utility. Next will be making the changes in OP2Archive. After that, I would like to attempt compiling OP2MapImager in X64 and see how it goes.

---

The stream library continues to improve. It has (or will will soon have):

 * expanded helper functions allowing auto reading/writing of vectors and strings
 * Allows StreamWriter to consume the contents of a StreamReader automatically
 * The ability to slice existing MemoryStreams and FileStreams into an independent substreams.

We are looking at adding options to FileStream on construction that allow for things like truncating existing files, not overwriting existing files, etc.

---

OP2Archive and OP2Utility now contain CMake project generation files. CMake can be used to quickly auto-generate valid project files for both MSVC and Linux. Adapting it for use on Macintosh may not be difficult, although no one has explored the topic. The original Visual Studio solution and Linux makefile still exist in the repositories, so using CMake to generate the project files is currently optional.

In general, I have been disappointed with the CMake results. The resulting Visual Studio solution works, but does not really proximate what I come to expect as standard in a Visual Studio project that would be created without CMake.

The CMake documentation is very dry, no outstanding books on the subject seem to exist, and web tutorials are a mixed bag of old or not recommended techniques.

We are approaching a crossroads where we need to decide what to do with CMake. If we want to continue supporting it, we need to sink a lot more time and effort into it and probably delete the other project files so to force its use. Currently I just use the built in Visual Studio project files since they are easier to use and do not contain all the CMake added oddities. Deleting them would force everyone to generate a CMake project before coding instead of relying on the existing makefile and Visual Studio project.

At this point, I would be okay deleting the CMake projects from the repository and just using a native makefile and Visual Studio solution. It would allow others to use the projects without having to download and use CMake for project generation.

I would be curious to hear how others feel about CMake being present or being removed from the projects. Both from the standpoint of people who have experience with it and from people who do not have experience but would be forced to use it.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4536
Re: Cross Platform Outpost 2 Utility Library
« Reply #4 on: August 08, 2018, 07:08:12 AM »
Indeed, the library now works for both Windows and Linux, and compiles for both 32-bit and 64-bit.

Initial attempts at a 64-bit build on Windows produced a number of compiler warnings. Recent work has cleaned that up, and looks ready to be merged into the master branch. On Linux, the library builds cleanly for either 32-bit or 64-bit.



The stream library has a few new additions that make processing files much easier. The new helper methods shorten and simplify code needed to parse or write generic binary files, while also providing a number of safety checks by default.



I think you kind of hit the nail on the head about the CMake experience. So far it hasn't been very compelling. It seems like for everything we can make it do, there's another equivalent problem that crops up earlier in the CMake system. It's just moving the problems, rather than solving them.

I could go either way on removing the CMake stuff. I think maybe I'm leaning slightly towards removing it, though right now, I want to leave it in for a little while longer, just to make sure we're not simply running from the discomfort of using a new system.

Perhaps we should set a deadline for a decision so we don't leave this hanging open.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 724
Re: Cross Platform Outpost 2 Utility Library
« Reply #5 on: August 21, 2018, 02:40:53 PM »
I wanted to tag onto this that I'm glad that I now know how to generate platform specific build systems using CMake. I'm also glad that we tried it out. Just not so sure about its future in the project.

---

Attached are some partially mangled decompressed contents from Sheets.vol. It was produced by updates to the Archive code in OP2Utility in case anyone wants to look at it for troubleshooting what we changed that affected it so negatively.

-Brett

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 724
Re: Cross Platform Outpost 2 Utility Library
« Reply #6 on: August 31, 2018, 04:32:09 PM »
Another update on OP2Utility / OP2Archive / OP2MapImager as we have made good progress.

We are routinely building code on x86, x64, Windows, and Linux that operates in the same capacity. There are no warnings in the code base for standard MSVC compilations, and only a few for Clang/g++. I think the code base is now fairly efficiently cross platform, especially for a newbie like me working on the project.

Hooman added automated builds to OP2Utility. This means when someone opens a new pull request, the code automatically compiles for both Linux and Windows in x86 and x64 mode. So even though I do not develop in Linux, I would see quickly (well sort of, the free packages build pretty slowly) that it doesn't work on Linux and vice versa for Hooman working on Linux. We are using AppVeyor for the Windows builds and TravisCI and CircleCI for the Linux builds. I'd like the Windows version to also compile in Debug and Release configuration, but this is really extending the time it requires to perform the automated build, so we will see.

For unit testing, we have chosen Google Test since it is popular, well documented, and has been around for a long time. We have a smattering of tests written and are working towards getting all the tests to run as part of the automated build process. With this in place, you would know that pushed code compiled properly and passed all unit tests on both Windows and Linux, which should add peace of mind when modifying the source code.

There have been several changes proposed to improve the Stream library, but these changes would not be testable using the current OP2Utility code as they are not required in archives or OP2 maps code. Having unit tests would add a way to test them.

I think the near future probably holds continuing to write unit tests and some minor improvements to the stream library.

I am curious, is there anything anyone in the community would like to see incorporated into these projects?

Thanks for reading,
Brett
« Last Edit: August 31, 2018, 04:34:14 PM by Vagabond »

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 2021
    • LairWorks Entertainment
Re: Cross Platform Outpost 2 Utility Library
« Reply #7 on: August 31, 2018, 06:33:27 PM »
Just wanted to put out there that your continued involvement in the project is very much appreciated. You're also far from a newbie now, you've made great strides in your use of C++. Major pat on the back! :D
- Leeor
LairWorks Entertainment

Titanum UFO's

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4536
Re: Cross Platform Outpost 2 Utility Library
« Reply #8 on: September 01, 2018, 12:03:12 AM »
One slight correction: It's only the g++ builds that show warnings, and that's due to an acknowledged g++ compiler bug concerning the warning it outputs. The Clang builds are clean, with no warnings or errors.

We could probably do MacOS builds as well if we could find a compiler and standard library that supports the new <filesystem> extension. From my initial attempt, that was the only sticking point.

----

Leeor is right, you've advanced quite quickly in your C++ coding abilities.

----

In terms of what I want to see incorporated into the OP2Utility project, that would be sprite support. The library currently handles the tilesets. I want it to also handle the unit sprites. The current tools for extracting and manipulating sprites are a bit hacky to use, so it'd be nice to have a good tested library with an easy to use API to support working with sprites.

In particular, I want a utility that can extract sprite sheets from the game. The basic sprite extraction and handling could go into OP2Utility. I think the actual sprite sheet extraction would be a new utility program that uses the OP2Utility library.

Having a decent set of unit images, and animated sheets, could really help spruce up the website and wiki. Plus, they would be available to anyone working on Outpost themed games.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 724
Re: Cross Platform Outpost 2 Utility Library
« Reply #9 on: September 01, 2018, 11:04:27 AM »
Well, we currently have Cynex's designed extractor which has aged well I think. No source code to play with though. There is also this GitHub project written in Python that allows repacking: https://github.com/phossy/op2art.

Do you think these applications do not meet the need?

Another project I was thinking of was trying to integrate OP2Utility into NetFix to allow playing with network streams if you felt there would be room in NetFix for this to be helpful? It would allow working on both the stream library and Outpost 2 simultaneously.

Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4536
Re: Cross Platform Outpost 2 Utility Library
« Reply #10 on: September 01, 2018, 12:08:13 PM »
Perhaps we could use those tools, though I feel like things will be a bit hacked together, and mixing languages. What I want is not simply raw data extraction, though that is the part I was thinking of for OP2Utility. Rather, I wanted that functionally to be wrapped up in a utility that re-organizes the data into sprite sheets, and organizes them by unit type and purpose for the animation.

Currently we don't have all the data available to organize by unit type and purpose. We might need to extract some of the data from the exe, or perhaps manually sort through it.

I think such a task would be easier if all tools were integrated. But then, I may have an aversion to using existing external libraries.  :-[



I'm having a bit of trouble envisioning how OP2Utility would relate to NetFix. As for the Streams concept, NetFix uses UDP packets, not TCP streams, so the stream interface might not be a good fit.

With that said, I was looking to enlist your help on a few NetFix upgrades.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 724
Re: Cross Platform Outpost 2 Utility Library
« Reply #11 on: September 02, 2018, 10:07:25 AM »
Even with the Python code available to extract, I think there is some value in getting it in C++ form within OP2Utility. There are some interesting but not too difficult challenges to organizing the sprite sheet. How to maximize space on a canvas with different sized sprites, picking a script such as XML to store the position of each sprite and animation, etc.

I was hoping there would be a way to work the Streams constructively into NetFix, but I didn't actually look. Oh well.

I think from a community perspective, improving NetFix is probably more important than easing modding of the game art. Although this would mean a lot less work on OP2Utility while the NetFix was going on.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4536
Re: Cross Platform Outpost 2 Utility Library
« Reply #12 on: September 02, 2018, 10:15:42 AM »
Indeed. The original game graphics have animations, which consists of frames, which consist of multiple layers. I don't want to be storing a lot of metadata. I think it might be better to produce a slightly less optimized but more regular graphical format, rather than overload the consumer with loads of metadata to parse and use. In particular, I want a bounding box to be calculated for each animation, based on the offsets and sizes of all layers within all frames, and then build a regular sized sprite sheet for that one animation. For now I want to focus on individual animations. Perhaps at some point they can be merged into unit specific sprite sheets, depending on what works best. In terms of web graphics, I think individual animations would be most convenient to consume. As for meta data, the number of frames would be sufficient. Frame width is bitmap width, and frame height is image height / number of frames.



You're right though, improving NetFix might be a more important community priority.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 724
Re: Cross Platform Outpost 2 Utility Library
« Reply #13 on: October 06, 2018, 09:26:55 AM »
Wow, it has already been a month since this post was active. Guess it is time for another update.

Hooman and I have developed an ArtFile class that representing the Outpost 2 .prt file in memory. It also allows for reading and writing of the .prt file. The PRT file contains all the metadata involved in Outpost 2 sprites and animations.

In the process, we discovered some minor new aspects of the .prt file that I think still need officially documented in the SVN repository. Additionally, there is a chunk of data at the end of the .prt file that is not documented at all. We haven't teased out the real purpose or structure of this data yet, so the prt Writer may not properly form data for the game yet. (This data may not be needed, but I haven't tested yet).

I also wrote an IndexedBmpWriter. It is a C++ platform agnostic way to write a bitmap file from in data memory. It cannot handle bitmaps with a bit count over 8 (non-indexed). I've been working on a BitmapReader as well, although it hasn't been committed yet.

Hooman has been providing a lot of code review for me and technical help. Especially with bitwise operations and improving naming of functions and tips on improving unit test code.

An OP2BitmapLoader allows loading the Outpost 2 specific art assets and then transfers them into a generic bitmap for saving to disk. It works on the images right now, but does not yet form actual frames from the data. All platform agnostic.

Hooman also submitted a decent number of improvements to Outpost2Utility in general for the Archive code and Stream library.

The next steps are probably commiting the BitmapReader to the library, teasing out the purpose of the data at the end of the .prt file, drawing frames from multiple images, etc.

-Brett