Outpost Universe Forums

Projects & Development => Projects => Topic started by: Vagabond on June 21, 2018, 11:22:13 PM

Title: Cross Platform Outpost 2 Utility Library
Post by: Vagabond on June 21, 2018, 11:22:13 PM
Hey everyone,

Currently there is a C++ Utility Library for Outpost 2 that lives on GitHub here https://github.com/OutpostUniverse/OP2Utility. It is a library to assit in Outpost 2 related tasks such as opening and saving map files, save files, clm files, and volume (vol) files. OP2Utility is also the backend for OP2Archive and OP2MapImager. Someday, I wouldn't mind seeing it used in a future map editor to facilitate Outpost 2 specific tasks.

There is a fairly robust Stream library that could easily be lifted for use in non Outpost 2 projects, which will likely continue to be improved on in the future.

OP2Utility currently only compiles for Windows x86.

Hooman and I have been putting a lot of time into improving the library and removing the Windows specific code. A couple of open branches will make all the size_t calls Clang compliant once merged. We are probably only a few lines of code from removing the Windows.h header altogether, which means it should compile on Linux and Windows very soon. Nothing should keep the library from compiling on Macintosh if anyone is interested?

Once it is compiling on both systems, I'd like to look at removing the Visual Studio (Windows) and Make (Linux) specific project files. Instead we can use cMake to build the platform specific project files. This would have the benefit of standardizing compiler settings and allowing updating new source code files in only one place, cMake.

When someone wants to compile the code, cMake generates the proper platform specific tooling. This means someone who uses Linux doesn't have to worry about how to add new files and settings to a Visual Studio project/solution that they cannot check on Linux and the same applies to the Windows users figuring out the Make file.

The downsides are you have to download and install cMake on your computer and learn the commands for generating the platform specific tooling. It also means if you want to make major edits to the project structure, you would have to dig into cMake enough to understand how to make those changes.

The end goal will be compiling OP2Archive and OP2MapImager for both Windows and Linux. I'd like to look at compiling them both for x86 and x64 operating systems. And finally keep a small symbols library for both (.PDB files on Windows, and I think usually .debug files for Linux) that match up with each release cycle. So you could drop in the symbols for full debugging no matter which operating system you are using.

This means both OP2Archive and OP2MapImager will have to switch to cMake for their build tool generation to hook into OP2Utility. The good news is both of these projects should already be almost fully Linux compatible already.

I've been happy working on this project, as it is a great opportunity to continue learning and practicing C++ in a project. I recognize there are probably limited gains to the community (unless you are a Linux fan!).

Anyways, I don't know how others feel about how this is all moving forward. We are happy to have help from others if anyone else is interested in checking it out. Open to suggestions as well if people love/hate cMake or are generally ambivalent to this entire discussion. I'd like to discuss a standardized file structure for the projects in the near future as well.

Title: Re: Cross Platform Outpost 2 Utility Library
Post by: leeor_net on June 22, 2018, 09:54:42 AM
You and Hooman have done absolutely amazing work! I've been following development on GitHub and an pleased to see how this progressed.

When I eventually pick up on my map editor I'll pull the NAS2D filesystem and use your stream library instead. I think it's quite well built and I look forward to being able to drop the dependency (though having ZIP and 7z support is still attractive).

Anyway, this should make it a lot easier to build high quality, robust tools.

Again, awesome work!
Title: Re: Cross Platform Outpost 2 Utility Library
Post by: Hooman on June 24, 2018, 07:09:17 AM
Thank you Leeor. And a very big thank you to Brett for all the work he's put into it.

As of today, the project now compiles on Linux.

There might still be a few rough edges, as some of the changes still need to settle into the master branch, and a few warnings may need to be addressed.

I'm particularly excited about the Streams portion of it, as I've been thinking about that part for quite some time. Though I admit there is so much more that can be done there. There may be some API changes coming to address a few shortcomings recently encountered, but likely mostly in the form of additions or restrictions on bad usage to prevent accidental bugs. I doubt changes will break existing good code from a user perspective. No promises yet for people implementing derived classes though.

Using CMake could be cool. We've talked about using that here for a while now. This just might be the project that forces us to do it. I'm still a little apprehensive about having to learn it, though now is probably a good time.

Overall, this project has been a lot of fun. It's also been a great learning opportunity. Both about tooling, such as using GitHub, and about the additions from the newer C++ standards. The C++ language has really changed in recent years, and it's received a lot of much needed improvement.
Title: Re: Cross Platform Outpost 2 Utility Library
Post by: Vagabond on August 07, 2018, 09:06:40 PM
I wanted to post an update on the progress OP2Utility, OP2Archive, and OP2MapImager have made. We have both OP2Archive and OP2MapImager compiling and running on Linux now.

Also, we have successfully compiled OP2Archive for x64. It appears capable of creating and extracting from Clm and VOL archive files equally whether compiled in x86 or x64. This was very encouraging. Although there were quite a few compiler warnings, especially on MSVC that should be addressed.

Most of these warnings deal with casting integer values that are represented differently based on which architecture they are compiled for. Addressing these warnings will mostly protect from edge case scenarios such as attempting to pack extremely large numbers of files into the archive. In the process, other minor improvements are being made to the code base in general.

These changes are mostly complete for OP2Utility. Next will be making the changes in OP2Archive. After that, I would like to attempt compiling OP2MapImager in X64 and see how it goes.


The stream library continues to improve. It has (or will will soon have):

 * expanded helper functions allowing auto reading/writing of vectors and strings
 * Allows StreamWriter to consume the contents of a StreamReader automatically
 * The ability to slice existing MemoryStreams and FileStreams into an independent substreams.

We are looking at adding options to FileStream on construction that allow for things like truncating existing files, not overwriting existing files, etc.


OP2Archive and OP2Utility now contain CMake project generation files. CMake can be used to quickly auto-generate valid project files for both MSVC and Linux. Adapting it for use on Macintosh may not be difficult, although no one has explored the topic. The original Visual Studio solution and Linux makefile still exist in the repositories, so using CMake to generate the project files is currently optional.

In general, I have been disappointed with the CMake results. The resulting Visual Studio solution works, but does not really proximate what I come to expect as standard in a Visual Studio project that would be created without CMake.

The CMake documentation is very dry, no outstanding books on the subject seem to exist, and web tutorials are a mixed bag of old or not recommended techniques.

We are approaching a crossroads where we need to decide what to do with CMake. If we want to continue supporting it, we need to sink a lot more time and effort into it and probably delete the other project files so to force its use. Currently I just use the built in Visual Studio project files since they are easier to use and do not contain all the CMake added oddities. Deleting them would force everyone to generate a CMake project before coding instead of relying on the existing makefile and Visual Studio project.

At this point, I would be okay deleting the CMake projects from the repository and just using a native makefile and Visual Studio solution. It would allow others to use the projects without having to download and use CMake for project generation.

I would be curious to hear how others feel about CMake being present or being removed from the projects. Both from the standpoint of people who have experience with it and from people who do not have experience but would be forced to use it.

Title: Re: Cross Platform Outpost 2 Utility Library
Post by: Hooman on August 08, 2018, 07:08:12 AM
Indeed, the library now works for both Windows and Linux, and compiles for both 32-bit and 64-bit.

Initial attempts at a 64-bit build on Windows produced a number of compiler warnings. Recent work has cleaned that up, and looks ready to be merged into the master branch. On Linux, the library builds cleanly for either 32-bit or 64-bit.

The stream library has a few new additions that make processing files much easier. The new helper methods shorten and simplify code needed to parse or write generic binary files, while also providing a number of safety checks by default.

I think you kind of hit the nail on the head about the CMake experience. So far it hasn't been very compelling. It seems like for everything we can make it do, there's another equivalent problem that crops up earlier in the CMake system. It's just moving the problems, rather than solving them.

I could go either way on removing the CMake stuff. I think maybe I'm leaning slightly towards removing it, though right now, I want to leave it in for a little while longer, just to make sure we're not simply running from the discomfort of using a new system.

Perhaps we should set a deadline for a decision so we don't leave this hanging open.