Thoughts on the future of OP2Editor (Mapper backend)

[Vagabond]

Since Hooman and I spent some time in the OP2Editor code recently, I wanted to record some thoughts I had on the project.

While this is the backend of the map editor, it could be (is?) a lot bigger in scope. It provides reading and writing of vol files, decoding OP2 specific BMP files, loading and saving map files, loading tile sets into memory, etc.

I made the OP2Utility project to be a utility library for handling the various parts of Outpost 2 related to maps, tilesets, archives, etc. The reality is, this already existed in better form in the OP2Editor. In the long run, I would be happy if OP2Utility was removed and OP2Editor was referenced by OP2MapImager and OP2Archive.

However, some improvements exist in OP2Utility that would need to be folded into OP2Editor. In particular, there is some better error handling for vol and clm files that Hooman and I wrote and the ability to read and extract from LZH compressed vol files (which Hooman wrote I think). I don't think it would be difficult to fold these changes into OP2Editor.

OP2Editor currently fully supports COM. VS2017 has no problem out of the box compiling the COM related code. So while I don't think COM is really in active development by Microsoft, they seem intent on continuing support of it. So currently, OP2Editor could be used in any language that can leverage COM, like C#, Visual Basic, F# (I think?), etc.

There are other ways to support cross language code calling. In particular, we could create buffer .h/.cpp files that expose the object oriented OP2Editor code in a way that could be called procedurally by C. There seems to be a lot of support in many languages for calling C code. Like CTypes in Python. So this might facilitate future ease of cross language support by exposing an external C compatible API. But the code is already fully COM compatible, so this would be a lot of work to change when there are already ways to call it outside of C++.

Of all our projects, OP2Editor would probably be the best candidate for setting up legitimate unit tests. It is a backend library only, so no user interface to deal with when unit testing. Also, OP2Editor could handle the backend work for OP2MapImager, OP2Archive, the mapper, etc. This would facilitate unit testing large portions of all these applications.

While it would be nice to see OP2Editor cross platform, there are a lot of hurdles. All the file system code is written Windows specific. CLM reading and writing relies heavily on Windows specific libraries for music file manipulation. The custom bitmap code uses windows specific code for image manipulation. Plus the COM interface would need to be ripped out. That would amount to a lot of work, and specialized code replacement for the music and bitmap manipulation.

I'm still a fan of using C++17 filesystem, just wish it would leave experimental status sooner. :/

Anyways, I'm not in a spot to work any of this right now. Perhaps in months to come some of these ideas could be implemented? If we ever get around to making a new mapper, any work put into OP2Editor would pay off well.

-Brett

[Hooman]

Yes, the OP2Utility and OP2Editor projects both had rather similar goals. I don't know if choosing OP2Editor over the other is necessarily the best option, because as you've already pointed out, it's a mess of Windows platform specific code, and COM. The COM part being interesting because allows object oriented code libraries to be used by multiple languages, while also being fairly Microsoft specific and tying things to that platform.

One of the choices for using COM, was to interface with Visual Basic, since at the time, there were quite a few other projects using Visual Basic. By using COM, the library could be usable by both C++ and VB projects. Now that VB use here is largely dead, that's no longer a very motivating concern.


If we did want to consider using COM, the project could likely be cleaned up by using more of the COM helper macros, rather than implementing everything from scratch. A lot of the AddRef/Release/QueryInterface code is trivial boilerplate code that could be handled by macros. I was still learning COM when I wrote that code, so I was avoiding the magic of macros.


A common problem with a lot of the library attempts is code based on streams, which the standard C++ library seems to provide a rather poor foundation on which to build. This led to a lot of reliance on the Windows API, which at the time seemed nicer than the standard C++ library. The other big issue was the use of memory mapped files, which led to more Windows API specific calls. If data can be read/written in a standard platform agnostic way, that would go a long ways towards making the libraries platform independent.

Bitmap drawing is also platform specific. The OP2Editor project has some support for drawing, which ties it to Windows. It might make sense to have it load data, while removing the drawing responsibility, though part of that coupling was due to efficiency, and bitmap data being loaded directly into a drawing surface, rather that copied there later. Perhaps a design update could address this, while maintaining efficiency.

The music dependency is less problematic, since it mostly just relies on imported structures, which could be defined in the project itself. Most of the difficulty in porting the music code would actually be the file access code using streams or memory mapped files.


It might be possible to design the COM interface as the most derived class type. I vaguely remember the design of ATL and its macros worked that way. That might be an option for a library that provides core functionality with simple standard C++ classes, which then have a COM layer added on top when building a COM DLL. The COM part might be Windows specific, while the underlying support code could be made platform independent.


Lots of ideas here. Though yes, updating this project could be a lot of work.

[leeor_net]

As mentioned in this other post, I had some difficulty with the original OP2Editor code -- pulling out all of the COM and Windows specific stuff took some effort. I've since pulled all of the COM and Win32 specific code and replaced it with NAS2D calls that really just wrap PhysicsFS -- but those interfaces can be replaced with C++17 functions. What I changed works with streams in memory with only a couple of calls that actually write said stream to disk.

It would probably be more helpful as a sort of guide than as direct drop-in code but I'll leave that to you and Hooman to decide.