Cross Platform Outpost 2 Utility Library

[Vagabond]

I think this behavior is a bug. I may have forgotten to add an include to the master header and will look at it this evening.

Thanks for pointing it out.

-Brett

[Vagabond]

@TechCor,

You should be able to pull the master for OP2Utility with the issue you pointed out fixed.

-Brett

[TechCor]

I'm having trouble getting the bitmap stored with ResourceManager. It returns null.

I do something like this (Greatly simplified):

Code: [Select]

ResourceManager resourceManager = new ResourceManager(Outpost2GameDirectory);

Code: [Select]

std::unique_ptr<Stream::BidirectionalReader> stream = resourceManager->GetResourceStream("well0004", true);
// NOTE: I've also tried "well0004.bmp"
		
if (stream == nullptr)
    return 0; // This happens. Stream is null.

return stream->Length();

Maybe I'm misunderstanding what ResourceManager does. I assumed it would search all the archives in the game directory for "well0004.bmp".

I can successfully get the BMPs using VolFile(Outpost2GameDirectory + "/art.vol") with ReadFileByName.

[Hooman]

The internal filenames within the VOL files for the tilesets includes the ".bmp" extension. You will need to specify the name exactly, including extension.

The ResourceManager object might be sensitive to things like having a trailing slash at the end of the path. If the path used does not include a trailing slash, try adding one. From your example of using VolFile directly, it looks like the Outpost2GameDirectory value does not include a trailing slash.

Nothing else jumps out at me as possibly wrong. I could check in more detail if you need.

[TechCor]

Hey guys, I'm going to work around this, but there's something I'd like to point out.

You have these very nice convenience functions. I use them to render the map:

Code: [Select]

std::size_t GetTileMappingIndex(std::size_t x, std::size_t y) const;
CellType GetCellType(std::size_t x, std::size_t y) const;
bool GetLavaPossible(std::size_t x, std::size_t y) const;

Only one problem - there aren't any setters!

It appears I have to copy the GetTileIndex implementation to my wrapper so I can modify the tiles.

I assume setters for those methods are all that are needed, and that the other bits of data in the Tile are for save games or something.

Anyway, you need at least some way of modifying tiles by XY coordinates.

[Hooman]

Haven't gotten around to implementing it yet.

The library was of course developed to support OP2MapImager (and OP2Archive), and so read access had a high priority. Those utilities had no reason to write map data.

I suppose there will be more of a push to develop write capabilities, now that two map editor projects are starting to use OP2Utility.

[Vagabond]

TechCor, I'm working to add the basic setter functions for CellType and LavaPossible. Please keep the requests coming on what you want to see added, changed, or further explained.

GetLavaPossible PR: https://github.com/OutpostUniverse/OP2Utility/pull/330

I would like to add some tile range checks to the code, but it became very complicated very quickly.

[Vagabond]

Helper functions Map::SetLavaPossible and Map::SetCellType and now implemented in master. there is also a minor update to Stream::FileWriter to improve edge case error messaging.

[TechCor]

I'm in the process of moving my projects to VS 2019. No problem with the mission SDK, but OP2 Utility is giving me a deprecation error:

Code: [Select]

Error	C1189	#error:  The <experimental/filesystem> header providing std::experimental::filesystem is deprecated by Microsoft and will be REMOVED. It is superseded by the C++17 <filesystem> header providing std::filesystem. You can define _SILENCE_EXPERIMENTAL_FILESYSTEM_DEPRECATION_WARNING to acknowledge that you have received this warning.	

OP2Utility	C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\include\experimental\filesystem	30

[Vagabond]

TechCor,

Thanks for reporting this. I've noticed in the past for some reason the Visual Studio build of OP2Utility is defaulting to the experimental build of filesystem instead of the final build.

We have a preprocessor directive as an include guard that should be incorporating the newer header on a VS2019 toolset compilation. We need to look into why it is not finding the final build and forcing selection of the experimental build for MSVC.

-Brett

[Vagabond]

TechCor,

I just created a Pull Request for OP2Utility that should fix the issue. https://github.com/OutpostUniverse/OP2Utility/pull/334

-Brett

[TechCor]

I've created a pure C# port of OP2 Utility. It's no longer a wrapper around the C++ one.

https://github.com/TechCor8/OP2UtilityDotNet

I've confirmed that map export/import, clm/vol extraction and creating archives works (via mission editor capabilities).

Works on both Windows and Linux.

This will make things easier for devs who want to make C# tools.

I'll also be expanding the animation support in C# as I want to be able to simulate the OP2 renderer.

[Vagabond]

I'm interested in checking out the animation support when you get there. This looks like a lot of work porting everything to C# instead of just wrapping it.

-Brett

[TechCor]

Hey there. The existing BMP support is looking quite good!

I'm able to view a large number of images, but there seems to be a problem with quite a few of them. I get the exception:

Code: [Select]

Too many colors listed on the indexed palette

Looking into it. It is because the "bitCount" of some images is 1, which, when calculated by ImageHeader.CalcMaxIndexedPaletteSize, is less than the palette length of 256 (1 << bitCount == 2). Every palette seems to be 256 in length.

Closer inspection of the bitCount shows that it can be either 1 or 8. 1 if the image is considered a "shadow", it is set to 1. It seems that "shadow" images need to have some kind of special treatment. Maybe a 2 color palette.

The image I've been debugging is at index 652.

I'm curious if anyone has any additional information or if I am entering unexplored territory.

EDIT: Confirmed that the issue is shadow images via ArtViewer by checking the failed indexes.

While I'm posting, I might as well also mention a possible bug I noticed when porting the Unit Tests:

Code: [Select]

TEST(ImageHeader, Create)
{
	...
	EXPECT_EQ(ImageHeader::DefaultPlanes, imageHeader.DefaultPlanes);
	...
	EXPECT_EQ(ImageHeader::DefaultImageSize, imageHeader.DefaultImageSize);
	EXPECT_EQ(ImageHeader::DefaultXResolution, imageHeader.DefaultXResolution);
	EXPECT_EQ(ImageHeader::DefaultYResolution, imageHeader.DefaultYResolution);
	EXPECT_EQ(ImageHeader::DefaultUsedColorMapEntries, imageHeader.DefaultUsedColorMapEntries);
	EXPECT_EQ(ImageHeader::DefaultImportantColorCount, imageHeader.DefaultImportantColorCount);
}

The above statements are referring to the same thing as far as I can tell. C# doesn't let you access class members this way, so I changed, for example, "imageHeader.DefaultPlanes" to "imageHeader.planes".

[TechCor]

Another rather cryptic issue I ran into:

ResourceManager.GetResourceStream()

When using this, I thought it was giving me the data that would be present in the extracted file, only in memory. Turns out this is a direct SliceStream to potentially compressed or altered data in the archive!

At the ResourceManager level, I don't know if this is actually useful, since the data format will be inconsistent at this point.

For example, I was trying to pull a .WAV file. If this is in the archive, the stream is a slightly altered (headerless?) WAV file. However, if the file is loose in the directory, the .WAV file will contain that header information.

I'm not sure what the correct solution should be if performance is a consideration.

C# side, I've extended the Archive classes to support ExtractFileToStream and ExtractFileToMemory, and changed GetResourceStream to return the stream from ExtractFileToMemory if it can't find a loose file.

I'm now able to play .WAV files in the app!

For those looking to backport these changes:

Added ExtractFileToMemory and ExtractFileToStream to Archive files: Link to Commit

ResourceManager OpenStream to ExtractFileToMemory: Link to Commit

[Vagabond]

TechCor,

You are spot on with the shadows being 1 bit. I was having difficulty understanding the packing rule differences between 1 and 8 bit bitmap pixels and this is basically where I left off of the work. I was unhappy overall with the design of the Bitmap class implementation in OP2Utility (my own fault).

Maybe there will be space to reverse your C# work back into C++ to finish up support for pulling out animations and shadows properly.

You are also correct about the ResourceManager. It allows pulling slices out of the volumes and clm files. Sometimes this is sufficient such as map files and sometimes not as in the music tracks. OP2Archive manually appended the headers to the slices to save a whole wav file to disc.

I like the idea of the extension you are mentioning about presenting a complete file in memory stream and I don't think it every materialized because we didn't have a real use case at the time.

If the art viewer portion of OP2Utility had matured, it would have been nice to create a program to extract animations into animated GIF files. But I don't think I'll come back to that anytime soon...

Glad to see OP2Utility is proving helpful to you though. Hooman and I put a lot of work into refining it.

-Brett

[TechCor]

Well. I managed to get all the animations rendering. Looks pretty neat.

Layer.frameIndex should probably be named layerIndex, but it is not really used anyway. Otherwise, the original code worked perfectly. Good job!

I'm curious as to how the frame rate is determined, but I'm sure it is all customized at the engine level.

I looked closely at the bitmap code to figure out what was going on with shadows.

First, BMPs were over-reading data. Took me a while to figure out this was going on.

imageMeta.scanLineByteWidth is too large and was being used to get the amount of data that needed to be read. I'm not sure what this property is actually used for, but the amount of data being read was 3-4 times past the end of the BMP for all BMPs.

Instead of doing this, I calculate the pitch and multiply the height by that to get the amount of data to read. Then, I iterate over the height and write the row using the pitch.

For the palette, I check if the image is a shadow, and set it to copy the first two colors.

Commit: Shadow fix
Commit: Merged method implementations that needed the shadow fix.

The only problem left is that the shadows are blue and lack transparency. I think the shadow palette must be handled at the engine level, ignoring the BMP palette. There doesn't appear to be any other explanation.

I also want to extend BitmapFile so that I can read the actual pixel color values in a usable format. The "pixels" property is the packed bytes and not easy to work with. Currently, I'm using a middleman to read the utility's output and load time is about 60 seconds 4 seconds to read all images when I don't render each one to the screen after loading .

[Vagabond]

Nice job with the animations. I'm unsure about framerate.

Yeah, you pretty much figured out the bitmap code I was writing. Hooman and I were discussing if custom padding was allowed in bitmaps and how to handle it when development stalled which led to the poorly formed functions inside the bitmap class.

I never investigated the proper color pallete settings for the shadows and suspect you are correct that they are custom set inside outpost 2 since there are only 2 colors involved.

Maybe someday we can look at porting your work back into the C++ project.

One of my goals for OP2Utility was to allow for developing some sort of command line frontend that would allow packaging custom sprites for use in game. So you could say replace the ConVec images or something. It gets pretty complicated though since different animations share the same color pallete and I'm not sure there was enough interest that someone would actually take the time to undertake drawing custom sprites. It would have complemented OP2Archive and OP2MapImager.

[Vagabond]

Just wanted to mention I had taken up working on OP2Utility again recently after the fairly long break.

Hooman and I were looking at creating a console application to Convert the custom tilesets (wells) into standard bitmaps. It would compliment the existing projects OP2Archive and OP2MapImager. We realized that we had not ported the code from OP2Editor to complete the task.

I'm happy to say that OP2Utility can now load the custom tilesets and reformat them as standard Windows Bitmap Files. The exception messages if something goes wrong is a bit goofy (does not provide fully truthful/helpful messages to the user), but this should be cleaned up before the end of the month.

A couple of smaller improvements came into the baseline before this, mainly:

 * Bug fixes to extracting in game sprites inspired by TechCor's port (thank you TechCor for leaving good notes on your changes)
 * Improvements to the BitmapFile class in OP2Utility. It is still a work in progress but is starting to clean up and become a bit more readable with a simpler interface.
 * Ability to read arbitrary complex types with Stream::Reader by defining a standard Read function.
 * Convenience Stream functions to seek to the beginning or end of a stream if the stream supports seeking backwards and/or forwards

Will post again when the new console application is up and running in some sort of beta form.

Thanks for Hooman providing code reviews of the work, resulting in a lot of improvements to its readability and use of the C++'s standard library features.

[Arklon]

It's worth mentioning the 1.4.0 installer already converts the tileset bitmaps.

[TechCor]

Good work. I had noticed a flurry of GitHub emails lately and was just wondering if you were going to post an update.