Author Topic: OP2Archive (Command line extraction and creation for .vol and .clm files)  (Read 1035 times)

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 684
OP2Archive is a 32bit Windows command line application for extracting, creating, and editing .vol and .clm archive files specific to Outpost 2. This tool allows for access to the in game music, maps, tilesets, datasheets, and more. Perfect if you wish to review or modify Outpost 2's internal data.

It is designed for easy use if you understand how to run command line (console) applications. Inside the download is a ReadMe.txt file that explains how to use it. If you have feedback or find bugs please post or message me.

OP2 Archive on Outpost Universe Wiki

Download Current Release: OP2Archive on GitHub

-Brett



ReadMe

OP2Archive - Outpost 2 Archive Access and Maintenance
Developed by Hooman and Brett208 (Vagabond)

Allows examining, creating, and extracting files from Outpost 2 .vol and .clm archives. Check the OP2Archive landing page for the newest version at https://wiki.outpost2.net/doku.php?id=outpost_2:helper_programs:op2archive.

+++ COMMANDS +++
  * OP2Archive LIST (archivename.(vol|clm) | directory)...
    * Lists the contents and size of all files contained in provided archives.

  * OP2Archive FIND filename...
    * Determines which archive contains the file. Stops searching at first instance.
    * All archives in the same directory as the provided filename are searched.

  * OP2Archive CREATE archivename.(vol|clm) [filename | directory]... [-q] [-o]
    * If no filename(s) or directory(s) provided,
      archives all contents of the default source directory (./archiveFilename).

  * OP2Archive EXTRACT archivename.(vol|clm) [filename]... [-q] [-d destDirectory] [-o]
    * If no filename(s) provided, extracts entire contents of archive.

  * OP2Archive EXTRACT directoryname [-q] [-d destDirectory] [-o].
    * Extracts entire contents of all archives in the provided directory.

  * OP2Archive ADD archiveName.[vol|clm] filename... [-q] [-o]

  * OP2Archive REMOVE archiveName.[vol|clm] filename... [-q] [-o]

+++ OPTIONAL ARGUMENTS +++
  -H / --Help / -?: Displays help information.
  -Q / --Quiet: [Default false] Prevents application from issuing console messages.
  -O / --Overwrite: [Default false] Allows application to overwrite existing files.
  -D / --DestinationDirectory: [Default is './']. Sets the destination directory for extracted file(s).

For more information about Outpost 2 visit the Outpost Universe (http://outpost2.net/).


+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +


+++ INSTALL DIRECTIONS +++

Unzip OP2Archive.exe using WINZIP compatible decompression algorithm. The typical place to put OP2Archive.exe is in the root OP2 install directory for ease of use, but this is not required.

Open a command prompt. Navigate to the directory containing OP2Archive and type 'OP2Archive -h' to see Usage Message.


+++ ARCHIVE (.VOL/.CLM) FILE CREATION BEST PRACTICES +++

If you wish to overwrite a file currently in an archive file, use the ADD command and include the optional argument -O / --Overwrite to allow overwriting the original file.

The ADD and REMOVE command will create a new temp directory with the name ./OP2ArchiveTemp-(RANDOM INT) to store the contents of the archive file while rebuilding it. The ADD and REMOVE command will also eventually delete the original archive file before rebuilding it. If certain fatal exceptions occur during this process such as a power loss, the original archive file may be lost and/or the temp directory may not be deleted. If you are performing ADD and REMOVE commands on a heavily modifyied archive file, consider keeping a backup of the file somewhere. In case of a fatal error in the critical steps for the ADD and REMOVE coomands, you should be able to just recover the archive file from the Outpost Universe website by redownloading the game.


+++ CLM FILES (AUDIO STORAGE) +++

Outpost 2 stores all music tracks except for the track that plays on the main menu in the archive file op2.clm.

If you wish to change out the sound tracks in Outpost 2, you must use specific settings and use the WAV file format.

As a starting point for manipulating audio tracks for Outpost 2, consider starting with the free program Audacity (http://www.audacityteam.org/).

When naming audio files for storage in a CLM archive, the filename will be clipped down to 8 characters. Without siginificant modifications to the Outpost 2 application, you must use the names of the audio tracks provided with the stock download of the game to get modified or new tracks to play in game. The game will then select the music tracks as it sees appropriate based on name. So, the Eden tracks will play for Eden missions, etc.

Outpost 2 audio tracks must be formatted as WAV files with the following settings:
 * Frequency = 22050 Hz
 * Channels = 1 (Mono)
 * BitsPerSample = 16

The file size of each WAV audio track (the actual wav data) must be a multiple of 32768 bytes (0x8000). Output 2 fills the music buffer in chunks that large, and doesn't zero fill if a chunk (at the end of a file) is too short. If the file is not a multiple of 32768 bytes (0x8000), some garbage data will be played in Outpost 2 before switches to the next track. The audio data must be zero padded to bring it up to the right size before adding to the .clm file.


+++ COMPRESSION +++

Outpost 2 contains references to 3 types of compression, RLE (Run - Length Encoded), LZ (Lempel - Ziv), and LZH (Lempel - Ziv, with adaptive Huffman encoding).

Only LZH was used in the final release of Outpost 2. Only one archive file was compressed, sheets.vol. In subsequent releases of Outpost 2 by the Outpost Universe community, sheets.vol was decompressed and included in the game download in uncompressed format.

OP2Archive is capable of reading and decompressing archives using LZH compression. However, it currently cannot CREATE archives or modify via the ADD or REMOVE file an archive using LZH compression.


+++ SOURCE CODE LOCATION AND COMPILIATION +++

Source code may be found at: https://github.com/OutpostUniverse/OP2Archive. See the file DeveloperReadMe.txt in the source code for specific instructions.
« Last Edit: January 16, 2018, 10:34:37 AM by Vagabond »

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 684
Just released version 1.0.2 of OP2Archive, which consisted of bugfixes. See the link in the first post for downloading.

Version 1.0.2 (16 Jan 2018)

 * Fix a fatal crash when attempting to extract a directory from a volume file. (Extracting directories from volume files is not possible, but the program should not crash if attempted).

Version 1.0.1 (07 Nov 2017)

 * Add License.txt to project and downloads.
 * Update ReadMe.txt and DeveloperReadMe.txt to reference License.txt.
 * Improve Post Build Script to include zipping PDB file.

-Brett

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 684
Just released version 1.1.0 of OP2Archive.

This one changes behavior of created Archives by forcing files to be inserted into the archive alphabetically by filename. Outpost 2 was missing files placed inside the archive if they were not placed alphabetically. Outpost 2 could accept archives not in alphabetical order, but the current code base to write the vol files assumes they are alphabetical. It makes more sense to me anyways that they appear this way when searching through the archives. See link in first post for new download.

Version 1.0.1 (07 Nov 2017)

 * Bug Fix and Feature Change: Sort all filenames alphabetically during CREATE command.
 * Error Handling: Attempting to CREATE a repository that would contain 2 files with the same name results in an error message.
 * Error Handling: Attempting to EXTRACT a directory from an Archive results in an error message.
 * General code cleaning.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 684
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #3 on: February 03, 2018, 09:03:55 PM »
I realized the sort algorithm I developed for the CREATE command improperly sorts based on letter casing. It places any uppercase letter above any lowercase letter. I must have tested with only lowercase filenames. I developed stole from the internet and then modified a new sort function that ignores case.

I'll release version 1.1.1 soon with the fixed sort command. I'll wait a bit to see if any other changes are noted.

[RANT]This is why I like working in the .net framework. It provides handy functions like string sorting/comparison that I don't have to home roll and figure all the details out about casing or where it sorts letters with accents, or Japanese characters or which localization the sort is performed in or anything else related to the task of sorting that are outside my specialty but pertinent to a good sort. Instead in C++ I either internet search a library to integrate with that performs the task which I'm completely unfamiliar with or search for a single function, then sift through 3 solutions where the authors are arguing about which is more efficient. Guess that is the price of not being attached to the Microsoft ecosystem.[/RANT]



Goof was helping me learn how to add paths to the PATH environmental variable on Windows to allow calling OP2Archive from any directory within the command prompt. This is useful as I was pasting copies of OP2Archive across my file structure to manipulate vol files. However, I still have to place the Vol and CLM template files in the directories I'm working in to use the CREATE/ADD/REMOVE command.

A solution would be to hard code the location of the template files and setup a legitimate installer to handle this. The installer could also handle adding the correct path to the PATH environment variable automatically. And then remove the setting on unistall. I would probably use InnoSetup to create the installer as I've used it with good success in the past. Creating an installer adds a layer of complexity though.

Another way to solve the problem would be to try to eliminate the template files by either somehow storing the templates in the executable or maybe as a resource or something. Or perhaps modifying the Archive code to allow easier creation of vol and clm files without an initial template.

I don't get the impression anyone is currently using OP2Archive besides myself. But if anyone has an opinion on the matter, I wouldn't mind hearing it.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4396
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #4 on: February 03, 2018, 09:33:44 PM »
Outpost 2 searches the VOL index entries using binary search, comparing strings using strcmp, which is case sensitive. It does a simple binary compare of the bytes, without regards to locale or other special sort considerations. As such, lowercase letters, which have a lower value ASCII code, must be sorted before uppercase letters.

If a different sort order is used, the binary search can get confused when encountering an out of place entry and search the wrong direction. That can result in files not being found, even though they are clearly listed in the index table. And to make matters confusing, the way the binary search jumps around the index table, it can make which files are not found seem random (though it is deterministic for each set of packed files).

I'm afraid you have no choice but to sort all uppercase filenames before all lowercase filenames.

Edit:
Upon closer examination of the code, it seems the function was misidentified as "strcmp". It's actually not so simple. There is a memory flag that controls whether a simple case insensitive compare is used, or a or more complicated comparison is done, which looks like it might remap character codes. Maybe it's actually strcoll, or perhaps _stricmp. It appears the default is a simple case insensitive compare. If the characters are different, it attempts to remap upper case characters so they would compare equal to their lower case equivalents.

I suppose this means there are multiple possible sort orders, depending on the memory flag affecting the compare. That would of course completely break VOL files if it ever changed. As such, I'd go with a function that does a simple case insensitive compare.


Quote
Another way to solve the problem would be to try to eliminate the template files by either somehow storing the templates in the executable or maybe as a resource or something. Or perhaps modifying the Archive code to allow easier creation of vol and clm files without an initial template.

Agreed. These would be better solutions, particularly not needing template files.

I would recommend against using a hardcoded path, even with an installer, as people sometimes choose to install to a different path. Plus it's nice to not need an installer. Hardcoded absolute paths are almost always a bad idea.

Of course this problem stems from building on badly structured code that I wrote.  :-[
« Last Edit: February 03, 2018, 10:17:19 PM by Hooman »

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 684
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #5 on: February 04, 2018, 06:49:56 PM »
Quote
Upon closer examination of the code, it seems the function was misidentified as "strcmp". It's actually not so simple. There is a memory flag that controls whether a simple case insensitive compare is used, or a or more complicated comparison is done, which looks like it might remap character codes. Maybe it's actually strcoll, or perhaps _stricmp. It appears the default is a simple case insensitive compare. If the characters are different, it attempts to remap upper case characters so they would compare equal to their lower case equivalents.

I suppose this means there are multiple possible sort orders, depending on the memory flag affecting the compare. That would of course completely break VOL files if it ever changed. As such, I'd go with a function that does a simple case insensitive compare.

Thanks for putting time into researching this.

I think right now I am approximating an Invariant Culture (English language w/o an attached culture/nationality) search. However, it doesn't handle letters with accents properly. I could have OP2Archive refuse filenames containing characters past ASCII decimal 126. (https://en.wikipedia.org/wiki/ASCII). This would reduce the chances of anyone finding a way to break the sort with random untested characters. Probably good enough. I could document that it is an approximation in the readme as well.

Maybe a more correct solution would be exposing the Outpost 2 sort code and statically linking against it. Not sure this is really a possible solution though.

Quote
I would recommend against using a hardcoded path, even with an installer, as people sometimes choose to install to a different path. Plus it's nice to not need an installer. Hardcoded absolute paths are almost always a bad idea.

Hardcoded was a bad word choice. I meant I would put the location in a settings file/resource and allow the user to specify its location on initial install.

I want to learn more about resources, so I'm leaning towards storing the templates as resources attached to the executable and foregoing the installer.

Quote
Of course this problem stems from building on badly structured code that I wrote.

The code reconstructs an undocumented custom archive format that includes compression. And I was able to follow what was going on as a new C++ programmer. Maybe not perfect code, but I count this as a big win.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4396
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #6 on: February 07, 2018, 04:22:54 AM »
Quote
I think right now I am approximating an Invariant Culture (English language w/o an attached culture/nationality) search. However, it doesn't handle letters with accents properly. I could have OP2Archive refuse filenames containing characters past ASCII decimal 126. (https://en.wikipedia.org/wiki/ASCII). This would reduce the chances of anyone finding a way to break the sort with random untested characters. Probably good enough. I could document that it is an approximation in the readme as well.

From my brief debugging session, it didn't look like Outpost 2 tried to do anything special with accented characters. The code path that was actually used only took special consideration for the 26 uppercase characters.

Quote
Hardcoded was a bad word choice. I meant I would put the location in a settings file/resource and allow the user to specify its location on initial install.

And where would the settings file be located?  :P

I think the resource idea would work well. Might be able to use a MemmoryStream to load from a resource. Though ideally we should look at making the create function static. That could get a bit messy from what I remember of my code.