Author Topic: OP2Archive (Command line extraction and creation for .vol and .clm files)  (Read 10626 times)

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
OP2Archive is a 32 and 64 bit command line application for extracting, creating, and editing .vol and .clm archive files specific to Outpost 2. This tool allows for access to the in game music, maps, tilesets, datasheets, and more. Perfect if you wish to review or modify Outpost 2's internal data. Windows releases are provided, but the source code is fully compatible for Linux source code builds.

It is designed for easy use if you understand how to run command line (console) applications. Inside the download is a ReadMe.txt file that explains how to use it. If you have feedback or find bugs please post or message me.

OP2 Archive on Outpost Universe Wiki

Download Current Release: OP2Archive on GitHub

-Brett



ReadMe


OP2Archive - Outpost 2 Archive Access and Maintenance
Developed by Hooman and Brett208 (Vagabond)

Allows examining, creating, and extracting files from Outpost 2 .vol and .clm archives. Check the OP2Archive GitHub release page for the newest version at https://github.com/OutpostUniverse/OP2Archive/releases.

OP2Archive is released under a MIT license. See License.txt for details.

+++ COMMANDS +++
  * OP2Archive LIST (archivename.(vol|clm) | directory)...
    * Lists the contents and size of all files contained in provided archives.

  * OP2Archive FIND filename...
    * Determines which archive contains the file. Stops searching at first instance.
    * All archives in the same directory as the provided filename are searched.

  * OP2Archive CREATE archivename.(vol|clm) [filename | directory]... [-q] [-o]
    * If no filename(s) or directory(s) provided, creates an empty archive.

  * OP2Archive EXTRACT archivename.(vol|clm) [filename]... [-q] [-d destDirectory] [-o]
    * If no filename(s) provided, extracts entire contents of archive.

  * OP2Archive EXTRACT directoryname [-q] [-d destDirectory] [-o].
    * Extracts entire contents of all archives in the provided directory.

  * OP2Archive ADD archiveName.[vol|clm] filename... [-q] [-o]

  * OP2Archive REMOVE archiveName.[vol|clm] filename... [-q] [-o]

+++ OPTIONAL ARGUMENTS +++
  -H / --Help / -?: Displays help information.
  -Q / --Quiet: [Default false] Prevents application from issuing console messages.
  -O / --Overwrite: [Default false] Allows application to overwrite existing files.
  -D / --DestinationDirectory: [Default is './']. Sets the destination directory for extracted file(s).

For more information about Outpost 2 visit the Outpost Universe (http://outpost2.net/).


+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +


+++ INSTALL DIRECTIONS +++

Unzip OP2Archive.exe using a WINZIP compatible decompression algorithm. The typical place to put OP2Archive.exe is in the root OP2 install directory for ease of use, but this is not required.

Open a command prompt. Navigate to the directory containing OP2Archive and type 'OP2Archive -h' to see Usage Message.


+++ ARCHIVE (.VOL/.CLM) FILE CREATION BEST PRACTICES +++

If you wish to overwrite a file currently in an archive file, use the ADD command and include the optional argument -O / --Overwrite to allow overwriting the original file.

The ADD and REMOVE command will create a new temp directory with the name ./OP2ArchiveTemp-(RANDOM INT) to store the contents of the archive file while rebuilding it. The ADD and REMOVE command will also eventually delete the original archive file before rebuilding it. If certain fatal exceptions occur during this process such as a power loss, the original archive file may be lost and/or the temp directory may not be deleted. If you are performing ADD and REMOVE commands on a heavily modified archive file, consider keeping a backup of the file somewhere. For a stock archive, if a fatal error in a critical step of the ADD and REMOVE commands, you can recover the archive file from the Outpost Universe website by redownloading the game.


+++ CLM FILES (AUDIO STORAGE) +++

Outpost 2 stores all music tracks except for the main menu track in the archive file op2.clm.

If you wish to change the sound tracks in Outpost 2, you must use the WAV file format with specific settings.

As a starting point for manipulating audio tracks for Outpost 2, consider the free program Audacity (http://www.audacityteam.org/).

When naming audio files for storage in a CLM archive, the filename (excluding extension) must be 8 characters or less. The names of the audio tracks provided with the stock download of the game must be used. The game will then select the music tracks as it sees appropriate based on name.

Outpost 2 audio tracks must be formatted as WAV files with the following settings:
 * Frequency = 22050 Hz
 * Channels = 1 (Mono)
 * BitsPerSample = 16

The file size of each WAV audio track (the actual wav data) must be zero padded to a multiple of 32768 bytes (0x8000). Output 2 fills the music buffer in 0x8000 size chunks and will not zero fill if a chunk at the end of a file is too short. If the file is not a multiple of 32768 bytes (0x8000), some garbage data will be played in Outpost 2 before switches to the next track.


+++ COMPRESSION +++

Outpost 2 contains references to 3 types of compression, RLE (Run - Length Encoded), LZ (Lempel - Ziv), and LZH (Lempel - Ziv, with adaptive Huffman encoding).

Only LZH was used in the final release of Outpost 2. Only one archive file was compressed, sheets.vol. In subsequent releases of Outpost 2 by the Outpost Universe community, sheets.vol was decompressed and included in the game download in uncompressed format.

OP2Archive is capable of reading and decompressing archives using LZH compression. However, it currently cannot CREATE archives or modify via the ADD or REMOVE an archive using LZH compression.


+++ SOURCE CODE LOCATION AND COMPILIATION +++

Source code may be found at: https://github.com/OutpostUniverse/OP2Archive. See the file DeveloperReadMe.txt in the source code for specific instructions.


+++ CHANGE LOG +++

Ver 1.2.2
 * Bug Fix: Make the ADD and REMOVE command work again.
 * Bug Fix: Allow the CREATE command to work when adding the contents of a directory to a new volume.
 * Bug Fix: Prevent adding a volume to itself.
 * Bug Fix: Make error messages show filename when problems arise parsing files for inclusion in a CLM.

Ver 1.2.1
 * Bug Fix: Allow creating archives in new directories
 * Bug Fix: Stop attempting to open directories that appear to have proper extensions (IE a directory named archive.vol)
 * Bug Fix: Stop attempting to package subdirectories into an archive (The subdirectory itself was being added, not the contents of the subdirectory)
 * Numerous other directory traversal fixes brought in from updated OP2Utility

Ver 1.2.0
 * Provide both an x86 and x64 Windows build.
 * Remove Windows specific dependencies from code base (may now be compiled for use on Linux).
 * Remove .clm and .vol template files from application download (no longer required).
 * Bug Fix: Allow providing filenames out of alphabetical order when CREATE is used.
 * Bug Fix: Align the internal name sorting algorithm to align with the Outpost 2 binary search.
 * Bug Fix: Reject filenames that contain non-ASCII characters (prevents sort discrepancies with Outpost 2 binary search).
 * Generally improve error handling messaging.
 * Rewrite sections of the ReadMe.

Ver 1.1.1
 * Bug Fix: During CREATE command, sort all filenames case insensitive.

Ver 1.1.0
 * Bug Fix and Feature Change: Sort all filenames alphabetically during CREATE command.
 * Error Handling: Attempting to CREATE a repository that would contain 2 files with the same name results in an error message.
 * Error Handling: Attempting to EXTRACT a directory from an Archive results in an error message.
 * General code cleaning.

Ver 1.0.2
 * Fix bug where program crashes when attempting to extract a directory from a vol file.

Ver 1.0.1 (07Nov2017)
 * Added MIT License to project in file Licence.txt.

Ver 1.0.0 (07Oct2017)
 * Initial Release
« Last Edit: November 16, 2019, 08:11:26 PM by Vagabond »

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Just released version 1.0.2 of OP2Archive, which consisted of bugfixes. See the link in the first post for downloading.

Version 1.0.2 (16 Jan 2018)

 * Fix a fatal crash when attempting to extract a directory from a volume file. (Extracting directories from volume files is not possible, but the program should not crash if attempted).

Version 1.0.1 (07 Nov 2017)

 * Add License.txt to project and downloads.
 * Update ReadMe.txt and DeveloperReadMe.txt to reference License.txt.
 * Improve Post Build Script to include zipping PDB file.

-Brett

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Just released version 1.1.0 of OP2Archive.

This one changes behavior of created Archives by forcing files to be inserted into the archive alphabetically by filename. Outpost 2 was missing files placed inside the archive if they were not placed alphabetically. Outpost 2 could accept archives not in alphabetical order, but the current code base to write the vol files assumes they are alphabetical. It makes more sense to me anyways that they appear this way when searching through the archives. See link in first post for new download.

Version 1.0.1 (07 Nov 2017)

 * Bug Fix and Feature Change: Sort all filenames alphabetically during CREATE command.
 * Error Handling: Attempting to CREATE a repository that would contain 2 files with the same name results in an error message.
 * Error Handling: Attempting to EXTRACT a directory from an Archive results in an error message.
 * General code cleaning.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #3 on: February 03, 2018, 10:03:55 PM »
I realized the sort algorithm I developed for the CREATE command improperly sorts based on letter casing. It places any uppercase letter above any lowercase letter. I must have tested with only lowercase filenames. I developed stole from the internet and then modified a new sort function that ignores case.

I'll release version 1.1.1 soon with the fixed sort command. I'll wait a bit to see if any other changes are noted.

[RANT]This is why I like working in the .net framework. It provides handy functions like string sorting/comparison that I don't have to home roll and figure all the details out about casing or where it sorts letters with accents, or Japanese characters or which localization the sort is performed in or anything else related to the task of sorting that are outside my specialty but pertinent to a good sort. Instead in C++ I either internet search a library to integrate with that performs the task which I'm completely unfamiliar with or search for a single function, then sift through 3 solutions where the authors are arguing about which is more efficient. Guess that is the price of not being attached to the Microsoft ecosystem.[/RANT]



Goof was helping me learn how to add paths to the PATH environmental variable on Windows to allow calling OP2Archive from any directory within the command prompt. This is useful as I was pasting copies of OP2Archive across my file structure to manipulate vol files. However, I still have to place the Vol and CLM template files in the directories I'm working in to use the CREATE/ADD/REMOVE command.

A solution would be to hard code the location of the template files and setup a legitimate installer to handle this. The installer could also handle adding the correct path to the PATH environment variable automatically. And then remove the setting on unistall. I would probably use InnoSetup to create the installer as I've used it with good success in the past. Creating an installer adds a layer of complexity though.

Another way to solve the problem would be to try to eliminate the template files by either somehow storing the templates in the executable or maybe as a resource or something. Or perhaps modifying the Archive code to allow easier creation of vol and clm files without an initial template.

I don't get the impression anyone is currently using OP2Archive besides myself. But if anyone has an opinion on the matter, I wouldn't mind hearing it.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #4 on: February 03, 2018, 10:33:44 PM »
Outpost 2 searches the VOL index entries using binary search, comparing strings using strcmp, which is case sensitive. It does a simple binary compare of the bytes, without regards to locale or other special sort considerations. As such, lowercase letters, which have a lower value ASCII code, must be sorted before uppercase letters.

If a different sort order is used, the binary search can get confused when encountering an out of place entry and search the wrong direction. That can result in files not being found, even though they are clearly listed in the index table. And to make matters confusing, the way the binary search jumps around the index table, it can make which files are not found seem random (though it is deterministic for each set of packed files).

I'm afraid you have no choice but to sort all uppercase filenames before all lowercase filenames.

Edit:
Upon closer examination of the code, it seems the function was misidentified as "strcmp". It's actually not so simple. There is a memory flag that controls whether a simple case insensitive compare is used, or a or more complicated comparison is done, which looks like it might remap character codes. Maybe it's actually strcoll, or perhaps _stricmp. It appears the default is a simple case insensitive compare. If the characters are different, it attempts to remap upper case characters so they would compare equal to their lower case equivalents.

I suppose this means there are multiple possible sort orders, depending on the memory flag affecting the compare. That would of course completely break VOL files if it ever changed. As such, I'd go with a function that does a simple case insensitive compare.


Quote
Another way to solve the problem would be to try to eliminate the template files by either somehow storing the templates in the executable or maybe as a resource or something. Or perhaps modifying the Archive code to allow easier creation of vol and clm files without an initial template.

Agreed. These would be better solutions, particularly not needing template files.

I would recommend against using a hardcoded path, even with an installer, as people sometimes choose to install to a different path. Plus it's nice to not need an installer. Hardcoded absolute paths are almost always a bad idea.

Of course this problem stems from building on badly structured code that I wrote.  :-[
« Last Edit: February 03, 2018, 11:17:19 PM by Hooman »

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #5 on: February 04, 2018, 07:49:56 PM »
Quote
Upon closer examination of the code, it seems the function was misidentified as "strcmp". It's actually not so simple. There is a memory flag that controls whether a simple case insensitive compare is used, or a or more complicated comparison is done, which looks like it might remap character codes. Maybe it's actually strcoll, or perhaps _stricmp. It appears the default is a simple case insensitive compare. If the characters are different, it attempts to remap upper case characters so they would compare equal to their lower case equivalents.

I suppose this means there are multiple possible sort orders, depending on the memory flag affecting the compare. That would of course completely break VOL files if it ever changed. As such, I'd go with a function that does a simple case insensitive compare.

Thanks for putting time into researching this.

I think right now I am approximating an Invariant Culture (English language w/o an attached culture/nationality) search. However, it doesn't handle letters with accents properly. I could have OP2Archive refuse filenames containing characters past ASCII decimal 126. (https://en.wikipedia.org/wiki/ASCII). This would reduce the chances of anyone finding a way to break the sort with random untested characters. Probably good enough. I could document that it is an approximation in the readme as well.

Maybe a more correct solution would be exposing the Outpost 2 sort code and statically linking against it. Not sure this is really a possible solution though.

Quote
I would recommend against using a hardcoded path, even with an installer, as people sometimes choose to install to a different path. Plus it's nice to not need an installer. Hardcoded absolute paths are almost always a bad idea.

Hardcoded was a bad word choice. I meant I would put the location in a settings file/resource and allow the user to specify its location on initial install.

I want to learn more about resources, so I'm leaning towards storing the templates as resources attached to the executable and foregoing the installer.

Quote
Of course this problem stems from building on badly structured code that I wrote.

The code reconstructs an undocumented custom archive format that includes compression. And I was able to follow what was going on as a new C++ programmer. Maybe not perfect code, but I count this as a big win.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #6 on: February 07, 2018, 05:22:54 AM »
Quote
I think right now I am approximating an Invariant Culture (English language w/o an attached culture/nationality) search. However, it doesn't handle letters with accents properly. I could have OP2Archive refuse filenames containing characters past ASCII decimal 126. (https://en.wikipedia.org/wiki/ASCII). This would reduce the chances of anyone finding a way to break the sort with random untested characters. Probably good enough. I could document that it is an approximation in the readme as well.

From my brief debugging session, it didn't look like Outpost 2 tried to do anything special with accented characters. The code path that was actually used only took special consideration for the 26 uppercase characters.

Quote
Hardcoded was a bad word choice. I meant I would put the location in a settings file/resource and allow the user to specify its location on initial install.

And where would the settings file be located?  :P

I think the resource idea would work well. Might be able to use a MemmoryStream to load from a resource. Though ideally we should look at making the create function static. That could get a bit messy from what I remember of my code.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #7 on: November 04, 2018, 06:38:05 AM »
Version 1.2.0 of OP2Archive has been released! It took a year to get this release out. That wasn't expected.

This release includes both a x64 and win32 (x86) version. Also, the code base is completely compatible with Linux. There isn't a Linux release, but it should be easy to compile if desired. The vol and clm template file are no longer required.

The new release can be found at: https://github.com/OutpostUniverse/OP2Archive/releases

Changelog:

Version 1.2.0 (04 Nov 2018)

  • Provide both an x86 and x64 Windows build.
  • Remove Windows specific dependencies from code base (may now be compiled for use on Linux).
  • Remove .clm and .vol template files from application download (no longer required).
  • Bug Fix: Allow providing filenames out of alphabetical order when CREATE is used.
  • Bug Fix: Align the internal name sorting algorithm to align with the Outpost 2 binary search.
  • Bug Fix: Reject filenames that contain non-ASCII characters (prevents sort discrepancies with Outpost 2 binary search).
  • Generally improve error handling messaging.
  • Rewrite sections of the ReadMe.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #8 on: November 04, 2018, 11:34:38 PM »
I just gave the update a try on Linux. It compiles painlessly with a single command, and the built in help makes it quite intuitive to use. No path issues when working on archives in a different directory. I'm quite pleased with the result.

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #9 on: November 05, 2018, 05:04:49 PM »
Hooman,

Thanks for the final Linux test!

If anyone is using OP2Archive, I'd appreciate hearing feedback for better or worse!

-Brett

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Hey everyone,

I finally finished version 1.2.1 of OP2Archive. This release was driven by many direcotry traversal bugs. The download link is in the first post. As usual it is posted on GitHub.

Thanks to Hooman for helping track, document, fix, and write unit tests for the directory bugs in OP2Utility, which made this release possible.

Version 1.2.1 (19 Aug 2019)

  • Bug Fix: Allow creating archives in new directories
  • Bug Fix: Stop attempting to open directories that appear to have proper extensions (IE a directory named archive.vol)
  • Bug Fix: Stop attempting to package subdirectories into an archive (The subdirectory itself was being added, not the contents of the subdirectory)
  • Numerous other directory traversal fixes brought in from updated OP2Utility


Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: OP2Archive (Command line extraction and creation for .vol and .clm files)
« Reply #11 on: November 16, 2019, 08:10:49 PM »
Another bug fix release for OP2Archive. The ADD and REMOVE commands had some pretty big bugs that are now cleaned up among some minor improvements. See the top post with the Github release link if interested in trying out!

Ver 1.2.2
 * Bug Fix: Make the ADD and REMOVE command work again.
 * Bug Fix: Allow the CREATE command to work when adding the contents of a directory to a new volume.
 * Bug Fix: Prevent adding a volume to itself.
 * Bug Fix: Make error messages show filename when problems arise parsing files for inclusion in a CLM.