Author Topic: OP2Archive Application Development  (Read 196 times)

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 388
OP2Archive Application Development
« on: August 04, 2017, 05:34:24 PM »
I was thinking we should create a finished version of a console application that allows for manipulation of Outpost 2 Archive files. We have the GUIs Vol Creator and Vol Extractor that work (http://wiki.outpost2.net/doku.php?id=outpost_2:helper_programs). Unfortunately, as they have aged they now require registering some components with newer versions of the Windows operating system that are not present by default. They also do not work with the clm file (I think they cannot anyways?). Also, the code for these two GUIs does not appear to be in the repository.

As an added benefit, we may eventually be able to get the application to compile for Linux or other platforms.

I was thinking a console application because they are simple to program and usually age very well. I'm also not experienced with GUI work in C++ which would take me a really long time to spool up on.

We have a console application called VolDecompress that Hooman made, which works well for extracting archive files. I was thinking OP2Archive might be a better name for this application since it will do more than extracting. Hooman has already written the code to extract, create, and modify both .vol files and .clm files.



Command line example convention syntax: https://www.ibm.com/support/knowledgecenter/SSZJPZ_8.5.0/com.ibm.swg.im.iis.common.doc/common/command_conventions.html

Below are the console commands I was thinking it should support:

OP2Archive CONTENTS archiveName.[vol|clm]
  * Lists the contents of an archive file.

OP2Archive FIND filename
  * Lists the archive filename that contains the specified file. Or lets you know the file is not archived.

OP2Archive CREATE archiveName.[vol|clm] [filename...]
  * Creates an archive with the given name. Adds the listed files to the archive. Also allows creating an empty archive.

OP2Archive ADD archiveName.[vol|clm] filename...
  * Adds the specified file to the archive.

OP2Archive REMOVE archiveName.[vol|clm] filename...
  * Removes the specified file from the archive.

OP2Archvie EXTRACT archiveName.[vol|clm]... [filename]...
  * Extracts the specified file from the archive. If no filenames provided, extracts the entire archive.



File Whitelisting

I was thinking we could whitelist file types that are supported by Outpost 2 loading from a given archive type. (It looks like maybe anything could be stuffed in a .vol file and loaded by Outpost 2, so I'm a bit on the fence about this). I don't want to get too wrapped up in checking files though. In general the user will need to be familiar with what is appropriate and not to stuff in the archives.

 * .vol file include whitelist: .bmp .map .prt .raw .txt .wav .rtf
 * .clm file include whitelist: .wav



The initial program will be written in C++ using VS2017.

Any thoughts or additions to this are appreciated. I haven't started any actual coding yet, so it is easy to change. : )

Once this is done, we can look at removing windows specific code from the OP2Utility library archive section and test the results to ensure everything is working properly.

-Brett
« Last Edit: August 04, 2017, 05:37:48 PM by Vagabond »

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 1534
    • LairWorks Entertainment
Re: OP2Archive Application Development
« Reply #1 on: August 04, 2017, 11:38:00 PM »
GUI work in C++ is extremely painful. It would be better to build a console app in C++ (as you suggested) and then, if demand exists, develop a GUI front-end in either C# or VB.Net.

The old programs didn't age well because they were written in VB. I'm all for replacing them with better open-source programs.

Want to state again that all of your work is very much appreciated!
- Leeor
LairWorks Entertainment

Titanum UFO's

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3857
Re: OP2Archive Application Development
« Reply #2 on: August 05, 2017, 07:48:05 PM »
I would love to have a console application like this that could run on Linux.


The whitelist makes sense for the CLM files. The CLM files are only really useful for WAV data, and all data must have the same header parameters. The header is stored once in the CLM. The WAV data is stored in a stripped form.

The VOL files are more generic, and can basically store any kind of data. Though with the way Outpost 2 works, it wouldn't make sense to store the DLLs inside the VOL files, since the Windows loader wouldn't be able to access them there, and that's needed for the levels to load and be playable. They might show up in a list of available games if they were in a VOL, but they wouldn't run. Hence the VOL files might have a blacklist for DLL files. Not sure if you'd really want to enforce this though, or maybe just give a warning higher up in the user interface. Nothing wrong with transporting a DLL in a VOL file, though there is no reason to do this.


Something I've learned from experience, the ADD and REMOVE functionality is a pain to implement in an efficient way, and often not very useful in practice. It might be nice to have them for a sense of completeness, but I think they should get lower implementation priority. The issue is that added files means updating the VOL header, which might cause expansion of the header if there isn't sufficient blank space, and so you're left copying the contents to a later point in the file to make space. In effect, it becomes a CREATE operation, where some of the source files are an existing VOL file. Similarly, if you want to remove a file and don't want dead space in the VOL header, you're again copying data back towards the front, again effectively doing a CREATE operation. Hence why I figure the CREATE and EXTRACT methods are the most useful, and you can get away with just those.

Might consider renaming CONTENTS to LIST. I think that would be more standard.

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 388
Re: OP2Archive Application Development
« Reply #3 on: August 08, 2017, 06:24:07 AM »
Leeor, thank you for the encouragement! Hooman, Thank you for the input!

Quote
The whitelist makes sense for the CLM files. The CLM files are only really useful for WAV data, and all data must have the same header parameters. The header is stored once in the CLM. The WAV data is stored in a stripped form.

Sounds good. For the first iteration I think we can just cerr if something besides a .wav file is attempted to be added to .clm. Maybe work up to actually error checking the contents of the .wav file if we continue far enough with the project.

Quote
The VOL files are more generic, and can basically store any kind of data. Though with the way Outpost 2 works, it wouldn't make sense to store the DLLs inside the VOL files

I agree. We can put a note in the Readme about the problems of storing dlls in the .vol file.



Quote
Something I've learned from experience, the ADD and REMOVE functionality is a pain to implement in an efficient way, and often not very useful in practice. It might be nice to have them for a sense of completeness, but I think they should get lower implementation priority. The issue is that added files means updating the VOL header, which might cause expansion of the header if there isn't sufficient blank space, and so you're left copying the contents to a later point in the file to make space. In effect, it becomes a CREATE operation, where some of the source files are an existing VOL file. Similarly, if you want to remove a file and don't want dead space in the VOL header, you're again copying data back towards the front, again effectively doing a CREATE operation. Hence why I figure the CREATE and EXTRACT methods are the most useful, and you can get away with just those.

Good to know. For modifying and creating archive files, I think we could implement what you are saying by encouraging the workflow below:

  • Use OP2Archive to extract all contents of an archive file to a directory. Extracting all contents could default to creating a relative directory with the name of the archive file. IE ./sheets or ./maps01. You could change the default extract directory with a -d or --DestinationDirectory
  • Manually outside of OP2Archive add/delete the files from the new relative folder as required.
  • Use OP2Archive to re-create the archive by providing the directory where all the files are stored. OP2Archive will archive all files in the provided directory. It would still be available to create an archive by specifying each file manually if desired.

What if for ADD/REMOVE we:

  • Extract all contents to a temp directory. Maybe OP2ArchiveTemp[GUID].
  • ADD/REMOVE files for temp folder as directed.
  • Delete the original archive file.
  • Recreate the archive.
  • Delete the temp folder.

The downsides to this approach is that a CREATE/ADD command are going to be more expensive than one would intuitively think and if an unhandled exception occurs the temp folder may remain or the original archive file be deleted and gone. Hmm, I could see this maybe not being a good idea to implement.



Optional Arguments

I was thinking about the following optional arguments

    -H / --Help / -?: Displays Help File
    -Q / --Quiet: [Default false] Add switch to run application without issuing console messages. (Not available for LIST/CONTENTS)
    -O / --Overwrite: [Default false] Add switch to allow application to overwrite existing files. (Not available for LIST/CONTENTS)
    -C / --Compression: [Default None]. Allows None|JPG|BMP. Sets the compression alghorithim used in archive. (Only available for CREATE.)
    -D / --DefaultDirectory: [Default to a directory called archiveName for extraction of an entire archive file or the current working directory for a single file]. Allows changing where the extracted contents of an archive file (either all contents or a single file) are placed.



Help / Usage Statements

We can have a short initial usage statement and then have the user type -h on the given command for details on the command. For example, 'OP2MapImager CREATE -h' for help on the CREATE command.



Format of LIST command

I was thinking about the following format for the LIST command. I think size might be the compressed size of the file inside the archive if compression is used instead of the uncompressed size for ease of implementation? It would be the results of ArchiveUnpacker::GetInternalFileSize. Either way, it would be useful for seeing the relative size of each file.

Contents of maps01.vol (12 files)
ID  NAME          SIZE (KB)
--------------------------------------------------
00  blue.bmp      x,xxx
01  color.bmp     xxx,xxx
02  eden04.map    xxx
03  op2_art.prt   xxx
04  tutorial.map  xxx
05  virmask.raw   xxx
06  well0000.bmp  xxx




Quote
Might consider renaming CONTENTS to LIST. I think that would be more standard.

I was on the fence about this. Since you are mentioning it, probably best to go with LIST.
« Last Edit: August 09, 2017, 04:36:08 AM by Vagabond »

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 388
Re: OP2Archive Application Development
« Reply #4 on: August 09, 2017, 04:39:29 AM »
Quick C++ memory management question.

Can I write the code below, or will the created VolFile become a memory leak? The underlying function listContents requires a pointer to a VolFile class.

Code: [Select]
if (XFile::extensionMatches(filename, ".vol"))
archiveConsoleListing.listContents(new VolFile(filename.c_str()));

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3857
Re: OP2Archive Application Development
« Reply #5 on: August 09, 2017, 06:56:36 AM »
Quote
Sounds good. For the first iteration I think we can just cerr if something besides a .wav file is attempted to be added to .clm. Maybe work up to actually error checking the contents of the .wav file if we continue far enough with the project.

The existing CLM packing code will already error on this condition. The packing code will try to load the WAVE header from all input files, which will error out if they are not proper .wav files. It also then compares the headers to make sure they are all the same format, and errors out if they are not.


To repack an archive, you can read the files right out of the old archive, same as the game can, while writing them into a new temporary file. Once the new file is packed, you'd want to do an atomic rename to the original filename. If the OS supports atomic renames, the original filename should at all times remain valid, and either point to the original archive, or the new fully packed and ready archive. Should be safe. Failing that, a quick delete and rename at the end is unlikely to be caught, and if the operation is interrupted after the delete, but before the rename, the user could always just complete that operation manually themselves. Again, that's fairly safe.

Quote
I think size might be the compressed size of the file inside the archive if compression is used instead of the uncompressed size for ease of implementation? It would be the results of ArchiveUnpacker::GetInternalFileSize.

I agree. The VOL structure doesn't make it easy to get the unpacked size without actually doing the decompression. They could have made it easy by using the unpacked size in the index entry, and the VBLK size as the packed size, but they didn't. They are both the packed size.

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 388
Re: OP2Archive Application Development
« Reply #6 on: August 10, 2017, 04:30:50 AM »
Quote
The existing CLM packing code will already error on this condition. The packing code will try to load the WAVE header from all input files, which will error out if they are not proper .wav files. It also then compares the headers to make sure they are all the same format, and errors out if they are not.

Perfect. I'll just display the error uses cerr for the user.


I'll upload my progress shortly. It will live under the GameResources directory in the repository. So far, I've hard coded the commands LIST (earlier CONTENTS) and FIND. Next step will be to design the command argument parser and hook LIST and FIND to command arguments. I'll base it off the code I wrote for OP2MapImager. The usage statement is also partially written.

I was surprised how easy it was to do formatting with cout. It breaks the rule that most everything is harder with C++. :)



Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 1534
    • LairWorks Entertainment
Re: OP2Archive Application Development
« Reply #7 on: August 10, 2017, 01:47:48 PM »
A lot of the new features in C++11 have made C++ considerably easier to use. I really love the new additions... not that this affects what you did but still. ;D

Anyway, I'm looking forward to having a CLI tool to do the packing/unpacking. I had to extract some map and graphics files and the VOLExtractor program is reasonably okay but it requires VB6 OCX files and I was somewhat unhappy that I had to register it as a COM object in the registry. Super not happy about that. :-\

So anyway yeah, I'm very much looking forward to that.

Separate note -- maybe now is a good time to start migrating some of the code from SVN to Git? I've started a new project forked from another one of mine and uploaded it to the OPU GitHub repository.
- Leeor
LairWorks Entertainment

Titanum UFO's

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3857
Re: OP2Archive Application Development
« Reply #8 on: August 12, 2017, 02:28:12 AM »
Quote
Separate note -- maybe now is a good time to start migrating some of the code from SVN to Git?

I was thinking the same thing. I'm enjoying the experience with GitHub. Let me know if you'd like help with moving things. I think git-svn can be used to make the transition easier, while preserving history.


Nice looking output. Does it handle long filenames well? Do long filenames show in full, or get truncated?

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 1534
    • LairWorks Entertainment
Re: OP2Archive Application Development
« Reply #9 on: August 12, 2017, 03:22:42 AM »
Quote
Separate note -- maybe now is a good time to start migrating some of the code from SVN to Git?

I was thinking the same thing. I'm enjoying the experience with GitHub. Let me know if you'd like help with moving things. I think git-svn can be used to make the transition easier, while preserving history.

There's a few things that we need to pull out of the SVN before we can move it over (namely the gamerelease directory) but otherwise git-svn should be fairly straight forward.

End Thread Hijack
- Leeor
LairWorks Entertainment

Titanum UFO's

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 388
Re: OP2Archive Application Development
« Reply #10 on: August 13, 2017, 08:08:12 AM »
Code: [Select]
Nice looking output. Does it handle long filenames well? Do long filenames show in full, or get truncated?

Thanks Hooman. Really long file names currently jack up the formatting. I'll try to fix this in the near future by setting a max filename character length before ignoring the length on the filename column width. This way really long filenames will only mess up their row on the table and none of the others.


I've programmed the LIST and FIND commands to accept command line arguments. They seem pretty robust. I'm partially through the EXTRACT command.

Below is a draft of the usage statement. I'm trying to go for a POSIX syntax. Let me know if it is confusing or out of line. () means required and [] means optional and ... means multiple allowed.


-Brett

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 388
Re: OP2Archive Application Development
« Reply #11 on: August 13, 2017, 06:01:18 PM »
Just uploaded code to the repository. I fixed the LIST command to play nicely(ish) with too long of filenames. See screenshot for what I mean. Basically, the normal list will cap out at 35 character length for the filename including extension. If the filename length is greater than 35 characters, it allows the longer filename to extend beyond the width of the file size column without affecting the rest of the table.

I think this should suffice unless anyone has better ideas on the matter. I would prefer if no one uses filenames that are so large though for my own sanity.  ;)


Also, I feel like the smiley face choices change every other time I post. I am with Hooman that they are better not being animated.

Back to work on the EXTRACT command...

-Brett