OP2Archive Application Development

[Vagabond]

I was thinking we should create a finished version of a console application that allows for manipulation of Outpost 2 Archive files. We have the GUIs Vol Creator and Vol Extractor that work (http://wiki.outpost2.net/doku.php?id=outpost_2:helper_programs). Unfortunately, as they have aged they now require registering some components with newer versions of the Windows operating system that are not present by default. They also do not work with the clm file (I think they cannot anyways?). Also, the code for these two GUIs does not appear to be in the repository.

As an added benefit, we may eventually be able to get the application to compile for Linux or other platforms.

I was thinking a console application because they are simple to program and usually age very well. I'm also not experienced with GUI work in C++ which would take me a really long time to spool up on.

We have a console application called VolDecompress that Hooman made, which works well for extracting archive files. I was thinking OP2Archive might be a better name for this application since it will do more than extracting. Hooman has already written the code to extract, create, and modify both .vol files and .clm files.


Command line example convention syntax: https://www.ibm.com/support/knowledgecenter/SSZJPZ_8.5.0/com.ibm.swg.im.iis.common.doc/common/command_conventions.html

Below are the console commands I was thinking it should support:

OP2Archive CONTENTS archiveName.[vol|clm]
  * Lists the contents of an archive file.

OP2Archive FIND filename
  * Lists the archive filename that contains the specified file. Or lets you know the file is not archived.

OP2Archive CREATE archiveName.[vol|clm] [filename...]
  * Creates an archive with the given name. Adds the listed files to the archive. Also allows creating an empty archive.

OP2Archive ADD archiveName.[vol|clm] filename...
  * Adds the specified file to the archive.

OP2Archive REMOVE archiveName.[vol|clm] filename...
  * Removes the specified file from the archive.

OP2Archvie EXTRACT archiveName.[vol|clm]... [filename]...
  * Extracts the specified file from the archive. If no filenames provided, extracts the entire archive.


File Whitelisting

I was thinking we could whitelist file types that are supported by Outpost 2 loading from a given archive type. (It looks like maybe anything could be stuffed in a .vol file and loaded by Outpost 2, so I'm a bit on the fence about this). I don't want to get too wrapped up in checking files though. In general the user will need to be familiar with what is appropriate and not to stuff in the archives.

 * .vol file include whitelist: .bmp .map .prt .raw .txt .wav .rtf
 * .clm file include whitelist: .wav


The initial program will be written in C++ using VS2017.

Any thoughts or additions to this are appreciated. I haven't started any actual coding yet, so it is easy to change. : )

Once this is done, we can look at removing windows specific code from the OP2Utility library archive section and test the results to ensure everything is working properly.

-Brett

[leeor_net]

GUI work in C++ is extremely painful. It would be better to build a console app in C++ (as you suggested) and then, if demand exists, develop a GUI front-end in either C# or VB.Net.

The old programs didn't age well because they were written in VB. I'm all for replacing them with better open-source programs.

Want to state again that all of your work is very much appreciated!

[Hooman]

I would love to have a console application like this that could run on Linux.


The whitelist makes sense for the CLM files. The CLM files are only really useful for WAV data, and all data must have the same header parameters. The header is stored once in the CLM. The WAV data is stored in a stripped form.

The VOL files are more generic, and can basically store any kind of data. Though with the way Outpost 2 works, it wouldn't make sense to store the DLLs inside the VOL files, since the Windows loader wouldn't be able to access them there, and that's needed for the levels to load and be playable. They might show up in a list of available games if they were in a VOL, but they wouldn't run. Hence the VOL files might have a blacklist for DLL files. Not sure if you'd really want to enforce this though, or maybe just give a warning higher up in the user interface. Nothing wrong with transporting a DLL in a VOL file, though there is no reason to do this.


Something I've learned from experience, the ADD and REMOVE functionality is a pain to implement in an efficient way, and often not very useful in practice. It might be nice to have them for a sense of completeness, but I think they should get lower implementation priority. The issue is that added files means updating the VOL header, which might cause expansion of the header if there isn't sufficient blank space, and so you're left copying the contents to a later point in the file to make space. In effect, it becomes a CREATE operation, where some of the source files are an existing VOL file. Similarly, if you want to remove a file and don't want dead space in the VOL header, you're again copying data back towards the front, again effectively doing a CREATE operation. Hence why I figure the CREATE and EXTRACT methods are the most useful, and you can get away with just those.

Might consider renaming CONTENTS to LIST. I think that would be more standard.

[Vagabond]

Leeor, thank you for the encouragement! Hooman, Thank you for the input!

The whitelist makes sense for the CLM files. The CLM files are only really useful for WAV data, and all data must have the same header parameters. The header is stored once in the CLM. The WAV data is stored in a stripped form.

Sounds good. For the first iteration I think we can just cerr if something besides a .wav file is attempted to be added to .clm. Maybe work up to actually error checking the contents of the .wav file if we continue far enough with the project.

The VOL files are more generic, and can basically store any kind of data. Though with the way Outpost 2 works, it wouldn't make sense to store the DLLs inside the VOL files

I agree. We can put a note in the Readme about the problems of storing dlls in the .vol file.

Something I've learned from experience, the ADD and REMOVE functionality is a pain to implement in an efficient way, and often not very useful in practice. It might be nice to have them for a sense of completeness, but I think they should get lower implementation priority. The issue is that added files means updating the VOL header, which might cause expansion of the header if there isn't sufficient blank space, and so you're left copying the contents to a later point in the file to make space. In effect, it becomes a CREATE operation, where some of the source files are an existing VOL file. Similarly, if you want to remove a file and don't want dead space in the VOL header, you're again copying data back towards the front, again effectively doing a CREATE operation. Hence why I figure the CREATE and EXTRACT methods are the most useful, and you can get away with just those.

Good to know. For modifying and creating archive files, I think we could implement what you are saying by encouraging the workflow below:

• Use OP2Archive to extract all contents of an archive file to a directory. Extracting all contents could default to creating a relative directory with the name of the archive file. IE ./sheets or ./maps01. You could change the default extract directory with a -d or --DestinationDirectory
• Manually outside of OP2Archive add/delete the files from the new relative folder as required.
• Use OP2Archive to re-create the archive by providing the directory where all the files are stored. OP2Archive will archive all files in the provided directory. It would still be available to create an archive by specifying each file manually if desired.

What if for ADD/REMOVE we:

• Extract all contents to a temp directory. Maybe OP2ArchiveTemp[GUID].
• ADD/REMOVE files for temp folder as directed.
• Delete the original archive file.
• Recreate the archive.
• Delete the temp folder.

The downsides to this approach is that a CREATE/ADD command are going to be more expensive than one would intuitively think and if an unhandled exception occurs the temp folder may remain or the original archive file be deleted and gone. Hmm, I could see this maybe not being a good idea to implement.


Optional Arguments

I was thinking about the following optional arguments

    -H / --Help / -?: Displays Help File
    -Q / --Quiet: [Default false] Add switch to run application without issuing console messages. (Not available for LIST/CONTENTS)
    -O / --Overwrite: [Default false] Add switch to allow application to overwrite existing files. (Not available for LIST/CONTENTS)
    -C / --Compression: [Default None]. Allows None|JPG|BMP. Sets the compression alghorithim used in archive. (Only available for CREATE.)
    -D / --DefaultDirectory: [Default to a directory called archiveName for extraction of an entire archive file or the current working directory for a single file]. Allows changing where the extracted contents of an archive file (either all contents or a single file) are placed.


Help / Usage Statements

We can have a short initial usage statement and then have the user type -h on the given command for details on the command. For example, 'OP2MapImager CREATE -h' for help on the CREATE command.


Format of LIST command

I was thinking about the following format for the LIST command. I think size might be the compressed size of the file inside the archive if compression is used instead of the uncompressed size for ease of implementation? It would be the results of ArchiveUnpacker::GetInternalFileSize. Either way, it would be useful for seeing the relative size of each file.

Contents of maps01.vol (12 files)
ID  NAME          SIZE (KB)
--------------------------------------------------
00  blue.bmp      x,xxx
01  color.bmp     xxx,xxx
02  eden04.map    xxx
03  op2_art.prt   xxx
04  tutorial.map  xxx
05  virmask.raw   xxx
06  well0000.bmp  xxx

Might consider renaming CONTENTS to LIST. I think that would be more standard.

I was on the fence about this. Since you are mentioning it, probably best to go with LIST.

[Vagabond]

Quick C++ memory management question.

Can I write the code below, or will the created VolFile become a memory leak? The underlying function listContents requires a pointer to a VolFile class.

Code: [Select]

if (XFile::extensionMatches(filename, ".vol"))
	archiveConsoleListing.listContents(new VolFile(filename.c_str()));

-Brett

[Hooman]

Sounds good. For the first iteration I think we can just cerr if something besides a .wav file is attempted to be added to .clm. Maybe work up to actually error checking the contents of the .wav file if we continue far enough with the project.

The existing CLM packing code will already error on this condition. The packing code will try to load the WAVE header from all input files, which will error out if they are not proper .wav files. It also then compares the headers to make sure they are all the same format, and errors out if they are not.


To repack an archive, you can read the files right out of the old archive, same as the game can, while writing them into a new temporary file. Once the new file is packed, you'd want to do an atomic rename to the original filename. If the OS supports atomic renames, the original filename should at all times remain valid, and either point to the original archive, or the new fully packed and ready archive. Should be safe. Failing that, a quick delete and rename at the end is unlikely to be caught, and if the operation is interrupted after the delete, but before the rename, the user could always just complete that operation manually themselves. Again, that's fairly safe.

I think size might be the compressed size of the file inside the archive if compression is used instead of the uncompressed size for ease of implementation? It would be the results of ArchiveUnpacker::GetInternalFileSize.

I agree. The VOL structure doesn't make it easy to get the unpacked size without actually doing the decompression. They could have made it easy by using the unpacked size in the index entry, and the VBLK size as the packed size, but they didn't. They are both the packed size.

[Vagabond]

The existing CLM packing code will already error on this condition. The packing code will try to load the WAVE header from all input files, which will error out if they are not proper .wav files. It also then compares the headers to make sure they are all the same format, and errors out if they are not.

Perfect. I'll just display the error uses cerr for the user.


I'll upload my progress shortly. It will live under the GameResources directory in the repository. So far, I've hard coded the commands LIST (earlier CONTENTS) and FIND. Next step will be to design the command argument parser and hook LIST and FIND to command arguments. I'll base it off the code I wrote for OP2MapImager. The usage statement is also partially written.

I was surprised how easy it was to do formatting with cout. It breaks the rule that most everything is harder with C++.

[leeor_net]

A lot of the new features in C++11 have made C++ considerably easier to use. I really love the new additions... not that this affects what you did but still.

Anyway, I'm looking forward to having a CLI tool to do the packing/unpacking. I had to extract some map and graphics files and the VOLExtractor program is reasonably okay but it requires VB6 OCX files and I was somewhat unhappy that I had to register it as a COM object in the registry. Super not happy about that.

So anyway yeah, I'm very much looking forward to that.

Separate note -- maybe now is a good time to start migrating some of the code from SVN to Git? I've started a new project forked from another one of mine and uploaded it to the OPU GitHub repository.

[Hooman]

Separate note -- maybe now is a good time to start migrating some of the code from SVN to Git?

I was thinking the same thing. I'm enjoying the experience with GitHub. Let me know if you'd like help with moving things. I think git-svn can be used to make the transition easier, while preserving history.


Nice looking output. Does it handle long filenames well? Do long filenames show in full, or get truncated?

[leeor_net]

Separate note -- maybe now is a good time to start migrating some of the code from SVN to Git?

I was thinking the same thing. I'm enjoying the experience with GitHub. Let me know if you'd like help with moving things. I think git-svn can be used to make the transition easier, while preserving history.

There's a few things that we need to pull out of the SVN before we can move it over (namely the gamerelease directory) but otherwise git-svn should be fairly straight forward.

End Thread Hijack

[Vagabond]

Code: [Select]

Nice looking output. Does it handle long filenames well? Do long filenames show in full, or get truncated?

Thanks Hooman. Really long file names currently jack up the formatting. I'll try to fix this in the near future by setting a max filename character length before ignoring the length on the filename column width. This way really long filenames will only mess up their row on the table and none of the others.


I've programmed the LIST and FIND commands to accept command line arguments. They seem pretty robust. I'm partially through the EXTRACT command.

Below is a draft of the usage statement. I'm trying to go for a POSIX syntax. Let me know if it is confusing or out of line. () means required and [] means optional and ... means multiple allowed.

-Brett

[Vagabond]

Just uploaded code to the repository. I fixed the LIST command to play nicely(ish) with too long of filenames. See screenshot for what I mean. Basically, the normal list will cap out at 35 character length for the filename including extension. If the filename length is greater than 35 characters, it allows the longer filename to extend beyond the width of the file size column without affecting the rest of the table.

I think this should suffice unless anyone has better ideas on the matter. I would prefer if no one uses filenames that are so large though for my own sanity. 

Also, I feel like the smiley face choices change every other time I post. I am with Hooman that they are better not being animated.

Back to work on the EXTRACT command...

-Brett

[Vagabond]

I've been plugging away at designing OP2Archive. The EXTRACT command is mostly designed but not really tested.

I have the CREATE command actually taking files and saving them in a new archive.

It currently requires opening an existing vol file before I have access to saving a new vol file. A cursory look at the ClmFile class code makes it look like it also requires opening an existing clm file in order to create a new clm file.

@Hooman,

How do you feel about & how hard do you think it would be to add a static function to the VolFile and ClmFile class that allows creating a the respective archive file without opening an existing archive?

If this isn't feasible, we can include a VolTemplate.vol and a ClmTemplate.clm with the executable to reference when creating new archives.

-Brett

[Hooman]

Nice output. Looks like you've got the long filenames handled. Good that they don't get truncated. Better to be correct than pretty. Also good they don't mess up the formatting of other rows.

The "KB" in the header, but not the individual rows seems weird to me. Something about a file of size "2", with no units, and it not being bytes feels odd. If using human readable units (B, KB, MB, GB), I'd suggest they change based on the row data, and hence be included for each row. Either that or fix it at bytes for all rows. Looks like there is sufficient space available to display bytes, and the result may be more meaningful.

How do you feel about & how hard do you think it would be to add a static function to the VolFile and ClmFile class that allows creating a the respective archive file without opening an existing archive?

This sounds like a very good plan. I actually regret some of the design choices of the old code. The class to read from a Vol or CLM file doesn't have much in common with code to create them. The multiple inheritance design was bad. A static method for archive creation makes way more sense.

I think the old design is bad for the same reason that "bidirectional streams" are a bad design. It really makes no sense.

[Vagabond]

The "KB" in the header, but not the individual rows seems weird to me. Something about a file of size "2", with no units, and it not being bytes feels odd. If using human readable units (B, KB, MB, GB), I'd suggest they change based on the row data, and hence be included for each row. Either that or fix it at bytes for all rows. Looks like there is sufficient space available to display bytes, and the result may be more meaningful.

Hmmm, I just ran the 'dir' command for windows in a command prompt. It defaults to showing data in bytes. I'm assuming a similar command in Linux would also display file size in bytes? So, probably best for consistency to switch to bytes. Bytes always seemed too small to me when comparing file sizes, but I'd rather be uniform.

It also occurred to me the sizes should be right aligned instead of left aligned.

I'll plan to make another pass on the LIST command's output at some point and make these changes.

This sounds like a very good plan. I actually regret some of the design choices of the old code. The class to read from a Vol or CLM file doesn't have much in common with code to create them. The multiple inheritance design was bad. A static method for archive creation makes way more sense.

I think the old design is bad for the same reason that "bidirectional streams" are a bad design. It really makes no sense.

I'm just really glad there is boiler plate code that can handle archive file access. This is something that would have been well beyond my ability. I don't mind if we want to rewrite the archive code to cut out the multiple inheritance.

I noted that when trying to create a vol file, the current code packs the files properly into temp.vol, but doesn't create a vol file with the passed filename.

Below is the function in question. I think if I just change the hard coded "temp.vol" to be volumeName from the function's argument, it will work properly.

Code: [Select]

bool VolFile::CreateVolume(const char *volumeName, int numFilesToPack,
		const char **filesToPack, const char **internalNames)
	{
		CreateVolumeInfo volInfo;

		volInfo.numFilesToPack = numFilesToPack;
		volInfo.filesToPack = filesToPack;
		volInfo.internalNames = internalNames;

		// Make sure files were specified
		if (numFilesToPack < 1 || filesToPack == NULL || internalNames == NULL) return false;
		// Open a file for output
		if (OpenOutputFile("Temp.vol") == 0) return false;	// Return false on error

		// Open input files and prepare header and indexing info
		if (PrepareHeader(volInfo) == false)
		{
			CloseOutputFile();
			return false;
		}


		// Write volume contents
		// ---------------------
		// Write the header
		if (WriteHeader(volInfo) == false)
		{
			CleanUpVolumeCreate(volInfo);
			return false;
		}
		if (WriteFiles(volInfo) == false)
		{
			CleanUpVolumeCreate(volInfo);
			return false;
		}

		CleanUpVolumeCreate(volInfo);

		return true;
	}

My current goal will be just getting the console interface working properly. Once that is done, I guess we can discuss what we want to do to the archive library code.

I was noticing that we are passing Boolean success values out right now for CreateArchive and Repack functions. I'm just adding a cerr message saying creating the archive failed right if the CreateArchive function passes back false. I'm thinking passing the Boolean success out is probably driven by using the windows specific file handles. Perhaps when we switch the code over to something less operating system specific we can switch to more of a throwing exceptions error handling than Boolean? I think we could probably provide a little more fine grained report to the user on why the action failed.

[leeor_net]

I would highly highly highly recommend switching

Code: [Select]

const char*

to

Code: [Select]

const std::string&

and

Code: [Select]

const char**

to

Code: [Select]

const std::vector<std::string>&

Perhaps

Code: [Select]

typedef std::vector<std::string> stringlist;
const stringlist&;

(really wish there was an inline code tag, hope all the above makes sense)

[Vagabond]

Leeor,

I understand you wanting to switch to C++11 strings, but there are currently 13 files of code involved in manipulating archive files. I'd have to switch them all from char* to std::string, which would be a fair commitment of time. I've avoided any C++11 includes in the archive code since this is how Hooman left it. I'd probably want his thought on it before changing it all.

I do agree with you in the long term of wanting to move away from char* where not required to interface with legacy Outpost 2 code.


I did some experimentation trying to create empty vol and clm files. I had to remove an if statement from the volFile's code to allow it to create an empty vol file (the if statement was returning false if no files provided to pack). After this, I was able to create and manipulate the empty vol file without issue. I'll plan to re-add part of the if statement to still check for improper values, just not reject empty values.

The ClmFile class allowed creating an empty CLM file. After creation, the file could not be loaded though. I suspect this is because a ClmFile requires at least one source audio file to pull valid audio parameters from. I'll plan to add a check to the ClmFile that returns false if no files are included to keep from creating the invalid ClmFile. 

Curiously, the VolFile class allowed overwriting an existing file while the ClmFile class did not. I fixed ClmFile to allow overwriting a file.

To summarize:
 * You can now create empty vol files
 * I plan to disable ability to create empty clmfiles since they are currently badly formed
 * clm files can now be written over when creating new ones

My plan was to create both an empty vol and clm template to use when creating new archives. Unfortunately, I'll have to pack at least one audio file in the clm file for it to be valid (which will be 9mb). An alternative would be clipping the majority of the contents of the audio file to get it down in size. I'd have to research how to create an audio file to do this (I'm sure the data is floating around the forum).

Also, I noticed while you can get the compression type used on a vol file, there is no way to set the compression on a vol file when created. It might be as simple as changing the compression in the CreateVolume function, but I haven't tested this yet. Otherwise, we can just support reading compressed files and only support creating uncompressed vol files.

-Brett

[leeor_net]

Just a note, std::string is not a C++11 deal, that's been in there since C++98.

I recommend it as it's a lot safer than using raw C strings. You can make small changes to the interfaces as you go. Except for the std::vector<std::string>, because of the way std::string is built, passing a const char* as a paramter for a std::string& will work just fine (under the hood it creates a std::string class, passes a reference to that, then destroys said string so there are some efficiency issues until everything is converted to std::string, but the point is it'll work). Using a std::vector<> though would complicate matters by requiring changes to any code that uses the original form (array of const char*).

Anyway, the intent behind it is to produce better code that's harder to break. By using objects that encapsulate pointers and handles C strings for the user, you provide some insulation against pointer mistakes. Having worked with C strings for a couple of months now, I've discovered how much of a headache they are.

As a side note, for functions not yet converted, you can use std::string::c_str() to get a const char*.

[Hooman]

It also occurred to me the sizes should be right aligned instead of left aligned.

Good call, I missed that.

Multiple inheritance is the work of the devil 

I noted that when trying to create a vol file, the current code packs the files properly into temp.vol, but doesn't create a vol file with the passed filename.

I vaguely remember code to pack stuff into a temporary file, and then rename it to the proper file name. That avoided problems of corrupt files if the program or computer crashed while packing an archive. In particular, it was safer if you were repacking the current archive, in which case you wouldn't want to corrupt it before you had a reliable replacement, and you wouldn't want to replace it if you were still reading from it. Though from what you've pointed out, it does indeed look like a bug if the parameter is ignored. Perhaps the code was badly structured.

I was noticing that we are passing Boolean success values out right now for CreateArchive and Repack functions. I'm just adding a cerr message saying creating the archive failed right if the CreateArchive function passes back false. I'm thinking passing the Boolean success out is probably driven by using the windows specific file handles. Perhaps when we switch the code over to something less operating system specific we can switch to more of a throwing exceptions error handling than Boolean? I think we could probably provide a little more fine grained report to the user on why the action failed.

Indeed, exceptions would provide better error messages. I was mostly just following an older code style at the time. Also somewhat following how Outpost 2 seemed to work, which seemed to have exceptions disabled. The choice was not related to Windows or file handles. Back then, I tended to avoid exceptions since even having them enabled in the compiler, would cause less efficient code to be output, even if you weren't using exceptions. I didn't see the value in them at the time. I also tended to avoid any sort of runtime inefficiency, even when other concerns may have warranted it. Some of those concerns may have been dealt with in future compilers. At any rate, it would make sense to use exceptions here.


It may also be worth switching things over to using string and vector. I used to avoid a lot of standard library stuff for little good reason.

The ClmFile class allowed creating an empty CLM file. After creation, the file could not be loaded though. I suspect this is because a ClmFile requires at least one source audio file to pull valid audio parameters from. I'll plan to add a check to the ClmFile that returns false if no files are included to keep from creating the invalid ClmFile.

You're probably correct here. Maybe set a default WaveFormat if no input files are given, and see what happens?

It's good that you're testing corner cases.

My plan was to create both an empty vol and clm template to use when creating new archives.

I'm not sure I follow here. Was this to work around some issue I've forgotten about now? Was this for repacking or something? I think there must be a better way than relying on empty template files.

Also, I noticed while you can get the compression type used on a vol file, there is no way to set the compression on a vol file when created. It might be as simple as changing the compression in the CreateVolume function, but I haven't tested this yet. Otherwise, we can just support reading compressed files and only support creating uncompressed vol files.

Yes, we never wrote any compression code. We've only ever supported repacking of uncompressed data. The compression scheme was complex enough that I never quite wrapped my head around how to write a compressed stream. Plus there was little reason to work on the problem. I suspect it can be done, and shouldn't be too hard given decompression code already exists, but it may take some thought and a detailed look at the decompression code.

I can't remember now if the game included any compression code. I don't think it did, or if it did, I don't think it got documented in the disassembly efforts.

[Vagabond]

Alright, the LIST command is fixed up for right aligned and showing results in bytes. I think LIST is pretty much done.

[Vagabond]

Besides fixing up the LIST command as indicated in the last post, I spent some time reviewing the Archives source code and creating more clm and vol files.

I committed changes to the repository. Below is the overview of changes.

OP2Archive:
 * Implemented the CREATE command.
 * Fixed bug with ClmFile that did not allow over writing an existing CLM file.
 * Allowed VolFile to create empty Vol Files.
 * Added error checking to ClmFile and VolFile that stop file creation if NULL pointers provided to filenames.
 * Fixed Archive LIST command to show file sizes right aligned and in bytes.


I think there is sort of a crossroads here on where I continue work. I can either spend time refactoring the archive code or leave the archive code alone and have some funny peculiarities in the OP2Archive console application.


Currently, the archive library code requires loading an existing vol or clm file into memory before being allowed to create a new file. To instantiate either a ClmFile class or VolFile class, you must provide the filename of the Vol or Clm file to load. Additionally, both classes reference memory locations to pull information out of the archive. (Sorry, if I'm not explaining the well.) Basically, when Hooman and I made up the OP2 Map Structure, we wrote up a structure that is then filled up with data when the stream is loaded. In the ClmFile and VolFile case, the code references the memory location in the raw loaded data, and sometimes also records important data to actual variables.

My first thought was to create a constructor taking no input parameters that represented an empty file. Then I could write a static function that creates the archive using the parameterless constructor. But, because the full contents of the archive are not contained in a structure, I cannot just go in and set reasonable default values without doing what looks to me like a very major overhaul of the archive code. To complicate this, I'm not familiar with the design of audio files, and I would want to bone up on this subject before really overhauling the ClmFile class anyways.

Currently, I don't really plan on spending enough time on this project to do a major overhaul of the archive library code and write a decent console application to use it. Someone with a better understanding of the Windows specific file handles and Windows specific audio file code could probably do it faster than me?

Anyways, after all that info, the archive library code works fine, just not how I was expecting to use it. I have noticed a couple of small bugs, but nothing that was not easy to patch with one line of code change here or there.

So, using the current archive library code means that OP2Archive will require a vol archive and clm archive template file in order to construct new archives. Since I can create an empty vol archive, this will be about 2kb I think. The problem is that a clm file will not save in good form without at least one audio file included to pull the proper audio meta-data out of. The smallest audio file included in OP2 is about 9 megabytes. OP2 also has some weird rules on how to properly format an audio file. See http://forum.outpost2.net/index.php?topic=3906.0. If I can figure out how to create a short 1 second clip that is well formed, I will do this, but I may just leave it at 9 megabytes for now.

Also, I verified that there is no plumbing in VolFile for creating a compressed archive. I didn't continue digging into the LZH code to see if it would support the operation. I think if we are content keeping all the files uncompressed in the archives, there probably isn't a lot of reason to dig down this rabbit hole. I was pretty curious how much hard drive space we could save by compressing it all. I'll plan to take the compression argument out of the usage statement.

To recap: I'm planning on finishing OP2Archive in a way that requires a template of each archive in order to create a new file. If we get around to reformatting the archive library code, it will be easy to go back through the OP2Archive application code and change how the archive files are creating.


List of things to consider in improving Archive library code sort of in order of importance in my eyes

 * Remove Windows Specific file manipulation code (windows.h).
 * Remove Windows Specific audio file code from ClmFile class (windows.h).
 * Create a way to make a default, empty vol and clm file with a parameterless constructor.
 * Add static Create functions to both ClmFile and VolFile.
 * Consider switching fileName to filename (or change all the filename in my code to read FileName )
 * Consider switching from char* to std::string.

Thanks for everyone's help so far on this project!

-Brett

[Vagabond]

Quick update. I used Audacity to crop the EDEN11 audio track down to a half second or less and then exported it. This made the ClmTemplate.clm only about 2kb in size. the file formed well and was able to act as a template when encoding a new op2.clm file that in turn was able to play the in game videos without problem. So, no more worry about a 9 MB template file.

I'll plan on mentioning Audacity as a starting point in the ReadMe for formatting audio files in an Outpost 2 specific format. Unless someone else has a better free tooling solution?

My only worry is that I don't have sound in the main menu anymore? Not sure what caused this. I'm trying to download the original clm file from OPU to test if that fixes it, but it wants about an hour and a half to download. I doubt it is a problem with how the clm archive was created though or it would probably manifest in all the in game music and not just on the title track...

-Brett

[leeor_net]

I'm actually going to be doing something very similar with a mapper project for the same sorts of reasons. There is information that we don't yet know how to fill exactly so we need an existing file to get that going.

[Hooman]

Audacity is a wonderful tool.

... was able to play the in game videos without problem

I assume you mean in game music? I would be very surprised if you got anything else working from a CLM file.

The main menu music is a separate wav file. It's not packed in the CLM.


I still feel like this template file idea is kind of going down the wrong path.

* Remove Windows Specific file manipulation code (windows.h).
 * Remove Windows Specific audio file code from ClmFile class (windows.h).
 * Create a way to make a default, empty vol and clm file with a parameterless constructor.
 * Add static Create functions to both ClmFile and VolFile.
 * Consider switching fileName to filename (or change all the filename in my code to read FileName )
 * Consider switching from char* to std::string.

These are all good ideas. I'm thinking getting rid of the memory mapped files would make conversion to platform independent code easier, though it's a change I kind of dread. Another change is getting rid of the multiple inheritance. Maybe that can be done independently, and I think it would solve your template file issue.

I'm actually going to be doing something very similar with a mapper project for the same sorts of reasons. There is information that we don't yet know how to fill exactly so we need an existing file to get that going.

The maps do contain a large amount of stock data. I think this should be extracted into special data files that get shipped with the editor. Or perhaps compiled into the editor. Most useful is the tile set information, which should probably get packaged into a tileset meta info file, and the tile group info, which can also be a part of the tileset meta info. This doesn't need to be in the same format as the current map files. In fact, I think we could probably do something more useful with a custom format. Like adding tileset transition information, or marking tile groups as either palettes (take single tiles from these groups) or doodads (paste the group as a whole).

At any rate, in either case, I don't think a full template file is necessary, nor desirable. Though I do understand the desire to just get something working for now.


Edit: I'm thinking the CreateVolume function can be made static instead of virtual. That may allow you to create archives without instantiating an object first.

[Hooman]

Came across this:
filename or fileName

The argument seems to be whether you consider "filename" to be one word or two. I've gone both ways on this in the past, though I tend to stick with the two word convention now.

The two word convention seems to match the Microsoft convention listed in the answer, and leads to consistent casing of similar items: fileName, fileSize

A counter argument is that some people consider "file name" and "filename" to have subtly different meanings.

Here's a question: Would you write "directoryname" as one word? How about "foldername"?