Python script for extracting .vol files; questions regarding art.vol

[pente]

Out of an intense desire to avoid doing anything useful this morning I decided to poke around at mapping / tiles stuff a bit. I couldn't figure out how to compile OP2Archive on linux so I wrote a python script for extracting .vol files, which I've pasted below.

I looked around at the tile set graphics and the well*.bmp files don't seem to follow the file format described in the documentation, but instead start with "BM" and the file length in bytes, followed by data. I didn't see any information on the file virmask.raw, and was unclear on how op2_art.prt worked (it looked similar to the file format described for the well*.bmp files, but not quite a match). There are also mystery files blue.bmp and color.bmp in the art.vol volume. I read on the forum that "fixed" .bmp files not in the OP2 format are floating around somewhere, is there a link pointing to that?

I am under the understanding that some of the tiles are animated, is there information on how that is represented? I assume existing utilities that work with tile image data are just taking one of the frames from the animation, or some such?

I recall reading that someone made progress on analyzing the various .map files to figure out automatically which tiles can be adjacent to which other tiles; is this a done thing or still a work in progress?

Code: [Select]

import sys
import struct
import os
import os.path

class VolReader:
    def __init__(self, data):
        self.data = data
        self.index = 0
        self.N = len(data)

        self.read_all()

    # static
    def read_file(filename):
        with open(filename, 'rb') as stream:
            data = stream.read()
        return VolReader(data)

    ## Read a .vol file ##

    def read_all(self):
        self.read_section_heading('VOL ')
        assert self.read_section_heading('volh') == 0
        self.read_filenames()
        self.read_index_table()

    ## Helper functions for reading a .vol file ##

    def next_k(self, k):
        b = self.data[self.index : self.index + k]
        self.index += k
        assert self.index <= self.N
        return b

    def read_int4(self):
        # little endian
        int4, = struct.unpack('<i', self.next_k(4))
        return int4

    def read_byte(self):
        return self.next_k(1)[0]

    def read_section_heading(self, tag):
        assert self.next_k(4) == bytes(tag, 'ascii')
        l = self.read_int4()
        assert (l & 0x80000000) > 0
        return l & 0x7fffffff

    def read_filenames(self):
        size = self.read_section_heading('vols')

        start = self.index + 4
        end = self.index + size

        assert start == 28

        self.filenames = {}
        filename_start = start
        for i in range(start, end):
            # check null-terminated
            if self.data[i] == 0:
                if i > filename_start:
                    filename = self.data[filename_start : i].decode('ascii')
                    self.filenames[filename_start - start] = filename
                filename_start = i + 1

        self.index = end

    def read_index_table(self):
        size = self.read_section_heading('voli')

        self.numfiles = (size // 14)

        assert size == (14 * self.numfiles)

        self.files = []
        for i in range(self.numfiles):
            filename = self.filenames.get(self.read_int4(), '(none)')
            fileoffset = self.read_int4()
            filesize = self.read_int4()
            compression = self.read_byte()
            valid = self.read_byte()

            assert compression in [0, 1, 2, 3]
            assert valid in [0, 1]
            self.files.append((filename, fileoffset, filesize, compression, valid))

    def print_file_table(self):
        compression_name = ['Uncomp.', 'RLE', 'LZ', 'LZH']
        valid_name = ['Invalid', 'Valid']

        print("{} files, {} bytes".format(len(self.files), self.N))
        for filename, fileoffset, filesize, compression, valid in self.files:
            print("{: <30s} {: <10d} {: <10d} {: <10s} {: <10s}".format(
                filename, fileoffset, filesize,
                compression_name[compression], valid_name[valid]))

    def extract_files(self, dest_dir):
        for filename, fileoffset, filesize, compression, valid in self.files:
            if valid and filesize > 0:
                assert compression == 0
                self.extract_file(fileoffset, filesize, os.path.join(dest_dir, filename))

    def extract_file(self, fileoffset, filesize, dest):
        assert fileoffset + 8 + filesize <= self.N
        filedata = self.data[fileoffset + 8 : fileoffset + 8 + filesize]
        with open(dest, 'wb') as stream:
            stream.write(filedata)

def run(filename):
    assert os.path.isfile(filename)

    vol = VolReader.read_file(filename)
    vol.print_file_table()

    dest_dir = os.path.join(
            os.path.dirname(filename),
            'extracted_' + os.path.basename(filename))
    os.makedirs(dest_dir, exist_ok = True)
    vol.extract_files(dest_dir)

if __name__ == "__main__":
    filename = sys.argv[1]
    run(filename)

[Vagabond]

pente,

Sounds like you are proficient with Python. I would recommend reviewing the following project: https://github.com/phossy/op2art. It is a Python script capable of parsing the op2art.prt/bmp files.

As for the wells (tilesets), the original versions had 2 of the colors in opposite order from the typical standard. Since the game was capable of reading the more standard format, we just reordered the RGB values to the more standard variant and started packing it with Outpost 2. This way you could open the wells using any normal image editor and mod them. You can extract them from art.vol. Try using the packaged release of OP2Archive at https://github.com/OutpostUniverse/OP2Archive/releases. If you cannot get it to work with Wine, let me know and I'll just zip and send you the wells.

Unfortunately, no one finished any sort of automapper that can figure out contiguous terrain. It has been talked about a lot, but no one has but the time in. It is a complex problem to solve due to the many complicated transitions in tiles in Outpost 2.

My knowledge of how the animation data is stored is scare. Something that I wanted to work on but gave up on to do other things like help more with NetFix and the current mission SDK.

Maybe if you post the specific errors you are getting when trying to compile OP2Archive, someone with Linux experience could help solve.

-Brett

[pente]

I already have the well* files I extracted from art.vol using the script above -- it turns out my problem was something else. When I tried to view the images I got the error "unknown or unsupported file type", which I thought meant they were still in the OP2 format, but from your explanation I realized actually meant my viewer couldn't display bmp files. I have now used imagemagick to convert them to png and they display fine.

What kind of information is in op2_art.prt? I guess if all I care about is the tilesets than I can ignore it....

Looking at the wells, I gather that only well0007 has animations in it, and it is pretty easy to see which images are frames belonging to the same animation, and there are few enough of them that if the information is needed for some purpose it could be reconstructed by hand in a few minutes. (However each tile seems to have 6, 11, or 12 frames, which makes me suspect some of the ones with 11 frames have a 12th frame in well0004.) So I guess that mostly sorts that out, and I can move on to looking at the .map files if I feel motivated to do so later.

color.bmp seems to be a single pixel (black I think?) and blue.bmp is oddly enough also black.

[Hooman]

The OP2Archive project comes with a Linux makefile. I believe it's setup to use Clang by default. You can override that at the command line.

Code: [Select]

make CXX=g++

Maybe I should take out the Clang default.

I don't believe OP2Archive requires any dependent libraries.


The .prt file contains sprite data. It's needed for unit graphics, not for tileset data. The blue.bmp should actually be blue, though it's basically used as an unfilled tile marker, so maybe it has transparency set. The color.bmp might be used to store palette data?

The only animated tiles I remember, are the lava tiles. The data within the .map files contains the animation info for the tilesets it has loaded.

[Hooman]

Slight off topic update:
I was just looking back at this thread and realized I still hadn't yet removed the explicit Clang default. It has now been changed. The project will build with whatever the environment default compiler is, which should typically be GCC.


Curious if there has been any updates to pente's Python extraction code, or if there is a GitHub repository somewhere.

[pente]

I haven't done anything of note that I haven't posted here, I occasionally think about automatic map generation but haven't come up with a method that I like. (The idea is to look at existing maps to see which tiles tend to be adjacent, and then randomly build a map following the same distribution, but there are a lot of difficulties.)

[Hooman]

Hmm, sounds a bit like an old project of mine that never quite made it to a useful state:
Auto Map Maker Source

After scanning the original maps to find adjacency lists, I quickly discovered the original maps had quite a lot of graphical errors, so the scanned data wasn't very good. Using it in such a simplistic manner didn't always produce the best results.

Plus the filling algorithm I wrote kind of sucked, and was responsible for a whole new set of artifacts.

[Frank C., ASCE]

Hello i have seen a good script on reading volume files here.
Can any file be read by that code above.
Am newbie to python