Serialization / Deserialization

[leeor_net]

So... this is a problem I didn't think I'd have but I could use some insight from other programmers.

Serializing data to disk is simple enough. I'm planning on using XML for now (because I'm familiar with it and because I have built-in support in NAS2D already). The question is... what is a good way to design the serializers/deserializers?

My Redmine tasks initially describe building classes to serialize particular objects. This is fine but leads to new objects.

On the other hand, I'm currently building the code such that the objects in question pretty much serialize themselves... but this couples them with two more dependencies (one on NAS2D, two on TinyXML). This is something I was hoping to avoid but it's also the simplest method as of right now.

Thoughts?

[Hooman]

You can have your game objects depend on a simple serialization interface, without having them depend on full libraries. That would be just a header file import for the serializer, which doesn't need to bring in any other header files. The implementation file for the serializer can depend on the external libraries.

Here's how a serializer might be used in client code:

Code: [Select]

void Object::Serialize(Stream& stream)
{
  stream.Serialize(m_field1);
  stream.Serialize(m_field2);
  // ...
  m_complicatedSubObjectField.Serialize(stream);  // Yes the order is reversed here. Think about it.
  // ...
}

Also note there is no distinction between saving and loading, other than the stream's direction. You can reuse the same save/load code for fixed sized data. For variable sized data, you can check the stream's direction to determine if memory allocation is required.

Code: [Select]

void Object::Serialize(Stream& stream)
{
  stream.Serialize(m_arraySize);
  if (stream.Direction() == StreamRead) {
    m_array = new SubObject[m_arraySize];
  }
  stream.Serialize(m_array, m_arraySize*sizeof(m_array[0]));
}

To make this work, both the stream reader, and the stream writer classes should derive from the same interface that provides the Serialize method. The virtual function dispatch of the interface will call the appropriate Serialize method in either a StreamReader or a StreamWriter subclass, allowing the same client code to serialize for both directions. (The stream knows if it's reading or writing the data, the client code doesn't care, it just tells the serializer what data needs to be either saved or loaded). The client code only needs to know the direction of the stream for memory allocation with dynamic sized data.

Note that I was a bit lazy in the code above about indicating data size. The virtual method should probably take a pointer and a size as parameter. I omitted the size parameter except for the line where an array is saved/loaded. You can make things easier for scaler types by providing a template method that forwards calls to the virtual method with the appropriate sizeof value for the second parameter. Such a template method won't be virtual, but would be a nice convenience method to call the virtual method, and can be made to compile inline for performance.

Here's the idea in likely not syntactically correct code:

Code: [Select]

template <T> void Serialize(T& data) { Serialize(data, sizeof(data)); }

Essentially this allows for:

Code: [Select]

int data;
BigLargeStruct moreData;

stream.Serialize(data);  // => stream.Serialize(data, sizeof(data));
stream.Serailize(moreData);  // => stream.Serialize(moreData, sizeof(moreData));

[leeor_net]

It's funny because I want to do this well but also seem stuck in low-level thinking. So far I have the objects themselves serializing/deserializing themselves. Seems like a simple enough implementation, I just worry about future maintenance with it.

[Hooman]

They do serialize themselves. They have to if they have private fields. A stream class only needs to know how to serialize fundamental types, or fixed size buffers really. The rest is built on top of such primitives.

[leeor_net]

I'm still doing the derp thing but I think I have a reasonable way of doing it in the code.

Since I'm outputting to XML, I figured why not just have the objects set up the XML structure on their own, right? (also yes, I noticed the spelling mistakes, will be fixing that in one of the next revisions).

Code: XML [Select]

<OutpostHD_SaveGame version="0.01">
    <properties planetname="Planet Name -- DUMMY" sitemap="maps/mars_04" sitemapname="Sitemap Name -- DUMMY" tset="tsets/mars.png" diggingdepth="4" />
    <view_parameters currentdepth="0" viewlocation_x="183" viewlocation_y="30" />
    <mines>
        <mine x="33" y="121" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="126" y="76" age="0" depth="0" active="0" exhausted="0" yield="0" />
        <mine x="273" y="124" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="10" y="55" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="255" y="49" age="0" depth="0" active="0" exhausted="0" yield="0" />
        <mine x="113" y="21" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="283" y="101" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="270" y="103" age="0" depth="0" active="0" exhausted="0" yield="2" />
        <mine x="123" y="32" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="76" y="60" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="105" y="36" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="79" y="48" age="0" depth="0" active="0" exhausted="0" yield="1" />
        <mine x="260" y="95" age="0" depth="0" active="0" exhausted="0" yield="2" />
        <mine x="81" y="110" age="0" depth="0" active="0" exhausted="0" yield="1" />
    </mines>
    <tiles>
        <tile x="184" y="36" depth="0" index="0" />
        <tile x="186" y="34" depth="0" index="0" />
        <tile x="189" y="36" depth="0" index="0" />
        <tile x="189" y="37" depth="0" index="0" />
        <tile x="189" y="38" depth="0" index="0" />
    </tiles>
    <structures>
        <structure x="185" y="35" depth="0" id="64185312" age="48" state="1" type="2" direction="1" />
        <structure x="185" y="36" depth="0" id="64412400" age="0" state="0" type="13" direction="1" />
        <structure x="186" y="37" depth="0" id="111595704" age="0" state="0" type="13" direction="1" />
        <structure x="187" y="37" depth="0" id="111596032" age="48" state="1" type="11" direction="1">
            <production common_metals_ore="196" common_minerals_ore="30" rare_metals_ore="8" rare_minerals_ore="16" common_metals="0" common_mminerals="0" rare_metals="0" rare_mminerals="0" energy="0" food="0" />
        </structure>
        <structure x="257" y="8" depth="0" id="111596360" age="35" state="2" type="8" direction="1">
            <production common_metals_ore="0" common_minerals_ore="12" rare_metals_ore="8" rare_minerals_ore="8" common_metals="0" common_mminerals="0" rare_metals="0" rare_mminerals="0" energy="0" food="0" />
            <storage common_metals_ore="130" common_minerals_ore="154" rare_metals_ore="96" rare_minerals_ore="120" common_metals="0" common_mminerals="0" rare_metals="0" rare_mminerals="0" energy="0" food="0" />
        </structure>
        <structure x="187" y="35" depth="0" id="111599096" age="48" state="1" type="0" direction="1" />
        <structure x="186" y="36" depth="0" id="113860920" age="48" state="1" type="6" direction="1" />
        <structure x="188" y="36" depth="0" id="114327688" age="25" state="1" type="12" direction="1" />
        <structure x="186" y="35" depth="0" id="114353432" age="0" state="0" type="13" direction="1" />
        <structure x="242" y="53" depth="0" id="114428832" age="41" state="1" type="8" direction="1">
            <storage common_metals_ore="0" common_minerals_ore="0" rare_metals_ore="0" rare_minerals_ore="8" common_metals="0" common_mminerals="0" rare_metals="0" rare_mminerals="0" energy="0" food="0" />
        </structure>
        <structure x="187" y="36" depth="0" id="114429208" age="0" state="0" type="13" direction="1" />
    </structures>
    <factories>
        <factory x="185" y="37" depth="0" id="114429536" age="48" state="1" type="3" direction="1" production_completed="0" production_type="0" />
    </factories>
</OutpostHD_SaveGame>

Which is precisely what I expect and want. This is prone to breaking though as I'm using implicit conversions of enumerated types to integers, something which can cause subtle (and not so subtle) errors if the enumerations change. E.g., if I have a structure enumerator as such:

Code: [Select]

enum StructureType
{
    STRUCTURE_AGRIDOME,
    STRUCTURE_MINE_FACILITY,
    STRUCTURE_WHATEVER
};

And later on down the line add a new enumerator in between what's already there:

Code: [Select]

enum StructureType
{
    STRUCTURE_AGRIDOME,
    STRUCTURE_ATMOSPHERE, // new enumeration
    STRUCTURE_MINE_FACILITY,
    STRUCTURE_WHATEVER
};

Mine Facility and Whatever will now be assigned different integer values and loading in an older save game will lead to the wrong structures being instantiated.

I see three solutions, none of which I particularly like. I can either explicitly define the numeric value of the enumeration (don't like this idea as the point of the enumeration is to be able to treat it as a type vs. a distinct value), I can only add new enumerations to the end of the enumerator list (still don't like this idea because I'm really picky about alphabetizing enumerations) and the final which is what I'm probably going to settle on, every time the enumerations change, update the version string and have the game either fail to load the game or tell the player that their game will behave funny.

I think the last of the options is the least of the evils because it avoids dealing with first two problems but also allows for 'conversion' tools to be written to 'upgrade' a savegame from an older version of the game, something I hate doing (because what if someone saved a game in v0.6.0 but never picked it back up until 1.2.1? Undoubtedly there would be LARGE changes between all those versions so you'd have to run the save game through a chain of upgrade tools).

There's also a fourth option of defining all structure I think I may use or just setting several aside as 'reserved', but I really don't care for that either.

I dunno... thoughts?

[Hooman]

I was thinking my way doesn't work so well for XML files, since you lose the field name. It also doesn't group attributes together in any well defined way, which can make it difficult for a serializer to generate proper tags from the function calls I provided. I suppose you could add the extra parameter for the field name when writing attributes, and a few extra calls to mark the end of the attributes (the end of the opening tag), and the end of an object (the ending tag), allowing any sub-objects to be nested inside. Writing arrays would also need some kind of array name, but that's basically just a field name, and then a bunch of sub-objects.

I suspect X-Macros, and possibly templates could be used to generate a lot of the code. What you're doing is very regular. There should be a way to automate it.

[leeor_net]

You'd think so right?

What you proposed works really well if I'm just going to save information out as a blob of data (generally efficient, WAY smaller file size). I just always liked the idea of savegame data being human readable. JSON is a good option as well but I'm unfamiliar with its syntax and I'm very familiar with how TinyXML works (and it's kind of included in NAS2D).

This is one of those cases where I feel like there should be a cleaner way to do this without tight coupling with a particular XML library but I suppose there really isn't unless I write my own XML Wrapper library and that's genuinely beyond the scope of this project.

Perhaps one day I'll do that for NAS2D to allow for other XML back ends to be used?

[Hooman]

The example I gave originally doesn't quite fit nicely, but adapting it according to my update you could potentially do something along these lines:

Code: [Select]

class BinaryStreamReader : public Serializer
class BinaryStreamWriter : public Serializer
class XmlStreamReader : public Serializer
class XmlStreamWriter : public Serializer
class JsonStreamReader : public Serializer
class JsonStreamWriter : public Serializer

Depending on the file format you want to deal with, you change what type of Stream to pass in to the objects being serialized/deserialized.

You have to use an interface that passes along any needed data. For a binary file, you'd generally ignore field names and just store things positionally. For a text format, you'd need to ensure field names were included in the stream interface and get written to the file. My original example didn't include field names, but if you added them, they could be used by the XML and JSON serializers, and ignored by a binary serializer. The pattern still works if you can abstract away the data that needs to be saved from the format it's being saved in.

[leeor_net]

Was actually thinking that as I finally got some sleep.

I'll probably do something along those lines later on down the line. It's a great feature list.

Only problem with a binary format is that you need to know the order in which things are written. JSON and XML allow for greater flexibility but if you have tools doing the reading and writing, it almost doesn't matter.

[Hooman]

Exactly. It doesn't matter, so long as it's consistent. And it should be consistent, since this is all done in code.

Also consider that the saving and loading will use the same serialization code (but different stream objects), so it will always use a consistent ordering between saving and loading, no matter what odd ordering you choose. A good default is often to just dump a full struct in the same order it was declared in. This can potentially be done with a template or X-Macro.

It's only versioning issues that creep in, where fields are added, removed, or have their semantics changed. That's a pain point no matter what file format you use.

[gubid]

The only additional recommendation here would be that you need to ensure a version of save file is included.  The save/load routine could then simply load the objects in the order for that given version.  Chances are you're not going to be changing this order every time you save/load anyway.  More like.. I also need to save this, and tag it onto the end of the save file. 

And for conversation here, I think I would go with a GameData class and simply implement save/load functions for it.  I agree with Hooman for the method of save though.  Utilizing a 'interfaced' object instead of otherwise has its benefits of exchanging it for something else at just about any time. JSON btw, is actually pretty easy and straight forward.  Less verbose than XML, but it has a couple of special rules that are annoying.  Specifically when dealing with binary.  Although, no worse than dealing with XML.

[Hooman]

Agreed on the JSON. I would take JSON over XML any day.

[lordpalandus]

Can JSON be modified with a text file like XML can, or does it require a special program to modify it?

Generally for modifying data I've found XML the easiest to work with. Though, if you, leeor, don't care for modding, then go for the one that works best for you.

[Goof]

Yes JSON is easy to modify in a text editor.

For a small amount of data, it might work well. but on a huge amount of data it seems hard to apprehend without a parser.

I think that an easier way to downsize the backup would be to simply compress it with zip/gzip/7zip/any algorithm.

With a backup of 58 893 octets

Original XML file	58 893 octets	100,000%
Zip compressed	5 138 octets	8,723%
gZip compressed	5 015 octets	8,515%
7Zip compressed	4 063 octets	6,899%

Not so easy to modify with a text editor. (so not so easy to cheat on it)

[leeor_net]

I really don't care much for JSON but I'm not entirely opposed to using it either. I've already implemented save/load with XML since TinyXML comes prepackaged with NAS2D.

Goof has a point -- save files can just be compressed via gzip. In this case there isn't too much point because the save files aren't really that big even with hundreds of structures. I have considered zipping and encoding the ZIP'd info directly into the XML but this seems needlessly complicated.

Savegame version tag has already been implemented. Since this game is still under development I doubt I'll really need to build converters. Later structures can just be appended to the end of the in-code lists or I can go with specified identifiers.

TO THAT END

The save/load code works but it's not very good. It's implemented in several forms in the GameState and in several objects. That alone indicates bad code.

I've opted to focus on other more important tasks at the moment but one of the things I want to do is clean up the save/load, make it consistent across all of the code and pull all serialize/deserialize code out of individual objects. The interface idea is a good one but I'm not very familiar with it.

When I take on the task of refactoring this code I'll look at several options and go from there.

[Vagabond]

Just curious, but why not save the data to disk in binary with a simple BinaryWriter/Reader?

This will make the save data more difficult to edit by the user then XML or JSON and can still be easily compressed.

I would be curious if the save data was smaller in binary then in XML or JSON.

[leeor_net]

It would be WAY smaller in Binary than in XML.

Thing is, the way I see it, in single player modes, if a player wants to cheat, let them cheat. It's their choice if they don't want to be intellectually honest with themselves and others.

There are two reasons I chose XML:

1) It's easy to read and debug as a non-machine human like myself
2) I don't have to worry about portability issues. It's all just plain-text.

Despite this, a few of the caveats of using XML:

1) Much larger file size
2) MUCH slower

Easy counter points: disk space is no longer a premium so a 150KiB XML file is a non-issue these days.

In release builds, speed for processing an XML file takes less time than loading the graphics files so it's also a non-issue.

[Hooman]

2) I don't have to worry about portability issues. It's all just plain-text.

OMG, don't even get me started on that one. What's the newline convention on your OS? Carriage Return + Line Feed? Only Carriage Return? Only Line Feed? Line Feed + Carriage Return? What about the encoding? ASCII? Latin-1? UTF-8? UTF-16?

"Plain-text" is a lie!


I believe JSON would be smaller and faster to work with than XML, and still easy to understand and work with in a text editor. In some ways easier to read than XML because it's not so verbose and awful (subjective term). Natural indentation would make it easier to read, which is true regardless of it being JSON or XML.


Binary files would be the smallest and fastest to work with. Compression would probably make them all roughly equal in terms of size. It's the same data being stored in all cases.

In short, do what you want.

[leeor_net]

2) I don't have to worry about portability issues. It's all just plain-text.

OMG, don't even get me started on that one. What's the newline convention on your OS? Carriage Return + Line Feed? Only Carriage Return? Only Line Feed? Line Feed + Carriage Return? What about the encoding? ASCII? Latin-1? UTF-8? UTF-16?

"Plain-text" is a lie!

Yes yes, I know. The good thing is that a well written, mature library can handle all of that transparently. Hence TinyXML (though I understand TinyXML 2 is a lot better). Point is, it's much more transparent unless I build/use a library to read/write binary files transparently to the system in question.

I believe JSON would be smaller and faster to work with than XML, and still easy to understand and work with in a text editor. In some ways easier to read than XML because it's not so verbose and awful (subjective term). Natural indentation would make it easier to read, which is true regardless of it being JSON or XML.

It would be probably 20% lighter in terms of pure character count. Maybe. Ultimately it doesn't really matter which 'format' I use, the only real comparison is between binary and non-binary.

Binary files would be the smallest and fastest to work with. Compression would probably make them all roughly equal in terms of size. It's the same data being stored in all cases.

In short, do what you want.

Exactly.

[Hooman]

It would be probably 20% lighter in terms of pure character count. Maybe. Ultimately it doesn't really matter which 'format' I use, the only real comparison is between binary and non-binary.

XML needs "</ClosingTags>", whereas JSON needs a closing brace "}". The overhead depends heavily on the relative size of field names to field values, the length of those names, and if you're storing data in attributes, or nested tags. Attributes have less name overhead, similar to JSON, without quoting, but enforce a flat structure. If you want nested data, you need to use nested tags.

For some numbers, which is arguably a bit skewed for XML since they chose nested tags rather than attributes, see JSON vs XML: Some Hard Numbers.

When using nested tags to store values, the generated XML was 84% larger than the JSON, took 20% longer to compress, and was about 10% larger after compression.


Another perspective, is what are the two formats used for? Some people choose JSON for data, and XML for documents.

[Vagabond]

Goof and Leeor_net,

Thank you for posting the info on compressing saved data. I just implemented this in one of my projects using gzip and it dropped the save file size from around 750kb to 15kb! In my programming amateurishness, it didn't even occur to me that save data is easily compressed. Also, there is no noticeable delay in saving/loading with the compression/decompression added so big win.

Sorry for derailing the post a little. I'm happy to see all the progress on OutpostHD!

-Brett

[lordpalandus]

One extra negative that isn't being mentioned about JSON is the requirement of needing JavaScript to use. Not everyone has JavaScript installed on their machine as its one of those languages that can be easily abused by online hackers and deliver a virus to you via a JavaScript supported program. Thus, if you used JSON, they may not be able to play your game period, as I think by utilizing JSON it would force the user to have JavaScript installed to run the program at all, much less try to modify the contents of the file.

While with XML, the requirements are much less. Something to consider.

[Goof]

One extra negative that isn't being mentioned about JSON is the requirement of needing JavaScript to use. Not everyone has JavaScript installed on their machine as its one of those languages that can be easily abused by online hackers and deliver a virus to you via a JavaScript supported program. Thus, if you used JSON, they may not be able to play your game period, as I think by utilizing JSON it would force the user to have JavaScript installed to run the program at all, much less try to modify the contents of the file.

While with XML, the requirements are much less. Something to consider.

There is some JSON parser for C++, C, PHP, Java, etc.
Even the french IDE we use at work with his own language could create/read JSON data.

It's not because it start with a J that it could only be used with Java/Javascript.

Like everything starting with an 'i' isn't related with apple (even if they try to do it ... they even patented a paper bag)

First found on G :
https://github.com/nlohmann/json

Goof and Leeor_net,

Thank you for posting the info on compressing saved data. I just implemented this in one of my projects using gzip and it dropped the save file size from around 750kb to 15kb! In my programming amateurishness, it didn't even occur to me that save data is easily compressed. Also, there is no noticeable delay in saving/loading with the compression/decompression added so big win.

Sorry for derailing the post a little. I'm happy to see all the progress on OutpostHD!

-Brett

De nada

[leeor_net]

One extra negative that isn't being mentioned about JSON is the requirement of needing JavaScript to use.

As Goof stated, this is not a requirement at all. This is just how JavaScript objects are notated... hence Java Script Object Notation. This does not mean that JavaScript is required to parse these objects as they are in a text format.

Additionally, JavaScript is not a runtime environment. It's a scripting language. Java requires a runtime environment but Java and JavaScript are not the same thing. They aren't even the same language (JavaScript derives from LiveScript which was created by Netscape).

[lordpalandus]

If its called JavaScript Object Notation, then good chance it requires JavaScript. Yes JavaScript is a scripting language, but all scripting languages REQUIRE an interpreter of some kind on the system to use it. If you don't have the interpreter you can't use JavaScript. When people don't install JavaScript on their machine, they are purposefully trying to avoid auto-activating scripts and the only surefire way to avoid them is to not have a JavaScript interpreter or any of the runtimes installed. Hope I've made things clear enough.

Also wonder which came first, Java or JavaScript.