Outpost Universe Forums

Projects & Development => Outpost 2 Add On Missions => Topic started by: Vagabond on March 31, 2016, 09:35:43 PM

Title: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on March 31, 2016, 09:35:43 PM
Hey everyone,

I am releasing a Plymouth Campaign Scenario called Evacuation Under Fire. It should be a shorter scenario to play and has some unique play characteristics. The briefing is worth reading to know what is going on, even though it is kind of long. It displays before the level is loaded, so you don't have to read it here. Hopefully it is the first of a mini-campaign. The second scenario is basically done and should be released soon...

Compatibility
The scenario is compatible with Windows and WINE. Thanks to Hooman for testing compatibility with WINE.

Download: https://github.com/Brett208/OP2MissionEvacuationUnderFire/releases

Install Directions

 * Place the .map, .dll, and research tree (other .txt file) in your root Outpost 2 install directory.
   Some scenarios share maps (.map) and research trees (.txt). You should not need to install over already
   downloaded versions of maps and tech trees unless the map or tech tree is being patched for all scenarios
   using it.
 * Open Outpost 2 and search for the new scenario with the other colony games.
 * Good luck, and have fun!

Source Code: https://github.com/Brett208/OP2MissionEvacuationUnderFire

Bugs

* If you save and then load the mission before emptying vehicles in the garage, sometimes attempting to release a vehicle from the garage will crash the game. Unknown if bug is in mission logic or game logic.

Mission Briefing
Commander,

Unfortunately, your mission to reinforce Outpost Delta is no longer viable. Eden broke through Outpost Delta's defenses and destroyed many key buildings. You are now on a rescue mission for the surviving colonists.

Eden found a way to manipulate our vehicle command network. They can scramble all vehicle command algorithm processors linked to the local network but cannot take direct control. Outpost Delta lost control of most all their units before attacking. Our units are still affected and appear mostly stationary, clustered in different areas around the damaged base. Occasionally the scrambled initiate patrols, self-destruct, or commit other random commands.

After rendering our defensive units ineffective, Eden entered the base and destroyed both Command Centers. Once the command centers were destroyed, their forces pulled back across the Eastern ridge. We believe the retreat is an effort to reduce their casualties. Eden left 2 tank squadrons in the area and we believe is waiting for the eventual exhaustion of our scrambled units before reentering Outpost Delta. Based on their previous actions, they will likely level the remaining buildings on rentering the base. Your orders are to exploit the lull in their advance and organize an evacuation for the remaining colonists.

Eden's new Tiger tank squadrons are combat operations. We are unable to match their firepower. Stay clear of their tank positions. If Eden locates your non-scrambled units in vicinity of the base, they will certainly retransmit their signal into our vehicle command nets. We are sending an attached file archive that contains a software patch for your vehicles' command net. The Savant with you can employ the patch to prevent complete control loss of your tank squadron. However, our vehicles will require maintenance at a garage to fully shield them from the effects.

We are betting on Eden's imagery satellites remaining busy imaging our main base's defensive positions and not available to notice your vehicle activity. However, Eden will certainly be tipped on your presence if your vehicles come within visual range of their tank squadrons. Their wide thermal grid will likely pick up new building activity, so do not construct additional buildings. There is a small field of fumaroles West of the base that should be able to mask the heat signature of the construction and operation of a new command center.

Outpost Delta was preparing an evacuation convoy in case they were overrun, but the convoy was not completed before the vehicles were manipulated. There are several ConVecs in the evacuation staging area West of the Outpost. Hopefully one of the ConVecs contains a Command Center so you can restore enough of the base systems to complete evacuation preparations. When a spider reprograms one of our scrambled units, it should be able to reboot the vehicle's scrambled command processor.

Move quickly and get our colonists out before Eden re-attacks. Eden's main battle line continues to advance toward our main colony. You will not have the firepower to break through Eden's attack line and return to our main base. After evacuating the colonists, head West and establish a temporary resupply base. We will deal with Eden's approaching forces.

Good Luck Commander.
The survival of Outpost Delta's colonists rests in your hands.

You have to be logged into the forum to see the attachment.

----
Edit 01APR: Fixed typo in briefing. Credit to lordpalandus for finding it.
Edit 04APR: Fixed source code bug not allowing code to be built in Debug Configuration. Doesn't affect release dll.
Edit 15APR: Removed dependency on Visual C++ Redistributable 2015 from mission DLL. Attachment updated.
Edit 09JUN: Moved download to Github. Updated to new version that sets HFLInit properly when loading a saved game.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on March 31, 2016, 11:43:30 PM
Neat. Works under Wine.

I like how your initial vehicles start off moving into the map, rather than just sitting at the start point. That definitely adds some urgency to the mood.


Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: lordpalandus on April 01, 2016, 02:15:11 PM
Typo in =  Based on their previous actions, they will likely level the remaining buildings on rentering the base. Your orders are to exploit the lull in their advance and organize an evacuation for the for the remaining colonists.



Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on April 15, 2016, 05:19:36 PM
Hey Everyone,

Just like Rising From the Ashes, this scenario DLL has been updated and no longer requires the Visual C++ 2015 Redistributable package. Thanks for the proofreading help lordpalandus!

The new DLL can be downloaded from the first post in the topic.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Commander on October 01, 2016, 10:09:08 PM
Vagabond, just wanted to thank you and let you know that this mission is AWESOME.  8) Really cool ideas with the Eden scrambler link, the CC at the geothermal sources, and superb level design too - better than many of the official missions! It took me five restarts, but I eventually made it. Now busy beating "On the run" - keep up the good work!
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on October 02, 2016, 06:51:48 PM
Commander,

Thank you for the feedback. I'm glad you enjoyed the scenario and happy it worked without any bugs!

I'm planning on rebuilding the Outpost Monopoly multiplayer code. After that, I would like to come back and code the 3rd mission in the series.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on May 28, 2018, 05:47:57 AM
Thanks for making a great series of missions. Unfortunately I got a crash with this mission. I happened to save 2 seconds before it crashed so I've uploaded the save file. The crash can be reproduced by clicking on either garage and pressing S.

(I'm playing on Outpost 1.3.7 under wine.)
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on May 28, 2018, 08:22:17 AM
Thanks for the saved game file. That really helps to reproduce and investigate bugs.

Technical:
The crash seems to be happening in code marked as CommandPaneView:BuildingStorageBays. Unfortunately the routine is not documented, so it may take a bit of effort to figure out what's going on. It seems the code is trying to call a virtual function with an invalid object pointer, which leads to a crash when it tries to dereference the virtual function table.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on May 28, 2018, 11:17:15 AM
Interesting. If it matters, I believe that earlier in that same game I was able to view the garage contents without crashing, and that the usual vehicles were present. I also believe that shortly before the crash some of the neutral-ish units had just received new orders to attack my stuff -- maybe a garaged unit got such an order and got confused? Pure speculation of course.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on May 28, 2018, 12:53:15 PM
Dang, I hate to see a bug here. I was being unorthodox and loaded an opponent's units into the player's garage. Not sure if this is causing any issues. I could rebalance the scenario so these units didn't exist, although I thought it was a fun surprise.

pente, thanks for the complement on the new scenarios! I had a lot of fun making them.

I loaded your saved game on my machine and both garages crashed as reported. So it is not just a Wine problem as I'm running Windows 10.

I tried saving a new round of the scenario a couple of times, exiting Outpost 2, and then loading the game. The garages still worked fine when doing this. So something besides simply saving/loading has to happen to trigger the crash.

When I have some more time, I'll try manually reviewing the code and seeing if the Garage is referenced anywhere else in a suspicious manner. I may also try loading debug symbols with the scenario in play. However, since the crash is happening inside the Outpost 2 side and not in the scenario code, this may not reveal anything.

Thanks for checking the bug out Hooman.

If anyone else wants to take a crack on the bug, the source code is linked in the first post of this topic.

-Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on May 29, 2018, 12:36:27 PM
Hmm, interesting trick, loading opponent vehicles into a garage. I like the deviousness of that. ;)


I spent some time analyzing the code where the crash occurred. The data being accessed are the cargo bays, which should be an array of 6 Unit pointers. The pointers do seem to point into unit data, however, the referenced units appear to be dead. The flags and key values that are checked for dead units are all set to indicate the unit is dead. Further, the field for the virtual function table pointer does not contain a valid pointer. That's the actual cause of the crash, when it tries to dereference the virtual function table. Perhaps the field was cleared when the unit died.

Interestingly, the field for the virtual function table pointer is not simply cleared to zero, but contains a low valued number, like an index. I've noticed during game save/load, the virtual function table fields can be temporarily translated to an index (since saving/loading a pointer to disk is generally a bad idea). Perhaps the low valued indexes are related to the translations done during saving/loading the game. Maybe they aren't translated back to virtual function table pointers for dead units. This is all just speculation of course.

As to how you ended up with dead units in the garage, I have no idea.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: lordpalandus on May 29, 2018, 01:53:28 PM
Perhaps it is how units are stored in enemy garages... as dead units, that only become alive when they are removed. As I've never seen an enemy store a unit in one of their own garages, whether in campaign or colony missions. So it may be some kind of weird interaction with how enemy garages work and how player garages work, that creates the issue.

I'd check to see how an enemy garage handles it's units, particularly garages that start with units inside them (ie that one Plymouth attack mission, where you attack the Eden colony).
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on May 29, 2018, 05:11:42 PM
Hmm, I wonder if it's related to dock damage.

If you move a vehicle onto an enemy dock, the vehicle will take continuous damage. Might also damage allied vehicles too. Keeps you from blocking other people's docks, especially early game. I once tried that, running a scout or robo surveyor over someone's structure factory dock. If you stay there, your vehicle dies. Though if you repeatedly run it across without staying, you take less damage, and you can still bump their Convecs off the dock. That can really upset people's build flow.  ;) It also really irritates people. To the point there was a rule against that the next game.  :-[
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on May 30, 2018, 02:32:10 AM
I'm not sure about the dock damage. The vehicles exit the bay fully healed even when they are an opponent's unit in your garage. Also, the health bar in the garage displays fully green and doesn't reduce over time. I tried waiting sufficient time for the units to die if they were taking damage, saving the game, and then loading. Everything worked fine and the units exited the garage at full health.

Hooman, do you think that vehicles placed in a garage would need to be recorded in ScriptGlobal for save/load purposes? Currently, I am not tracking these units in ScriptGlobal like all the other units. If this is needed, I guess I would have assumed that the game would crash more consistently after loading?

pente, could you verify which difficulty the saved game you sent is using?

Below are the 2 helper functions to load vehicles in the garage that I use. I am not passing the vehicles outside of this function, so I'm pretty sure none of them should be somehow populating into a fight group or patrol or anything. Excuse the bad code as I was still very new to C++ and wrote this for my own consumption only.

Code: [Select]
void VehicleBuilder::CreateVehicleInGarage(Unit &garage, int bayIndex, map_id vehicleType, map_id cargo)
{
UnitDirection oldDirection = unitDirection;
unitDirection = UnitDirection::West;

Unit vehicle;
CreateVechLightsOn(vehicle, vehicleType, garage.Location(), cargo);
vehicle.PutInGarage(bayIndex, garage.Location().x, garage.Location().y);

unitDirection = oldDirection;
}

void VehicleBuilder::CreateVehicleInGarage(Unit &garage, int bayIndex, Truck_Cargo truckCargo, int cargoAmount)
{
UnitDirection oldDirection = unitDirection;
unitDirection = UnitDirection::West;

Unit vehicle;
CreateVechLightsOn(vehicle, map_id::mapCargoTruck, garage.Location(), map_id::mapNone);

if (cargoAmount > 0)
{
vehicle.SetTruckCargo(truckCargo, cargoAmount);
}

vehicle.PutInGarage(bayIndex, garage.Location().x, garage.Location().y);

unitDirection = oldDirection;
}

Thanks,
Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on May 30, 2018, 08:01:32 AM
I was able to create another corrupted save file (normal difficulty). The only orders I issued in game were moving the units slightly. (I checked the two saves that were previous to the save I uploaded before, and even the first one was corrupted at mark 31 before I had engaged the enemy.) I did the following steps, I believe:

1. wait until mark 30
2. check garage (fine)
3. save game
4. check garage (fine)
5. load game
6. check garage (fine)
7. save game (attached)
8. check garage (crash)
9. load game from step 7
10. check garage (crash)

So maybe loading doesn't trigger it, rather saving might (perhaps saving triggers some kind of garbage cleaning internally). I tried following the steps above multiple times and wasn't able to get it to happen again. I tried saving over and over but wasn't able to trigger it that way either. I am still suspicious that saving/loading might be responsible but it is certainly not reliably so.

I also tried restarting the mission about 30 times and just checking the garage. Every time it was fine at the start of the mission.

Across all four corrupted save files that I have, both garages are corrupted.

Nice trick with the scouts. Almost a shame it was banned... almost :)
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on May 30, 2018, 09:45:23 AM
I forgot to say, I think the first save file I uploaded was hard difficulty, but I don't remember for sure.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on May 30, 2018, 01:30:03 PM
Ok, even simpler: I started a new game, immediately saved four times (no loading, no touching anything, no waiting) and the garage crashed. So loading is definitely not necessary -- saving is sufficient. Either that, or there is just some small random chance of corruption at the beginning of the game.

Again, on multiple re-tries I have not been able to replicate this procedure, even with restarting OP2 between tries. Funny how both this time and the previous time I got a corruption on the first attempt and then couldn't get it again.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on May 30, 2018, 02:48:46 PM
Well, doesn't sound like it's related to dock damage then.

I doubt it's related to ScriptGlobal at all. That shouldn't have any impact on this issue.

And my gosh, you're using a global unitDirection  :o
 ;)

I don't see any obvious issues with the code. Might be something else entirely.

Maybe we can set a breakpoint to monitor for unit death, or for writes to the unit vtbl field for units stored in the garage. Would help to know how to reliably reproduce the problem though, so it can be caught in the act. The saved games were very useful for the initial diagnosis, though they are a snapshot from after the corruption happened. Being able to reliably reproduce the point in time where the corruption happens would help further diagnose the cause.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on June 01, 2018, 01:50:47 AM
pente,

Thanks for the continued testing. On easy mode, the stored vehicles actually belong to the player as opposed to being the enemy's. If you don't mind taking a few minutes and trying to reproduce the crash on the easy difficulty setting, I would appreciate it. If we cannot reproduce on easy, it would point strongly to the problem being associated with the enemy units being stored in the player's garages. In this case, I can just make the units friendly at all difficulty levels to keep the crash from happening. However, I'm not aware of any other fan made scenarios that actually use garages, so I don't necessarily want to assume the problem is caused by the enemy unit in player's garage situation.

Hooman,

No making fun of my random global variables that actually make coding the scenario more difficult.  :-\
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on June 02, 2018, 12:39:39 AM
I was able to get the crash on easy by saving four times immediately after starting the mission.

Again, I got it on my first attempt, and restarting OP2 and retrying the exact same thing 7 more times failed to trigger the bug again. It is hard to imagine what is going on in my system that causes the bug to show up only if OP2 has been closed for some time while I do other tasks, and then is triggered by specifically saving. Maybe some kind of memory management bug inherent to OP2, which you would never see in the normal course of the game (there is the plymouth mission with loaded garages, but you can't select them). I guess pre-loaded garages is just dangerous territory for mission authors, unfortunately? I would be curious if anyone else can replicate the bug.

I also went ahead and actually tried to beat the mission, and finally got it and the next one on hard. (Having already beat the third on medium I feel no need to repeat it on hard.) I have no idea how you were supposed to get to 80 colonists on the second mission before the tigers show up -- I pulled every trick I thought of (even built a GORF!) and still depended on a healthy amount of luck to get there.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on June 02, 2018, 11:28:04 AM
Hmm, that's some good debugging info. In particular, it's good to know the crash happens even on easy when the vehicles are owned by the same player as the garage. Perhaps there is something else in the code base which is interacting with the garages in an unexpected way. My guess would be Trigger or FightGroup related.

Pente, you seem to be doing a pretty thorough job trying to reproduce this bug. Would you be at all interested in trying some advanced debugging tools?



I'm taking a peek at the code.

This is probably unrelated, but in ReviewRogueFightGroupStatus, the logic here looks reversed. Is this to recycle the fight groups? I assume if TotalUnitCount() == 0 then it should set RogueFightGroupsAvailable = true. In particular, there is no code anywhere in the project aside from the initialization that sets RogueFightGroupsAvailable = true, meaning once used, the groups can never be used again.
Code: [Select]
if (scriptGlobal.RogueFightGroups[i].TotalUnitCount() == 0)
{
scriptGlobal.RogueFightGroupsAvailable[i] = false;
continue;
}

On another note, if TotalUnitCount() == 0 does mean the group is available for use, that can be used directly to track group availability rather than relying on a secondary array.

Of course, other code I saw seems to suggest the group tracking might be more of a countdown until some event happens, with no reuse. The intention of the code is not immediately obvious.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on June 02, 2018, 12:51:32 PM
pente,

Thanks so much for the testing. Really good to know that the problem is independent of which player's units are loaded in the garage. Has anyone else used garages in a colony game or have advice on how to load them without a crash? I just tried saving 5 times on my computer, closing Outpost 2, and reopening. Didn't reproduce the crash.

For scenario number 2, it is designed on hard so that you have to play really well and still be in progress of evacuating your base when the enemy shows up. It actually makes me really happy you could barely beat it, and only as the enemy were descending on your facilities.

I have a 4th scenario in the series partially created. Would you be interested in playing it if I take the time to finish programming it? For these scenarios I try pushing the boundaries of the API in different ways to make them a bit unique from the typical Outpost 2 campaign game or colony game.


Hooman,

In this scenario, each fight group will only be activated one time. The code you highlighted exists to check if the fight group hasn't been wiped out by a natural disaster or the player attacking it before tasking it with a patrol or self-destructing it. Technically I don't think the code would crash if you tasked an empty fight group, but it makes for less action.

I made some minor documentation updates around that part of the code and committed to the repository so it should be clearer to others. I didn't expect the code to get so much attention.

I could rewrite the scenario to not have vehicles in the garage or we could just note the bug in the change log. Since the garage isn’t used much, I was happy incorporating it but nothing like ruining a series of scenarios than by repeatedly crashing in the first installment. I’m a little split here on what to do.

Thanks,
-Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on June 02, 2018, 02:58:59 PM
I am game for further testing, within the time constraints of my various obligations. And subject to the fact that I still can't /reliably/ reproduce the bug on demand. You could try making a very simple map -- nothing but a bunch of garages and units, variously with or without the scenario-specific scripting in it.

I'd suggest that it is sufficient to just warn people about the potential bug (and ask them to report it if they encounter it -- maybe in the scenario briefing itself? I don't know what fraction of your players actually check the forums) until at least one other person has confirmed that they have seen it.

For mission 2, the only way I found that worked involved taking the bottom-right beacon, but on the first few attempts the enemy literally spawned in my base, so there is a strong element of pure luck in whether that happens. (I also had to very carefully place the provided kits in the correct order, train almost no scientists until almost the end, do the usual morale stuff, and build walls and scouts to block and distract the attackers.) I had enough extra resources to put some supernova lynx where the tigers spawned, but they didn't auto-target the enemy. If I had been willing to commit to winning the fight against the tigers I could have built a bunch of decoy tokamaks around the map and supernova'd the tigers when they were emp'd. (This would give me more time to get to 80 colonists, so I could have taken the left beacon, which removes the element of luck of enemies spawning in my base.) That would feel very cheap though, if it even worked at all. Does the game spawn more enemies if the big attack is killed off?
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on June 03, 2018, 08:28:53 AM
I don't think the garages need to be modified at this point. We don't even know the exact cause of the bug yet. It might be elsewhere, and the garage problem is just a symptom of it.


If I can find the time to do it, I'd like to load up the level under OllyDbg. After the vehicles are loaded into the garage in InitProc, I want to set some hardware breakpoints on the vehicle data, such as HP, flags, or the next pointer (3 fields related to unit death), to watch for when those units get killed off. This is likely well before the virtual function table pointer gets overwritten.

My guess is, the virtual function table pointer is overwritten during saving, and not restored during loading because the units are dead, and so the records are considered free. However the garage is still referencing the dead units since it's not designed to clear out units that get killed inside. And it's not designed to clear out units that get killed inside because there should be no way to kill units inside. If I had to guess how they got killed, I'd suspect a memory management error concerning groups or triggers, though don't have any particular place in mind to start looking. Hence why I'm hoping a hardware breakpoint will tell me where it's happening.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on June 10, 2018, 05:08:22 PM
I've done a bit more digging. Not quite at a direct cause yet, but I suspect this is a bug in the game engine.

The game stores several pointers in the Unit objects. Pointers aren't very meaningful when written to files, since allocated memory tends to get moved around, with no guarantee an object will be at the same location in a future instance of the program. To avoid this issue, the game translates between Unit pointers, and Unit indexes. Just before it prepares to save a game, it runs through objects in memory, and translates these values in place. It then writes the images of all the objects to disk. Afterwards, it has to restore the memory to proper pointers so it can continue operating normally. That part of the code is run directly after saving a game to disk, as well as after loading a saved game file. The result should be all the indexes are translated back to pointers.

However, the game doesn't seem to translate the values correctly for units stored in garages. During a save, the field gets corrupted. The reason why is a bit odd, and why it doesn't cause problems more often is something I still need to look more into. I think by virtue of not much code accesses units in garages means there is a limited opportunity for things to go wrong when the values are in a corrupt state. Further, when you unload units, it resets the field to sensible data.

In a Unit object, the first 4 fields are union values. What they mean depends on context. The first field is normally the object's virtual function table pointer. A virtual function table pointer can be a handy way of distinguishing the exact type of an object. In a somewhat natural correspondence, this field gets translated to a map_id unitTypeIndex when generating a saved game, and translated back to a virtual function table pointer afterwards or upon loading. The next 3 fields store Unit pointers. Normally they mean something like next, prev, playerNext. When generating a saved game, these fields are translated to corresponding uint unitIndex values.

Now here's where things get weird. Normally you'd think a Unit* value would either point to a valid Unit object, or would be set to the 0, the nullptr. A uint unitIndex might be set to 0..2047 (for a unit limit of 2048 units), where 0 is kind of a dummy non-existent unit that never gets saved to disk. You might expect a dummy object like that, or something like -1 or UINT_MAX used to denote no valid unit. Instead, when the game is operating normally, and the field should represent a Unit*, it uses the value -1 to represent no valid unit. In particular, when a unit is dead, its next pointer is set to -1. The game has many special detects for -1 values sprinkled throughout the code, much like you might expect of checks against nullptr. As such, a value of -1 in this field, although odd, is not a problem.

When a unit is loaded into a garage, the next field is set to -2. There are no checks against -2, so it's assumed to be a valid pointer value, which clearly it's not.

When the game is saved, the -2 is taken as a valid unit pointer, and a calculation is done to determine a unit index from the unit address. It uses a pretty standard formula for converting array element addresses into array indexes:
 index = (elementAddress - arrayBaseAddress) / sizeof(ElementType)
That's all fine when the element actually is within the array. However, implicit in this formula is an assumption:
 elementAddress is within the bounds of the array and an integer multiple of sizeof(ElementType) past the arrayBaseAddress
This is not true for -2, and so the result is a garbage value.

What I think might be happening, is through some as yet unknown code path, this garbage value somehow becomes -1. I don't think this can happen directly, which is perhaps why the game doesn't typically crash when accessing a garage after saving. Instead, I think what happens is the garbage value gets translated to and from pointer and index form until it ends up set to -1. At this point, the checks for that sentinel value are hit, and the unit is considered dead. During the post-save and load code, when it translates indexes back to pointers, it sees the -1 sentinel value and skips over index to pointer translation. In particular, it doesn't bother to translate the map_id unitType field back to the virtual function table pointer. Trying to access the virtual function table while the unitType index value is still loaded is the direct cause of the crash.

There's more weirdness too. When a unit is loaded into the Garage, it is removed from the list of active units, and the prev pointer is set to uint gameTick. I suspect this is used to track when a unit died. It might also be related to repair progress, but I think that's handled by another field the gameTick gets copied into in more garage specific code. The game also seems to clear this value to 0 at times.

I'd like to do a bit more work to find out how the -2 gets translated to a -1, and see if there is a way to reliably reproduce the bug.

We could probably patch this. Though I'd want to know a bit more about the problem path before trying to devise a patch.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on June 15, 2018, 12:11:59 PM
I'm impressed at how it's possible to analyze so much of the OP2 internals without having access to the OP2 source code. Good luck with hunting down this problem the rest of the way.

Is there something different between the player loading a unit into the garage or a unit already being loaded at the beginning of the scenario that causes only the latter to be vulnerable to corruption?
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on June 19, 2018, 08:28:18 PM
Should be no difference in regards to loading vehicles into a garage at the start of the level versus during game play. In terms of how the DLL interfaces with the game, the vehicle is actually created outside the garage first, and then loaded into it.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on June 08, 2019, 11:21:22 PM
I posted a new version of the mission and moved the downloads to the Github release page. See the first post for the new location of the downloads.

The new version properly sets HFLInit in DLLMain. In previous versions, HFLInit would not be initialized when loading a game, only when the game is first started. I'm not sure if this could affect the garage bug.

With the new version, old saved games will still exhibit the garage bug if they are present previously.

I tried saving/loading a half dozen times and was unable to reproduce the bug. Don't know if HFL being present changed anything or if I was just lucky and didn't encounter the bug. Not sure HFL even plays in this bug at all.

-Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on July 18, 2019, 11:12:27 PM
It took me a few tries, but with v1.0.1 off of github I managed to duplicate the garage bug (see dump below).

I have played around some more to see if I can find a way to reliably reproduce it. The steps are to start a new mission, save twice, and then click on a garage and press S. Saving only once never works, and more than twice doesn't help if twice wasn't enough. Loading is not necessary.

If Outpost has been closed for more than 5 minutes when I start it, and I immediately try to trigger the bug, the bug will happen. If Outpost crashed due to this bug, restarting Outpost and immediately trying the bug will always succeed. However if Outpost has been running normally for a while, the bug will usually not happen, even if I close and restarting Outpost.

I continue to think there is something funny going on in my system to explain this behavior.

I've attached two saves, which were made immediately after each other. In SGAME2 the bug does not happen, in SGAME3 it does. (Normal difficulty.)

Code: [Select]
Unhandled exception: page fault on read access to 0x00000001 in 32-bit code (0x004634cb).
Register dump:
 CS:0023 SS:002b DS:002b ES:002b FS:0063 GS:006b
 EIP:004634cb ESP:0034f9e0 EBP:04261c50 EFLAGS:00010202(  R- --  I   - - - )
 EAX:00000001 EBX:00000000 ECX:04261cc8 EDX:00456ce0
 ESI:00565d30 EDI:005663d4
Stack dump:
0x0034f9e0:  00000000 00565d30 00565d30 004d4fb0
0x0034f9f0:  00565d34 00000001 00456ce0 00565690
0x0034fa00:  04261cb0 0056d250 ffffffff 0047efb9
0x0034fa10:  0047efb9 04a04978 ffff0000 00000000
0x0034fa20:  0049d244 00000000 00577ee8 59960000
0x0034fa30:  00570010 00463153 00575d18 0045d1f7
Backtrace:
=>0 0x004634cb EntryPoint+0xffffffff() in outpost2 (0x04261c50)
  1 0x04265850 (0x004cfca0)
  2 0x0043ada0 EntryPoint+0xffffffff() in outpost2 (0x0041d630)
0x004634cb EntryPoint+0xffffffff in outpost2: call *0x0(%eax)
Modules:
Module Address Debug info Name (40 modules)
PE   370000-  3e0000 Deferred        out2res
PE   400000-  5bc000 Export          outpost2
PE 10000000-10011000 Deferred        op2ext
PE 11000000-1102d000 Deferred        c_euf
PE 20000000-2001a000 Deferred        odasl
PE 7b430000-7b5ed000 Deferred        kernel32
PE 7bc30000-7bc34000 Deferred        ntdll
PE 7cde0000-7cde4000 Deferred        dsound
PE 7d120000-7d123000 Deferred        winealsa
PE 7d160000-7d164000 Deferred        mmdevapi
PE 7d2f0000-7d2f8000 Deferred        oleaut32
PE 7d520000-7d523000 Deferred        api-ms-win-core-localization-l1-2-1
PE 7d550000-7d553000 Deferred        api-ms-win-core-fibers-l1-1-1
PE 7d590000-7d593000 Deferred        api-ms-win-core-synch-l1-2-0
PE 7d6c0000-7d6c3000 Deferred        msvcr120
PE 7d7b0000-7d7b4000 Deferred        uxtheme
PE 7d9d0000-7d9d3000 Deferred        concrt140
PE 7da10000-7da14000 Deferred        winex11
PE 7dce0000-7dce3000 Deferred        api-ms-win-crt-heap-l1-1-0
PE 7dcf0000-7dcf3000 Deferred        api-ms-win-crt-stdio-l1-1-0
PE 7dd10000-7dd13000 Deferred        api-ms-win-crt-string-l1-1-0
PE 7dd20000-7dd23000 Deferred        api-ms-win-crt-runtime-l1-1-0
PE 7dd30000-7dd33000 Deferred        vcruntime140
PE 7dd80000-7dd84000 Deferred        ucrtbase
PE 7de90000-7de94000 Deferred        msvcrt
PE 7df90000-7df93000 Deferred        msvcp140
PE 7e080000-7e084000 Deferred        iphlpapi
PE 7e0b0000-7e0b4000 Deferred        ws2_32
PE 7e0e0000-7e0e4000 Deferred        wsock32
PE 7e100000-7e104000 Deferred        imm32
PE 7e130000-7e133000 Deferred        usp10
PE 7e1b0000-7e203000 Deferred        comctl32
PE 7e300000-7e309000 Deferred        msacm32
PE 7e340000-7e3bd000 Deferred        winmm
PE 7e430000-7e434000 Deferred        rpcrt4
PE 7e4f0000-7e518000 Deferred        ole32
PE 7e650000-7e654000 Deferred        advapi32
PE 7e6f0000-7e6f7000 Deferred        gdi32
PE 7e850000-7e950000 Deferred        user32
PE 7efe0000-7efe4000 Deferred        version
Threads:
process  tid      prio (all id:s are in hex)
00000008 (D) Z:\home\user\games\outpost2\test\137\Outpost2.exe
00000036   15
0000002d   15
0000002c    0
0000002b    0
00000009    0 <==
0000000e services.exe
00000021    0
0000001a    0
00000013    0
00000010    0
0000000f    0
00000011 plugplay.exe
00000017    0
00000016    0
00000012    0
00000018 winedevice.exe
0000001c    0
0000001b    0
00000019    0
0000001d explorer.exe
00000027    0
00000026    0
00000025    0
0000001e    0
0000001f winedevice.exe
00000024    0
00000023    0
00000022    0
00000020    0
System information:
    Wine build: wine-4.9
    Platform: i386
    Version: Windows 7
    Host system: Linux
    Host version: 4.20.4-gentoo
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on July 25, 2019, 09:13:17 PM
pente,

Thank you for retesting and the detailed debug data. I've left the bug as outstanding in the upcoming Outpost 2 release. Unfortunately, I'm not sure that I have the skills to debug on my own.

-Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on July 27, 2019, 11:52:18 AM
I am mighty curious what is causing this bug, but so long as I'm the only person who's ever run into it I don't think anyone should be losing any sleep over it -- I've beaten this mission enough times that this bug is no concern to me. Still probably my favorite Outpost 2 mission though!
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on July 28, 2019, 07:07:08 PM
I've managed to reproduce the bug independently in the past, although not nearly as reliably as you have. So not just limited to your computer. I think maybe most people don't bother to report it or are not saving until after they deal with the garage contents.

Thanks for the compliment on the mission. I may pick up work on Mesa Missions again in the near future, the next mission in the series.

-Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on August 19, 2019, 11:27:43 PM
I took a look at the new saved game files. I haven't really found much in terms of a cause, other than what we already know.

The garage I clicked on had 3 units inside. The bay section of the Garage unit had 3 valid looking pointers to 3 unit records. Each of those 3 unit records had the virtual function table pointer replaced with the unit type index (enum mapId). It's standard practice for the game to replace the virtual function table pointer of each unit with the corresponding mapId value when saving, and then restore it afterwards (to continue playing after saving), or when loading. As to why it's not being properly replaced with the virtual function table pointer, I don't know.

The double save requirement to reproduce this bug seems interesting. I wonder if that might be related to the translation done during save/load. It seems a little too suspect.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on August 21, 2019, 01:26:59 PM
Was there any difference between the two save files? I saved them as quickly after each other as I could press the button, so they should be almost the same game state, except that one is broken and one not. I didn't get the two saves on the same tick, so it's not as easy as looking at a hexdump and seeing which bytes changed (I checked just in case :) ) but maybe you have a more sophisticated means of analysis.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on August 25, 2019, 02:11:26 AM
Hmm, interesting thought. It made me realize we don't really have the tools to do a proper saved game file compare. I tried just opening them in a hex editor, but the two files are different sizes, and show a lot of differences right from after the initial header. We do have code to load and parse the map data into memory, but not the saved game specific data, such as unit data where the problem lies along with any likely clues. Even if we did though, we don't have any code that would compare them with each other. It would all need to be written.

With that said, the in-memory analysis has already determined which field has gone bad, so I'm not sure a comparison of the saved game data would give additional information. We already know which field, we just don't know why that field went bad.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on August 25, 2019, 01:24:50 PM
I spent some time on this with OllyDbg. Unfortunately I'd forgotten much of what I'd already posted on June 10th in this thread, and so mostly just reconfirmed it.

The bug appears when a garage stored unit has it's next pointer set to -1. This is generally a marker of an unused unit record. When the game is loaded, some patch up code is run to convert certain index fields back to pointers, as well as converting the unit type to a virtual function pointer. This patch up code skips over unit records that have a next pointer set to -1. This means units with a next pointer of -1 won't have a proper virtual function table pointer set. If any virtual method calls are made for that unit, the game will crash. Normally this is fine, as unit records with a next pointer of -1 are supposed to be unused, and so should never have any virtual methods called on them.

In the case of the garage bug, the stored units have their next pointer set to -1, yet the garage still references them. Hence there are referenced unit records which have been marked as unused, and have no valid virtual function table pointer set. When you view the contents of the garage, it attempts to call a virtual method on the stored units. The virtual method is to get a pointer to unit type information, so it can extract the name of the unit type for display purposes.

In the crashing saved game (saved_twice), the stored units have their next pointer set to -1. In the good saved game (saved_once), the stored units have a valid next pointer set.



Methods of interest:
Code: [Select]
00435CD0  Function: Map.UnitIndexesToPtrs()  [[vtbl] and [next, prev, playerNext]]
0041CF40  Function: Unit:Building:Garage.OnLoad()
004632B0  Function: CommandPaneView:BuildingStorageBays.???()  [DrawStorageBayPane]



It hasn't been observed how the stored unit records are marked as unused. There is however a bug in the code relating to some unique garage behavior.

When a unit is stored in a garage, the next pointer is set to -2. The game has lots of sentinel value checks for -1, but none for -2. Normally, if the field is not -1, it is expected to be a valid pointer to another unit record. In particular, the patch up code tries to convert pointers to indexes for all next pointers not set to -1. This calculation is not valid for the value -2, and so this corrupts the field when saving. The reverse conversion will also produce an unexpected and corrupted value.

It's not known if this corruption leads to further problems, or if it somehow gets corrected. In particular, I suspect the next pointer field would be restored to a proper value when a unit is unloaded from the garage. The field should remain largely unused while the unit remains in the garage, so the corrupted value may not be noticeable.



Given that this bug has only presented for this one level, and given that it does so consistently, there's likely something level specific that triggers this behavior. It may be caused in part by the corrupted next pointer field, or it may be caused through some other means. In particular, the garage stored units that end up with a next pointer field of -1 also have their flags field set to indicate an unused or dead unit. This suggests there may be some other mechanism involved that sets both the next pointer field and the flags field to indicate an unused unit record.

The problem may or may not relate to the corruption from the general bug, and most likely to something level specific. It's unlikely the problem -1 value would come from corruption alone.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on August 26, 2019, 07:27:49 PM
Thanks for taking so much time on this issue again. I had been hoping that, since it takes two saves for the crash to occur, there would be some form of visible corruption in the first save (as after all, *something* must go wrong in the first save to put it into a state where the second save corrupts it), but I appreciate your point about the difficulty of doing that kind of analysis.

I guess we'll just have to remain in the dark about this bug until one of these Outpost remake projects gets far enough along that we have a playable successor game. And hopefully they don't replicate this particular bug :D
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on August 27, 2019, 02:45:11 PM
I would actually like to keep working on this. My current problem though is that I'm unable to reproduce the problem on my own. Your saved game exhibits the crash, which provided the info for the initial investigation. It doesn't quite tell how the unit data got corrupted or units got killed within the garage though. So far I've been unable to reproduce the transition from good unit data stored in garages to bad unit data stored in garages. If I could reliably reproduce that, there would be more I could determine.

I've tried restarting the level and saving twice multiple times, and then checking the garage, but so far no problems. I've tried restarting from within the level, from going back to the main menu, and from restarting Outpost 2. I also tried loaded up the bad save to get a crash first, then reloading the game and trying again. Nothing seems to cause problems with the garage. I'm not sure how often the problem occurs, though I probably tried maybe a couple dozen times.

Is there anything more you can determine about reproducing the problem from a good state? How often does it fail?
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on August 28, 2019, 06:37:53 AM
Hooman, if you are interested, we can pair program and use my machine to reproduce the bug more reliably. I would generally be available this Monday at the earliest.

Thanks,
Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on August 28, 2019, 02:13:08 PM
That would be quite helpful. I was becoming increasingly tempted to start giving pente instructions on how to use the OllyDbg debugger. ;)  (The offer still stands) :P

I plan to be busy Monday, though if it's done early, or late, I may be able to sneak it in.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on August 29, 2019, 12:18:07 PM
How about next Friday after 6pm Eastern or Saturday from 8am to 10pm Eastern?

If you have a reference for me to read on beforehand, it might save some explaining. As all I remember from our last session is EAX means something about copying a variable... I was trying to load the udd file into ollydbg the other day and couldn't figure it out although I didn't try too hard. I think you may have posted how to do it on the forum somewhere. I should look that up beforehand.

Happy if pente wants to join or if you two want to work without me. There are plenty of things to do.

Thanks,
Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on September 01, 2019, 04:17:55 AM
It looks like OllyDbg is a windows program? I'd be happy to try out any experiments you want if you can give me step-by-step instructions that work on linux.

I don't expect to be available for live debugging but if I remember I'll check back here on the forum before Friday.

I haven't done anything of substance with assembly since 2005 so I'm going to be struggling to understand any disassembled output.

The bug seems to be pretty consistent (>75% ?) under the conditions I described above. However it does seem to be sensitive to how long it's been between closing Outpost 2 and starting it, which makes trying to trigger the bug an unnerving process.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on September 01, 2019, 10:12:43 AM
OllyDbg works just fine on Linux under Wine. That's how I use it these days.

If you want to try it out, I highly recommend downloading the OllyDbg comments file for the Outpost 2 executable. It contains years of reverse engineering comments and code labels. You can find Outpost2.udd at:
https://svn.outpostuniverse.org:8443/!/#outpost2/view/head/OllyDbg
(The site's certificate may be out of date, so the browser may show a warning about an insecure connection)

Important OllyDbg settings:
Options -> Appearance -> Directories : UDD Path    (Copy Outpost2.udd to this folder)
Options -> Debugging Options -> Security :
  Ignore path and extension
  Ignore timestamp
  Ignore CRC of code section
Check those boxes, or OllyDbg may ignore the .udd file and then overwrite it with a fresh analysis.

To check if the comments file has been loaded, open the Names window (shortcut: Ctrl+n), and look for entries with Type set to User. These are the user defined labels we set while reverse engineering. There should be about 4500+ user defined labels. If the comments file was not loaded, you'll only see the imports and exports (about 750+).



@Brett: The MOV command copies a value. Values can be stored in registers, such as EAX. :P

I'm afraid I don't have much in the way of a reference to point people to, though perhaps I should prepare something.

I may be free on Monday after all. I won't be free on Friday. Leaving early to go camping for the weekend.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on September 01, 2019, 06:22:52 PM
Hooman, Pente,

It took me about a 18 tries, but Outpost 2 finally crashed. Just like pente described, if I save twice in a row and then go to garage and press 's', it will eventually crash. Not sure why it is so much more prevelant in the Linux/Wine combo than Win 10, but it is not isolated to one machine or operating system.

Hooman,

I can be available at 7PM Eastern time tomorrow (2 SEP) for about 2.5 hours if that works for you. Since I thought you were busy, I made plans earlier in the day to do some pair programming on a multiplayer mission. Hopefully my brain isn't jelly by the end of the earlier session. :)

If you are good with it and available, primary Team Viewer, secondary Discord for screen sharing / audio. I'll jump into the chat room prior, but feel free to poke me on Team Viewer chat as well.

I found an undergraduate book on x86 assembly programming / debugging that I started reading through the intro. Probably not far enough in to be of practical help at this point but maybe in the future? I've got ollydbg and HxD already on my CPU. Also, ollydbg is successfully loading the .udd comments, so first step accomplished!

Hopefully we can get it to crash eventually and it doesn't take more than 18 times. The good news is, it is fairly quick to attempt crashing.

-Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on September 02, 2019, 01:40:41 PM
I will be available at that time.

I've got Team Viewer and Discord working on my old desktop. I'm having trouble finding a decent microphone that works, so the audio quality might suffer a bit. Still working on some options.

The amount of assembly knowledge required should be fairly minimal. The compiler mostly generates the same few instructions, so it's easy to learn the bulk of it quite quickly. Plus the best way to learn is often to be exposed to it.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on September 02, 2019, 05:02:23 PM
I cannot join today.

I did try OllyDbg and was able to run Outpost 2 with it. I'm not sure what you had in mind to do, so I tried to duplicate the crash and see if anything interesting would happen.

I was unable to cause the crash from within OllyDbg. I repeatedly alternated between using OllyDbg and running Outpost 2 directly, and was able to trigger the bug every time from Outpost 2 directly, and never from within OllyDbg. This makes one suspect some kind of race condition, but there are other possible explanations of course.

Also when using OllyDbg, I got an error message when launching Outpost 2 before the menu comes up:

"Bad or unknown format of 32-bit executable file 'C:\windows\system32\oleaut32.dll'"

I did not get that message when running Outpost 2 directly.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on September 02, 2019, 05:15:39 PM
Well, I am unexpectedly freed up and expect to have time to join for at least a little bit at the suggested time. Let me know the best way to get in contact with you and I'll see if I can do so.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on September 02, 2019, 10:17:24 PM
Ahh gee, I didn't see your post until after we finished.

So the OllyDbg session went well. We were able to reproduce the bug while watching things unfold with OllyDbg. We spent a bit of time on initial setup getting the values we needed, and setting breakpoints at reasonable places. Brett did a quick run without OllyDbg to see if the bug could be reproduced by restarting the level, without having to go through the main menu, as that would save us quite a bit of time. After about 15-20 tries we found that was indeed possible. We then got setup in OllyDbg, and repeated the process. As the unit array potentially moved around each time the level was restarted, we found we had to clear and reset the hardware breakpoint on the field we were watching each round. This was a very tedious process. Brett managed to successfully reproduce the bug under OllyDbg after about 15-20 rounds. We kind of noticed something peculiar was up slightly before we witnessed what we were looking for. The next pointer on the unit record of a vehicle in a garage was first unexpectedly cleared to 0, and then shortly after was set to the -1 value we were looking for. As expected, the game crashed when we examined the garage contents after this.

The two writes before the crash were done in Map.UnitIndexToPtrs() and Map.SaveUnits. These were both functions that had been previously analysed, and were responsible for the translation between unit indexes and unit pointers during saving/loading. It was also related to the corruption of the value mentioned in earlier posts. Units have their next pointer set to -2 when loaded into a garage. This -2 value is not handled correctly by the conversion done during saving/loading. With a bit of luck, this corruption can lead the value to be converted back to 0. Normally a value of -1 is special cased to map to 0, and no valid value should map to 0. This means that during the reverse process, the 0 is special cased and maps to -1. Hence the -2 value is corrupted, and then converted 0, which is then mapped to -1, which indicates a dead unit, and so the game stops mapping the unit type back to a virtual function table, which leads to the crash.

To exhibit this bug, the -2 value must be corrupted in such a way that the reverse process maps the corrupted value back to 0. For this to happen, it depends on the address of the unit array. Each time the level is run, the unit array is reallocated, and may move around in memory. Certain addresses will trigger corruption in a way that the value gets mapped to 0. Hence the bug does not always exhibit itself every run.

This confirms the bug comes directly from the conversion process of the -2 value. There are no other factors at play here. This also shows the bug is in the game itself, and not part of the level. Any level with a garage, with units loaded into it, which is saved twice, runs the risk of corrupting the unit records of the stored units, and whether or not this happens depends on where in memory the unit array happened to be allocated.

The good news is, we can patch this. It shouldn't be too hard to write a replacement function that includes special casing for the -2 value as well as the -1 value. That should stop the corruption from happening during saving/loading. It won't fix already corrupted saved games, but it should prevent the problem from happening again. Perhaps we can roll something out in the next update of Outpost 2.




I may post further details later on about the exact sequence of events and values, and sample problem memory locations for the unit array. For now though, I'll leave the addresses of the two final writes that lead to values of 0 and -1 in the next field.

In Map.UnitIndexesToPtrs() the value 0 is unexpectedly stored at:
Code: [Select]
00435D34  MOV DWORD PTR SS:[EBP],EAX    ;  unit.ptr[j]* = [Unit*]nextUnit

In Map.SaveUnits(StreamIO* savedGameFile) the value -1 is stored at:
Code: [Select]
00435BC9  MOV DWORD PTR DS:[EDI],-1    ;  Unit.ptr[j] = -1


Of particular note, is how unexpected the write of 0 was. The larger code sequence is:
Code: [Select]
00435D1B  |/MOV EAX,DWORD PTR SS:[EBP]                  ;  EAX = [Unit*]currentUnit.ptr[j]*
00435D1E  ||CMP EAX,-1                                  ;  Check if ([Unit*]ptr == -1)  [IsDead]
00435D21  ||JE SHORT Outpost2.00435D39                  ;  -> Set unit.next* = 0
00435D23  ||SHL EAX,3                                   ;  [EAX = unitIndex * 8]
00435D26  ||LEA EDX,DWORD PTR DS:[EAX+EAX*2]            ;  [EDX = unitIndex * 24]
00435D29  ||LEA EAX,DWORD PTR DS:[EDX+EDX*4]            ;  [EAX = unitIndex * 120]
00435D2C  ||MOV EDX,DWORD PTR DS:[<Map.unitArray[]*>]   ;  EDX = Map.unitArray[]*
00435D32  ||ADD EAX,EDX                                 ;  EAX = &unitArray[unitIndex]
00435D34  ||MOV DWORD PTR SS:[EBP],EAX                  ;  unit.ptr[j]* = [Unit*]nextUnit
00435D37  ||JMP SHORT Outpost2.00435D40                 ;  -> Skip instruction
00435D39  ||MOV DWORD PTR SS:[EBP],0                    ;  unit.ptr[j]* = 0  [null]
00435D40  ||ADD EBP,4                                   ;  EBP = &ptr[j+1]
00435D43  ||DEC ECX                                     ;  numPtr--
00435D44  |\JNZ SHORT Outpost2.00435D1B

The bad write of 0 occurs at 00435D34. This is a calculated 0 value. It occurs when the corrupted field value is such that &Map.unitArray[corruptedIndex] == 0. For that to happen, we must have corruptedIndex * 120 = -&Map.unitArray, which is a rather bizarre thing to have happen. (Here sizeof(Unit) == 120). This is in contrast to the special cased write of 0 that would normally happen at 00435D39, which happens when an index value of -1 is converted to a nullptr.

Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: pente on September 03, 2019, 01:16:00 AM
Well done on deciphering it. I guess that mystery is finally solved! That equality is very strange indeed, I guess these values are 32 bit, so for this equality to happen by coincidence would be fantastically unlikely.

I am a little unclear on which part of the corruption takes place in the first vs the second time the game is saved.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Vagabond on September 03, 2019, 01:59:08 PM
Pente, the game set the value to 0 first, then at a later check it translated the 0 to mean a null pointer, so it set the valid ID to -1. I believe it is a union field that can be either an ID or a pointer based on context and where it is used.

While random, I was only seeing about a dozen values come up like 68, 10, etc. I suspect by random there are still only s few choices available. I wonder if the specific hardware implementation could affect the likelihood of different values coming up.

It may still be awhile before the fix is implemented.

Brett
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on September 18, 2019, 03:31:24 AM
Yeah, we're doing some work to overhaul the module code in op2ext. Once that's done, we'll probably write up the patch as an internal module within op2ext.

As for the likelihood of getting that 0 value, it's probably not all that unlikely. Based on experience, it was roughly 1/15 tries. I haven't yet analysed exact details, but the math involves division by the unit size, which is 120 bytes. I suspect there is some modulo effect, which is why we were typically seeing small values in a limited range. Couple that with memory alignment requirements for the unit array, which is likely at least 4 bytes, we are further limiting possible values. Actually, if the memory alignment requirements were 8 bytes, which would be quite reasonable for an array of large objects, and that worked in conjunction with a modulus of 120, we might expect a given outcome to take on any of 120 / 8 = 15 values. Highly suspicious.

As things like memory alignment requirements are architecture specific, yes, there would be some degree of hardware dependence, though that choice would be made at compile time, and I wouldn't expect it to vary based on end user's machines. Granted, memory allocator's sometime defer responsibility to the OS's memory allocator, so there could be some difference in alignment possibilities if a particular OS/version imposes a larger minimum alignment than the language requires.
Title: Re: Evacuation Under Fire - Ply Campaign Scenario
Post by: Hooman on November 17, 2019, 06:48:36 AM
Quick update here, the garage save/load bug has now bubbled to the top of the priority list and is getting some traction. Some discussion can be found at:
Create Built In Module to address garage data corruption during saving (https://github.com/OutpostUniverse/op2ext/issues/204)

The conditions and mechanism of the bug now seem to be understood, and are detailed in the issue report. The next step is to create a reproducer level that contains a bit of memory hacking to ensure the problem case always occurs. That will help verify the conditions and mechanism are actually well understood and correct. It will also help with patch verification when the time comes. I expect the patch will be written sometime within the next few days. Hopefully a new release of op2ext containing the patch will be out within the next couple of weeks.