Evacuation Under Fire - Ply Campaign Scenario

[Vagabond]

Hey everyone,

I am releasing a Plymouth Campaign Scenario called Evacuation Under Fire. It should be a shorter scenario to play and has some unique play characteristics. The briefing is worth reading to know what is going on, even though it is kind of long. It displays before the level is loaded, so you don't have to read it here. Hopefully it is the first of a mini-campaign. The second scenario is basically done and should be released soon...

Compatibility
The scenario is compatible with Windows and WINE. Thanks to Hooman for testing compatibility with WINE.

Download: https://github.com/Brett208/OP2MissionEvacuationUnderFire/releases

Install Directions

 * Place the .map, .dll, and research tree (other .txt file) in your root Outpost 2 install directory.
   Some scenarios share maps (.map) and research trees (.txt). You should not need to install over already
   downloaded versions of maps and tech trees unless the map or tech tree is being patched for all scenarios
   using it.
 * Open Outpost 2 and search for the new scenario with the other colony games.
 * Good luck, and have fun!

Source Code: https://github.com/Brett208/OP2MissionEvacuationUnderFire

Bugs

* If you save and then load the mission before emptying vehicles in the garage, sometimes attempting to release a vehicle from the garage will crash the game. Unknown if bug is in mission logic or game logic.

Mission Briefing
Commander,

Unfortunately, your mission to reinforce Outpost Delta is no longer viable. Eden broke through Outpost Delta's defenses and destroyed many key buildings. You are now on a rescue mission for the surviving colonists.

Eden found a way to manipulate our vehicle command network. They can scramble all vehicle command algorithm processors linked to the local network but cannot take direct control. Outpost Delta lost control of most all their units before attacking. Our units are still affected and appear mostly stationary, clustered in different areas around the damaged base. Occasionally the scrambled initiate patrols, self-destruct, or commit other random commands.

After rendering our defensive units ineffective, Eden entered the base and destroyed both Command Centers. Once the command centers were destroyed, their forces pulled back across the Eastern ridge. We believe the retreat is an effort to reduce their casualties. Eden left 2 tank squadrons in the area and we believe is waiting for the eventual exhaustion of our scrambled units before reentering Outpost Delta. Based on their previous actions, they will likely level the remaining buildings on rentering the base. Your orders are to exploit the lull in their advance and organize an evacuation for the remaining colonists.

Eden's new Tiger tank squadrons are combat operations. We are unable to match their firepower. Stay clear of their tank positions. If Eden locates your non-scrambled units in vicinity of the base, they will certainly retransmit their signal into our vehicle command nets. We are sending an attached file archive that contains a software patch for your vehicles' command net. The Savant with you can employ the patch to prevent complete control loss of your tank squadron. However, our vehicles will require maintenance at a garage to fully shield them from the effects.

We are betting on Eden's imagery satellites remaining busy imaging our main base's defensive positions and not available to notice your vehicle activity. However, Eden will certainly be tipped on your presence if your vehicles come within visual range of their tank squadrons. Their wide thermal grid will likely pick up new building activity, so do not construct additional buildings. There is a small field of fumaroles West of the base that should be able to mask the heat signature of the construction and operation of a new command center.

Outpost Delta was preparing an evacuation convoy in case they were overrun, but the convoy was not completed before the vehicles were manipulated. There are several ConVecs in the evacuation staging area West of the Outpost. Hopefully one of the ConVecs contains a Command Center so you can restore enough of the base systems to complete evacuation preparations. When a spider reprograms one of our scrambled units, it should be able to reboot the vehicle's scrambled command processor.

Move quickly and get our colonists out before Eden re-attacks. Eden's main battle line continues to advance toward our main colony. You will not have the firepower to break through Eden's attack line and return to our main base. After evacuating the colonists, head West and establish a temporary resupply base. We will deal with Eden's approaching forces.

Good Luck Commander.
The survival of Outpost Delta's colonists rests in your hands.

You have to be logged into the forum to see the attachment.

----
Edit 01APR: Fixed typo in briefing. Credit to lordpalandus for finding it.
Edit 04APR: Fixed source code bug not allowing code to be built in Debug Configuration. Doesn't affect release dll.
Edit 15APR: Removed dependency on Visual C++ Redistributable 2015 from mission DLL. Attachment updated.
Edit 09JUN: Moved download to Github. Updated to new version that sets HFLInit properly when loading a saved game.

[Hooman]

Neat. Works under Wine.

I like how your initial vehicles start off moving into the map, rather than just sitting at the start point. That definitely adds some urgency to the mood.

[lordpalandus]

Typo in =  Based on their previous actions, they will likely level the remaining buildings on rentering the base. Your orders are to exploit the lull in their advance and organize an evacuation for the for the remaining colonists.

[Vagabond]

Hey Everyone,

Just like Rising From the Ashes, this scenario DLL has been updated and no longer requires the Visual C++ 2015 Redistributable package. Thanks for the proofreading help lordpalandus!

The new DLL can be downloaded from the first post in the topic.

[Commander]

Vagabond, just wanted to thank you and let you know that this mission is AWESOME.  Really cool ideas with the Eden scrambler link, the CC at the geothermal sources, and superb level design too - better than many of the official missions! It took me five restarts, but I eventually made it. Now busy beating "On the run" - keep up the good work!

[Vagabond]

Commander,

Thank you for the feedback. I'm glad you enjoyed the scenario and happy it worked without any bugs!

I'm planning on rebuilding the Outpost Monopoly multiplayer code. After that, I would like to come back and code the 3rd mission in the series.

[pente]

Thanks for making a great series of missions. Unfortunately I got a crash with this mission. I happened to save 2 seconds before it crashed so I've uploaded the save file. The crash can be reproduced by clicking on either garage and pressing S.

(I'm playing on Outpost 1.3.7 under wine.)

[Hooman]

Thanks for the saved game file. That really helps to reproduce and investigate bugs.

Technical:
The crash seems to be happening in code marked as CommandPaneView:BuildingStorageBays. Unfortunately the routine is not documented, so it may take a bit of effort to figure out what's going on. It seems the code is trying to call a virtual function with an invalid object pointer, which leads to a crash when it tries to dereference the virtual function table.

[pente]

Interesting. If it matters, I believe that earlier in that same game I was able to view the garage contents without crashing, and that the usual vehicles were present. I also believe that shortly before the crash some of the neutral-ish units had just received new orders to attack my stuff -- maybe a garaged unit got such an order and got confused? Pure speculation of course.

[Vagabond]

Dang, I hate to see a bug here. I was being unorthodox and loaded an opponent's units into the player's garage. Not sure if this is causing any issues. I could rebalance the scenario so these units didn't exist, although I thought it was a fun surprise.

pente, thanks for the complement on the new scenarios! I had a lot of fun making them.

I loaded your saved game on my machine and both garages crashed as reported. So it is not just a Wine problem as I'm running Windows 10.

I tried saving a new round of the scenario a couple of times, exiting Outpost 2, and then loading the game. The garages still worked fine when doing this. So something besides simply saving/loading has to happen to trigger the crash.

When I have some more time, I'll try manually reviewing the code and seeing if the Garage is referenced anywhere else in a suspicious manner. I may also try loading debug symbols with the scenario in play. However, since the crash is happening inside the Outpost 2 side and not in the scenario code, this may not reveal anything.

Thanks for checking the bug out Hooman.

If anyone else wants to take a crack on the bug, the source code is linked in the first post of this topic.

-Brett

[Hooman]

Hmm, interesting trick, loading opponent vehicles into a garage. I like the deviousness of that.


I spent some time analyzing the code where the crash occurred. The data being accessed are the cargo bays, which should be an array of 6 Unit pointers. The pointers do seem to point into unit data, however, the referenced units appear to be dead. The flags and key values that are checked for dead units are all set to indicate the unit is dead. Further, the field for the virtual function table pointer does not contain a valid pointer. That's the actual cause of the crash, when it tries to dereference the virtual function table. Perhaps the field was cleared when the unit died.

Interestingly, the field for the virtual function table pointer is not simply cleared to zero, but contains a low valued number, like an index. I've noticed during game save/load, the virtual function table fields can be temporarily translated to an index (since saving/loading a pointer to disk is generally a bad idea). Perhaps the low valued indexes are related to the translations done during saving/loading the game. Maybe they aren't translated back to virtual function table pointers for dead units. This is all just speculation of course.

As to how you ended up with dead units in the garage, I have no idea.

[lordpalandus]

Perhaps it is how units are stored in enemy garages... as dead units, that only become alive when they are removed. As I've never seen an enemy store a unit in one of their own garages, whether in campaign or colony missions. So it may be some kind of weird interaction with how enemy garages work and how player garages work, that creates the issue.

I'd check to see how an enemy garage handles it's units, particularly garages that start with units inside them (ie that one Plymouth attack mission, where you attack the Eden colony).

[Hooman]

Hmm, I wonder if it's related to dock damage.

If you move a vehicle onto an enemy dock, the vehicle will take continuous damage. Might also damage allied vehicles too. Keeps you from blocking other people's docks, especially early game. I once tried that, running a scout or robo surveyor over someone's structure factory dock. If you stay there, your vehicle dies. Though if you repeatedly run it across without staying, you take less damage, and you can still bump their Convecs off the dock. That can really upset people's build flow.  It also really irritates people. To the point there was a rule against that the next game. 

[Vagabond]

I'm not sure about the dock damage. The vehicles exit the bay fully healed even when they are an opponent's unit in your garage. Also, the health bar in the garage displays fully green and doesn't reduce over time. I tried waiting sufficient time for the units to die if they were taking damage, saving the game, and then loading. Everything worked fine and the units exited the garage at full health.

Hooman, do you think that vehicles placed in a garage would need to be recorded in ScriptGlobal for save/load purposes? Currently, I am not tracking these units in ScriptGlobal like all the other units. If this is needed, I guess I would have assumed that the game would crash more consistently after loading?

pente, could you verify which difficulty the saved game you sent is using?

Below are the 2 helper functions to load vehicles in the garage that I use. I am not passing the vehicles outside of this function, so I'm pretty sure none of them should be somehow populating into a fight group or patrol or anything. Excuse the bad code as I was still very new to C++ and wrote this for my own consumption only.

Code: [Select]

void VehicleBuilder::CreateVehicleInGarage(Unit &garage, int bayIndex, map_id vehicleType, map_id cargo)
{
	UnitDirection oldDirection = unitDirection;
	unitDirection = UnitDirection::West;

	Unit vehicle;
	CreateVechLightsOn(vehicle, vehicleType, garage.Location(), cargo);
	vehicle.PutInGarage(bayIndex, garage.Location().x, garage.Location().y);

	unitDirection = oldDirection;
}

void VehicleBuilder::CreateVehicleInGarage(Unit &garage, int bayIndex, Truck_Cargo truckCargo, int cargoAmount)
{
	UnitDirection oldDirection = unitDirection;
	unitDirection = UnitDirection::West;

	Unit vehicle;
	CreateVechLightsOn(vehicle, map_id::mapCargoTruck, garage.Location(), map_id::mapNone);

	if (cargoAmount > 0)
	{
		vehicle.SetTruckCargo(truckCargo, cargoAmount);
	}

	vehicle.PutInGarage(bayIndex, garage.Location().x, garage.Location().y);

	unitDirection = oldDirection;
}

Thanks,
Brett

[pente]

I was able to create another corrupted save file (normal difficulty). The only orders I issued in game were moving the units slightly. (I checked the two saves that were previous to the save I uploaded before, and even the first one was corrupted at mark 31 before I had engaged the enemy.) I did the following steps, I believe:

1. wait until mark 30
2. check garage (fine)
3. save game
4. check garage (fine)
5. load game
6. check garage (fine)
7. save game (attached)
8. check garage (crash)
9. load game from step 7
10. check garage (crash)

So maybe loading doesn't trigger it, rather saving might (perhaps saving triggers some kind of garbage cleaning internally). I tried following the steps above multiple times and wasn't able to get it to happen again. I tried saving over and over but wasn't able to trigger it that way either. I am still suspicious that saving/loading might be responsible but it is certainly not reliably so.

I also tried restarting the mission about 30 times and just checking the garage. Every time it was fine at the start of the mission.

Across all four corrupted save files that I have, both garages are corrupted.

Nice trick with the scouts. Almost a shame it was banned... almost

[pente]

I forgot to say, I think the first save file I uploaded was hard difficulty, but I don't remember for sure.

[pente]

Ok, even simpler: I started a new game, immediately saved four times (no loading, no touching anything, no waiting) and the garage crashed. So loading is definitely not necessary -- saving is sufficient. Either that, or there is just some small random chance of corruption at the beginning of the game.

Again, on multiple re-tries I have not been able to replicate this procedure, even with restarting OP2 between tries. Funny how both this time and the previous time I got a corruption on the first attempt and then couldn't get it again.

[Hooman]

Well, doesn't sound like it's related to dock damage then.

I doubt it's related to ScriptGlobal at all. That shouldn't have any impact on this issue.

And my gosh, you're using a global unitDirection 
 

I don't see any obvious issues with the code. Might be something else entirely.

Maybe we can set a breakpoint to monitor for unit death, or for writes to the unit vtbl field for units stored in the garage. Would help to know how to reliably reproduce the problem though, so it can be caught in the act. The saved games were very useful for the initial diagnosis, though they are a snapshot from after the corruption happened. Being able to reliably reproduce the point in time where the corruption happens would help further diagnose the cause.

[Vagabond]

pente,

Thanks for the continued testing. On easy mode, the stored vehicles actually belong to the player as opposed to being the enemy's. If you don't mind taking a few minutes and trying to reproduce the crash on the easy difficulty setting, I would appreciate it. If we cannot reproduce on easy, it would point strongly to the problem being associated with the enemy units being stored in the player's garages. In this case, I can just make the units friendly at all difficulty levels to keep the crash from happening. However, I'm not aware of any other fan made scenarios that actually use garages, so I don't necessarily want to assume the problem is caused by the enemy unit in player's garage situation.

Hooman,

No making fun of my random global variables that actually make coding the scenario more difficult. 

[pente]

I was able to get the crash on easy by saving four times immediately after starting the mission.

Again, I got it on my first attempt, and restarting OP2 and retrying the exact same thing 7 more times failed to trigger the bug again. It is hard to imagine what is going on in my system that causes the bug to show up only if OP2 has been closed for some time while I do other tasks, and then is triggered by specifically saving. Maybe some kind of memory management bug inherent to OP2, which you would never see in the normal course of the game (there is the plymouth mission with loaded garages, but you can't select them). I guess pre-loaded garages is just dangerous territory for mission authors, unfortunately? I would be curious if anyone else can replicate the bug.

I also went ahead and actually tried to beat the mission, and finally got it and the next one on hard. (Having already beat the third on medium I feel no need to repeat it on hard.) I have no idea how you were supposed to get to 80 colonists on the second mission before the tigers show up -- I pulled every trick I thought of (even built a GORF!) and still depended on a healthy amount of luck to get there.

[Hooman]

Hmm, that's some good debugging info. In particular, it's good to know the crash happens even on easy when the vehicles are owned by the same player as the garage. Perhaps there is something else in the code base which is interacting with the garages in an unexpected way. My guess would be Trigger or FightGroup related.

Pente, you seem to be doing a pretty thorough job trying to reproduce this bug. Would you be at all interested in trying some advanced debugging tools?


I'm taking a peek at the code.

This is probably unrelated, but in ReviewRogueFightGroupStatus, the logic here looks reversed. Is this to recycle the fight groups? I assume if TotalUnitCount() == 0 then it should set RogueFightGroupsAvailable = true. In particular, there is no code anywhere in the project aside from the initialization that sets RogueFightGroupsAvailable = true, meaning once used, the groups can never be used again.

Code: [Select]

if (scriptGlobal.RogueFightGroups[i].TotalUnitCount() == 0)
{
	scriptGlobal.RogueFightGroupsAvailable[i] = false;
	continue;
}

On another note, if TotalUnitCount() == 0 does mean the group is available for use, that can be used directly to track group availability rather than relying on a secondary array.

Of course, other code I saw seems to suggest the group tracking might be more of a countdown until some event happens, with no reuse. The intention of the code is not immediately obvious.

[Vagabond]

pente,

Thanks so much for the testing. Really good to know that the problem is independent of which player's units are loaded in the garage. Has anyone else used garages in a colony game or have advice on how to load them without a crash? I just tried saving 5 times on my computer, closing Outpost 2, and reopening. Didn't reproduce the crash.

For scenario number 2, it is designed on hard so that you have to play really well and still be in progress of evacuating your base when the enemy shows up. It actually makes me really happy you could barely beat it, and only as the enemy were descending on your facilities.

I have a 4th scenario in the series partially created. Would you be interested in playing it if I take the time to finish programming it? For these scenarios I try pushing the boundaries of the API in different ways to make them a bit unique from the typical Outpost 2 campaign game or colony game.


Hooman,

In this scenario, each fight group will only be activated one time. The code you highlighted exists to check if the fight group hasn't been wiped out by a natural disaster or the player attacking it before tasking it with a patrol or self-destructing it. Technically I don't think the code would crash if you tasked an empty fight group, but it makes for less action.

I made some minor documentation updates around that part of the code and committed to the repository so it should be clearer to others. I didn't expect the code to get so much attention.

I could rewrite the scenario to not have vehicles in the garage or we could just note the bug in the change log. Since the garage isn’t used much, I was happy incorporating it but nothing like ruining a series of scenarios than by repeatedly crashing in the first installment. I’m a little split here on what to do.

Thanks,
-Brett

[pente]

I am game for further testing, within the time constraints of my various obligations. And subject to the fact that I still can't /reliably/ reproduce the bug on demand. You could try making a very simple map -- nothing but a bunch of garages and units, variously with or without the scenario-specific scripting in it.

I'd suggest that it is sufficient to just warn people about the potential bug (and ask them to report it if they encounter it -- maybe in the scenario briefing itself? I don't know what fraction of your players actually check the forums) until at least one other person has confirmed that they have seen it.

For mission 2, the only way I found that worked involved taking the bottom-right beacon, but on the first few attempts the enemy literally spawned in my base, so there is a strong element of pure luck in whether that happens. (I also had to very carefully place the provided kits in the correct order, train almost no scientists until almost the end, do the usual morale stuff, and build walls and scouts to block and distract the attackers.) I had enough extra resources to put some supernova lynx where the tigers spawned, but they didn't auto-target the enemy. If I had been willing to commit to winning the fight against the tigers I could have built a bunch of decoy tokamaks around the map and supernova'd the tigers when they were emp'd. (This would give me more time to get to 80 colonists, so I could have taken the left beacon, which removes the element of luck of enemies spawning in my base.) That would feel very cheap though, if it even worked at all. Does the game spawn more enemies if the big attack is killed off?

[Hooman]

I don't think the garages need to be modified at this point. We don't even know the exact cause of the bug yet. It might be elsewhere, and the garage problem is just a symptom of it.


If I can find the time to do it, I'd like to load up the level under OllyDbg. After the vehicles are loaded into the garage in InitProc, I want to set some hardware breakpoints on the vehicle data, such as HP, flags, or the next pointer (3 fields related to unit death), to watch for when those units get killed off. This is likely well before the virtual function table pointer gets overwritten.

My guess is, the virtual function table pointer is overwritten during saving, and not restored during loading because the units are dead, and so the records are considered free. However the garage is still referencing the dead units since it's not designed to clear out units that get killed inside. And it's not designed to clear out units that get killed inside because there should be no way to kill units inside. If I had to guess how they got killed, I'd suspect a memory management error concerning groups or triggers, though don't have any particular place in mind to start looking. Hence why I'm hoping a hardware breakpoint will tell me where it's happening.

[Hooman]

I've done a bit more digging. Not quite at a direct cause yet, but I suspect this is a bug in the game engine.

The game stores several pointers in the Unit objects. Pointers aren't very meaningful when written to files, since allocated memory tends to get moved around, with no guarantee an object will be at the same location in a future instance of the program. To avoid this issue, the game translates between Unit pointers, and Unit indexes. Just before it prepares to save a game, it runs through objects in memory, and translates these values in place. It then writes the images of all the objects to disk. Afterwards, it has to restore the memory to proper pointers so it can continue operating normally. That part of the code is run directly after saving a game to disk, as well as after loading a saved game file. The result should be all the indexes are translated back to pointers.

However, the game doesn't seem to translate the values correctly for units stored in garages. During a save, the field gets corrupted. The reason why is a bit odd, and why it doesn't cause problems more often is something I still need to look more into. I think by virtue of not much code accesses units in garages means there is a limited opportunity for things to go wrong when the values are in a corrupt state. Further, when you unload units, it resets the field to sensible data.

In a Unit object, the first 4 fields are union values. What they mean depends on context. The first field is normally the object's virtual function table pointer. A virtual function table pointer can be a handy way of distinguishing the exact type of an object. In a somewhat natural correspondence, this field gets translated to a map_id unitTypeIndex when generating a saved game, and translated back to a virtual function table pointer afterwards or upon loading. The next 3 fields store Unit pointers. Normally they mean something like next, prev, playerNext. When generating a saved game, these fields are translated to corresponding uint unitIndex values.

Now here's where things get weird. Normally you'd think a Unit* value would either point to a valid Unit object, or would be set to the 0, the nullptr. A uint unitIndex might be set to 0..2047 (for a unit limit of 2048 units), where 0 is kind of a dummy non-existent unit that never gets saved to disk. You might expect a dummy object like that, or something like -1 or UINT_MAX used to denote no valid unit. Instead, when the game is operating normally, and the field should represent a Unit*, it uses the value -1 to represent no valid unit. In particular, when a unit is dead, its next pointer is set to -1. The game has many special detects for -1 values sprinkled throughout the code, much like you might expect of checks against nullptr. As such, a value of -1 in this field, although odd, is not a problem.

When a unit is loaded into a garage, the next field is set to -2. There are no checks against -2, so it's assumed to be a valid pointer value, which clearly it's not.

When the game is saved, the -2 is taken as a valid unit pointer, and a calculation is done to determine a unit index from the unit address. It uses a pretty standard formula for converting array element addresses into array indexes:
 index = (elementAddress - arrayBaseAddress) / sizeof(ElementType)
That's all fine when the element actually is within the array. However, implicit in this formula is an assumption:
 elementAddress is within the bounds of the array and an integer multiple of sizeof(ElementType) past the arrayBaseAddress
This is not true for -2, and so the result is a garbage value.

What I think might be happening, is through some as yet unknown code path, this garbage value somehow becomes -1. I don't think this can happen directly, which is perhaps why the game doesn't typically crash when accessing a garage after saving. Instead, I think what happens is the garbage value gets translated to and from pointer and index form until it ends up set to -1. At this point, the checks for that sentinel value are hit, and the unit is considered dead. During the post-save and load code, when it translates indexes back to pointers, it sees the -1 sentinel value and skips over index to pointer translation. In particular, it doesn't bother to translate the map_id unitType field back to the virtual function table pointer. Trying to access the virtual function table while the unitType index value is still loaded is the direct cause of the crash.

There's more weirdness too. When a unit is loaded into the Garage, it is removed from the list of active units, and the prev pointer is set to uint gameTick. I suspect this is used to track when a unit died. It might also be related to repair progress, but I think that's handled by another field the gameTick gets copied into in more garage specific code. The game also seems to clear this value to 0 at times.

I'd like to do a bit more work to find out how the -2 gets translated to a -1, and see if there is a way to reliably reproduce the bug.

We could probably patch this. Though I'd want to know a bit more about the problem path before trying to devise a patch.