OP2 Scenario Project for C#

[TechCor]

Hey all, and thanks Leeor for marking my account as confirmed.

I've been playing Outpost 2 for the last couple months and completed both campaigns and Eden Starship on hard. Still wanting more, I started looking around at the various GitHub projects and how to make new scenarios.

The number one thing I noticed is that making scenarios is incredibly tedious, compounded by the fact that it is done in C++ which I haven't used seriously in 10 years, and had completely forgotten the pains of char* strings, header files and linker errors.

So last week I decided to move it all to C#. I wanted to make sure the community had access to the source, so you can access the project here:

https://github.com/TechCor8/OP2DotNetMissionSDK

It also reads JSON for the initial setup and creating triggers and disaster zones. The goal being to move scenario development into an external editor.

UPDATE [2019-8-22]:

StateSnapshot and AsyncPump added.

Features exclusive to the C# SDK:

• JSON Reader - Optionally perform initial game setup through a JSON file.
  
• BaseGenerator - Automatically creates a base from a list of units with assigned distances.
• Pathfinder - Navigates obstacles on the map to reach a destination. Can also find closest open locations based on a set of rules.
• PlayerCommandMap - Tracks a player's command center connections/tubing. Can find closest connections and generate paths to them.
• StateSnapshot - A single access point for read-only Outpost 2 related data at a specific TethysGame.Time().
• GameState - A static lookup table for live Outpost 2 players and units for issuing commands and modifying data.
• AsyncPump - A multithreaded solution that runs a method in parallel and then calls a completion callback at a set TethysGame.Time(). By using StateSnapshot in the async method, and GameState in the completion method, you can maintain predictable simulations.
• BotPlayer - AI that works in any scenario without cheating. (Currently first draft / incomplete)

Things that are still missing in C# that are in the C++ SDK:

• HFL Panes
• OP2 Helper
• odasl Library

Miscellaneous functions that are missing:

• Trigger.CreateSetTrigger
• TethysGame.CreateUnitBlock
• Group Pinwheel class
• TethysGameEx.MsgBox
• UnitInfo.CreateUnit

[leeor_net]

Glad you were able to sign on!

This is wonderful! I've always loved C++ but recently learned how much easier it can be to develop with C#. I still prefer C++ because I'm a masochist but this could help to make mission coding more accessible. There has allegedly been an effort toward making it accessible via python scripting (glares at Arklon and BlackBox) but that seems to be an echo on the wind.

We've been pushing to migrate everything to GitHub to be as public as possible with it. Would you consider doing the same? Or would you prefer that we integrate this into the OutpostUniverse organization on GitHub?

[Hooman]

Wow, what? I'm both surprised and intrigued.

I've manged to create simple DLLs with other compilers before, that the game sees as a level, and can load them, but they've never been able to do anything useful, like create initial units. The name mangling used by the C++ functions that Outpost2.exe exports always got in the way. The data exports (from the DLL) were simple enough to get the game to recognize the DLLs, but the function exports (from Outpost2.exe) limited the DLL from doing anything useful.

Yet you say this works? What about saving and then loading a game? Outpost 2 requires all data to be accessible in a fixed sized buffer, and there's no notification of when it's going to save or load data to/from that buffer, so it needs to be the live data.


I would love to see this on GitHub.

[Arklon]

There has allegedly been an effort toward making it accessible via python scripting (glares at Arklon and BlackBox) but that seems to be an echo on the wind.

That's still in development, but it's also larger in scope than just for writing missions. As far as missions goes, it's basically done, except for the parts of the public API we never bothered to document like Pinwheel groups, and exposing more non-public API game internals (though quite a bit is already exposed).

I've manged to create simple DLLs with other compilers before, that the game sees as a level, and can load them, but they've never been able to do anything useful, like create initial units. The name mangling used by the C++ functions that Outpost2.exe exports always got in the way. The data exports (from the DLL) were simple enough to get the game to recognize the DLLs, but the function exports (from Outpost2.exe) limited the DLL from doing anything useful.

He worked around this problem by just making extern C wrappers of all the OP2 API functions. I remember someone telling me that C++.NET/CLR is fully ABI compatible with C++ though (they're both still built using the MS toolchain so that wouldn't be a shock), so there's probably a better way to do this. That said, the general idea of using C++.NET as an interop shim between C++ and C# is actually a pretty good practice, and also the only real good use case for C++.NET.

[TechCor]

We've been pushing to migrate everything to GitHub to be as public as possible with it. Would you consider doing the same? Or would you prefer that we integrate this into the OutpostUniverse organization on GitHub?

I'll put it on GitHub. I thought about that as I was posting, but was too lazy to do it right then.

What about saving and then loading a game? Outpost 2 requires all data to be accessible in a fixed sized buffer, and there's no notification of when it's going to save or load data to/from that buffer, so it needs to be the live data.

I haven't tested the saving yet, but the general idea is to reserve a static buffer larger than what is needed for most missions in the unmanaged DLL, and forward the buffer to C# through GetSaveRegions where it can be written with a memory stream. It is a bit inefficient, but so is everything else about this. Hopefully, the data is loaded by the time InitProc is called. If not, I could test in a loop for some init flag that is always saved to the buffer.

-- Thinking about what you said, I may have misunderstood what GetSaveRegions does. Is it just getting a pointer to the buffer at initialization? This whole saving process is really arcane. I would have had Save()/Load() procs... If that's the case, I'll probably have to send the buffer at init and use some kind of indexing wrapper class in C# for accessing and modifying the buffer with live data.

This makes me wonder then, is it enough to create the buffer dynamically at DLL attach based on the size reported by C#? For example, in C# there could be a SaveBuffer class with GetBufferSize which is based on the total size of the variables in the class. Use that to new char[] the buffer and pass in GetSaveRegions.


It really is just a lot of wrapper boilerplate. My eyes started to glaze over while doing triggers.

[Vagabond]

Hi TechCor,

Welcome to the forums and glad to here you are enjoying Outpost 2 20 years after its creation.

It sounds like you have a great start on the programming. One note on making scenarios is the DLL has to be loaded into the proper base address. Otherwise, if you save/load a game, the mission objectives will become garbled. If you haven't seen it yet, there are 2 posts in the wiki that go over a lot of the C++ settings details (just glance over the SVN part I guess). https://wiki.outpost2.net/doku.php?id=op2_sdk:projectcreation

Within the SVN repository you can find a decent number of the custom missions that have been added to Outpost 2 over the years for coding examples. Up to you to sift through the good/bad though. :| https://svn.outpostuniverse.org:8443/!/#outpost2/view/head/LevelsAndMods/trunk/Levels

If you want to move past basic level design, you may want HFL and/or IUnit to give some of the more advanced commands. HFL relies more on memory hacking. You can also add a modal dialog to the beginning of your scenario using odasl. Couple of examples in missions I've made and Plymouth Cold War.

There are a lot of undocumented bugs in the mission development SDK. Both internal to Outpost 2 and in Outpost2DLL and HFL. It would be good to get things fixed or at least documented, but it is all volunteer work.

I've been trying to split my time between improving the mission SDK and template on GitHub and trying to help improve NetFix, but am happy to try and answer questions about the SDK.

If you run through all the colony games and fan added colony games, there are some multiplayer coop missions with AI. You can bring up two instances of Outpost 2 on your machine and attempt them solo for a tough challenge. I think Danger Zone is barely doable single player. If you want to play multiplayer, I'm willing to schedule some time out, but mostly play coop missions now. The deathmatches never did much for me (probably because I'm a mediocre player).

Anyways, happy playing and coding.

-Brett

[Hooman]

Yeah, I'm liking this idea of using C# for mission development. I guess it makes sense there would be some kind of interop.

I was looking through the code a bit the other night. Looks like the C# classes are proxies for the C++ classes. The C++ classes are in turn proxies for internal Outpost2.exe classes. I'm wondering if we can do something more direct there. Perhaps there's a way that adds less boilerplate, or that might make the Save/Load thing easier. The game really could have made use of Save/Load methods, or at least callback notifications. It would have made the system much more flexible. I'd love to see this up on GitHub so we could play around with this C# idea some more. 

And Triggers, oh boy.

[TechCor]

OK. I read the base address thing 5 or 6 times, and I think I finally get it.

So I assume Outpost 2 is doing something like:

GetSaveRegions to get the buffer address and size.
Saves the struct ADDRESS to the save file and copies the buffer.
At some point OP2 loads the buffer from the save file back to that saved address.

If the DLL or save buffer were dynamic addresses, it would end up copying the data back to the wrong place.

If this is true, the "oversized static save buffer" method with a C# wrapper that indexes properties will be the way to go.

What threw me off, is I would think it would just query GetSaveRegions for the most up-to-date address before loading the data.


--
Playing two copies of OP2 at the same time might be tough. I am more of a "versus AI" type, but especially enjoy the morale aspects of the game. Otherwise, I'd play Starcraft. I never did like competitive multiplayer, but maybe that's because the Starcraft guys were just too good.

Didn't realize there were projects on the SVN repo. I thought everything was on GitHub. Will look into it when I have time.

[TechCor]

Yeah, I'm liking this idea of using C# for mission development. I guess it makes sense there would be some kind of interop.

I was looking through the code a bit the other night. Looks like the C# classes are proxies for the C++ classes. The C++ classes are in turn proxies for internal Outpost2.exe classes. I'm wondering if we can do something more direct there. Perhaps there's a way that adds less boilerplate, or that might make the Save/Load thing easier. The game really could have made use of Save/Load methods, or at least callback notifications. It would have made the system much more flexible. I'd love to see this up on GitHub so we could play around with this C# idea some more. 

And Triggers, oh boy.

I'm not sure if it can be made more direct, but certainly if someone knows a way, let me know, and I'll update it. For now, "working" is better than not.

The trigger problem (where it will call the native DLL) is definitely a tough one. I am really hoping that HasFired() works properly. In which case, I will save all the triggers to a list and poll HasFired() to execute the result.

Also, it is now on GitHub. Check the first post.

[Hooman]

Ok, I've cloned the GitHub repo. I'll have to take some time to play with it.

I agree that working is better than not-working/nice/"perfect".

The thought I had the other night, is the C# code seemed to be using the C++ classes in much the same way as native C# classes. That's what made me think maybe it could be more direct. I admit my C# knowledge is pretty weak though, so perhaps I'm missing something obvious. I know C# has some differences in terms of memory management, and these sorts of wrappers could be to paper over the memory management differences. Though in the case of Outpost 2's exported classes, they are mostly proxy objects with simple value like semantics, so that might negate some of the memory management differences. You could basically treat those classes like you'd expect of structs. Hard to say at this point, just a sneaking suspicion I had.

Internally, Outpost 2 uses function pointers for triggers. If we use the memory hacking APIs, we could gain direct access to those pointers, and setup triggers without going through the named DLL function export/lookup process. Though I assume C# must have a way of exporting plain functions from DLLs, so maybe that wouldn't be needed.

As for the saved games, Outpost2.exe calls GetSaveRegions just after loading the DLL. It calls it before InitProc when starting a new mission. When loading a mission it calls it before resuming the game cycle. It stores the result, and never asks again. Whenever a player saves the game, Outpost 2 copies the contents of that buffer to the saved game file. There is no notification to the mission DLL, so the mission has no idea when a game has been saved. Similarly, it doesn't know when it's been loaded. It simply loads the DLL, calls GetSaveRegions to find the buffer, and then copies data from the saved game file to that buffer, then resumes the game cycle.

You could potentially write a detect for the load, using an initialized global/static variable outside of the saved region, and modified at the end of InitProc. If it has the original initialized value, rather than the InitProc set value, the game has been loaded since InitProc ran. There would be no way to write a detect for saving the game though. At least nothing short of hooking into the code section of the game. Hence that static buffer must always contain current live data, or you risk it not being stored when a game is saved. Well, minus the small window of time within a game cycle, when the DLL has complete control to run AIProc, or a trigger callback. Once it returns to the game engine though, the game engine can do whatever in terms of saving/loading. You would need to update the static buffer before returning if things were modified outside the save region.

[TechCor]

Thanks for clearing all that up!

Saving and loading appears to be working.

I went ahead and made it a packed C# class for storing variables + a native memory wrapper. Checks if "isLoaded" in the update loop, and calls Load if it isn't. Then saves at the end. Glad to know the buffer doesn't need to be declared in the native plugin.

I'll see what I can do for triggers next.

---
On the topic of bypassing the native wrapper, besides C++ name mangling, C# does not appear to support __fastcall which is used on some of the functions, most notably TethysGame. Not worried about it right now though.

[TechCor]

UPDATE:

Triggers have been added. It works a little differently in C# to avoid interop passthrough.

The process is as follows:

• Create the trigger as normal through the C# TriggerStub interface. Ex: TriggerStub.CreateVehicleCountTrigger(). Ensure that the trigger has a unique trigger ID (first parameter)
• Add the trigger to the C# TriggerManager.
• When the trigger has fired, TriggerManager calls an onTriggerFired event.
• Use the triggerStub.id to identify the trigger. Call appropriate actions.

Of course, you can also specify all triggers and their IDs through the JSON file. Currently, you can't specify actions in JSON, but you can still reference the ID and perform the actions via C#.

---
The C# DLL can now be referenced under different file names.

The NativePlugin has a #define USE_CUSTOM_DLL in LevelMain.cpp for toggling loading.

USE_CUSTOM_DLL true = [NativePluginName]_DotNet.dll
USE_CUSTOM_DLL false = DotNetMissionSDK.dll (JSON only library).

By default, this is set to false for JSON only missions.

---
UPCOMING:

• Create a CustomLogic.cs file to abstract custom mission code from the SDK system.
• Try to eliminate some unmanaged calls to new to eliminate the need to dispose C# objects.
• Add support for groups.

Unfortunately, groups are kind of a mystery to me. The wiki page does not exist, and there isn't much documentation to be found elsewhere. Might take some experimentation.

[Arklon]

Okay, I looked at your code again, it actually wasn't using C++.NET like I thought before. I did some more research on the C++.NET interop shim technique, and yeah, it's fully ABI compatible with C++ (you need to do native function calls in unsafe { } blocks) including all calling conventions etc., and it compiles to a DLL that's a .NET assembly that you can drop into C# and it just works, so it would serve both of the roles that the native and .NET layers do currently.

[TechCor]

Hmm. There were 2 interop layers and 1 C# layer before. I just merged the 2 interop layers (NativeInterop and DotNetInterop) into 1 layer (DotNetInterop). C++ can have both managed and unmanaged code in the same DLL by setting file properties.

I'm thinking of trying this technique on the NativePlugin layer to call C# directly. I didn't try this initially in case Outpost 2 had trouble with it.

If I understand you correctly, you are saying DotNetInterop should wrap the Outpost2 calls with managed classes instead of extern C. DotNetMissionSDK (C#) could then reference DotNetInterop instead of using DllImport.


EDIT:
Tried to get NativePlugin to work with the CLR on just one file. Linker complains of flag incompatibility, so I don't think this is an option. I would have to make project wide flag changes which apparently some of them are required for Outpost 2.
This means there will be a circular reference problem in DotNetInterop if I change the externs to managed classes. Referencing the C# DLL may need to be replaced with reflection to accomplish this.

I'm out of time now. Will look at it later.

[Hooman]

Ok, this is getting interesting.

Have you updated GitHub? I don't see any recent changes.

[TechCor]

There were 6 commits yesterday.

You can check activity here:

https://github.com/TechCor8/OP2DotNetMissionSDK/commits/master

UPDATE:

I just switched the dependencies. It actually makes sense now. C++ no longer depends on C#. C# now references the C++ DotNetInterop and implements the C++ managed interface for MissionEntry.

Should be possible to wrap all the Outpost 2 functionality inside of DotNetInterop as managed C++ classes now.

[Hooman]

Ahh, I see what happened. It seems the public history was rewritten. Perhaps a rebase was run or something. When I updated, it pulled down new commits, but then refused to merge them to my local master branch, since the histories had diverged. Hence when I checked the log of my local master branch, I didn't see the updates.

It's such a tiny little error message, in a paragraph of text, and no highlighting.

Ok, yes, I see you've been busy.

[TechCor]

Yes, I rewrote the history because I accidentally pushed the first few commits with the wrong author info. Was hoping nobody would notice.

[BlackBox]

This looks really promising!

Thinking about this a bit further, it might be useful to join forces with the Python SDK that Arklon and I have been working on (this isn't dead, we've been quietly working on it in the background, hoping to have something more or less ready for general consumption soon-ish).

It'd probably be possible to eliminate the need for the unmanaged wrapper/proxy DLL that loads the assembly.
For the Python SDK (which is implemented as a mod DLL loaded through op2ext/the INI file) we already hook/replace quite a bit of UI code related to the list of missions in the various menus (in order to show Python missions in the list). We also hook a whole bunch of code relating to loading the mission, as well as trigger callbacks and loading/saving the game (same reason, since the mission itself is no longer its own DLL).

Since we're already hooking these places it probably wouldn't be a huge amount of work to detect whether a mission is a .NET assembly, and if so, load it via CLR hosting (i.e. call directly into the assembly without needing the C++ proxy DLL). I'd expect the assembly that handles interop between the Outpost2 API and .NET would probably still be needed.

Also, regarding loading and saving from earlier in the thread, the game actually internally handles this by passing a file-like object (StreamIO) to the internal class that represents a mission DLL. This in turn calls the GetSaveRegions exported function and then reads/writes the buffer contents from/to this file.
Since we similarly can't assume a fixed buffer for Python missions (since we don't control where the interpreter places things in memory, etc) our current plan is to expose the low-level Load/Save API to the python code and the mission is responsible for loading/saving whatever it needs to reconstruct its state from a saved game. We can probably simplify this further by having the mission provide an object such as a dict that we pickle/unpickle to/from the file. I would guess something similar would work for C# as well.

[TechCor]

It would be nice to get rid of the extra dll per mission, but since full compatibility is possible, I will continue to support that.

If the mod calls the DotNetMissionEntry interface, it would be possible to bypass the unmanaged DLL. Would just need to add something for level details. This way the C# DLL can be used with or without the mod.

If the mod wants to support JSON missions without the unmanaged DLL, it would need to check for the JSON files, and call into the special/uncustomized C# DLL's DotNetMissionEntry with the appropriate file path.

If you don't want to use the CLR in the mod, you can just call DotNetInterop's C functions which will do it for you. That DLL will be required regardless, and I don't think the slight overhead is worth worrying about. Also, the DotNetInterop will handle switching to JSON for you. You just have to set the bool in Attach for whether to load a C# mission or JSON.

[TechCor]

Decided to put some time into things I can see. One of the goals is to be able to rapidly set up generic scenarios (two hours or less time invested).

I've just added Autolayout. AutoLayout lets you place a colony without worrying about tiles, collisions, tubes, etc. It's very simple to use and quite customizable. I was having fun toying with the build order and distances to see what kind of bases it would come up with.

There is still some work to be done on it with regards to tube placement and structures being placed too far away (crow flies vs walking distance). I'll be adding A* pathfinding to the project and using that to solve those issues.

Here is an example of the usage (JSON AutoLayouts section):

Code: [Select]

"PlayerID":0,
"BaseCenterPt": { "X":27, "Y":38 },

"Units":
[
{ "TypeID":"CommandCenter",		"MinDistance":1 },
{ "TypeID":"StructureFactory",		"MinDistance":0 },
{ "TypeID":"ConVec", "CargoType":"CommandCenter", "MinDistance":1, "SpawnDistance":3 },
{ "TypeID":"ConVec", "CargoType":"None","MinDistance":1, "SpawnDistance":3 },
{ "TypeID":"CommonOreSmelter",		"MinDistance":0 },
{ "TypeID":"Agridome",			"MinDistance":0 },
{ "TypeID":"Agridome",			"MinDistance":1 },
{ "TypeID":"GORF",			"MinDistance":0 },
{ "TypeID":"Residence",			"MinDistance":1 },
{ "TypeID":"StandardLab",		"MinDistance":0 },
{ "TypeID":"Nursery",			"MinDistance":0 },
{ "TypeID":"Residence",			"MinDistance":1 },
{ "TypeID":"University",		"MinDistance":0 },
{ "TypeID":"RobotCommand",		"MinDistance":0 },
{ "TypeID":"Residence",			"MinDistance":1 },
{ "TypeID":"MedicalCenter",		"MinDistance":1 },
{ "TypeID":"DIRT",			"MinDistance":0 },
{ "TypeID":"Agridome",			"MinDistance":1 },
{ "TypeID":"MedicalCenter",		"MinDistance":1 },
{ "TypeID":"RecreationFacility",	"MinDistance":0 },
{ "TypeID":"RecreationFacility",	"MinDistance":1 },
{ "TypeID":"DIRT", 			"MinDistance":2 },
{ "TypeID":"VehicleFactory",		"MinDistance":1,	"CreateWall":true },
{ "TypeID":"Tokamak",	"MinDistance":3 },
{ "TypeID":"Tokamak",	"MinDistance":2 },
{ "TypeID":"GuardPost",	"CargoType":"Microwave","MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost",	"CargoType":"EMP",	"MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost",	"CargoType":"Microwave","MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost",	"CargoType":"ESG",	"MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost",	"CargoType":"RPG",	"MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost",	"CargoType":"EMP",	"MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost",	"CargoType":"RPG",	"MinDistance":2, "CreateWall":true },
{ "TypeID":"Tokamak",	"MinDistance":3 }
]

And the base it generates:



EDIT:

Added some BFS and A* pathing for structures and tubes. Base now stays together properly. Autolayout vehicles spawn last to prevent structures from being affected by their placement. Wall placement should probably be moved to after structure placement.

Image above has been updated to reflect changes.

I think I'll add an option for single or multi tube setups. Image shows multi. Single tube is closest only. Multi is closest + all structures within minDistance.

[Crow!]

So with this, we could make a map which, at start-up, randomly selects a subset of buildings, and assigns that same set of buildings to all players, without the map needing to explicitly write out how to connect things with different sizes together.

Can you add buildings to the map via some method other than this base creator first, and let autolayout avoid collisions with that building's placement, or does autolayout only know the positions of map obstructions and its own base layout?

[TechCor]

It will avoid map obstructions and units it knows about.

The base generator has a constructor for taking existing units. You can also set the "ignoreLayout" flag for a UnitData which uses manual placement when the generator gets to it.

Each call to Generate keeps the list of created units, so the number of bases generated isn't a concern either.

[Hooman]

Very neat. 

Sounds fairly similar to the BaseBuilder section of the OP2Helper project. Define a base as data, and allow people to create duplicate copies of it at various locations. BaseBuilder didn't have any auto layout though. All coordinates were specified relative to some fixed/center point.


Curious, would this project build with the .NET Core API? Seeing as how it's C#, the build system is a little more cross platform than C++, so I thought I'd give it a try on Linux. Unfortunately the .NET Framework API is Windows only. The .NET Core API is cross platform.

[TechCor]

I just converted DotNetMissionSDK to .Net Standard 2.0. Hopefully that is portable. Please give it a try and report back.

I don't think the rest of the project can be made platform independent, however. Tips are always welcome.