Outpost Universe Forums

Projects & Development => Projects => Topic started by: TechCor on March 02, 2019, 11:29:15 PM

Title: OP2 Scenario Project for C#
Post by: TechCor on March 02, 2019, 11:29:15 PM
Hey all, and thanks Leeor for marking my account as confirmed.

I've been playing Outpost 2 for the last couple months and completed both campaigns and Eden Starship on hard. Still wanting more, I started looking around at the various GitHub projects and how to make new scenarios.

The number one thing I noticed is that making scenarios is incredibly tedious, compounded by the fact that it is done in C++ which I haven't used seriously in 10 years, and had completely forgotten the pains of char* strings, header files and linker errors.

So last week I decided to move it all to C#. I wanted to make sure the community had access to the source, so you can access the project here:

https://github.com/TechCor8/OP2DotNetMissionSDK (https://github.com/TechCor8/OP2DotNetMissionSDK)

It also reads JSON for the initial setup and creating triggers and disaster zones. The goal being to move scenario development into an external editor.


UPDATE [2019-8-22]:

StateSnapshot and AsyncPump added.

Features exclusive to the C# SDK:

Things that are still missing in C# that are in the C++ SDK:
Miscellaneous functions that are missing:
Title: Re: OP2 Scenario Project for C#
Post by: leeor_net on March 03, 2019, 02:55:22 AM
Glad you were able to sign on!

This is wonderful! I've always loved C++ but recently learned how much easier it can be to develop with C#. I still prefer C++ because I'm a masochist but this could help to make mission coding more accessible. There has allegedly been an effort toward making it accessible via python scripting (glares at Arklon and BlackBox) but that seems to be an echo on the wind.

We've been pushing to migrate everything to GitHub to be as public as possible with it. Would you consider doing the same? Or would you prefer that we integrate this into the OutpostUniverse organization on GitHub?
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 03, 2019, 12:14:12 PM
Wow, what? I'm both surprised and intrigued.

I've manged to create simple DLLs with other compilers before, that the game sees as a level, and can load them, but they've never been able to do anything useful, like create initial units. The name mangling used by the C++ functions that Outpost2.exe exports always got in the way. The data exports (from the DLL) were simple enough to get the game to recognize the DLLs, but the function exports (from Outpost2.exe) limited the DLL from doing anything useful.

Yet you say this works? What about saving and then loading a game? Outpost 2 requires all data to be accessible in a fixed sized buffer, and there's no notification of when it's going to save or load data to/from that buffer, so it needs to be the live data.


I would love to see this on GitHub.
Title: Re: OP2 Scenario Project for C#
Post by: Arklon on March 03, 2019, 12:58:10 PM
There has allegedly been an effort toward making it accessible via python scripting (glares at Arklon and BlackBox) but that seems to be an echo on the wind.
That's still in development, but it's also larger in scope than just for writing missions. As far as missions goes, it's basically done, except for the parts of the public API we never bothered to document like Pinwheel groups, and exposing more non-public API game internals (though quite a bit is already exposed).

I've manged to create simple DLLs with other compilers before, that the game sees as a level, and can load them, but they've never been able to do anything useful, like create initial units. The name mangling used by the C++ functions that Outpost2.exe exports always got in the way. The data exports (from the DLL) were simple enough to get the game to recognize the DLLs, but the function exports (from Outpost2.exe) limited the DLL from doing anything useful.
He worked around this problem by just making extern C wrappers of all the OP2 API functions. I remember someone telling me that C++.NET/CLR is fully ABI compatible with C++ though (they're both still built using the MS toolchain so that wouldn't be a shock), so there's probably a better way to do this. That said, the general idea of using C++.NET as an interop shim between C++ and C# is actually a pretty good practice, and also the only real good use case for C++.NET.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 03, 2019, 08:01:41 PM
Quote
We've been pushing to migrate everything to GitHub to be as public as possible with it. Would you consider doing the same? Or would you prefer that we integrate this into the OutpostUniverse organization on GitHub?

I'll put it on GitHub. I thought about that as I was posting, but was too lazy to do it right then.

Quote
What about saving and then loading a game? Outpost 2 requires all data to be accessible in a fixed sized buffer, and there's no notification of when it's going to save or load data to/from that buffer, so it needs to be the live data.

I haven't tested the saving yet, but the general idea is to reserve a static buffer larger than what is needed for most missions in the unmanaged DLL, and forward the buffer to C# through GetSaveRegions where it can be written with a memory stream. It is a bit inefficient, but so is everything else about this. Hopefully, the data is loaded by the time InitProc is called. If not, I could test in a loop for some init flag that is always saved to the buffer.

-- Thinking about what you said, I may have misunderstood what GetSaveRegions does. Is it just getting a pointer to the buffer at initialization? This whole saving process is really arcane. I would have had Save()/Load() procs... If that's the case, I'll probably have to send the buffer at init and use some kind of indexing wrapper class in C# for accessing and modifying the buffer with live data.

This makes me wonder then, is it enough to create the buffer dynamically at DLL attach based on the size reported by C#? For example, in C# there could be a SaveBuffer class with GetBufferSize which is based on the total size of the variables in the class. Use that to new char[] the buffer and pass in GetSaveRegions.


It really is just a lot of wrapper boilerplate. My eyes started to glaze over while doing triggers.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on March 03, 2019, 09:56:09 PM
Hi TechCor,

Welcome to the forums and glad to here you are enjoying Outpost 2 20 years after its creation.

It sounds like you have a great start on the programming. One note on making scenarios is the DLL has to be loaded into the proper base address. Otherwise, if you save/load a game, the mission objectives will become garbled. If you haven't seen it yet, there are 2 posts in the wiki that go over a lot of the C++ settings details (just glance over the SVN part I guess). https://wiki.outpost2.net/doku.php?id=op2_sdk:projectcreation

Within the SVN repository you can find a decent number of the custom missions that have been added to Outpost 2 over the years for coding examples. Up to you to sift through the good/bad though. :| https://svn.outpostuniverse.org:8443/!/#outpost2/view/head/LevelsAndMods/trunk/Levels

If you want to move past basic level design, you may want HFL and/or IUnit to give some of the more advanced commands. HFL relies more on memory hacking. You can also add a modal dialog to the beginning of your scenario using odasl. Couple of examples in missions I've made and Plymouth Cold War.

There are a lot of undocumented bugs in the mission development SDK. Both internal to Outpost 2 and in Outpost2DLL and HFL. It would be good to get things fixed or at least documented, but it is all volunteer work.

I've been trying to split my time between improving the mission SDK and template on GitHub and trying to help improve NetFix, but am happy to try and answer questions about the SDK.

If you run through all the colony games and fan added colony games, there are some multiplayer coop missions with AI. You can bring up two instances of Outpost 2 on your machine and attempt them solo for a tough challenge. I think Danger Zone is barely doable single player. If you want to play multiplayer, I'm willing to schedule some time out, but mostly play coop missions now. The deathmatches never did much for me (probably because I'm a mediocre player).

Anyways, happy playing and coding.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 03, 2019, 11:47:46 PM
Yeah, I'm liking this idea of using C# for mission development. I guess it makes sense there would be some kind of interop.

I was looking through the code a bit the other night. Looks like the C# classes are proxies for the C++ classes. The C++ classes are in turn proxies for internal Outpost2.exe classes. I'm wondering if we can do something more direct there. Perhaps there's a way that adds less boilerplate, or that might make the Save/Load thing easier. The game really could have made use of Save/Load methods, or at least callback notifications. It would have made the system much more flexible. I'd love to see this up on GitHub so we could play around with this C# idea some more.  :)

And Triggers, oh boy.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 03, 2019, 11:53:05 PM
OK. I read the base address thing 5 or 6 times, and I think I finally get it.

So I assume Outpost 2 is doing something like:

GetSaveRegions to get the buffer address and size.
Saves the struct ADDRESS to the save file and copies the buffer.
At some point OP2 loads the buffer from the save file back to that saved address.

If the DLL or save buffer were dynamic addresses, it would end up copying the data back to the wrong place.

If this is true, the "oversized static save buffer" method with a C# wrapper that indexes properties will be the way to go.

What threw me off, is I would think it would just query GetSaveRegions for the most up-to-date address before loading the data.


--
Playing two copies of OP2 at the same time might be tough. I am more of a "versus AI" type, but especially enjoy the morale aspects of the game. Otherwise, I'd play Starcraft. I never did like competitive multiplayer, but maybe that's because the Starcraft guys were just too good.

Didn't realize there were projects on the SVN repo. I thought everything was on GitHub. Will look into it when I have time.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 03, 2019, 11:58:32 PM
Yeah, I'm liking this idea of using C# for mission development. I guess it makes sense there would be some kind of interop.

I was looking through the code a bit the other night. Looks like the C# classes are proxies for the C++ classes. The C++ classes are in turn proxies for internal Outpost2.exe classes. I'm wondering if we can do something more direct there. Perhaps there's a way that adds less boilerplate, or that might make the Save/Load thing easier. The game really could have made use of Save/Load methods, or at least callback notifications. It would have made the system much more flexible. I'd love to see this up on GitHub so we could play around with this C# idea some more.  :)

And Triggers, oh boy.

I'm not sure if it can be made more direct, but certainly if someone knows a way, let me know, and I'll update it. For now, "working" is better than not.

The trigger problem (where it will call the native DLL) is definitely a tough one. I am really hoping that HasFired() works properly. In which case, I will save all the triggers to a list and poll HasFired() to execute the result.

Also, it is now on GitHub. Check the first post.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 04, 2019, 01:45:22 AM
Ok, I've cloned the GitHub repo. I'll have to take some time to play with it.

I agree that working is better than not-working/nice/"perfect".

The thought I had the other night, is the C# code seemed to be using the C++ classes in much the same way as native C# classes. That's what made me think maybe it could be more direct. I admit my C# knowledge is pretty weak though, so perhaps I'm missing something obvious. I know C# has some differences in terms of memory management, and these sorts of wrappers could be to paper over the memory management differences. Though in the case of Outpost 2's exported classes, they are mostly proxy objects with simple value like semantics, so that might negate some of the memory management differences. You could basically treat those classes like you'd expect of structs. Hard to say at this point, just a sneaking suspicion I had.

Internally, Outpost 2 uses function pointers for triggers. If we use the memory hacking APIs, we could gain direct access to those pointers, and setup triggers without going through the named DLL function export/lookup process. Though I assume C# must have a way of exporting plain functions from DLLs, so maybe that wouldn't be needed.

As for the saved games, Outpost2.exe calls GetSaveRegions just after loading the DLL. It calls it before InitProc when starting a new mission. When loading a mission it calls it before resuming the game cycle. It stores the result, and never asks again. Whenever a player saves the game, Outpost 2 copies the contents of that buffer to the saved game file. There is no notification to the mission DLL, so the mission has no idea when a game has been saved. Similarly, it doesn't know when it's been loaded. It simply loads the DLL, calls GetSaveRegions to find the buffer, and then copies data from the saved game file to that buffer, then resumes the game cycle.

You could potentially write a detect for the load, using an initialized global/static variable outside of the saved region, and modified at the end of InitProc. If it has the original initialized value, rather than the InitProc set value, the game has been loaded since InitProc ran. There would be no way to write a detect for saving the game though. At least nothing short of hooking into the code section of the game. Hence that static buffer must always contain current live data, or you risk it not being stored when a game is saved. Well, minus the small window of time within a game cycle, when the DLL has complete control to run AIProc, or a trigger callback. Once it returns to the game engine though, the game engine can do whatever in terms of saving/loading. You would need to update the static buffer before returning if things were modified outside the save region.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 04, 2019, 05:57:20 AM
Thanks for clearing all that up!

Saving and loading appears to be working.

I went ahead and made it a packed C# class for storing variables + a native memory wrapper. Checks if "isLoaded" in the update loop, and calls Load if it isn't. Then saves at the end. Glad to know the buffer doesn't need to be declared in the native plugin.

I'll see what I can do for triggers next.

---
On the topic of bypassing the native wrapper, besides C++ name mangling, C# does not appear to support __fastcall which is used on some of the functions, most notably TethysGame. Not worried about it right now though.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 10, 2019, 11:56:57 AM
UPDATE:

Triggers have been added. It works a little differently in C# to avoid interop passthrough.

The process is as follows:

Of course, you can also specify all triggers and their IDs through the JSON file. Currently, you can't specify actions in JSON, but you can still reference the ID and perform the actions via C#.

---
The C# DLL can now be referenced under different file names.

The NativePlugin has a #define USE_CUSTOM_DLL in LevelMain.cpp for toggling loading.

USE_CUSTOM_DLL true = [NativePluginName]_DotNet.dll
USE_CUSTOM_DLL false = DotNetMissionSDK.dll (JSON only library).

By default, this is set to false for JSON only missions.

---
UPCOMING:


Unfortunately, groups are kind of a mystery to me. The wiki page does not exist, and there isn't much documentation to be found elsewhere. Might take some experimentation.
Title: Re: OP2 Scenario Project for C#
Post by: Arklon on March 10, 2019, 03:31:21 PM
Okay, I looked at your code again, it actually wasn't using C++.NET like I thought before. I did some more research on the C++.NET interop shim technique, and yeah, it's fully ABI compatible with C++ (you need to do native function calls in unsafe { } blocks) including all calling conventions etc., and it compiles to a DLL that's a .NET assembly that you can drop into C# and it just works, so it would serve both of the roles that the native and .NET layers do currently.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 10, 2019, 07:58:28 PM
Hmm. There were 2 interop layers and 1 C# layer before. I just merged the 2 interop layers (NativeInterop and DotNetInterop) into 1 layer (DotNetInterop). C++ can have both managed and unmanaged code in the same DLL by setting file properties.

I'm thinking of trying this technique on the NativePlugin layer to call C# directly. I didn't try this initially in case Outpost 2 had trouble with it.

If I understand you correctly, you are saying DotNetInterop should wrap the Outpost2 calls with managed classes instead of extern C. DotNetMissionSDK (C#) could then reference DotNetInterop instead of using DllImport.


EDIT:
Tried to get NativePlugin to work with the CLR on just one file. Linker complains of flag incompatibility, so I don't think this is an option. I would have to make project wide flag changes which apparently some of them are required for Outpost 2.
This means there will be a circular reference problem in DotNetInterop if I change the externs to managed classes. Referencing the C# DLL may need to be replaced with reflection to accomplish this.

I'm out of time now. Will look at it later.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 11, 2019, 02:06:42 AM
Ok, this is getting interesting.

Have you updated GitHub? I don't see any recent changes.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 11, 2019, 11:09:33 AM
There were 6 commits yesterday.

You can check activity here:

https://github.com/TechCor8/OP2DotNetMissionSDK/commits/master

UPDATE:

I just switched the dependencies. It actually makes sense now. C++ no longer depends on C#. C# now references the C++ DotNetInterop and implements the C++ managed interface for MissionEntry.

Should be possible to wrap all the Outpost 2 functionality inside of DotNetInterop as managed C++ classes now.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 11, 2019, 01:00:46 PM
Ahh, I see what happened. It seems the public history was rewritten. Perhaps a rebase was run or something. When I updated, it pulled down new commits, but then refused to merge them to my local master branch, since the histories had diverged. Hence when I checked the log of my local master branch, I didn't see the updates.

It's such a tiny little error message, in a paragraph of text, and no highlighting.

Ok, yes, I see you've been busy.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 11, 2019, 02:25:15 PM
Yes, I rewrote the history because I accidentally pushed the first few commits with the wrong author info. Was hoping nobody would notice.
Title: Re: OP2 Scenario Project for C#
Post by: BlackBox on March 11, 2019, 02:54:08 PM
This looks really promising!

Thinking about this a bit further, it might be useful to join forces with the Python SDK that Arklon and I have been working on (this isn't dead, we've been quietly working on it in the background, hoping to have something more or less ready for general consumption soon-ish).

It'd probably be possible to eliminate the need for the unmanaged wrapper/proxy DLL that loads the assembly.
For the Python SDK (which is implemented as a mod DLL loaded through op2ext/the INI file) we already hook/replace quite a bit of UI code related to the list of missions in the various menus (in order to show Python missions in the list). We also hook a whole bunch of code relating to loading the mission, as well as trigger callbacks and loading/saving the game (same reason, since the mission itself is no longer its own DLL).

Since we're already hooking these places it probably wouldn't be a huge amount of work to detect whether a mission is a .NET assembly, and if so, load it via CLR hosting (i.e. call directly into the assembly without needing the C++ proxy DLL). I'd expect the assembly that handles interop between the Outpost2 API and .NET would probably still be needed.

Also, regarding loading and saving from earlier in the thread, the game actually internally handles this by passing a file-like object (StreamIO) to the internal class that represents a mission DLL. This in turn calls the GetSaveRegions exported function and then reads/writes the buffer contents from/to this file.
Since we similarly can't assume a fixed buffer for Python missions (since we don't control where the interpreter places things in memory, etc) our current plan is to expose the low-level Load/Save API to the python code and the mission is responsible for loading/saving whatever it needs to reconstruct its state from a saved game. We can probably simplify this further by having the mission provide an object such as a dict that we pickle/unpickle to/from the file. I would guess something similar would work for C# as well.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 12, 2019, 10:46:21 AM
It would be nice to get rid of the extra dll per mission, but since full compatibility is possible, I will continue to support that.

If the mod calls the DotNetMissionEntry interface, it would be possible to bypass the unmanaged DLL. Would just need to add something for level details. This way the C# DLL can be used with or without the mod.

If the mod wants to support JSON missions without the unmanaged DLL, it would need to check for the JSON files, and call into the special/uncustomized C# DLL's DotNetMissionEntry with the appropriate file path.

If you don't want to use the CLR in the mod, you can just call DotNetInterop's C functions which will do it for you. That DLL will be required regardless, and I don't think the slight overhead is worth worrying about. Also, the DotNetInterop will handle switching to JSON for you. You just have to set the bool in Attach for whether to load a C# mission or JSON.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 18, 2019, 05:02:13 PM
Decided to put some time into things I can see. One of the goals is to be able to rapidly set up generic scenarios (two hours or less time invested).

I've just added Autolayout. AutoLayout lets you place a colony without worrying about tiles, collisions, tubes, etc. It's very simple to use and quite customizable. I was having fun toying with the build order and distances to see what kind of bases it would come up with.

There is still some work to be done on it with regards to tube placement and structures being placed too far away (crow flies vs walking distance). I'll be adding A* pathfinding to the project and using that to solve those issues.

Here is an example of the usage (JSON AutoLayouts section):

Code: [Select]

"PlayerID":0,
"BaseCenterPt": { "X":27, "Y":38 },

"Units":
[
{ "TypeID":"CommandCenter", "MinDistance":1 },
{ "TypeID":"StructureFactory", "MinDistance":0 },
{ "TypeID":"ConVec", "CargoType":"CommandCenter", "MinDistance":1, "SpawnDistance":3 },
{ "TypeID":"ConVec", "CargoType":"None","MinDistance":1, "SpawnDistance":3 },
{ "TypeID":"CommonOreSmelter", "MinDistance":0 },
{ "TypeID":"Agridome", "MinDistance":0 },
{ "TypeID":"Agridome", "MinDistance":1 },
{ "TypeID":"GORF", "MinDistance":0 },
{ "TypeID":"Residence", "MinDistance":1 },
{ "TypeID":"StandardLab", "MinDistance":0 },
{ "TypeID":"Nursery", "MinDistance":0 },
{ "TypeID":"Residence", "MinDistance":1 },
{ "TypeID":"University", "MinDistance":0 },
{ "TypeID":"RobotCommand", "MinDistance":0 },
{ "TypeID":"Residence", "MinDistance":1 },
{ "TypeID":"MedicalCenter", "MinDistance":1 },
{ "TypeID":"DIRT", "MinDistance":0 },
{ "TypeID":"Agridome", "MinDistance":1 },
{ "TypeID":"MedicalCenter", "MinDistance":1 },
{ "TypeID":"RecreationFacility", "MinDistance":0 },
{ "TypeID":"RecreationFacility", "MinDistance":1 },
{ "TypeID":"DIRT", "MinDistance":2 },
{ "TypeID":"VehicleFactory", "MinDistance":1, "CreateWall":true },
{ "TypeID":"Tokamak", "MinDistance":3 },
{ "TypeID":"Tokamak", "MinDistance":2 },
{ "TypeID":"GuardPost", "CargoType":"Microwave","MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost", "CargoType":"EMP", "MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost", "CargoType":"Microwave","MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost", "CargoType":"ESG", "MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost", "CargoType":"RPG", "MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost", "CargoType":"EMP", "MinDistance":2, "CreateWall":true },
{ "TypeID":"GuardPost", "CargoType":"RPG", "MinDistance":2, "CreateWall":true },
{ "TypeID":"Tokamak", "MinDistance":3 }
]

And the base it generates:

(https://scoriastudios.com/myfiles/op2/Autolayout001.png)

EDIT:

Added some BFS and A* pathing for structures and tubes. Base now stays together properly. Autolayout vehicles spawn last to prevent structures from being affected by their placement. Wall placement should probably be moved to after structure placement.

Image above has been updated to reflect changes.

I think I'll add an option for single or multi tube setups. Image shows multi. Single tube is closest only. Multi is closest + all structures within minDistance.
Title: Re: OP2 Scenario Project for C#
Post by: Crow! on March 19, 2019, 11:51:19 AM
So with this, we could make a map which, at start-up, randomly selects a subset of buildings, and assigns that same set of buildings to all players, without the map needing to explicitly write out how to connect things with different sizes together.

Can you add buildings to the map via some method other than this base creator first, and let autolayout avoid collisions with that building's placement, or does autolayout only know the positions of map obstructions and its own base layout?
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 19, 2019, 02:40:29 PM
It will avoid map obstructions and units it knows about.

The base generator has a constructor for taking existing units. You can also set the "ignoreLayout" flag for a UnitData which uses manual placement when the generator gets to it.

Each call to Generate keeps the list of created units, so the number of bases generated isn't a concern either.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 19, 2019, 02:47:30 PM
Very neat.  :)

Sounds fairly similar to the BaseBuilder section of the OP2Helper project. Define a base as data, and allow people to create duplicate copies of it at various locations. BaseBuilder didn't have any auto layout though. All coordinates were specified relative to some fixed/center point.



Curious, would this project build with the .NET Core API? Seeing as how it's C#, the build system is a little more cross platform than C++, so I thought I'd give it a try on Linux. Unfortunately the .NET Framework API is Windows only. The .NET Core API is cross platform.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 19, 2019, 07:19:33 PM
I just converted DotNetMissionSDK to .Net Standard 2.0. Hopefully that is portable. Please give it a try and report back.

I don't think the rest of the project can be made platform independent, however. Tips are always welcome.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on March 19, 2019, 10:10:25 PM
We have some minor changes brewing in Outpost2DLL, OP2Helper, and HFL. Would be nice if we could get them into OP2MissionSDK since it sounds like TechCor is actively using it.

Nice to see some more C# development going on. Make me feel like a bit of hack with how quickly this is moving compared to the C++ code I am slogging through.

I enjoyed seeing the auto base builder results.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 20, 2019, 02:00:59 AM
Thanks for the update!

Now it runs package restore, though fails at a later step (edited paths):
Code: [Select]
/bin/sh: 2: /tmp/tmpcac72cbbf94540508eaf575e65118431.exec.cmd: copy: not found
/.../OP2DotNetMissionSDK/DotNetMissionSDK/DotNetMissionSDK.csproj(8,5): error MSB3073: The command "copy "/.../OP2DotNetMissionSDK/DotNetMissionSDK/bin/Debug/netstandard2.0/DotNetMissionSDK.dll" "*Undefined*Outpost2\DotNetMissionSDK.dll"" exited with code 127.

Looks like you have a shell command as part of the Post Build step. Shell commands can easily lead to a few incompatibilities. Here, Linux uses "cp" rather than "copy". There also seems to be a strange "*Undefined*" in the path, which looks like use of an environment variable.

I checked DotNetMissionSDK.csproj and saw this:
Code: [Select]
  <Target Name="PostBuild" AfterTargets="PostBuildEvent">
    <Exec Command="copy &quot;$(TargetPath)&quot; &quot;$(SolutionDir)Outpost2\$
(TargetFilename)&quot;" />
  </Target>

It seems $(SolutionDir) was undefined. That makes sense, since I tried to run the build from the "DotNetMissionSDK/" folder to avoid errors with the other projects in the solution file.

Doesn't look like there is much in that project file in terms of build settings, or even references to source files. I'm guessing C# is a little smarter about finding the source files for a project.



The following might help for fixing the Post Build command:
Best way to make a project with post-build script work on MonoDevelop and Visual Studio? (https://stackoverflow.com/questions/13150733/best-way-to-make-a-project-with-post-build-script-work-on-monodevelop-and-visual)
The MSBuild Task answer looked promising.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 20, 2019, 02:48:02 AM
Nice! I didn't know it could be done that way!

I replaced line 8 in DotNetMissionSDK.csproj with:

Code: [Select]
<Copy SourceFiles="$(TargetPath)" DestinationFolder="$(SolutionDir)Outpost2" />
It seems to work. You will need to run from the solution however.

This one with the $(ProjectDir) also works and may be the better choice:

Code: [Select]
<Copy SourceFiles="$(TargetPath)" DestinationFolder="$(ProjectDir)../Outpost2" />
The C++ post builds did not work however. The project won't load when I do that. I'm sure those projects have a lot more problems than a convenience post-build step though.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 20, 2019, 03:53:04 AM
That was quick.


I tried out the first option, and when I built the specific project file, it generated a weird "*Undefined*" directory and copied to that.

I tried out the second option and when I built the specific project file, it worked.


For either option, when I tried to build the whole solution, I see what looks like some partial success, and then a whole bunch of errors from the other projects. The output file was not updated. I don't think the Post Build step ran. Perhaps the Post Build step didn't run due to errors in the other projects. I assume it only runs after a successful compilation.



Edit:
As for C++ projects, there are much deeper problems that prevent them from building on Linux. The whole project file structure ends up getting significantly changed. C++ uses a different compiler on Linux than for Windows, so the references to source files end up in different tags referring to a different compiler. The compiler options are also completely reworked. I've never seen a C++ project file that contained both Windows and Linux settings in the same file. All the examples were either one or the other.

C# seems to be much more standardized in terms of compiler and compiler options across platforms.



Edit 2:
From the solution folder, I tried running:
Code: [Select]
dotnet build DotNetMissionSDK

With the first option, it appears to compile successfully, though the output file is not updated. No strange folders observed.
With the second option, it appears to compile successfully, and the output file is updated as expected.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 21, 2019, 03:22:53 PM
The last thing to do for AutoLayout is get map wrap working, but I'm having a bit of trouble understanding the values I'm getting back for clip.

on6_01.map is a 128x128 map.

When I clip the map, I get an area of (32,0) to 160,127) inclusive as valid tiles.
This comes out to a size of 129x128. That can't be right.

Maybe it would help if I knew why maps can start at arbitrary points.

newworld.map gets an area of (0,0) to (511,255) which comes out to a 512x256 size. Which is fine.

Here's the code I'm using to calculate this:

Code: [Select]
// Calculate map size and determine if map wraps
// Clip to top left
LOCATION mapSize = new LOCATION();
mapSize.ClipToMap();

// Initialize to top left corner
area = new MAP_RECT(mapSize, mapSize);

for (int i=0; i < MaxMapSize+10; ++i)
{
int x = mapSize.x;
int y = mapSize.y;

++mapSize.x;
++mapSize.y;

mapSize.ClipToMap();

// Clamped map will pull mapSize.x back to x.
// The max value is the current and previous value.
if (mapSize.x == x)
{
area.maxX = x;
doesWrap = false;
}

// Unclamped map will wrap mapSize.x back to 0.
// The max value is the previous value.
if (mapSize.x == 0)
{
area.maxX = x;
doesWrap = true;
}

// Y is always clamped.
// When mapSize.y is pulled back to y, and we have our clamped or wrapped values, end search.
if ((mapSize.x == x || doesWrap) && mapSize.y == y)
{
area.maxY = y;
break;
}
}

EDIT:

Since the area on the wrap map is correct. I was able to get base generation working there. I tested generating the base near the edges of the clamped map to see if it also worked, and it does.

I'm going to assume that 129 width is expected unless someone says otherwise.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on March 21, 2019, 10:26:19 PM
Map starting position has two options. Nonwrap maps have offsets of (31,-1). Typically I use the constants X_ and Y_ defined in op2helper.h when making maps.

Check the usage note on cliprect in map.h from the project op2utility for a bit more detail. https://github.com/OutpostUniverse/OP2Utility/blob/master/src/Map/Map.h

Hooman could maybe give you a bit more technical details about it. Also in the sun repo there is a ollydbg section that contains detailed notes about different files, structures, and code related to Outpost 2. Maybe a good reference to help with understanding.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 22, 2019, 12:18:30 AM
That's good to know that those offsets don't change arbitrarily from map to map.

I'm thinking I could clip -1,-1 to determine if the map wraps and hard code the lower bounds. Then use the result for wrap upper bounds, or clip again at 5000,5000 for the upper bounds for clamped maps. (The for loop used above is a bit inefficient. ;) )

The question is why am I getting the values I am from clip, as I'm using that function everywhere to resolve clamp/wrap. It seems I can't trust the values coming from it.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 22, 2019, 06:07:18 AM
For non-around-the-world maps, there is an internal memory offset of one 32-column wide section of tiles for the map data. As the internal map coordinates are 0-based, but the displayed values are 1-based, they also need to be offset by 1. Hence you end up with display coordinates to internal coordinates being offset by (x + 31, y - 1).

I assume the offset is to add padding to the map data, so when units drive off the edge of the map, it doesn't crash the game. I don't think they did proper clipping and bounds checking, so that was probably a hack to prevent crashes. In terms of memory allocation, non-around-the-world maps actually allocate twice as much memory as necessary. That gives them room to have padding on both sides of the map, with possibly a lot of extra padding on the right side.

For around the world maps, the coordinates are always wrapped, so they'll never be out of bounds. Hence they have no need for that internal offset.

You can find some details in:
FileFormats: Read down to post about the Saved Game File Format (https://forum.outpost2.net/index.php/topic,1250.msg27376.html#msg27376). In particular, the ClipRegion struct (or ClipRect in some sources).

The around-the-world aspect is controlled by the map width. From the above, if lgMapWidth >= 9 (or equivalently, if mapWidth >= 512) then it's an around-the-world map.

Additionally, the bottom edge of the map is marked as impassible. I haven't verified if this is all maps, or only around-the-world maps, or just some maps. I assume that's probably also to deal with clipping problems. Hence the ClipRect may exclude the last row of tiles.



Edit: The 129 is probably not correct.
(160, 127) - (32, 0) = (128, 127)
Those values should be the size, which for 0-based indexing would mark a non-inclusive endpoint. Valid values are (0..127, 0..126) inclusive.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 22, 2019, 11:06:45 PM
You said "around the world" so many times I have ATC in my head.

So Clip() doesn't clamp to the playable area, but includes the max exclusive value for the x-axis unless the map wraps. I guess that's a bug? I changed everything to use the correct inclusive maximum. C# now uses its own clip instead of Outpost2DLL's.

When I saw the 32 value initially, I thought it was some cool map cropping feature that some games use to only show a portion of the map. Often completing objectives would "unlock" more playable area. What a let down.


I'm done with AutoLayout for now. The only thing that bothers me is the tube aesthetic isn't perfect, but it would be quite complex to cover all the cases to make it look nice. Might come back to it later.

Thanks for the help with tile offsets, guys. Here is your reward:  8)
(http://scoriastudios.com/myfiles/op2/AutoLayout002.jpg)
(AutoLayout at map edge)

Going to start making general-purpose AI.

I'm a little concerned about the limited API functionality.

It doesn't appear possible to get a structure's demand through Outpost2DLL or HFL, so I may just wing it with like pop / 25, but there could be other problems that can't be resolved. Time to see how well this goes.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 24, 2019, 04:25:24 AM
Lol @ATC  :P



Normally, when doing bounds checking, I generally expect some kind of Point + Size to determine a Rect, much like how a for loop will loop over arrays:
Code: [Select]
for (int i = 0; i < container.size(); ++i)

It's pretty easy and standard to write 0-based array code that is start point inclusive, and end point exclusive.

When looking at the Rect struct in Outpost 2, I assumed it would be the same. After reading your comment though, I decided to go check. In particular, there is the int Check(LOCATION& ptToCheck); method for hit testing if a point is in a Rect. It turns out to be a little more complicated and bizarre than I thought. There's a bit of code to determine if the Rect wraps around the edge of an around-the-world (ATC) map, but other than that complication it's basically checking: (y1 <= y < y2) and (x1 <= x <= x2). Note the end point conditions. It's exclusive for y, but inclusive for x. I'm thinking that's got to be some kind of bug.



I suppose you could still try modifying the clip rect, and try to unlock an area of the map. If you clip a section of the map, it's likely to still be visible, but the mouse cursor will likely change when you hover over an impassible area of the map. Maybe with a bit of day/night hacking, you could also make such a section of the map permanent night time until it is unlocked. Might not be quite what you were looking for, but perhaps close enough.



As for building requirements, I don't believe we currently have a way to get it, but I could probably find a way to get that info for you. I know where it is in memory, there's just no API written (yet) to access it. Same for the day/night position.



Oh and really cool image btw. Looks like it handles corners and restrictive complicated terrain quite well.  :)
Title: Re: OP2 Scenario Project for C#
Post by: Crow! on March 25, 2019, 11:26:04 AM
The biggest thing I'm thinking about for how I would actually use this function is that, in most scenarios, there will be one or two structures you want to demand are put in a particular place.  Smelters come to mind here, as well as enemy structures that are mission objectives in some way.  If you use the manual structure placement you mentioned before, will it know to build a tube from the final autolayout-ed base to wherever the manually placed structure is, or does it simply know not to cause unit overlaps with it?
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 26, 2019, 09:14:18 AM
The biggest thing I'm thinking about for how I would actually use this function is that, in most scenarios, there will be one or two structures you want to demand are put in a particular place.  Smelters come to mind here, as well as enemy structures that are mission objectives in some way.  If you use the manual structure placement you mentioned before, will it know to build a tube from the final autolayout-ed base to wherever the manually placed structure is, or does it simply know not to cause unit overlaps with it?
You make a good point.

I just added a "MaxTubes" attribute to UnitData that defaults to -1 (unlimited). This can be used to control the number of tubes to any structure passed to BaseGenerator. Tokamaks, Light Towers, Mines, etc will default to 0 if the value is -1, so you will be able to connect those by tubes as well, if desired. This attribute applies even with IgnoreLayout set to true. Same with CreateWall.

If all else fails, you can always use the PathFinder to generate a path between structures for creating tubes. It's a little more complicated, but can help create extra tubes if the BaseGenerator isn't putting them where you want them. All generated units are put in a list, so you can scan the list for the auto structure you want to connect.

Quote from: Hooman
It's pretty easy and standard to write 0-based array code that is start point inclusive, and end point exclusive.
[snip]
Saying this made me rethink using max inclusive. I looked at some other frameworks, and apparently you are right, it is standard to use max exclusive. It seems strange to me that a rect contains a point on its min edge, but not its max edge. However, I also think that a rect of zero size shouldn't be able to contain a point, even if it is infinitely small. Anyway, it's been standardized.




Just kind of want to outline how I'm going to set up the AI.

Managers
The AI is going to be set up as three managers. Managers assess the situation and prioritize goals using heuristics and goal weights. Goals are top-level tasks. Managers have no direct dependencies on each other. They assess the situation and perform their own goals to the best of their ability.

The managers are independent to prevent task blocking. For example, if the BaseManager needs a spaceport, but Space Program has not been researched, it will get stuck waiting for the research to complete rather than pursue other goals. CombatManager would block the other two managers if it's performing a raid, and so on.

BaseManager - Handles deployment, morale, and economic tasks. Also builds military.
ResearchManager - Chooses the next research topic for the colony. Base manager assigns the number of scientists to the lab.
CombatManager - Directs combat tactics with the military units available.

Tasks
Tasks represent what an AI must do to complete a goal. They are arranged as a tree. The current goal task is at the top. When a manager wants to achieve a goal, it calls PerformTask on it.

Tasks contain a list of dependent tasks underneath it that are required to complete the current task. Perform task first checks these tasks to see if they are complete (IsTaskComplete()). If all are true, the task performs itself. If any are false, those unfinished tasks get PerformTask called on them. This can continue all the way down to bottom-level tasks that have no dependencies.

Examples of top-level tasks include "LaunchSpaceportEvac", "CreateMilitaryUnits", "HarassEnemy", and "FixDisconnectedStructure".

If, for example, the goal task is "FixDisconnectedStructure", it would check IsTaskComplete on the "CreateEarthworker" task. "CreateEarthworker" would check "CreateVehicleFactory".

Some tasks can't be completed, and the AI needs to give up on the goal to pursue other things. PerformTask returns false if a task cannot be performed.

In the example above, "CreateVehicleFactory" may not be able to complete its task. If there is no structure factory (or convec with it) or Cybernetic Teleoperation is not researched, the task cannot be completed. The manager is informed via the return, and chooses a different goal.

Bot Types
Bot types or "personalities" will be based on goal weights. I intend to support the following predefined types:

Code: [Select]
PopulationGrowth,	// Bot focuses on growing population. Keeps enough defense to avoid being killed. Will build Recreation, DIRT and other optional structures.
LaunchStarship, // Bot focuses on launching starship. Keeps enough defense to avoid being killed.
EconomicGrowth, // Bot focuses on resource acquisition. Keeps enough defense to avoid being killed.
Passive, // Bot does not build new structures. Keeps enough defense to avoid being killed.
Defender, // Bot will build military units and defend itself and allies. Does not attack.
Balanced, // Bot will build military units and defend itself and allies. Attacks with best available strategy.
Aggressive, // Bot will build military units and won't defend itself or allies. Attacks with best available strategy.
Harassment, // Bot will build military units and harass trucks, power plants, and unescorted or poorly defended utility vehicles.
Wreckless, // Bot will build military units and send them to attack even against overwhelming odds.

A predefined type can be selected, and then the weights tweaked to get the desired effect. For example, you can choose "Aggressive" and tweak how much it focuses on population and economic growth to either confine it to its starting base, or have it attempt to build all over the map. You can also change these weights at any time, so a dormant AI can be triggered to suddenly turn Aggressive or Wreckless.



The greatest obstacle is going to be suboptimal access to player state. I'm thinking of creating a PlayerInfo class to start pooling information in an efficient manner at the beginning of each update.

For example, I am working on the LaunchStarshipTask, and I need to know if the evac module has been deployed. I believe I can iterate over the player units to find it, but I need to also check if the phoenix module has been deployed in another task, the orbital module in another, and... well.. you can see where this is going. Hundreds of loops to pull single items from the player state.

There are other things that are harder to get due to limited API - Beacon states (was it surveyed?), structure demand, and possibly many other things.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 26, 2019, 10:16:51 AM
Very neat. I'd started on a task and dependency management scheme years ago, but the efforts stalled out after I spent all my time tracing dependencies down rabbit holes. You can really take that stuff too far.

For instance, if any any stage, you don't have the immediate resources to do something, trace down to the way to get that resource:

Build structure Type at location (x, y) Depends on Moving ConVec with kit Type to (x, y)
Moving ConVec with Kit at (x, y) depends on having ConVec with kit type, (and (x, y) being move accessible)
Having ConVec with kit Type depends on Loading ConVec with kit Type at Structure Factory
Loading ConVec with kit Type at Structure Factory depends on having a ConVec docked at Structure Factory, and Structure Factory having kit Type in storage, and Structure Factory being enabled.
Having a ConVec docked at a Structure Factory depends on Moving a (free?) ConVec to dock location
Having kit Type at Structure Factory in storage depends on completing research to build kit Type, and having resources (common, rare) needed to build it, and on Structure Factory being enabled.
...
Moving a (free?) ConVec depends on having a (free?) ConVec
...
Having a ConVec depends on Having a Vehicle Factory, having completed research for building ConVecs, having the resources (common, rare) to build a ConVec, and the Vehicle Factory being enabled
...
Having a Vehicle Factory depends on building a Vehicle Factory ... (loop back to top)
...
Having resources (common, rare) depends on being able mine resources (common, rare)
Mining resources (common, rare) depends on ...


You get where that went. The resources were particular messy, as they effectively depended on themselves. If you didn't have a cargo truck, or smelter, or mine, or robo surveyor, you needed to build these things, and that depended on having resources (common, rare). Potentially things could deadlock with unsatisfiable requirements.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 26, 2019, 12:26:32 PM
Technically a player could receive an impossible task, too. No cargo trucks or metal and you're done.

Clearly if you have no convecs and no vehicle factory, you can't create a vehicle factory. Hence the need to declare the task impossible to complete. The question is at what point is it impossible.

Create Military Unit >
Need Vehicle Factory? Yes >
Need Structure Factory? Yes >
Need Convec with Structure Factory? Yes >
Fail out / Task impossible
...
Need Structure Factory? No >
Need free convec? Yes >
Fail out / Task impossible

But what if you have a vehicle factory? Well.. then you wouldn't need to go all the way down the tree to create a military unit.

Option 1:

Each goal can have its own set of tasks. Therefore, no circular dependencies. Downside? Massive code redundancy.

Option 2:

Each task tracks its parent. When you check if the task is possible, check if any prerequisites is an ancestor up the tree.

Create Military Unit >
Need Vehicle Factory? Yes >
Need Structure Factory? No >
Need free convec? Yes >
Is the VehicleFactoryTask prerequisite an ancestor of this task? Yes >
Fail out

Pseudo code:

Code: [Select]
CreateConvecTask : Task
{
    VehicleFactoryTask m_Prerequisite;
    Task m_Parent;

    if (!m_Prerequisite.IsTaskComplete())
    {
        if (IsAncestorTask(m_Prerequisite))
            return false; // Fail out
    }

    bool IsAncestorTask(prerequisite)
    {
        if (m_Parent == null)
            return false;

        if (prerequisite.GetType() == m_Parent.GetType())
            return true;

        return m_Parent.IsAncestorTask(prerequisite);
    }
}

I think that would terminate all circular tasks. Let me know if you think otherwise.

EDIT:

Here's Option 2 integrated into the current Task class.
Code: [Select]
public abstract class Task
{
private Task m_Parent;

protected Player m_Owner;
protected Task[] m_Prerequisites = new Task[0];


public Task(Player owner, Task parent)
{
m_Owner = owner;
m_Parent = parent;
}

public abstract bool IsTaskComplete();

public virtual bool PerformTask()
{
if (!PerformPrerequisites())
return false;

return true;
}

private bool PerformPrerequisites()
{
for (int i=0; i < m_Prerequisites.Length; ++i)
{
/* Skip completed tasks */
if (m_Prerequisites[i].IsTaskComplete())
continue;

/* If a prerequisite is an ancestor task, then we have a circular dependency. This task cannot be performed. */
if (IsAncestorTask(m_Prerequisites[i]))
return false;

/* Perform the task. Fail out if it can't be done. */
if (!m_Prerequisites[i].PerformTask())
return false;
}

return true;
}

private bool IsAncestorTask(Task task)
{
if (m_Parent == null)
return false;

if (task.GetType() == m_Parent.GetType())
return true;

return m_Parent.IsAncestorTask(task);
}
}

...And then presumably the manager picks a new top-level task if it fails.

I admit, the sheer amount of tasks that have to be considered is a PITA, but I don't think the architecture is flawed in itself.
Title: Re: OP2 Scenario Project for C#
Post by: lordpalandus on March 26, 2019, 05:43:14 PM
For my game I'm developing, I like to use something obvious to quickly cancel an activity, so that processing through various steps is quick, when failure is obvious. So for my game, instead of going through all of the code to setup variables for a technique, I first check if the player has enough energy to cast that technique in the first place. Energy is a single variable, so it is quick to check and if you lack energy, then it is quick failure state.

What I mean by this is:

In the case of construction of a military unit, the first thing that should be checked is cost. It is quick to check cost, as they are likely a globally accessible variable. If you lack the resources, there is no point checking all of the other steps.

Then, if you have the resources, the next step to check would be research. Why? Because I'd expect that all structures would be contained within a vector, and thus you'd have to search the entire vector to see if you have an active vehicle factory (as an inactive one would prevent them from building anything). If they lack the research to construct a unit, that then would be a quick check as well (even if research was stored in a vector, you'd likely be looking for a Boolean True for a researched topic)

Then check for an active vehicle factory. Then check to see if the vehicle factory isn't producing anything, and then build said unit.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 26, 2019, 10:16:15 PM
OK. Based on feedback, I've made a dependency graph for spaceship modules. The idea is to limit the decision tree so it doesn't sprawl and become unmanageable.

(https://scoriastudios.com/myfiles/op2/TaskDependencies.png)
NOTE: "IsEnabled" should be a CanPerform requirement for structures.

The top-level task order is not quite right. BaseManager should be performing a weighted situational assessment to determine priority in addition to whether or not the task can be performed.

So for example:

Choose DeployEvacModule, but if I don't have a smelter and trucks, prioritize that.
If common metal is below X amount for Y duration, prioritize CommonMetalExpansion instead.
If enemy is stronger than me, prioritize defense units over the spaceship.
If I have no structures at all, deploy a base.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 27, 2019, 01:02:11 PM
@lordpalandus: In terms of checking resources and failing early, that doesn't quite work with dependency management when you are trying to optimize time to goal completion. As a task may require many lengthy to perform dependencies, such as research, you shouldn't fail out on moving towards a goal because you don't currently have the metals to produce it. You still need to check what those lengthy dependencies are, such as doing research, or setting up mining operations, and if the prerequisites are satisfiable. It may also be that some prerequiste tasks can be done in parallel. For example, you can setup mining operations while doing research.



@TechCor: Will that algorithm work? I don't really know. I'm not sure I quite understand it yet.

As for the graph, I should point out land rush games. At the start, you don't have a Structure Factory, nor do you (usually) have empty ConVecs, but you do usually have structure kits loaded in ConVecs. As such, I think the graph needs an extra node for all structures, which is the single prerequisite of having a ConVec loaded with the proper kit. At that node, you would then fan out to having a Structure Factory, and having a ConVec. I suppose the ConVec need not be empty, since you would swap kits at the Structure Factory when loading a new kit, though generally speaking, if the ConVec wasn't empty, it was probably already given a different task.

I noticed you modelled that dependency for the Structure Factory, but not the Spaceport.

As for balancing priorities, that's a whole other ballpark. I don't really have a clear idea how to handle that.

I suppose if you get the dependency management right, you do get some minimal inbuilt priority system. Such as setting up mining to build the components for the spaceship. Such dependency tracking wouldn't be enough to shift resources between launching an evacuation module, or building an army for defense though. Or shifting research between starship components and weapons.



If you really want to complicate things, there are also maps that can get reinforcements from off the map. Perhaps a ConVec shows up with a needed kit at some delayed time. Though that's a complication that can probably be ignored starting out. There's no real way to know those sorts of things without advance knowledge. Would be nice to have custom pluggable inputs to let the algorithm know about such events though.



Looking more closely at the dependency loop detection, I'm thinking it may be more complicated. The cycle can close at any point, so you don't know where the head of the loop is. It might be 5 levels down before the start of the loop, and maybe another 10 levels until it closes. Those tasks may have seemingly very little to do with the target goal.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 27, 2019, 06:24:17 PM
I think you are looking at the tasks too granularly.

Let's look at how HaveSpaceport would work if we don't have a spaceport.

We start in HaveRocket and ask, HaveSpaceport.IsTaskComplete?

TaskComplete If:
Spaceport exists
Spaceport has empty bays (This should basically be always true)

Task is not complete, so we attempt to perform the task.

Prerequisites:
HaveStructureFactory
HaveConvec

If we don't meet prerequisites, perform those tasks first. If they fail, task fails, manager can decide what to do, if anything.

Now, we assume we meet all the criteria above and the task isn't complete, perform task:

If no CC, return FAIL (If a convec has a CC, the manager should be prioritizing SetupInitialBase task)
If spaceport inactive or idle, return FAIL (Fixing this issue is delegated to another task, BaseManager for tubes, EmploymentManager for worker/scientist distribution)

If spaceport is under construction, return
If spaceport in convec and convec is moving, return
If spaceport in convec and convec is NOT moving, build spaceport, return
If spaceport in structure factory and convec is empty and on dock, transfer cargo, return
If spaceport in structure factory and convec is empty and not on dock, move to dock, return
if spaceport in structure factory and convec is NOT empty and is moving, return (Wait for operation to complete)
if spaceport in structure factory and convec is NOT empty and is on dock, transfer cargo for spaceport
if spaceport in structure factory and convec is NOT empty and is not moving and not on dock, move to dock
If metals < cost) return FAIL (Wait for metal, or expand mining operations, decision to be made by manager)
If spaceport not in structure factory, build spaceport kit, return


In real life, tasks are delegated to the appropriate teams by a manager. Let's look at decision making:

Manager: "Deploy the satellite!"
Engineer: "Well, I need a spaceport, and I can build one for you, but I don't have the materials."
Manager: "Damn it, Bobby. Hurry up, blue collar guys!"
...
Manager: "Ok, you guys are taking too long gathering those metals, forget the satellite for now, ExpandCommonMetalSupply when you save up enough.
...
Manager: "Our tokamak just exploded! Change task to MeetPowerDemand! Priority one! If it fails, just wait for resources!

etc.

Managers decide what to do. The task tree is how to accomplish what the manager wants. If what the manager wants can't be done / is outside your field, tell him so he can update priorities as necessary.
Title: Re: OP2 Scenario Project for C#
Post by: lordpalandus on March 27, 2019, 11:38:34 PM
AI is computationally expensive. If you can provide a fail condition early in the AI tree, it will save you computational resources for other things. You could provide an event after failing to check back on the task in one minute (or 1800 frames -> 30FPS * 60SEC), and see if have resources then. Or fail out of the construction task, and create a new task to gather the resources necessary and then retry the original task when you have required resources.

AI for strategy games can get quite expensive fast, so creating ways of optimizing it and exiting out of a lengthy AI tree, when failure is ensured, will really help with that. If it cannot perform a task, there is no point in checking the other dependencies of the task.

Or if you want the AI to perform several mutually exclusive tasks at once (ie mine resources and research), you could provide several failure conditions to check and if any failed, it aborts the task and looks to turn the failed conditions into successful ones and then retry the task the moment all conditions read as successful. Think of trying to solve all failure conditions as a quest in an rpg; it is successful when you've "gathered" all the necessary components.

Ex.

If you want to build say a Vehicle Factory, it would have these mutually exclusive conditionals:
-> Have available adults
-> Have power
-> Have convec with kit
-> Have technology

If any of these fail, it aborts/postpones the task to construct the factory. Say, it fails on Adults, Power, Convec with Kit. So it generates three new tasks: Grow Worker Population, Construct Power Plant, and Build Kit. Once population reaches necessary amount, Adult task is complete. And so one... until all conditions are satisfied.

Then it produces a task to place the kit in a specific area. While it is being built, it checks to make sure that there is a tube connection, and if not, build one. If the building is built, and a power plant was destroyed, it will idle the structure. Etc...

Ughh... this got complicated fast. I'm going back to developing my roguelike... so much simpler.
Title: Re: OP2 Scenario Project for C#
Post by: leeor_net on March 28, 2019, 06:44:27 PM
AI doesn't have to be expensive -- in terms of games it's not a true AI, it's a fake AI. Have it run tasks in a separate thread maybe once every three to five frames and it need not be expensive. Many commercially successful games do this including World of Warcraft, StarCraft and Age of Empires (you can actually see the lag in AoE2 -- select a unit and direct it to move somewhere, you'll see it hesitate for a few frames before it actually starts to move). Quake III Arena does the same thing with its bots.
Title: Re: OP2 Scenario Project for C#
Post by: lordpalandus on March 29, 2019, 12:45:41 AM
Most things that get called AI are not true AI. No videogame in existence has a true AI... that would be way too computationally expensive to have. So, we fake it with something that appearances to have intelligence with decision trees, probabilities, build orders, etc... to create an illusion of an intelligent AI. Some games does it better than others; ie First Encounter Assault Recon (FEAR) had exceptionally smart "AIs" and as far as adaptability goes for multiplayer bots, Unreal Tournament still is the best with them.

The closest thing the world has seen of a true AI would be that "social experiment" back in 2017? with TayAI, which I think was a learning program based on a neural network fed information from the interactions the AI had with people and adapted their personality and knowledge-base based on the information provided to TayAI.

The world isn't ready for a true AI. If we cannot get along with eachother, avoiding wars, hatred and racism, then we stand no chance of coexisting with an AI. More likely the AI would take over, with either a "Matrix scenario" or a "Skynet scenario".
Title: Re: OP2 Scenario Project for C#
Post by: leeor_net on March 29, 2019, 03:43:14 AM
Thread is starting to derail. Philosophical discussion about what a true AI is and whether or not humanity is ready for it can be started in a new topic.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on March 29, 2019, 07:46:20 PM
TechCor, there is a new PR in OP2MissionSDK that pulls in new library changes. Wanted to bring it to your attention before merging since I think you are actively using the library. Unless you are using deprecated portions, I do not anticipate any issues (famous last words). Also opened a new forum thread if you want to post any concerns, additions, etc.

Thanks,
Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 30, 2019, 01:31:41 AM
TechCor, there is a new PR in OP2MissionSDK that pulls in new library changes. Wanted to bring it to your attention before merging since I think you are actively using the library. Unless you are using deprecated portions, I do not anticipate any issues (famous last words). Also opened a new forum thread if you want to post any concerns, additions, etc.

Thanks,
Brett
Sounds good. I'm watching the other thread and will get the changes when they are ready.


HFL is now in the C# project (with the exception of panes).

Starting AI now, we'll see it how it goes.  ;)

Updated OP to reflect the current status of the project - what's missing, what's new.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 31, 2019, 04:58:46 AM
Does anyone know how to transfer cargo to the spaceport rocket?

I can't seem to figure this one out. I tried DoTransferCargo, but I'm pretty sure that's for the dock.

Scanned types in Unit/Building enumerators and couldn't find an SULV.

--
Can't seem to get the status of a structure either. Kind of necessary to redirect to another task if disabled.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on March 31, 2019, 07:28:42 AM
Haven't tested yet, but have you tried from Outpost2DLL:

Code: [Select]
void SetFactoryCargo(int bay, map_id unitType, map_id cargoOrWeaponType);	// [StructureFactory, Spaceport]  [Note: If items is an SULV, RLV, or EMP Missile, it is placed on the launch pad instead of in the bay]

Might be able to use unitType SULV and cargoType of spaceship part. I can update the comment based on your results.

---

Just a speculation, but someone may have to go into UnitInfo or UnitEx on HFL to get building status and figure out which unknown it is? You might want to check IUnit-HFL to see if it has a building status field (I have never used it). Since the AI cheats, I guess I've never worried about it.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on March 31, 2019, 08:57:34 AM
It doesn't work. It will place an empty SULV on the pad.

I'll try dumping out the OP2Unit struct to a log file, disable a structure, and dump it again to see what changed.

EDIT:

This wasn't as easy as I was hoping it would be:

Active:
unknown5 = -2144927203
flags = 8449037
unknown7 = -2144927203
timerStickyfoam = 0

Disabled:
unknown5 = 2556445
flags = 8440844
unknown7 = 2556445
timerStickyfoam = 7

---
The log was printing continuously (no way to detect keypress AFAIK).
timerStickyfoam changed a few times in there.

timerStickyfoam started at 0.
Then it went to 519.
Then it went to 263.
Then the other values changed (I assume from becoming disabled) and it went to 7.

I'm thinking it might represent the structure animation state, but I am just speculating.



Decided to assume it has something to do with the flags and ran multiple tests for each disable type.

00000000 10000000 11101100 00001101 - Enabled
00000000 10000000 11001100 00001100 - Power
00000000 10000000 00101100 00001100 - Tubes
00000000 10000000 00001100 00001100 - Idle
00000000 10000000 01101100 00001100 - Scientists
00000000 10000000 00101100 00001100 - Workers and scientists
00000000 10000100 00101100 00001100 - Infected

It looks to me:

byte 2:
bit 6 = Is infected

byte 3:
bit 1 = Has scientists?
bit 2 = Has workers
bit 3 = Has power

byte 4:
bit 8 = Lights (Already defined in file)

There doesn't appear to be a "Connected to CC" flag which is very confusing. I checked for other differences between "disabled CC" and "disabled workers" and found none. They are considered identical in OP2Unit.



Tested spaceport with empty SULV and one with Phoenix module loaded. The cargo is very clearly placed into "unknown14", as the value gets set from 0 to 99 (Phoenix module map_id).
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on March 31, 2019, 08:33:47 PM
Good question about loading rockets. I'm not actually sure. I'm not sure if an AI ever does that. I only remember the AI launching EMP Missiles, where there is a special function for in TethysGame. In terms of direct Unit control, the only exported methods for manipulating specific units seems to be:
Code: [Select]
	// Specific Building
map_id GetObjectOnPad() const; // [Spaceport]
void DoLaunch(int destPixelX, int destPixelY, int bForceEnable); // [Spaceport]
void PutInGarage(int bayIndex, int tileX, int tileY); // [Garage]
int HasOccupiedBay() const; // [Garage, StructureFactory, Spaceport]
void SetFactoryCargo(int bay, map_id unitType, map_id cargoOrWeaponType); // [StructureFactory, Spaceport]  [Note: If items is an SULV, RLV, or EMP Missile, it is placed on the launch pad instead of in the bay]
void DoDevelop(map_id itemToProduce); // [Factory]  [Note: Sets weapon/cargo to mapNone, can't build Lynx/Panther/Tiger/GuardPostKits]
void ClearSpecialTarget(); // [Lab]

A few are Spaceport related, but none seem to load cargo into a rocket. My guess is, the original game didn't ever use this functionality for the AI, and so there are no exported methods to do it.

It would still be possible to construct a CommandPacket which matches the Player's command to do it, and issue that. The OP2Internal (https://github.com/OutpostUniverse/OP2Internal) project has a fair bit of info on CommandPackets. Check src/Game/CommandPacket.h. Unfortunately Outpost2DLL has no direct way to issue a CommandPacket. That functionality was split off into Outpost2App.h. It's in class TApp:
Code: [Select]
int PlaybackCommand(struct CommandPacket *, int);

We should look at merging Outpost2App.h into the same project as Outpost2DLL. Maybe as separate subfolders of the same project or something.



In terms of Unit fields, we have some text files in the SVN repository (currently offline?) in the OllyDbg/InternalData/ folder that describes a lot of detail about the Unit structure. It's messy though. It's basically a union of a bunch of different structs. Each Unit type can potentially have it's own layout. It does actually follow an inheritance hierarchy, though we've never actually worked out the full hierarchy. Fields that are common to many unit types come first.

The flags field is reasonably well understood and documented in there.

We should port those text files over to GitHub.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 01, 2019, 06:49:16 AM
Hell Yeah! I reinvented the wheel! The OllyDbg confirms all those flags and the launchPadCargo. I assume HFL used an earlier version or something. The "connected to CC" flag most def isn't there, either.

CommandPacket.h doesn't appear to mention anything about transferring launchpad cargo. The only thing that might be it is TransferCargo which has an unknown variable. I guess it isn't necessary. I set unknown14 to the launch cargo and it loads the cargo. I mimic the command by doing a swap with the bay.


So that leaves two issues:

1. How to tell that a structure is disconnected.

2. HFL doesn't give me enough power. Do I desync my version by adding functionality?
Title: Re: OP2 Scenario Project for C#
Post by: Crow! on April 01, 2019, 09:43:30 AM
What is the "tubes" disable type, if not a reference to CC connection?
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on April 01, 2019, 11:19:25 AM
You can always submit a pull request for HFL updates.

Additionally, OP2Internal can be used with level building. Though it does currently complicate linking, and make the module not relocatable. The relocatable thing isn't normally a problem in practice though, particularly not for level DLLs. I'm planning to address those issues soon.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on April 01, 2019, 05:08:34 PM
TechCor,

If you have the skill to create a function that allows loading rocket cargo via memory hacks, I would encourage you to attempt to add it to HFL via a pull request. We can review the code and hopefully incorporate it into the SDK.

Better not to diverge unless we cannot come to agreement on implementation and this seems like the right thing to add to HFL.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: Arklon on April 01, 2019, 08:40:38 PM
You can always submit a pull request for HFL updates.

Additionally, OP2Internal can be used with level building. Though it does currently complicate linking, and make the module not relocatable. The relocatable thing isn't normally a problem in practice though, particularly not for level DLLs. I'm planning to address those issues soon.

Level DLLs are actually the most susceptible to the not being relocatable thing, since they are loaded last.

Launch pad rocket is an int32 at +0x6E from the base of the internal unit structure, launch cargo (starship module loaded in rocket) is an int32 at +0x72.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 03, 2019, 09:29:35 PM
Launch pad rocket is an int32 at +0x6E from the base of the internal unit structure, launch cargo (starship module loaded in rocket) is an int32 at +0x72.
How did you know it was an int32? I was having problems building a SULV after launching hacked cargo. Turns out changing the value from short to int fixed the problem.

The OllyDbg says it's 2 bytes.


Anyway, I've run into a couple issues.

1. For some reason "DoDock" sends the convec off the map. However, "DoMove" works fine. This is mostly an aesthetic problem that I'm not too concerned about right now, as the convec will still transfer its cargo and move on.

2. Robominer won't deploy. I'm calling DoBuild(map_id.CommonOreMine, beacon.GetTileX(), beacon.GetTileY()). It seems to spin in place on the beacon. Maybe it's the wrong command?
Title: Re: OP2 Scenario Project for C#
Post by: Sirbomber on April 04, 2019, 05:53:13 PM
If you're using .DoBuild you need to issue the command 1 tile down and to the right of the beacon, as if the Robo-Miner (or GeoCon) were a ConVec.  It's a bit of a cheat, since if you watch it happen the miner appears to teleport onto the beacon when construction starts, but it's the easiest way to do it.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on April 04, 2019, 08:18:39 PM
Yeah, the game doesn't provide a proper method to directly build a mine. You either have to hack it with the (+1, +1) tile offset, using a build group (AI only, I believe), or issue a command packet directly.



For the Spaceport, I see this in the InternalData notes from SVN:
Code: [Select]
Unit:Building:Spaceport
-----------------------
0x6E 4 launchPadUnitType  [SULV/RLV/EMPMissile]
0x72 2 launchCargo
0x74 2 boolRLVLanding
Title: Re: OP2 Scenario Project for C#
Post by: Sirbomber on April 04, 2019, 08:35:38 PM
I forked HFL and am working on a DoDeployMiner method, but I'm not sure how to do the bitwise stuff to get the X and Y coordinate into the low/high byte, respectively.  Thoughts?
Code: [Select]
Type: (0x06) "ctMoBuild" Build a building (ConVec, RoboMiner, GeoCon)
------------
Size in bytes:     0x13 (19 bytes) minimum to deploy RoboMiner/GeoCon/ConVec building
Size in bytes:     0xD + numUnits*2 + numWayPoints*4
        (13 + numUnits*2 + numWayPoints*4)
Offset     Size     Description
------     ----     -----------
0x0     1     numUnits
0x1     2*X     unitID - array of size numUnits
0x3     2     numWayPoints (BYTE)
0x5     4*Y     wayPointLocation - (pixelX:15, pixelY:14)
0x9     2     buildArea.TopLeft.x (bulldozed tile)
0xB     2     buildArea.TopLeft.y (bulldozed tile)
0xD     2     buildArea.BottomRight.x (bulldozed tile)
0xF     2     buildArea.BottomRight.y (bulldozed tile)
0x11     2     (Stored to Unit.+0x6A - value 0xFFFF)
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on April 05, 2019, 03:27:25 AM
Do you mean the waypoint location?
Code: [Select]
(pixelX & 0x07FFF) | (pixelY & 0x03FFF) << 15;

Ehh, actually, that is pretty awful and unnatural looking. Assuming I got that right.
Title: Re: OP2 Scenario Project for C#
Post by: Sirbomber on April 05, 2019, 09:06:55 AM
Thanks, that gave me the push I needed to figure it out.  I also fixed DoDock while I was at it.  Take a look here (https://github.com/Sirbomber/HFL).
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 05, 2019, 02:36:03 PM
Nice work. I'll take a look at it when I'm back at a testable point.


Currently implementing a trio of tasks:

CreateCommonMiningBase
CreateCommonMine
SaturateCommonMine

A mine isn't saturated if it has less than X smelters assigned to it. If there are unsaturated mines, build a smelter.
A command center has a "control area". A CC owns beacons inside the control area. If there are beacons without mines, build a mine.
If every beacon has a mine and every mine is saturated, build a new CC near a beacon to increase access.


In order to place buildings, I need to know the "connection grid" to determine where tubes need to be placed, etc. I assume how OP2 does this is unknown. If that's the case, I'm going to create my own grid that can be used as a lookup. A bit of an expensive operation - BF search from the CC checking for tubes and buildings on tiles.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on April 06, 2019, 06:55:00 AM
Ahh sweet. Would be good to get new functions into the development libraries.


Just had a thought about the waypoint calculation. If you used bitfields in a struct, the compiler should be able to pack the values for you. It could also properly handle signed/unsigned extensions. (I'm not sure what is needed here in regards to signed/unsigned).
Code: [Select]
struct Waypoint {
  int x:15;
  int y:14;
  int reserved:3;
};

Waypoint waypoint{pixelX, pixelY};
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 06, 2019, 11:34:52 PM
Things are going pretty well.

The SVN is down, and I can't check the OllyDbg notes. I guess I should have downloaded it. There are a couple new things that I need:

1. Power output and demand. I assume this is in OP2Player.

2. UnitInfo about capacity for residences, medical centers, recreation facilities and DIRT.

I need this to determine the number of each structure that needs to be built.


Hopefully that information is there.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on April 07, 2019, 07:17:08 AM
I zipped the OllyDbg folder from my local copy of the SVN repository. It may be missing a couple of changes if Hooman has updated since I pulled. He tends to update them slowly over time when researching.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 09, 2019, 07:05:28 PM
Thanks for the OllyDbg zip. Found what I needed.

---
I would like to check if a player has a particular unit's required technology researched to determine if a task can be completed. For example, you need to research "Space Program" before you can make a Spaceport.

In the UnitInfo class in HFL, there is a function called GetResearchTopic(). The problem is I don't know what this is returning or if it is valid. It returns 69 for the Spaceport, but "Space Program" has tech ID of 5405.

I've tried looking at OllyDbg, but it doesn't appear to match what HFL is doing at all. I'm thinking it's a discrepancy between unit types (vehicles and structures). HFL seems to assume all units are the same size struct.

Can someone give me some clarity on this?


HFL UnitInfo:

https://github.com/TechCor8/OP2DotNetMissionSDK/blob/master/NativeMissionSDK/NativeSDK/HFL/Source/UnitInfo.cpp

I'm looking at DataStructure UnitTypeInfo.txt in OllyDbg.
Title: Re: OP2 Scenario Project for C#
Post by: Arklon on April 09, 2019, 09:28:38 PM
In the UnitInfo class in HFL, there is a function called GetResearchTopic(). The problem is I don't know what this is returning or if it is valid. It returns 69 for the Spaceport, but "Space Program" has tech ID of 5405.
That's because the tech index in that context is by array index, which boils down to the order in which techs are defined in the tech .txt, rather than tech ID.

Quote
I've tried looking at OllyDbg, but it doesn't appear to match what HFL is doing at all. I'm thinking it's a discrepancy between unit types (vehicles and structures). HFL seems to assume all units are the same size struct.
The unit entries in the internal unit array are all the same size, based on sizeof mapGeneralUnit (max map_id) I believe.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on April 10, 2019, 10:42:53 AM
We should get those txt files moved to GitHub.

A long while back, I was paring down the txt files as that information got added into what is now the OP2Internal (https://github.com/OutpostUniverse/OP2Internal) project. Effectively functional documentation, rather than simple text. It may be a good alternate source of information for some things. Though the Unit and UnitInfo stuff was not really cleaned up enough to port over effectively.

In terms of storage, the game seems to use arrays of unions of all sub-types, so you effectively get an array of fixed sized data, even though the fields within each chunk have variable meaning based on unit type. Each Unit has 120 bytes of storage, even though not all unit types use the full 120 bytes. More like the largest Unit type used up to 120 bytes. UnitType info has something similar going on.

As Arklon stated, the research numbers are index numbers into the research array data. OP2Internal likely has some good detail there.

Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on April 12, 2019, 07:49:57 PM
Sirbomber,

Did you want to move your new HFL code into a branch and issue a PR with HFL for review and merging (functions DoDeployMiner and DoDock). I can manually add them, but it would take the credit from you.

Don't want to derail topic, the fix was just posted in here.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: Sirbomber on April 12, 2019, 09:30:41 PM
I'll PM you about this, to not derail the thread.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 12, 2019, 09:41:23 PM
Just want to let you guys know that there are a bunch of HFL additions on the C# project including:

Research/TechInfo files (Research)
Player structure capacities and status info (PlayerEx)
DoBuildWall (UnitEx)
LaunchPadCargo (UnitEx)
Mining beacon data (UnitEx)

I won't be getting around to creating a proper fork for quite some time, so it could be useful to grab those whenever. I suppose it's possible only I will find those things useful.

--
While I am here, anybody have a good way of detecting when a unit gets stuck?

Sometimes I'll have a situation where a unit will want to move to an unreachable location. Maybe they were ordered to go somewhere, and now a building is in the way. Instead of giving up, it spins in place. I need to detect that this is happening and order it to stop so it can be reassigned.

---
TethysGame::GetRand appears to be returning 0 after the first tick. Is this expected behavior? I may switch to C# rand to get around this. I'm trying to randomize guard post types.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on April 15, 2019, 02:06:20 AM
If you're not using the build in Rand(), there's a pretty good chance multiplayer games will desync.

I believe Rand should return from 0 (inclusive) up to the max value (exclusive). Are you sure it's not behaving correctly? You're not messing with the seed, are you? The build in random number generator should auto seed at game start, based on the system time. If you try to re-seed it, you basically reset the random number sequence. Continually reseeding means it will continually return the same sequence of numbers.

Not sure about stuck units.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 15, 2019, 02:48:18 AM
Hmm. I went back and checked, and Tethys::GetRand works fine. Must have been really bad luck the handful of times I used it. Like flipping heads 15 times in a row... I have no idea.
Title: Re: OP2 Scenario Project for C#
Post by: lordpalandus on April 15, 2019, 10:58:16 AM
I use random number generation a lot in the video game I'm making, so some possible reasons it is returning 0:

1. You improperly set up the array / vector, so it is using an empty array/vector. Random number of 0 between the size of the array/vector would be 0, so it will always return 0.

2. It is choosing a random number as a float, and converted into an int. If the random number was 0.9, in floats that is nearly 1, but when converted to int it becomes 0. Some round up, some round down.

3. Some random number generators, like the one I use, utilizes different streams. If your random number generator requires a specific stream, that could be the issue.

Otherwise:

We will need to see what code you are trying to execute to provide more detailed assistance. Just because there is no syntax error, doesn't mean there isn't some unintentional logic error. Without the code in question, it is really hard to say.
Title: Re: OP2 Scenario Project for C#
Post by: leeor_net on April 17, 2019, 09:38:41 PM
Gonna go out on a limb and assume the core code is just using a really shit PRNG. Haven't looked too far into it but I'd suggest using literally any other PRNG than whatever OP2's engine provides (unless for some reason mission DLL's must use that function as Hooman has suggested).
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 19, 2019, 01:11:28 AM
Again, I don't know what happened. It's possible I coincidentally fixed some code the same time I swapped it out for C# Random. Tethys rand appears to be working, so I will leave it as is. It's easy enough to change the wrapper function to C# random later if needed.

---

I have need of accessing a lab's data for labor reassignment. Unfortunately, it appears that the lab variable addresses overlap the "bayItem" array, which are represented by different sizes. It's not simply a variable naming issue.

I assume then, that I need to put a union in here to make this work. Rather than hack a union in at that spot, what address marks the end of the base class and the beginning of each sub type?

Or am I approaching this entirely wrong?

--It just occurred to me while posting in another thread that instead of a union, I could create a separate struct and offset the address similar to how BeaconData is done. I will give this a shot.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on April 19, 2019, 01:37:14 AM
Correct about the union. Each unit type may have it's own specific unit related fields. That will make the non-common fields appear to overlap in memory, reusing the same offsets.

We don't have clear picture of the entire unit hierarchy, or how big each unit record is. We know a handful of them, and have some general sense of parts of the unit hierarchy, just not the full info. This is one of our long standing ToDo items.

In SVN: OllyDbg/InternalData/DataStructure Unit.txt, you can find many pieces of known memory layouts. Where base classes and hierarchy relationship are known, they are marked. Where the constructor sets a different set of flags (offset 0x44), they are marked. The meaning of flags may be dependent on unit type, and isn't fully detailed or separated out by unit type, though many details are known. Similarly which offsets and fields belong to which unit types or base types isn't fully worked out. The hierarchy has multiple levels, and is a bit lopsided, so there are many sizes and offsets involved in the unit structure breakdown. Many details are known about what values can be found at what offsets. It's the breakdown of those fields and offsets into unit types and base types that is lacking.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 24, 2019, 01:29:28 AM
I ended up going with a LabData struct with a direct pointer to the first variable's address.


Base, Labor and Research management are all at a nice first draft. Currently going through now and fixing bugs and adding extras such as DIRT, geocon, and magma well handling. Then I'll move onto task priorities, customization and finally combat management.

Looking forward to doing real world tests in the next month or so.

---
I really want to get this unit stuck bug fixed, so I no longer need to "help" the AI occasionally.

I'm thinking the best way is to find out where the unit is trying to move. If I knew that, I could check if it is occupied. Is there a way to get the adjacent tile that the unit is trying to move into? This would be different from its final destination. Then I could check if there is something in the way, and how to handle it.

The other "stuck" situation seems to be due to a bug in "UnloadCargo". 1 in 50 times, UnloadCargo will fail to trigger, and can't even be triggered manually in the agridome or smelter pane despite *not* being grayed out. I can't seem to figure out the pattern, but I've yet to see it happen while only a single cargo truck is trying to dock at the same port, making me think it happens when another truck is trying to push the one on the dock off. The truck on the dock reports "moDone", but building.UnloadCargo() is ignored. This bug is only an issue for Land Rush type games where you need to unload food and metals. Ore doesn't require UnloadCargo(), and therefore the bug never happens there.
Title: Re: OP2 Scenario Project for C#
Post by: Crow! on April 24, 2019, 06:45:47 AM
I've had the inability to unload cargo in the normal game from time to time.  The best way to fix it I've found is to order the truck to move onto the dock again, even if it's already on it.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on April 24, 2019, 03:44:05 PM
Yes, it seems issuing any command to the truck will resolve it. The issue is detecting that the problem is happening in the first place.

I'm thinking maybe adding a timer linked to the unit index to determine if they have been on the dock idling too long. Maybe a tenth of a second with "moDone" is all that is needed to determine that it's "gone bad". Not as elegant of a solution as I was hoping for, but any fix is better than no fix.

This solution might work for the movement problem as well, if pixelX/pixelY is a reliable way of telling if a unit is moving.

EDIT:

Checking for "spinning wheels" and waiting 200 ticks before issuing a stop command seems to work. Wish this kind of handling was built into the engine, but whatever.
Title: Re: OP2 Scenario Project for C#
Post by: Arklon on July 11, 2019, 08:48:13 PM
Has this been progressing lately?

The Python project has been slowly coming together, though as of late a combination of real life and making the mistake of getting Factorio has been getting in the way. Still, it is very close to being ready for 1.0 release - if we get back to crunching it like we were before, it'd be maybe a month or a couple months of work left. And seeing Factorio has given us ideas for how to improve OP2's mod system, but that will most likely be post-1.0.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on July 15, 2019, 10:22:29 PM
Hey,

The project stalled out as I needed to focus on getting work and making the monies. Things are starting to look good now. Perhaps too much work. ;) Let's not mention Anno 1800 is on my play list. I should be back on this project next month.

AI needs its CommandGrid optimized, some other Building AI related odds and ends, and the offensive strategies implemented.

These are nice things to have, but the SDK is plenty capable of full mission development. Check out the original post for things that are still missing that are in the C++ SDK - mostly minor stuff.
--

For the mod to support C#, these are the functions that need to be called. Pretty much what you would expect.

https://github.com/TechCor8/OP2DotNetMissionSDK/blob/master/NativeMissionSDK/DotNetInterop/CppInterface.h

Then you can tell me what extra functions you need to support registering the scenario parameters or other extended features.

--
As an aside, if anyone has a wishlist, feel free to add it here. I will look into it. If someone asks about something, it will get higher priority.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on July 31, 2019, 09:58:25 AM
Is it possible to modify the tile map at run-time? Or perhaps inject a map file at DLL init?

It would be nice to bypass the need for an external map file.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on August 01, 2019, 12:12:36 AM
TechCor,

Yes it is possible. It is typically done during mission startup (initproc). I've never attempted it outside of initproc though.

Try these functions.

Code: [Select]
GameMap::SetCellType(loc, CellTypes::cellRubble);
GameMap::SetTile(loc, changeSet.ReplacementTileID(TethysGame::GetRand(changeSet.ReplacementTileRange())));

Example code that creates destroyed buildings on map startup by changing cell types and tiles:
https://github.com/Brett208/OP2MissionEvacuationUnderFire/blob/master/RubbleMaker.cpp

I suspect Outpost 2 only supports loading maps through standalone files or from volume (vol) files. We typically pack all the maps for a release into maps.vol so they do not clutter the directory.

I guess it could be possible to hook into the Outpost 2 code and change the default behavior to allow loading maps through other streams or something.

Hope this helps,
Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 01, 2019, 11:57:59 PM
Thanks.

I know about those functions, but I didn't realize you could set the tile to any graphic simply by changing the index. The Map internal data seems to indicate some kind of Tile Source. This will be useful later.


Update on AI Combat.

Added the CombatManager which requests and populates CombatGroups with military units. The BaseManager receives the requested list of units required to fill the groups and builds them.

Also added a PlayerStrengthMap which calculates the total damage each player can generate on a tile given their current units' positions and weapon ranges. This map will be used to help the CombatGroups navigate in and around enemy units without taking fire, as well as determining the combat strength required to overtake an area. This map accounts for both enemies, self and allies, so that it can make a well-informed decision.

I will be adding a PlayerUnitMap tomorrow, which will be used in conjunction with the strength map to detect the closest vulnerable buildings and units to CombatGroups.

Both of these maps get run through the A* pathfinder as "tile cost".


The final architecture of the combat AI is starting to take shape. Management will be split into two major components.

First, the CombatManager will generate a list of objects called "ThreatZones". These zones represent different areas on the map that require CombatGroups to move into them. Zones are generated for things like enemy units attacking allied structures, vulnerable structures and units, and enemy convoys/combat groups. The zone tracks important information such as the combat strength required to eliminate the threat and the relative importance of the zone.

The CombatManager will then assign CombatGroups to ThreatZones based on the group's type and distance to the zone, the importance, and the strength required (only send as many groups as needed). Groups will use the maps/pathfinder to find the safest path to the ThreatZone.

For the second part, when a CombatGroup enters a ThreatZone, the group's combat tactics kick in. Basic tactics that apply to all groups are things like avoiding combat when the group's strength is lower than what is detected on the PlayerStrengthMap. CombatGroups with specific tactics will engage in their specific targeting and movement procedures when in close proximity to enemy units, or are otherwise in the ThreatZone.

CombatGroups will stay in or near a ThreatZone until the importance drops, or the threat (and ThreatZone) is completely eliminated.

Hoping to have a first draft ready for GitHub in a week or two.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on August 02, 2019, 05:15:50 PM
The AI Combat system sounds robust and flexible from what you describe.

You should be able to set to most any graphic. A few may cause problems like if you wanted to set a middle animation in a lava flow as I haven't tested things like that. As long as you are setting reasonable tiles though it should be okay.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 04, 2019, 01:52:50 AM
EDIT:
Cleaned up to summarize problem.

Exception thrown at 0x00472D78 in Outpost2.exe: 0xC0000005: Access violation reading location 0x080D4F63.

On subsequent runs, thrown address stays the same, read location changes, so it is the same fail point every time in HasTech.

Test code used to simplify the error:
Code: [Select]
public void Main.Update()
{
/* First thing called on the first update frame after mission init.
   Will crash inside Outpost2.exe::HasTechnology()*/
TethysGame.GetPlayer(TethysGame.LocalPlayer()).HasTechnology(1);
}
Any help appreciated.


***
Elsewhere in the remote world of Outpost 2 coding...
***
UnitInfo.GetOwnerFlags returned "Both Colonies" for map_id.laser. Is there a correct way of determining if a player can build a unit? Right now I have Eden producing RPGs. I supposed I could hard-code a list...
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on August 04, 2019, 05:42:09 PM
TechCor,

I'm not sure what to say about the HasTechnology bug.

If you are having trouble determining which colony a weapon belongs to, check out:

https://github.com/OutpostUniverse/OP2Helper/blob/master/ColonyType.cpp

I'd be willing to expand this file's functionality if you needed it presented in a different manner. Again, not sure about the underlying bug, but in this case might be easier just to use OP2Helper.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 04, 2019, 07:03:40 PM
I feel dumb now.

I just realized GetResearchTopic() returns the TechIndex, not the TechID (which is what HasTechnology uses). Arklon told me this before, and I did it right in code older than last week.

Need to add some comments about this...

Ran it 50 times and it didn't crash, so I'm going to call this fixed.

--
Since GetOwnerFlags doesn't work right, and OP2Helper is the recommended solution, I will replace GetOwnerFlags with a hard-coded list of colony types.

I can't really use OP2Helper directly because it is in C++, and interop would add unnecessary overhead. It's much easier to reimplement OP2Helper functionality in C#. That's the best option for anything open source that doesn't interface directly with Outpost2.exe.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 06, 2019, 02:38:39 AM
First draft of Combat System is now on GitHub. AI has the combat strategy of a cockroach, but it's a start.

The mission is set to start with 2 bots in Land Rush, 1v1. It was fascinating to watch, especially since the combat does not work as expected. I watched until there were too many units and the pathfinder slowed the game to a crawl. Yeah, gotta find some way to optimize without causing desync for multiplayer.

With base management, I can actually see what the AI is doing, but the combat system is so abstract it's an enigma. Is there a way to render rectangles and lines on the map so I can see zones and paths?

EDIT:
Made a workaround by creating/destroying markers at zones and path destinations.

Turns out the vehicle pathing had some bugs.

80% of the performance problems are coming from the pathfinder, and the algorithm is as fast as I can make it. Going to move pathfinding to a thread pool and use an "execution tick" set to a few ticks into the future. If the pathfinder finishes early, it waits for the "execution tick", if it takes too long, it blocks on the "execution tick".

For the moment though, going to try to improve attack strategy.


UPDATE:

Added an AsyncPump that delays execution for any tasks by some amount of TethysGame.Time(). This has been integrated into the Pathfinder. Already seeing substantial improvements in performance with one AI having over 60 combat units moving around without issue.

Not all SDK functionality has been made thread-safe, but I will add it to important classes. I haven't decided yet whether to lock the OP2 API (and the entire .exe), or instead flag those functions when calling from off the main thread. Locking the entire executable might be a significant bottleneck that async support for the OP2 API may not be worth it. In most cases, you can simply perform calculations in the worker thread, and call the API when it completes on the main thread.

Now that is out of the way, going to start improving combat strategies. I already see some problem points. In particular, there is an over-reliance on bombers. While one AI did achieve victory-by-starflare, it was quite a slow way of doing it.

EDIT:
It occurred to me that thread safety is not enough. A Time() desync will occur if, say, the PlayerUnitMap updates while in the middle of pathfinding. Instead of locking classes, I'm going to create a StateSnapshot class that captures the game state at a particular TethysGame.Time() as an immutable object. While more memory will be used, locks will no longer be necessary outside of the AsyncPump, and it is easier to use since all async code will rely on the StateSnapshot. Then later I will add asserts to everything else to prevent use outside of the main thread.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on August 16, 2019, 08:27:07 PM
This is getting pretty interesting.

Any chance of time lapse videos of AI versus AI?  :)
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 18, 2019, 02:45:21 PM
Major Update
I've finally completed the transition to StateSnapshot. This has resulted in a major overhaul of pretty much every part of the mission SDK. Some might call it DotNetMissionSDK version 2.0  ;).

State Snapshot
I needed an immutable, threadsafe state that could be used for AI async processing. Using the live Outpost2 state was not going to cut it.

Behold, StateSnapshot. This is the cornerstone of this update.

StateSnapshot is an immutable class object that is generated at the beginning of every frame. This object contains a wealth of read-only OP2 state information including easy to access player/gaia data, unit lists, unit info, and generated maps.

Units and UnitInfo have been broken into separate derived classes so that it belongs to its correct type. For example, you cannot access objectOnPad from a convec, you must use the SpaceportState object.

The snapshot never changes and exists until garbage collected, which is perfect for performing async operations over time without locks and while maintaining predictability.

Accessing data looks like this:
Code: [Select]
UniversityState university = stateSnapshot.players[0].units.universities[0];
if (stateSnapshot.players[0].commandMap.ConnectsTo(university.GetRect()))
    // university connects to a command center, do something

Game State
The static GameState class is the new access point for live data. The game state contains lookup tables for players and units. The idea is to use StateSnapshot to read data and perform calculations, and then access the live data with GameState when ready to issue commands and modify data.

Accessing and modifying data looks like this:
Code: [Select]
UniversityState university = stateSnapshot.players[0].units.universities[0];
if (stateSnapshot.players[0].commandMap.ConnectsTo(university.GetRect()))
    GameState.GetUnit(university.unitID).DoTrainScientists(Math.Min(stateSnapshot.players[0].workers, 10));

Async Pump
The static AsyncPump class allows for predictable multithreading.

When you want to run an async task, you call AsyncPump.Run from the main thread and pass the amount of TethysGame.Time() you want to wait to complete execution. The first callback parameter runs asynchronously, the second one is called from the main thread at the time requested. If your task finishes early, it waits for the target time. If the time is reached before the task finishes, the main thread will wait for it to complete.

Combined with StateSnapshot, you can achieve a predictable game state every time.

Accessing and modifying data asynchronously looks like this:
Code: [Select]
ThreadAssert.MainThreadRequired();

if (m_IsProcessing)
    return;

m_IsProcessing = true;

AsyncPump.Run(() =>
{
    List<Action> actions = new List<Action>();

    // Do async tasks
    UniversityState university = stateSnapshot.players[0].units.universities[0];
    if (stateSnapshot.players[0].commandMap.ConnectsTo(university.GetRect()))
    {
        // Add command to list for processing on main thread.
        // The ? in the command checks if unit is null before executing DoTrainScientists.
        // GameState.GetUnit will return null if the unit has been destroyed.
        // This may happen due to the Time() difference between the snapshot time and when the command is finally executed.
        actions.Add(() => GameState.GetUnit(university.unitID)?.DoTrainScientists(Math.Min(stateSnapshot.players[0].workers, 10)));
    }

    return actions;
},
(object returnState) =>
{
    m_IsProcessing = false;

    // Execute all completed actions
    List<Action> actions = (List<Action>)returnState;

    foreach (Action action in actions)
        action();
});

Thread Assert
ThreadAssert.MainThreadRequired() is used on various methods to prevent access from async tasks. This helps prevent desync and access violation related issues from multithreading. In other words, these methods are not "thread-safe".

AI Performance
With the new systems above, AI now has a lot of wiggle room when it comes to performance.

First, AI no longer has to perform a complete update on every frame, instead performing its updates over several frames/TethysGame.Time().

Second, AI is now almost entirely performed on separate threads. Each AI manager performs an async update, putting all of its unit commands into a list, which is then executed on the main thread's completion callback.


Upcoming


Will post a video of the AI soon for the lazy.  :D
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on August 19, 2019, 01:47:38 AM
I'm curious, isn't there a noticeable performance impact when creating snapshots every frame?
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 19, 2019, 02:37:12 AM
I mean, yes, it does come at a small performance cost, but it is negligible compared to the gains.

A lot of calculations and loops are eliminated by being able to iterate over exactly what you want as opposed to iterating the entire player unit list to find cargo trucks, then again later for convecs, etc. Also, there is room for compiler optimizations now that it is not performing interop for every bit of data access.

Performance Benchmarks
Snapshot time recorded with StopWatch. There are 6 bots building with full tech, all morale buildings satisfied, and combat units. DLLs are set to release mode. Hardware is an AMD Ryzen 5 1600X Six-Core processor.

128x128 map (on6_01.map)
1 ms - Game Start (Land Rush)
2 ms - Mid Game

512x256 (newworld.map)
8 ms - Game Start (Land Rush)
9 ms - Mid Game

Total CPU Breakdown - 512x256 (newworld.map)
27.47% - StateSnapshot
: 18.9% - TileMap > 13.53% - GetCellType > 11.45% P/Invoke
: 7.46% - PlayerState > 5.19% - CommandMap

Mid Game screenshot for reference
(https://scoriastudios.com/myfiles/op2/AIPerfTest.png)


Optimization Update
TileMap is now copied entirely on the C++ side in one method call. I couldn't figure out WTF HFL was doing, so it still makes a hundred thousand native function calls, but it is much faster now. Also removed an unneeded clear from PlayerCommandGrid.

512x256 (newworld.map)
1-3 ms - Game Start (Land Rush)
2-3 ms - Mid Game

Mid Game Total CPU Breakdown
12.39% - StateSnapshot
: 6.16% - TileMap > 6.16% - CopyTileMap
: 5.78% - PlayerState > 3.12% - CommandMap > 2.89% - Pathfinder::GetClosestValidTile


Garbage Collector spikes have been eliminated by pooling the top-level StateSnapshot and retaining the maps, opting to clear and reuse instead of reallocating. Inlined HFL method calls in CopyTileMap. PlayerCommandMap has now been merged for all players to a single map, and building processing for it has been delegated to PlayerUnitMap.

512x256 (newworld.map)
Game Start (Land Rush)
0-1 ms - Create
0-1 ms - Release

Mid Game
1-2 ms - Create
0-1 ms - Release

Mid Game Total CPU Breakdown
8.7% - StateSnapshot
: 7.53% - Create
: : 3.44% - PlayerState (All constructors)
: : 2.8% - PlayerCommandMap > 2.8% - Pathfinder::GetClosestValidTile
: : 0.82% - TileMap/CopyTileMap
: 1.17% - Release

Release mode appears to optimize the Create process, so constructors are probably over-represented in the CPU breakdown.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 20, 2019, 01:45:13 PM
Hey Hooman,

Is this enough optimization for ya? Took like 12 hours. I'll send the bill in the mail. ;)

Performance of StateSnapshot is 3-9 times faster, and GC spikes were eliminated.

I bet I could optimize CopyTileMap some more if I could figure out the format to switch from double nested loops to memcpy, but it is a micro-optimization at this point.

1/3 of the cost is from calculating the command map for all players mid-game, so I'd say it's pretty well optimized.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 21, 2019, 08:07:46 PM
Here's a quick video for 6 bot Land Rush FFA with Tech Level 13.

Outpost 2 AI Test or: How I learned to Stop Worrying and Love the Starflare


Still a lot of work to be done:
Research priorities, earthworker should complete tubes it starts, prioritize defense when enemies have combat units, combat priorities, ESG attacking buildings, etc, etc.

The infrastructure is there, however, and doesn't have any noticeable slowdowns.
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on August 22, 2019, 06:23:08 PM
Hey, looks pretty good. It looks like you are using the different waypoints to mark AI points of interest for debugging?

I am impressed by the progress!

What is the requirement for this to be embedded in a mission. Is it just the DLL? Does the DLL require a specific .net framework available to work?

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 22, 2019, 10:23:19 PM
For those just arriving, there is an AI demo video (https://forum.outpost2.net/index.php/topic,6245.msg88551.html#msg88551).


Yes, the debug points are from the local player's combat manager. The targets are the "Threat Zones" including defensive points. The DNA markers are the combat units' pathfinder destinations.

Project is using .NET Standard 2.0 built with VS 2017.

Here are the files required to play:

https://github.com/TechCor8/OP2DotNetMissionSDK/tree/master/Outpost2

cTest.dll - The native C++ plugin that contains the ExportLevelDetails.
cTest.opm - The JSON file with startup settings. You can configure player/unit settings and which players use the AI. No compile required. Reading of this file can be disabled in C#.
DotNetInterop.dll - The interface between the native plugin and C#, and C# and the Outpost 2 API.
DotNetMissionSDK.dll - The C# mission DLL. This is the generic version for use with JSON/editor missions. For building a custom C# mission, see the GitHub ReadMe (https://github.com/TechCor8/OP2DotNetMissionSDK).


You can easily integrate the AI into your existing C++ missions. Just add DotNetInterop as a reference and call the required functions:

https://github.com/TechCor8/OP2DotNetMissionSDK/blob/master/NativeMissionSDK/NativePlugin/LevelMain.cpp

That will hook C# into the C++ mission. From there, you can manually enable the bot in C# CustomLogic.cs if using a custom C# DLL, or use the JSON file to manage player setup (which will be "your_dll_name.opm").
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on August 23, 2019, 04:30:20 PM
Any idea how this would work on Linux using Wine? A decent subset of Outpost 2 players use Wine.

I was going to ask if you could statically compile against .net2.0. However since C# compiles into IL and not native, I suppose the real question is whether Wine would support running a compiled C# application?

Targeting .net 2.0 probably would improve compatibility, being around for as long as it has.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 23, 2019, 06:08:06 PM
I don't use Linux, so I don't know.

Hooman said he was able to get the C# project to successfully compile on Linux (https://forum.outpost2.net/index.php/topic,6245.msg88106.html#msg88106). However, the interop could be a problem point.

Someone would need to try it out and report back what any issues are.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on August 25, 2019, 12:04:30 AM
Hey, very cool. That was quite an impressive AI demo video. I assume it automatically adapts to the level? We've never had a general purpose AI before.

Would be really cool to deploy an AI for a level with just 1 line of setup code, or even as an in game option for multiplayer maps.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 26, 2019, 01:11:17 AM
Yes, it will work with any starting base or even no base at all (if you want to do combat only).


I've now added the weighted goal system. The AI previously had a hardcoded task order.

Each top-level task/goal calculates a 0-1 importance. These goals are then sorted with the most important goals executed first. Static weights are used to control how important a goal is overall. For example, maybe you don't want the AI to ever expand mining operations, so you set the weight to 0.

The goal system dramatically increases the bot's ability to adapt to the situation. Each goal can have its importance calculated from a variety of factors. For example, expanding mining is more important if reserves are running low. Same with food supply. Building combat units more important if the enemy has combat units.

Already, I am seeing a substantial change in AI behavior.

Instead of building agridomes until a surplus is reached, it instead decides to build a vehicle factory because it has enough metal in reserve. Perfecting morale is no longer mandatory, instead opting to build a handful to boost it a little. Combat units start appearing within a couple minutes, and the first player's CC went down in 5 minutes. I think I'll upload a new video of this to show the improvement.

A lot of additional goals will be added to tailor the AI's personalities. Convec counts, scouts, light towers, evac transports will all be weighted goals. Tweaking the values can give straight aggressive AIs, or dawdling ones that build all the optional stuff.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on August 26, 2019, 08:49:11 AM
Hmm, would be very neat if the AI could be dropped into multiplayer missions for the extra player slots. Might even be possible to make them playable single player against AI opponents.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on August 27, 2019, 03:16:47 PM
That would be cool, but outside my knowledge island.


AI Changes
The top-level "Fix Disconnected Structures" has been removed. That task used to connect the closest structures to an earthworker until all structures were connected. Instead there is now a ConnectStructureTask that is a (blocking) sibling to BuildStructureTask under the new MaintainStructureTask. MaintainStructureTask now controls the number of structures to maintain of a particular type. MaintainStructureTask also has a RepairStructureTask under it for making sure that structures are actually operable and not critically damaged.

The result of this is that structures will be connected and repaired based on importance, and some structures may even be abandoned if they are not important enough. Full repairs are still a top-level goal, but it will have lower importance compared to other goals.


Labor Management
The AI base management is adapting pretty well. Unfortunately, the labor manager isn't cooperating. The base manager builds a second smelter when metal runs low, but the labor manager decides to put workers towards morale instead.

I've been thinking of creating a labor task tree that would set the activation priority order that the labor manager uses. The issue that I haven't worked out is how to prevent labor from being improperly reassigned when importance drops. The smelters shouldn't shut down just because we have enough metal and need more combat units! (Or should they?)

I suppose I need to think about what an optimal labor strategy looks like.

EDIT:
It just occurred to me that, since the goals are not tied directly to the number of active structures, an ActivateStructure task just might work. The bot would start performing all sorts of micro. I can see it now! Morale goes to 90, residences shut down, factories turn on and start building. Morale drops to 70, factories turn off, residences turn back on. Ha!


Morale and Food
Also, I've realized that food production may impact morale. I've never thought about this before since I personally maintain a surplus. However, the bot has "figured out" that it can kill all opponents before needing a second agridome with the current starting food.

Where does food rank in morale importance? Is it better to build an agridome or a residence when it comes to morale?
Title: Re: OP2 Scenario Project for C#
Post by: Crow! on August 27, 2019, 04:44:53 PM
You can find morale.txt on this forum, and it shows the relative effect on morale of various things.

I'm not sure what constitutes "high" versus "med" levels of crowdedness, but food storage being in deficit appears to have the same effect as residences being highly crowded.

As far as I know, building excess residences than required to get below 100% does nothing for morale other than give you a backup should one get disabled for some reason.

I'm also a little unsure if everything actually works the way morale.txt would suggest; med centers, for example, seem to indicate their crowdedness doesn't affect morale, but it definitely does.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on September 08, 2019, 03:36:00 AM
Thanks, Crow! I added agridomes to the AI morale task.

Labor Manager now takes base manager's goal priorities into account. I've also made a number of fixes and tweaks to the AI to improve base management.

- AI will build structure factories at each command center (importance represented as a factory/cc ratio).
- AI will build kits at the closest factory to the deploy site.
- AI will enable/disable command centers depending on whether there are connected buildings that want to be enabled.


Some problems I've run into:

- Scorpions don't attack. They don't appear to have a weapon either (It is set to map_id.None).
- Can't get the structure factory kit being built. I did a memory dump and could not identify it. This would be nice for producing parallel structure kits without making the same type on accident.

I'm going to start working on research prioritization, so tech level will be set to 1 for testing. Eventually, I will get to much needed combat work.

Last video of 6p tech level 13:
Title: Re: OP2 Scenario Project for C#
Post by: leeor_net on September 08, 2019, 02:28:21 PM
This is very exciting! It's good to see that this is still being developed. Could make for some very interesting missions and make computer players in multiplayer a thing. :D
Title: Re: OP2 Scenario Project for C#
Post by: lordpalandus on September 08, 2019, 11:23:29 PM
Looks pretty awesome. Great job TechCor!
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on September 17, 2019, 05:52:52 AM
Thanks for the comments, guys!

Research is coming along. Almost time for a new video.

Current Changes:

I was originally thinking of randomly picking upgrades to research (Automated Diagnostics, etc), but I've come up with a good priority-based scheme. The goals that handle things like "MaintainArmy" and "MaintainMorale" will have a topic list. If the goal is high enough priority, that research takes precedence. It already does this for structure/vehicle required topics, so might as well keep it going!

I've noticed a bug where GetEdenCost() returns 0 for the Observatory's research topic. I have no idea why this happens, but it prevents Eden from doing required research. The AI is now building Meteor Defenses without an Observatory due to this.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on September 18, 2019, 03:46:53 AM
This has really taken shape quickly. I'm quite impressed. That AI actually looks quite good, and sounds like it's quite versatile.

The AI really seems to have a strange fascination with patrolling starflares though. Don't know if I'd want starflares patrolling my base. :P
Title: Re: OP2 Scenario Project for C#
Post by: lordpalandus on September 19, 2019, 12:56:31 AM
... Do starflares explode violently if shot at by the enemy? I can't remember if they do or not.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on September 19, 2019, 05:15:21 PM
I don't remember either. But assuming the enemy floods your base with an army, what are you going to do? Attack them?  ???
Title: Re: OP2 Scenario Project for C#
Post by: leeor_net on September 19, 2019, 05:16:10 PM
No, they can be neutralized before they detonate. Kind of a balancing trade-off me thinks. Gives the defending player an opportunity to minimize damage.

As for the flood, the strategy was EMP's and hoping to hell you got them all before they could move again.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on September 19, 2019, 05:19:36 PM
No no, we're talking about defending with starflares, not attacking with them. ;)
Title: Re: OP2 Scenario Project for C#
Post by: Vagabond on September 19, 2019, 09:37:44 PM
Well look at the bright side, it isn't defending bases with supernova tanks...

I'd really like to see this applied to multiplayer missions somehow so people could play multiplayer missions locally. Maybe it would make a nice teaming effort someday in the future to make a reality.

-Brett
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on September 20, 2019, 06:58:03 AM
Collateral damage? You'd have to be very careful with those supernovas. Or just pretend like you're Arnold Schwarzenegger, then all explosions are cool. :P

I agree about applying this to multiplayer missions. We'll have to hurry up with the op2ext updates and write a patch for it. I would love to have an AI module. :D
Title: Re: OP2 Scenario Project for C#
Post by: leeor_net on September 20, 2019, 10:40:10 AM
No no, we're talking about defending with starflares, not attacking with them. ;)

Yeesh, leave it to me to misunderstand.  :-[

But yeah, in that case, that would be somewhat dangerous.  :o
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on September 21, 2019, 02:37:46 AM
A bit off topic here, but someone should totally make a map based on that idea. Disable all weapons except for starflares, so the player is forced to defend with them. Then an AI can send wave after wave of their own tanks at the player's base.  ;)

Speaking of AI, could the AI code here handle attack waves coming in from off the side of the map?  :)
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on September 21, 2019, 04:42:35 AM
Wow, lots of casual talk in this thread! Here I thought the community was dead. :D

---
Figured out why Observatories weren't getting researched properly. Turns out Cost 0 means "Free" and -1 means "Can't Research". Probably a source of a lot of research problems I didn't notice.

Added in all the extra research options. The only thing that won't get researched is Consumerism because I don't have a task for the Consumer Factory yet. The majority of the optional topics are in MaintainPopulationGoal and MaintainArmyGoal, with a handful in ExpandRareMiningGoal, and the rest that didn't have a good place were put into MaintainResearchGoal. Topics required for units/structures, are already handled on an "as needed" basis.

There is a bug where goal priorities rapidly switch due to labor moving around. Going to make a video this weekend anyway, as I won't have time to work on this again until the holidays.


A bit off topic here, but someone should totally make a map based on that idea. Disable all weapons except for starflares, so the player is forced to defend with them. Then an AI can send wave after wave of their own tanks at the player's base.  ;)

Speaking of AI, could the AI code here handle attack waves coming in from off the side of the map?  :)
The AI will take control of anything it is given. Combat Management is separate from Base Management, so you don't need any structures for it to work. However, if they happen to have a base they might prioritize defending (which could be a good thing!).
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on September 21, 2019, 09:48:31 AM
Here's the research video; No more free tech.



If you are wondering why the AI is so slow to get weapons, it is because the MaintainArmy goal is so far down the list. This is because the AI is reactive not proactive. MaintainArmy's importance is based on enemy strength. If no one else builds weapons, it won't either, until there is nothing else to do. This will be changed at some point to increase importance based on perceived threat. For example, if the enemy has a vehicle factory, an advanced lab, or worse, BOTH, importance would increase.
Title: Re: OP2 Scenario Project for C#
Post by: leeor_net on September 21, 2019, 09:52:21 AM
Sounds like a fairly passive AI. Are there plans on having different 'personalities'? Age of Empires had several AI personalities that would range from extremely passive to extremely aggressive, it made for more interesting single-player and even multi-player campaigns.
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on September 21, 2019, 10:55:58 AM
Sounds like a fairly passive AI. Are there plans on having different 'personalities'? Age of Empires had several AI personalities that would range from extremely passive to extremely aggressive, it made for more interesting single-player and even multi-player campaigns.
Please understand this is a lot of work and not even close to done.

There are three major elements to the AI.

The first is the task tree that lets the AI do everything possible in the game. This is simply stuff like "I want a laser lynx, but first I need X tech and a vehicle factory, but before that I need Y tech and a convec" etc. This is about 90% complete. I need to add solar sats, consumer factories, EMP missiles, and some useless stuff like light towers. Then a lot of time will be spent fixing all the little issues that crop up during the execution of these tasks.

The second is the goal system, which processes the priorities for the task tree. "This is the most important thing to take care of right now, I should make a laser lynx to solve that problem...[enter task tree]". These goals are sorted by the 0.0 to 1.0 importance value, which is combined with a weight passed at startup. The goals include all sorts of things, but there are a few critical ones that are typically high in the priority list (ExpandCommon/Rare, MaintainPop, MaintainArmy). I still need to add a useless goal set for scouts, evac transports, and dozers, and then tweak the importance formulas.

Finally, the combat system is independently ran. This system assesses enemy state and creates zones that act as a combat plan. These are sorted based on importance, and groups are populated in this sorted order. Any zone groups that don't have units assigned are put into a list for Base Management to grab and process in MaintainArmy. While the core infrastructure is there, it has not been revised since the first draft, at all. I'd say it's probably 20% complete, if that. Still need to stop ESGs from shooting buildings, Scorpions from not firing, and generally fix all sorts of poor decisions it is currently making.

This is a pretty good summary of how the AI works and where I'm at with it. The key thing to point out here is how I'm working on it. Notice I start at the bottom (task tree) and work up to self-actualization (Destroy thy enemies).

So what does this have to do with your question?

Well, the goal system weights mentioned above affect goal priorities. For example, you can increase the MaintainArmy weight and decrease other weights to increase the focus on building troops. Or you could decrease ExpandCommon/Rare if the AI shouldn't build more bases/mines/smelters. You can also decrease MaintainPop and let the people be miserable to death, or decrease MaintainFood to allow them to starve. Combat goals will also have a similar thing, but it is barely on the radar at the moment. At some point there will be preset weights to choose from so that mission developers don't need to fiddle with the weights themselves, BUT the AI is not ready for that. EVERYTHING else must be complete first. Making presets now would be a waste of time since they will most certainly break as the core is developed further.

Customization is the very, very last thing to do. With all systems complete, it's all about doing things like disabling morale structures and useless units, setting weights, and some optional features (I want to add a base preplanner).


tl;dr, I haven't gotten to it yet.

Title: Re: OP2 Scenario Project for C#
Post by: lordpalandus on September 22, 2019, 02:28:37 PM
A few questions:

1. How does it decide to focus on building a specific combat unit. Ie should I build a laser lynx, a railgun lynx, or a starflare lynx? Each are similarly priced but each serves a different combat role. Or maybe decide to save up for a Laser Tiger instead?

2. What happens when no one focuses on military? Does no AI become proactive and start building a military or do they all stay in colony mode?

3. Do the AIs suffer from morale and thus from blowing up an enemy's Nursery and the massive morale penalty that follows? Or Morale in general, and thus the need to keep a food surplus, sufficient residences, or an active nursery/university?

4. Do the AIs spend any time on preparing for disasters? ie if an AI had resources for lava walls, would they build them to protect their base from getting melted?

5. Can the AI's people die? Either from starvation, or natural causes or unnatural causes such as a starflare blowing up their workplace?
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on September 22, 2019, 03:50:14 PM
1. How does it decide to focus on building a specific combat unit. Ie should I build a laser lynx, a railgun lynx, or a starflare lynx? Each are similarly priced but each serves a different combat role. Or maybe decide to save up for a Laser Tiger instead?

The combat manager creates a bunch of Threat Zones that contain priority targets. Each zone has an empty combat group assigned to it. The combat group's type (bomb squad, assault group, etc) is based on the type of zone (defend base, attack building, etc). The combat group's "desired strength" is set to something above the total enemy strength in that zone. The combat group then generates a list of unit slots base on the group type and desired strength. Each slot contains a list of "supported unit types" (panther laser, lynx supernova, etc).

The combat manager goes through the list of zones, assigns units that are already in a zone to that group's slots, if it fits. Then it assigns the remaining units to zone groups based on zone priority and whether the unit fits in the slot. Finally, the remaining empty slots are placed into a list (based on zone priority). This is grabbed by base manager when it starts its update and is processed by MaintainArmyGoal.

Right now, it picks the most important empty slot and randomly picks a supported unit type to build. If it can't build it, it will randomly pick a different type. This method has some issues such as not saving metal for more powerful units, and will need to be adjusted.

NOTE: All things combat related are not final!

2. What happens when no one focuses on military? Does no AI become proactive and start building a military or do they all stay in colony mode?

I want to tweak this formula, but the way it currently works, is that it checks all enemy players for combat units. It assigns an importance based on perceived threat. So for example, if all enemies have no combat units, it assigns a importance of about 0.5. If its combat strength is above the strength of each individual enemy, the importance drops lower than 0.5. If its combat strength is less than each enemy, its importance rises. I would like to add Advanced Lab and Vehicle Factory as bonus importance so that it doesn't wait until it's practically being whacked before starting.

When does it pursue building combat units? It depends on where the goal is in the priority list, and if it is being blocked by a higher priority goal. In a previous video, it built weapons right away because it did not need to research anything, and had plenty of metal. In the current one, MaintainPopulation blocked up the labs, preventing weapons research from getting done. It didn't start building combat units until MaintainPop finished all of its optional research. MaintainPopulationGoal is too aggressively important right now and needs to be tweaked.


3. Do the AIs suffer from morale and thus from blowing up an enemy's Nursery and the massive morale penalty that follows? Or Morale in general, and thus the need to keep a food surplus, sufficient residences, or an active nursery/university?
5. Can the AI's people die? Either from starvation, or natural causes or unnatural causes such as a starflare blowing up their workplace?

The AI runs as a human player, including all the pros and cons that provides. Disasters lower morale, people are needed to activate structures, food to prevent starvation, morale/nurseries/universities to grow population, training scientists to fill labs, etc.

4. Do the AIs spend any time on preparing for disasters? ie if an AI had resources for lava walls, would they build them to protect their base from getting melted?

AIs have no knowledge of disasters at the moment and would build in a lava flow if the mission developer put them in a poor location. They will attempt to recover from any disasters that don't kill them.
Title: Re: OP2 Scenario Project for C#
Post by: lordpalandus on September 23, 2019, 01:48:25 AM
Ok cool. Thanks for the indepth replies. Here are some other questions:

1. Does the AI use group tactics, or just sends individual units to a spot? From watching the video, it seems each individual unit is sent to a spot.

2. Where on the list of priorities, does repairing damage fall in? ie a meteor hits a residence; does the AI prioritize resources to building lynxes to fight the enemy pounding on their turrets or do they spend their resources repairing the residence?

3. Similarly, will units use a Garage when damaged?

4. Or a better question, will units retreat after taking X damage, so that they might survive to return to repair?

5. How does the Ai handle earning enemy units via spider hacking? As the unit wasn't built, and its allegiance used to be another player...
Title: Re: OP2 Scenario Project for C#
Post by: TechCor on September 23, 2019, 02:28:44 AM
1. Does the AI use group tactics, or just sends individual units to a spot? From watching the video, it seems each individual unit is sent to a spot.
Combat is not finished yet, but it will have group and individual tactics. Currently, groups stage outside a Threat Zone before entering, but it does not work properly at all. Group tactics are essentially "send this group of units to this zone through this path", individual tactics will be "ESG does not shoot structures", "starflares attack between groups of units", and "stay just out of range of enemy fire, if possible".

Quote
2. Where on the list of priorities, does repairing damage fall in? ie a meteor hits a residence; does the AI prioritize resources to building lynxes to fight the enemy pounding on their turrets or do they spend their resources repairing the residence?
AI prioritizes repairs to structures that have been critically damaged / disabled if it is required to complete another task. For example, if the task is to build a laser lynx, but the vehicle factory is crippled, it will prioritize repairing the factory. Same deal with tubing. If the residence was destroyed, cutting off tube access to the factory, it will prioritize reconnecting the factory. What gets repaired/reconnected/rebuilt first, if more than one building is crippled/disconnected/destroyed, is based on relative goal priority/importance.

There is a general purpose repair goal that does non-critical repairs. If repair units are not busy doing something else, and there is enough spare metal that other, higher priority tasks are not blocked, they will repair structures.

Quote
3. Similarly, will units use a Garage when damaged?
Garages have not been added to the task tree yet. They are not like other structures and need very special handling. It will likely end up as a separate goal.

Quote
4. Or a better question, will units retreat after taking X damage, so that they might survive to return to repair?
Combat is very rudimentary right now. I'd like to get them attacking properly before worrying about retreat/repair behavior. I think I have assault groups set up to take a repair unit for every 10th unit so they can repair during the assault, but individual tactics haven't been worked on yet.

Quote
5. How does the Ai handle earning enemy units via spider hacking? As the unit wasn't built, and its allegiance used to be another player...
No individual tactics yet. Spiders can't reprogram. Captured units, or combat units created by the mission developer, are processed by the combat manager like any other combat unit. Supported unit types for slots include both colonies. Cargo trucks, convecs, etc, are used by the base manager if it can find a use for them.

Generally speaking, any unit or structure that an AI gains possession of will be taken into account. A mission could spawn units and buildings out of thin air, and it will start using them.
Title: Re: OP2 Scenario Project for C#
Post by: Crow! on September 23, 2019, 02:29:28 PM
A bit off topic here, but someone should totally make a map based on that idea. Disable all weapons except for starflares, so the player is forced to defend with them. Then an AI can send wave after wave of their own tanks at the player's base.  ;)
Commander:
Our situation is dire.  We have spotted a fleet of what can only be described as bombs on wheels headed from Eden's base to ours.  Though we assigned our top scientists to defense projects months ago, they have squandered our resources on flights of fancy like "invisible lasers", "self-cleaning glue", and perhaps dumbest of all, a system that appears to simply be a joy buzzer dispenser.

Useless researchers notwithstanding, our vehicle factory workers can create a rough facsimile of the explosive weapons headed our way.  They may be the most awkward type of weapon imaginable for defense, but at this point we have no choice.

Remember, extinction is not an option.
Title: Re: OP2 Scenario Project for C#
Post by: Hooman on September 24, 2019, 06:01:53 AM
Lol


Well, I like that the AI can take control of any units that happen to get added to the game. It definitely sounds like this is going to have some pretty robust results.