OP2 Mission Hub

[TechCor]

An idea I've been thinking about for a while:

A new application called the "OP2 Mission Hub".

This would be a way to share missions by uploading your ".map" and ".opm" files along with a name and description. It would probably require a login as well as administrative capabilities for curating spam/inappropriate content. The app would be restricted to missions created with the OP2 Mission Editor as DLLs are a security risk.

On the player side of things, the app would list all missions on the server, with some filters for selecting content. This wouldn't require login. The app would generate the minimap images asynchronously, possibly during scroll. Essentially, it would download the mission files into RAM for generating the minimap.

When you choose to download the mission, the appropriate DLL is generated and the files are copied to your game directory. The app will auto-detect the mission in the game directory, and show a "delete" button for removing it.

For backwards compatibility, I will likely have the missions report which SDK version they use. The SDK DLLs will have the version number in the name so that you can have more than one in the game directory at a time. The DLLs could either be stored on the server (easier to update without changing the application), or stored in the hub app until a copy is needed.

There will be a config file for selecting the mission servers. Maybe outpost2.net will host it.
The server side will be written with PHP/mySQL.

Source:
https://github.com/TechCor8/OP2MissionHub

Releases:
https://github.com/TechCor8/OP2MissionHub/releases

UPDATE [2020-04-05]:

v0.1:
Initial test release available.

RoadMap:

• Minimap rendering
• Scalable scroll list

[Sirbomber]

The app would be restricted to missions created with the OP2 Mission Editor as DLLs are a security risk.

A custom mission database that disallows 100% of all custom missions created up until now?  Why bother?  No disrespect but while your new editor has an impressive UI and is far more stable than the old OP2Mapper ever was and will be great for setting up quick little skirmish maps, I doubt it will ever be able to replace the way we do things now.  How would you implement something like this through your script editor?

Code: [Select]

// Propagators effect
    Unit P;
    PlayerVehicleEnum Propagate(6);
    while (Propagate.GetNext(P))
    {
        if (P.GetType() == mapScorpion)
        {
            // Get all player-owned units adjacent to the current Scorpion
            Unit newP;
            LOCATION ploc = P.Location();
            LOCATION Adjacent[8] = {LOCATION(ploc.x-1,ploc.y-1),
                                    LOCATION(ploc.x  ,ploc.y-1),
                                    LOCATION(ploc.x+1,ploc.y-1),
                                    LOCATION(ploc.x-1,ploc.y  ),
                                    LOCATION(ploc.x+1,ploc.y-1),
                                    LOCATION(ploc.x-1,ploc.y+1),
                                    LOCATION(ploc.x  ,ploc.y+1),
                                    LOCATION(ploc.x+1,ploc.y+1)};
            for (int i = 0; i < 8; i++)
            {
                int pCType = GameMapEx::GetTileEx(Adjacent[i]).cellType;
                if (pCType == cellNormalWall ||
                    pCType == cellMicrobeWall ||
                    pCType == cellLavaWall)
                {
                    GameMapEx::GetTileEx(Adjacent[i]).wallOrBuilding = 0;
                    GameMapEx::SetCellType(Adjacent[i],cellDozedArea);
                    TethysGame::CreateUnit(newP, mapScorpion, Adjacent[i], 6, mapEnergyCannon, 0);
                    newP.DoSetLights(1);
                }

                else
                {
                    UnitEx ontile;
                    ontile.unitID = GameMapEx::GetTileEx(Adjacent[i]).unitIndex;
                    if (ontile.unitID != 0 && ontile.OwnerID() != 6 && ontile.IsLive())
                    {
                        if (ontile.IsVehicle())
                        {
                            ontile.DoPoof();
                            TethysGame::CreateUnit(newP, mapScorpion, Adjacent[i], 6, mapEnergyCannon, 0);
                            newP.DoSetLights(1);
                        }
   
                    }
                }
            }

        }
    }

[TechCor]

Yes. I've been thinking about this.

The main issue is letting random people upload DLLs, but this can probably work with a separate curated system.

It could be possible to have a separate "DLL Mission" tab for curated DLLs that are placed by a server administrator. The mission details could include additional information such as the server providing the mission, the fact that the mission comes with a custom DLL, and that it is a potential security risk. Then, when choosing to download, have a confirmation dialog that the user trusts DLLs coming from "Outpost2.net" or whatever.

The topic of how the new editor's "scripting" works is a bit complicated, and I'd rather explain it in the other thread when it's released. I don't doubt that it will not be able to do everything, but it is certainly more than a skirmish editor. I intend to make a sample mission to show off its capabilities (and test it). The style is similar to StarEdit, but with a lot more options.

[lordpalandus]

DLLs are a security risk sure, but as Outpost 2 is a cult classic with a small following, what level of risk is there really?

If there was a concern over infected DLLs, then someone could test that DLL in a virtual machine, and if it was infected, then it would be contained within the virtual machine. Otherwise, I think people can trust their fellow poster not to screw them over.

Now, if this was a mainstream title, like Call of Duty, I think there would be a serious concern over an infected DLL, but I don't see anyone going to the trouble here.

[Hooman]

I like the idea of a central hub to download new missions from.

I've noticed a similar trend with programming languages, where having a central repository for libraries and associated dependency management tools can really help quickly increase the usability and popularity of a language.

Of course, we don't currently have a whole lot of content to distribute. In that regards, this might not be the most impactful use of time. On the other hand, not being able to easily distribute content can create an extra barrier that may discourage creating new content.

Excluding missions with binary DLLs seems like a real hindrance. For such a system to be really useful, I think we need some way to allow binary distribution for DLLs.

One way we might approach security, is to somehow sign each version of the DLLs, so people can know who wrote it, and who declared it to be safe. That sort of matches how things are informally done now. Custom missions have started being packaged into the game download. For missions to be included in the game download, people responsible for packaging the download have to accept them, which generally implies a certain degree of review. We don't currently have a very formal vetting system, though mostly missions that are included are from long time members who have been around for years, and have accumulated a certain amount of trust. Source code helps with that.

If a new member wishes to distribute something, and they are unknown, they could potentially ask a long time member to review it, and sign off on a review. Before a level is downloaded, the user can be presented with summary information, such as who the author is, how long they've been a member at OPU (if at all), and who signed off on review (along with how long they've been a member). We could also present a link to the source code.

For review purposes, we might require the mission have source code posted on GitHub. That may increase the effectiveness of review. It may also increase the safety, as the binaries could then be built on independent Continuous Integration servers, based on the source code that was checked in to GitHub. That way the binary matches up with the source code provided for review. We could even manage this through GitHub releases, where the Continuous Integration servers publish the binary as a new release on GitHub. That could effectively be our central store for distribution purposes.

Anyway, that's a rough idea I have for how things could potentially be managed. It still needs details on how the list of missions is collated, how binaries are signed, and signing them without disturbing a release by trying to add new files to it (which could affect signing), how to revoke signing should a problem be discovered after the fact, and how users might be warned of newly announced problems.

Another part of that is Continuous Integration builds can also scan the code for potential problems, using linters, compiler warnings, or virus scanning of the result, and potentially auto signing for basic safety.

[TechCor]

Of course, we don't currently have a whole lot of content to distribute. In that regards, this might not be the most impactful use of time. On the other hand, not being able to easily distribute content can create an extra barrier that may discourage creating new content.

Yes, this has kind of been the goal from the beginning - to reduce friction for content creators. The hub also has the side-benefit of reducing friction for players that want to play custom content.

--
Before you posted, I was thinking about allowing "uncertified" uploads that admins could "certify", and this would be reflected on the mission summary. Sort of like a "seal of approval". The certification would change its status from "high-risk" to "trusted". By default, untrusted missions could be hidden from the list. For edits by the author, the certification would either be reset, or put into a "pending approval" state where the old version is still available.

However, as you pointed out, you really need source code and certainty that the DLL was compiled from it to really be sure a DLL is safe.

Perhaps the system won't allow direct DLL uploads, but instead require specifying a GitHub link. A first release could be a completely manual review process, where the admin builds and uploads the DLL and certifies the mission. Future releases could automate creating and uploading the DLL, if it's deemed worth it. I doubt reviewers will be overwhelmed.

Other file types are probably safe to upload as part of the mission without certification. Pre-approved types would be .map, .opm, .txt.

--
For clarity, content uploading, certification, and administration would all be done through the Hub interface, but code review and compilation would not.

[Hooman]

Well, the op2ext project already uses automatic GitGub releases. Push a tag to GitHub, and CI will build and upload binaries as a new GitHub release. It's surprisingly easy to automate that part.

Automating releases is largely copy/paste a block of AppVeyor config to get it working for other projects. The main thing that might need to change, is the security key, which is done per organization. If the projects are hosted by OPU, that's already setup. If they self host on their own account, they would need one time setup of their own security key.

Code review and signing would be the more manual and time intensive task. For it to be valid, it would need to be done per version of the software, not just once per project, otherwise it would be easy to insert malicious code after a review.

I wonder if we might be able to use a GitHub repository to manage signing and revocation. We could potentially have a text file with URLs to GitHub release pages, along with a hash of the download package, and signing info from an arbitrary number of reviewers. Might also want to include space for review notes, particularly for revoking access.

I don't know if we'd want to have clients access GitHub directly for signing info, or copy it to our own servers and have clients access it there. For reviewers, it would probably be most convenient to use GitHub, as that would probably be the tool for reviewing the code itself.

Though, I've been using GitHub a lot recently, so perhaps it's becoming my hammer.

[Vagabond]

I think more important than a hub we have lingering issues preventing people from reliably playing multiplayer matches even though NetFix and NetHelper have improved the situation.

I would be interested in improving the 'game lobby' screen to show a map screenshot and short description of mission / objectives which would fit into this hub idea.

I'm partial to using old C++ to make missions security concerns outstanding. I'm certain if someone goes through the trouble of designing a workable mission hub, we could tackle the security concerns reasonably. The starting place for a hub as far as code hooks into the game may be partially contained in the NetFixClient/Server projects on GitHub that Hooman developed.

A dedicated subset of Outpost 2 players use Wine on Linux. It would be nice to either ensure Wine support for the client side of the hub or agree if it was tenable by the community to not support Linux up front before a large amount of work went into it. I'm not actually sure how many people use the Macintosh installer (and it is behind the Windows release at this point)...

[Arklon]

DLLs are technically a real security risk, but it's also a pretty ubiquitous hypothetical problem across many game modding communities that hardly anybody even shrugs about. If we have a system where DLLs can be manually reviewed, and in the UI have unreviewed files have a warning next to the download button saying they are unverified and may cause damage etc. (rather than necessarily forcing them to get approval), that would probably be good enough, especially considering how niche the community is. Although if you have all DLLs be built through an automated system, and also are allowing DLLs awaiting approval to be built, then you'd need to prevent people from being able to abuse that somehow.

It would be nice if using VirusTotal to help automate screening DLLs was viable, but it looks like the code hook stuff some custom missions do sometimes causes one or two AV engines to complain.

[TechCor]

This is my proposed main UI design (draft 1):


The three sentences at top are referring to the first 3 tabs at the top. Might show a single line there, at the bottom, or just assume that the tabs are intuitive enough.

Missions certified by an admin would be marked as "Trusted".
Missions with no DLL are unmarked. .opm with no DLL is assumed to be a Standard Mission.
Uncertified missions with a DLL are marked as "Risky".

Uploading files will automatically uncert the mission.

The boxed mission area is a single item in a scrollable list box, so imagine it repeating where all the design notes are placed.

I forgot about the GitHub link. Probably place it under the mission description if a link was provided. Might also be worth including an upload date in the details section.

Search bar would exclude non-matching missions from the list for the selected tab (updated while typing).

File uploads would be restricted to the files listed in the image at probably a couple MB with a limiter of something like 1 mission per day.

Cross-platform support should not be too much of an issue.

--
Certification is just marking the mission as "trusted". Presumably, the certifying admin would have some kind of process in place before pushing the button, which could include building from source and uploading their own DLL.

If we want a more automated solution, it could be added afterwards. It's nice to have something to start with, and this app is trivial to write.

Except maybe using the forum for authentication. Does this forum use a template like phpBB?

[leeor_net]

This sort of idea has been kicked around for awhile and I'm a big proponent of it. Finding custom add-ons on the forums is ... well... it kinda sucks.

I would think that allowing for the DLL versions is acceptable BUT with the caveat that the source code be made available and that the DLL file is uploaded only after compiled through our own build system? Something along those lines that way we can allow for both types of mission files to be allowed.

[Vagabond]

I was assuming you meant to write this as a DLL module for Outpost 2 so it would launch from within Outpost 2. Are you suggesting it be a standalone application?

[TechCor]

I was assuming you meant to write this as a DLL module for Outpost 2 so it would launch from within Outpost 2. Are you suggesting it be a standalone application?

Hmm, I suppose it could be done that way. I'm not too skilled with the memory hacking, though. Also, I'd have to relearn win32, and learn how to do C++ web requests. It would take significantly longer than a standalone app.

The standalone client would only take a couple weekends, so I think I will go with that first. I don't really see any major benefits to having it built-in, except for some slight convenience. Many games have "launchers" now that do all sorts of download and update related tasks, so I don't see it as unusual to go this route.

A motivated win32 programmer could easily write a module that hooks into the same web backend.

The mod could be a "download-only" version. I'm not sure if it makes much sense to have upload and admin capabilities in the game itself. That would keep the mod simpler, as you eliminate the majority of the work (login, upload, editing, certifying), and keeps the focus on downloading and playing missions. Most players wouldn't need the full capabilities, and anyone else would already be downloading other tools.

[Vagabond]

There is likely an inbetween option where you write a module that launches the mission hub client. That would require adding a button somewhere in Outpost 2, which would either require memory hacking or editing the resource file that contains the menu layouts. It would minimize the amount of win32 programming required.

The module interface is written in C, so in theory you might actually be able to create a C# module for Outpost 2? I would be interested in the outcome if attempted.

I'm not trying to push one way or the other, just bringing it up. If interested, you can checkout the readme on the op2ext project on GitHub. One word of caution, Hooman and I have made significant updates to op2ext that have not made it into the Outpost2 official download. A minor breaking source code change to the interface and changes to the syntax used in the Outpost2.ini file for specifying an INI module.

[Hooman]

I was just thinking, it would potentially be done as a web app. If the web app was implemented using a REST API, that could potentially be re-used by an in-game module.

Though yes, if you're more familiar with standalone, that may be quicker for initial development.


In terms of certification, I'm kind of thinking an external process might be better. I wouldn't want to have to worry about security being controlled by a new custom app.

More to the point though, the process of certifying would involve examining and building the source code. Most of that stuff is done using GitHub and Continuous Integration servers now, so a process that integrated well with that might be best. I noticed CircleCI has support for manual review tasks, where part of a build can wait on a reviewer to click a button to continue. That might involve publishing some sort of review certificate. I'm currently uncertain how it might be cryptographically signed by the reviewer.

Another reason for doing things this way, is that to be certain the binary matches up with the source code, we'd need to do the build ourselves in an environment we control.

[TechCor]

Oi, peeps!

First draft of the Mission Hub is available!

https://github.com/TechCor8/OP2MissionHub/releases/tag/0.1

This is a test version only, so feel free to post garbage to get a feel for it. Feedback/Bug reports are appreciated!

The major missing feature is minimap rendering because of OP2 Utility. I decided not to use the current interop wrapper. Instead, I will do a C# port of OP2 Utility so that cross-platform compatibility is built-in. The mission editor will also start using that to eliminate cross-platform issues.

There is also a lack of scalability, but I'm not anticipating a huge supply of missions. It is not high priority at the moment.

Everything else though should be fully functional, so let me know if it isn't!

FYI on administration:
Basically, admin accounts are regular accounts that are "marked" as admins. Admins have the ability to modify any mission as if they were the author, have fewer restrictions (rate limiting), and can mark missions as "certified".

[Hooman]

Is there a project file for this?

I noticed you uploaded a Linux build, though I'm uncertain how to build this form Linux.

[TechCor]

New (minor) Release:

v0.2
https://github.com/TechCor8/OP2MissionHub/releases/tag/0.2

Linux is now supported.

Updated documentation to make required redistributables clear.

[Hooman]

How do you build it from Linux? 

[TechCor]

Unity is required to open and build the project. The hub (and editor) are currently using Unity 2019.3.10f1. It will probably end up on 2019.4.x in a few months. When opening with Unity, the repo root directory is the project - there is no dedicated project file. I believe there is a Linux version of Unity, but I have not used it.

The editor uses the OP2UtilityForC project which was compiled into a Shared Object using Visual Studio For Linux and an Ubuntu Desktop VM through VirtualBox as the remote compiler. That Shared Object was dropped into the editor project as a plugin. Eventually, the Hub will use this plugin as well.