Technically open source and free are two seperate things. It's like that whole "free as in speech, not free as in beer" things you occasionally run into. For example, Windows is not free, nor is it open source. Adobe Reader is free, but not open source. Most Linux distributions are both free and open source. However, I believe Red Hat Enterprise Linux is open source, but not free. If you really wanted it, I believe you could download the source and compile it yourself, but they don't have binaries up for download. You have to pay for the precompiled binaries.
Granted, non-free, open source software is somewhat rare compared to the other categories. Plus, the free, but not open source stuff is usually to promote a company or technology that has other non-free related software packages. So I can see open source and free generally being equated.
I'd given some thought into the general game engine with an Outpost specific mod myself actually. Although, if you wanted to sell it, you'd probably need to write a whole other game, and offer such a mod for free. Plus, I suspect if you wrote both the engine and the mod yourself, that might lead to some trouble. Particularly if the main selling point of the game, was to run the free mod. That's ignoring problems over the free mod idea itself too.