Indeed, I think their free option is sufficient for our needs.
AppVeyor has their own website, and their own build machines, though it integrates with GitHub in the form of commit hooks. A commit hook is basically just a URL that GitHub contacts once a commit has been pushed to the repo. AppVeyor is able to insert the commit hooks for you using GitHub's OAuth feature. I noticed AppVeyor has fairly locked down restrictions, in that it doesn't ask for more access than it needs. You can revoke access from the GitHub interface at any time.
There are 3 different solutions for a Linux based CI build. I'll look into setting up one of those next. I've played with Travis CI before for NAS2D, though I'm curious to explore the others. I remember struggling a bit with Travis CI because of the really old Ubuntu images they were using, and they didn't seem keen on upgrading or keeping them up-to-date. There also didn't seem to be any caching for Apt update and package installations, so that took really long. I remember over half the build time was re-setting up the environment before each compile of the actual project source. It made for somewhat slow turnaround time for compile/test results.
As for testing, there are different levels of tests. Unit tests are a fairly low layer, testing individual components. Higher up there are integration tests, which test how multiple units are composed and integrated together. This layer is a little more concerned with the interfaces between components, and making sure everything fits together in a reasonable manner. System testing is done on the complete application. This might involve running the application, giving it specific inputs, and checking if the required outputs are produced. For console apps, this might involve scanning for console messages to indicate success or errors, or checking for the existence and validity of output files. At a higher level are acceptance tests, which is more concerned with does the application work, do what's intended, and actually solve the user's problem.
We can go beyond automating just the unit tests.
In the case of OP2Archive, I would actually really like to have a few system tests. I'm not sure of the best way to do this, but one way is just to have a Makefile that runs the output program to pack/unpack a few archives, or give a few bad commands and grep for "Error" output.
