Author Topic: Continuous Integration - AppVeyor  (Read 204 times)

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4566
Continuous Integration - AppVeyor
« on: June 28, 2018, 10:59:22 AM »
I was looking through some of the Continuous Integration tools on GitHub. Continuous Integration tools can watch a repository for new commits/branches/pull requests, and automatically download each new commit to a fresh environment, build it, and log the results. It can also optionally run tests, if any are present. Further, it can update GitHub pull requests with a pass/fail status of the build. Nice quick feedback to let you know if you broke something by accident, and to let other people know if your commits build and pass tests before review or merging. Saves having to checkout the branch and compile or run it yourself.

Previously I was aware of tools for a Linux environment, such as TravisCI. Recently, I noticed AppVeyor, which is an automated Windows build environment.

I gave it a quick try tonight, and it was reasonably painless to setup. I needed to make two project configuration adjustments to get it working with OP2Archive:
 Build -> Visual Studio solution or project file: OP2Utility.vcxproj
 Environment -> Build worker image: Visual Studio 2017

Attached are some screenshots of a passing build, and a failing build.

AppVeyor website - Build passing


AppVeyor website - Build failing


GitHub - build failing


Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 744
Re: Continuous Integration - AppVeyor
« Reply #1 on: June 28, 2018, 08:08:03 PM »
Hooman, thank you for exploring and reporting on this.

I am a fan of continuous integration. It would give more confidence that code I wrote does not behave poorly on a Linux based compiler and vice versa for someone developing on Linux. I suppose even just a basic compilation is helpful in this sense. Making sure no compile time errors occur on the changed code on Linux's compilers interpretation of the C++ standard when perhaps it behaved differently on MSVC.

Running unit tests would build even more confidence. Then it would allow code written for one compiler to actually be executed in a comprehensive manner on a different compiler/system build.

Human testing is still important even with comprehensive unit tests though.

How does the installation of AppVeyor work? Is it somehow installed on GitHub or does GitHub transmit the data to a third party server for testing? It looks like the free version would easily meet the needs of projects on Outpost Universe (1 concurrent task with 1GB build cache).

Thanks,
Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4566
Re: Continuous Integration - AppVeyor
« Reply #2 on: June 29, 2018, 02:44:41 AM »
Indeed, I think their free option is sufficient for our needs.

AppVeyor has their own website, and their own build machines, though it integrates with GitHub in the form of commit hooks. A commit hook is basically just a URL that GitHub contacts once a commit has been pushed to the repo. AppVeyor is able to insert the commit hooks for you using GitHub's OAuth feature. I noticed AppVeyor has fairly locked down restrictions, in that it doesn't ask for more access than it needs. You can revoke access from the GitHub interface at any time.

There are 3 different solutions for a Linux based CI build. I'll look into setting up one of those next. I've played with Travis CI before for NAS2D, though I'm curious to explore the others. I remember struggling a bit with Travis CI because of the really old Ubuntu images they were using, and they didn't seem keen on upgrading or keeping them up-to-date. There also didn't seem to be any caching for Apt update and package installations, so that took really long. I remember over half the build time was re-setting up the environment before each compile of the actual project source. It made for somewhat slow turnaround time for compile/test results.

As for testing, there are different levels of tests. Unit tests are a fairly low layer, testing individual components. Higher up there are integration tests, which test how multiple units are composed and integrated together. This layer is a little more concerned with the interfaces between components, and making sure everything fits together in a reasonable manner. System testing is done on the complete application. This might involve running the application, giving it specific inputs, and checking if the required outputs are produced. For console apps, this might involve scanning for console messages to indicate success or errors, or checking for the existence and validity of output files. At a higher level are acceptance tests, which is more concerned with does the application work, do what's intended, and actually solve the user's problem.

We can go beyond automating just the unit tests.  ;)

In the case of OP2Archive, I would actually really like to have a few system tests. I'm not sure of the best way to do this, but one way is just to have a Makefile that runs the output program to pack/unpack a few archives, or give a few bad commands and grep for "Error" output.