CFEngine Policy Servers with Docker

Right now I’m in the middle of an internship in the HPC group at the Sanger Institute, and one of the things we’ve been working on is a better workflow for changes to configuration management with CFEngine.

One issue we’ve been really struggling with is how to test changes to CFEngine policy. There are two particular points which make things awkward:

• Our CFEngine policy files are difficult to test in isolation.
  • This is partly due to the inability of the CFEngine agent to run bundles, CFEngine’s conceptual unit of configuration, directly, but also due to the way our codebase is structured.
  • Update: Nick Anderson points out in the comments below that cf-agent does have a ‑‑bundlesequence option which can be used to run bundles that don’t take any arguments. And in fact, experimenting in the past week, it turns out it’s not too hard to create a script which uses a wrapper policy to call a bundle with arguments (e.g. cfengine-run-bundle) - so this one can be solved easily.
• Testing a complete set of configuration is painful and error-prone, involving copying files from the CFEngine Git repository to a global location¹, modifying configuration to prevent CFEngine from overwriting them, testing them, and putting them back into the Git repository without having made any mistakes.

So for a while, we used a simple setup with a separate policy server to test changes - that way, the entire set of configuration can be tested by just switching which policy server a given host asks for configuration. The issue then is keeping that separate policy server in sync with the main policy server, and allowing multiple people to test changes without trampling over each other’s work.

We realised that what we really needed was a separate policy server for each change made to the CFEngine policy. There came a light-bulb moment: what if we created policy servers on demand for each branch pushed to the repository? ² Then we could kill two birds with one stone: encouraging people towards a Git flow-like use of Git, and we get easier testing!

The virtual policy servers can be used by physical virtualised CFEngine clients, enabling both automated virtualised testing and tests on real hardware with specialised components (e.g. InfiniBand).

Implementation-wise, this is a perfect fit for Docker: the policy servers for each branch need to come up quickly to enable rapid development cycles, and don’t need to be running anything other than the CFEngine policy server daemon, cf-serverd. Management of containers is handled by post-receive hooks on the CFEngine Git repository.

If you use CFEngine and would like to give it a go, all the source is on GitHub at mrahtz/docker-cfengine-servers, together with setup instructions. The repository also contains Vagrantfiles for automatically bringing up a VM with Vagrant to show how everything fits together.