Puppetize the Cloud - Building & Maintaining Cloud Infrastructure

Skip navigation

Puppetize the Cloud

Puppetize the Cloud by xcheng

At Mendix we move fast to be able to cope with the high demand of our ever growing customer base. This blog post focusses on some technical challenges we face while building and maintaining the cloud. For the sake of this post we define the scope of the cloud to be the (infrastructural) environments where customer apps are being hosted.

How did it all start?

Like most stories on automation in the beginning most work was done manually. Servers were configured the old fashion way by hand with the aid of documentation. We call it the copy-paste-era. It may not be the most efficient way back then but by doing everything manually we sharpened our skills. We evaluate and learn the in’s and out’s of tools before applying them to customer environments.

Soon the challenge to scale started to lure around the corner: we were unable to keep up with the pace of applications being sold. To mitigate the problem we started building new environments using bootstrap scripts. These scripts were mostly written in python mixed with bash fragments distilled from our documentation.

Even though we can deliver environments much quicker it is still not optimal. Whenever an error occurs during the bootstrap we had to start over again. It doesn’t happen often but it could be a nuisance. So there is a robustness problem. As time went by maintenance chores reveal we have a bigger problem: how do we apply configuration changes? At first we started using fabtasks (Fabric) to roll out new releases. It’s utterly difficult to keep track of changes systematically. We knew there are tools that could solve most of these problems for us. So we spend a great amount of time trying out the different configuration management systems. We ended up with Puppet because it was reasonably understandable thanks to its Domain Specific Language. We also loved the fact we could do dry-runs.

How do we use Puppet?

We try to follow community best-practices. Sometimes we feel the need to come up with new approaches that best fit our ideals and workflow. Next we will outline some key decisions we made so far.

Security: Puppet master behind nginx

Our servers are all over the world it is pretty hard to protect the Puppet master against the evil internet by whitelisting in the firewall. As the Puppet system already uses client certificate to authenticate agents we might as well use that. We didn’t feel comfortable exposing the Puppet master directly on the internet. So we put Nginx in front to handle HTTP(s) and client authentication:

upstream puppetmasters {
    server   unix:/var/run/puppet/puppetmasterd.0.sock;
    server   unix:/var/run/puppet/puppetmasterd.1.sock;
    server   unix:/var/run/puppet/puppetmasterd.2.sock;

server {
    listen [::]:8140;
    server_name _;

    ssl                     on;
    ssl_certificate         /var/lib/puppet/ssl/certs/puppetmaster.pem;
    ssl_certificate_key     /var/lib/puppet/ssl/private_keys/puppetmaster.pem;
    ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;

    ssl_crl             /var/lib/puppet/ssl/ca/ca_crl.pem;
    ssl_verify_client   on;

    proxy_redirect      off;
    proxy_set_header    Host             $host;
    proxy_set_header    X-Real-IP        $remote_addr;
    proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header    X-Client-Verify  SUCCESS;
    proxy_set_header    X-Client-DN      $ssl_client_s_dn;
    proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
    proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;

    client_max_body_size 5M;

    location / {
        proxy_pass http://puppetmasters;

Seeing the recent security advisories like CVE-2013-1655 and CVE-2013-3567 confirm this setup is worth the effort.

Manual releases with daemon-less agent

During a puppet run we often see the agent consume over 200 MB of RAM. That may not sound much but if you consider a virtual machine with 1024MB that accounts for 20% usage. Recalling one of the selling points of puppet: dry-runs. The ability to dry-run a release grants us more predictability on what exactly is going to happen. Our releases are done using Puppy developed in-house. It enables us to manually review changes in vim by pre-processing all reports into a concise summary.

Avoid off-the-shelf modules

Modules picked from the internet are often designed to cover most general use-cases and to work in different Linux distributions and sometimes even Operating systems. At Mendix we use Debian exclusively in the cloud. As our deployments are pretty narrow we could reduce Lines of Code dramatically by reducing complexity of the different configuration structures across distributions and cover less use-cases of e.g. Nginx, postgresql, etc… Occasionally we study external modules for inspiration on coding style, structure and approach.

A typical module structure looks like:

├── files
│   └── etc
│       └── postfix
│           └── dynamicmaps.cf
├── manifests
│   ├── common.pp
│   ├── config.pp
│   ├── init.pp
│   ├── install.pp
│   ├── params.pp
│   └── service.pp
└── templates
    └── etc
        ├── aliases.erb
        ├── caretakr
        │   └── oinkwall.d
        │       ├── postfix-relay.puppet.yaml.erb
        │       └── postfix-satellite.puppet.yaml.erb
        └── postfix
            ├── main-relay.cf.erb
            ├── main-satellite.cf.erb
            ├── sasl_passwd.erb
            ├── sender_canonical_maps.erb
            └── virtual.erb

Conclusion and improvements

We are very proud of our current Puppet adoption. We are continuously improving the setup. In particular the module structuring part. Sometimes it feels like spaghetti code when resources dependencies are all over the place. The philosophy to run concurrently in Puppet is nice but often we need steps to happen in a specific order; not just in one module but across modules also.
Recently steps are being taken to introduce continuous integration. Using Vagrant and VirtualBox we can build the full Mendix Platform using Puppet as provisioner.

To summarize we love Puppet as it enables us to do releases predictably and systematically. This concludes the fairy tale on puppetizing the Mendix Cloud.

# vim: ft=markdown:wrap

Author Info