Puppet Scalability Notes
This document describes some of the issues when scaling puppet to provide service for a large number of nodes.
Please see Using Mongrel for information on how to set up Mongrel as was used in this test.
Master Performance
As of Puppet 0.22.1, the fileserver, CA methods, and configuration parsing methods all share resources within the puppetmasterd process. The consequence of this setup is that a manifest with a large number of fileserver requests will result in the fileserver methods capitalizing the resources.
Symptoms: puppetd calls to puppetmaster.getconfig or the CA methods seem to take an unreasonably long time, or timeout altogether.
Splitting off the Fileserver
Since each node will typically make a single call to get their configuration, which in turn produces a large number of file requests, it helps dramatically if the fileserver is running in a discrete process from the configuration server.
If you're using definitions for your remote file copies, this is relatively painless to configure. Just configure the default port to be something different.
I'll assume you're using something like this remotefile definition:
# JJM My preferred recipe to copy files from the server.
define remotefile($source = false,
$fileserver = false,
$port = false,
$basedir = "dist",
$owner = 0, $group = 0, $mode = 640,
$recurse = true,
$ignore = ".svn",
$backup = false) {
# Some creative use of selectors to allow overrides and defaults.
$source_real = $source ? { false => $name, default => $source }
$fileserver_real = $fileserver ? { false => "puppet", default => $fileserver }
case $port {
false: { $source_uri = "puppet://$fileserver_real/$basedir/$source_real" }
default: { $source_uri = "puppet://$fileserver_real:$port/$basedir/$source_real" }
}
file { $name:
source => "$source_uri",
ignore => $ignore,
mode => $mode,
owner => $owner,
group => $group,
recurse => $recurse,
backup => $backup
}
}
So long as we use this definition whenever we copy a file from the puppetmaster, we can easily reconfigure the system to copy files from a discrete puppetmaster process running on a different port:
Remotefile { port => 8145, fileserver => "puppet1.math.ohio-state.edu" }
We also need to start a second puppetmasterd on the port:
/usr/sbin/puppetmasterd --manifest=/etc/puppet/manifests/site.pp --logdest=/var/log/puppet/puppetmaster.8145.log --masterport=8145
With this configuration, remotefile objects will use port 8145 by default, greatly reducing load on the "main" puppetmaster process listening on 8140.
Centralised Puppet Infrastructure
As we are deploying puppet infrastructure in multiple sites it was important for us to have a centralized management for puppet, but a non centralized "service", we wanted to have the option to switch between puppet masters, and did not like the idea of a single CA for all of our infrastructure. The main reason against a common ca was, as we are really spread over the world and its common to install 50+ servers in a time frame of a few hours, we didn't want to introduce any type of dependencies.
Additionally, revoking works better this way, and well... we just wanted to make it work.
Using our solution, you could also use real root CA, your company root or self sign certificate, in some cases it could make sense not to use a self sign if you want to reuse the certificates for Apache, ldap etc.
Since all of our puppet masters are managed as well, we have one root puppet master (i.e. puppet master of the puppet masters), we called it the puppeteer. The puppeteer installation is like a regular puppet master installation.
Please note that webrick is at this time (0.24.4) unable to handle the certs in a correct way to get this setup working. As such, you will need to use something else to handle your SSL connections such as Apache. Also, you must be using a fairly recently version of puppet for the client to support it (0.24.4 works good).
We are using Apache + Mongrel: on all puppet masters you should have something like that in your Apache configuration (that's just the ssl part):
<VirtualHost *:8140>
SSLEngine on
SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
SSLCertificateFile /var/lib/puppet/ssl/certs/your.fqdn.com.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/your.fqdn.com.pem
SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 3
SSLOptions +StdEnvVars
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
<Location />
Then, let your second puppet master (the second level of the certificate chain) request a certificate from the puppeteer. Setup an openssl.cnf file (just store it somewhere) with the following content (adjust for your needs):
HOME = . RANDFILE = $ENV::HOME/.rnd [ ca ] default_ca = CA_default [ CA_default ] dir = /var/lib/puppet/ssl new_certs_dir = $dir/ca/signed crl_dir = $dir/ca database = $dir/index certificate = $dir/ca/ca_crt.pem serial = $dir/ca/serial crl = $dir/ca/ca_crl.pem private_key = $dir/ca/ca_key.pem RANDFILE = $dir/private/.rand x509_extensions = usr_cert unique_subject = no name_opt = ca_default cert_opt = ca_default default_crl_days= 30 default_days = 3650 default_md = sha1 preserve = no policy = policy_anything [ policy_match ] countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional [ policy_anything ] countryName = optional stateOrProvinceName = optional localityName = optional organizationName = optional organizationalUnitName = optional commonName = supplied emailAddress = optional [ req ] default_bits = 2048 default_keyfile = ./ca/ca_key.pem default_md = sha1 prompt = no distinguished_name = root_ca_distinguished_name x509_extensions = v3_ca string_mask = nombstr [ root_ca_distinguished_name ] commonName = XXXXXXXX [ usr_cert ] basicConstraints=CA:FALSE subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer:always nsCaRevocationUrl = https://puppeteer.your.domain.com/ca_crl.pem [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment [ v3_ca ] subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer:always basicConstraints = critical,CA:true keyUsage = keyCertSign, cRLSign [ crl_ext ] authorityKeyIdentifier=keyid:always,issuer:always
On the puppet master then copy this file to your new puppet master (e.g. /tmp):
puppetmaster=fqdn
/usr/bin/perl -p -i -e "s/XXXXXXXX/$puppetmaster/" /tmp/openssl.cnf
/usr/bin/openssl req -new -nodes -key /var/lib/puppet/ssl/ca/ca_key.pem -config /tmp/openssl.cnf -out /tmp/${puppetmaster}.csr -passin file:/var/lib/puppet/ssl/ca/private/ca.pass
Copy the ${puppetmaster}:/tmp/${puppetmaster}.csr back to the puppeteer. On the puppeteer:
touch /var/lib/puppet/ssl/index
# Sign this request with the puppeteer's CA keys
/usr/bin/openssl ca -config openssl.cnf -extfile openssl.cnf -extensions v3_ca -in ${puppetmaster}.csr -out ${puppetmaster}.pem -passin file:/var/lib/puppet/ssl/ca/private/ca.pass -batch
# Push the new certificate into place on the puppetmaster
scp ${puppetmaster}.pem ${puppetmaster}:/var/lib/puppet/ssl/ca/ca_crt.pem
In your installation process append the content of the puppeteer's ~puppet/ssl/ca/ca_crt.pem to /var/lib/puppet/ssl/certs/ca.pem on the client
Now you should be able to use any puppet master that was signed this way.
