Puppet: System Administration Automated

Puppet Training Schedule
Next Class July 27-29
New York, New York
Discount before July 1st

Using Stored Configurations

Puppet uses a technique called stored configuration, enabled with the storeconfigs configuration option, to store data in a database. This technique makes use of the Ruby on Rails framework and one of three database back-ends: SQLite (the default), MySQL or PostgreSQL. Additionally, a message queue can be used to improve a puppetmaster's client response time. Currently stored configurations are principally used by exported resources.

It is important to note that once a node is in the stored configuration database it will not be automatically purged - even if you remove the corresponding node from your configuration. If you wish to purge nodes from your database you will need to do this manually. There is a script attached to this page that allows you to remove a node by name from the database.

Installing Prerequisites

To make use of stored configuration you will need to install some prerequisites principally the database back-end you wish to use and the Rails framework.

Puppet should work fine with both Rails 1.x and 2.x.

We start by installing the rails Ruby Gem, either with the distribution's package management (e.g. Debian/Ubuntu have a 'rails' package) or as a ruby gem:

gem install rails

MySQL

For MySQL you will need to install a number of packages. For Red Hat/Fedora the typical packages required would be:

mysql
mysql-devel
mysql-server

Or for Debian/Ubuntu you would need the following packages. libmysql-ruby provides the mysql gem, so installing via gems is unneccesary:

mysql-server
libmysql-ruby

You may also need to install the mysql Ruby Gem (not required on Debian/Ubuntu). You may need to specify the --with-mysql-config option to tell Gems where to find your MySQL libraries and headers:

# gem install mysql -- --with-mysql-config=/usr/bin/mysql_config

You may require the ruby-devel package in order to install this gem.

A word of warning - it is possible to be using stored configurations with MySQL without installing the mysql gem. This is because rails includes with it a simple Ruby based MySQL connector. Relying on this connector is strongly discouraged, as it has been observed to leak filehandles, leave open network and database connections, and eventually cause puppetmasterd to become completely unresponsive.

In order to check if you are using the unreliable Ruby based connector, examine the rails.log file produced by puppetmasterd. If you are using the Ruby one, you will find a message like this:

WARNING: You're using the Ruby-based MySQL library that ships with Rails.
This library is not suited for production. Please install the C-based MySQL
library instead (gem install mysql).

If you find that warning, make sure you install the mysql gem, which is based on the native MySQL libraries, before you begin using stored configurations in production.

SQLite3

For SQLite you will need to install the database package. For Red Hat/Fedora the package required would be:

sqlite

For Ubuntu/Debian, the packages required are (libsqlite3-ruby contains the appropriate gem):

sqlite3
libsqlite3-ruby

You may also need to install the sqlite3-ruby Ruby Gem (not required on Debian/Ubuntu):

# gem install sqlite3-ruby

PostgreSQL

For PostgreSQL you will need to install a number of packages. For Red Hat/Fedora the typical packages required would be:

postgresql
postgresql-server
postgresql-devel

You will also need to install the postgres Ruby Gem:

# gem install postgres

If you fail to install the postgres gem then you will receive the following error:

err: Could not retrieve configuration: Uncaught exception no such file to load -- postgres in method puppetmaster.getconfig

For PostgreSQL you will also need to create the database and create an appropriate role and password.

Configuring basic storeconfigs

Now we need to configure stored configuration in Puppet itself. There are three types of database adapters available for us, the default sqlite3, MySQL and PostgreSQL. For the latter backends we also have available a number of options that let us specify a particular user, password and the location of a database socket. Let's walk through the steps required to configure the most basic SQLite backend.

  • Configure puppet.conf:

    [puppetmasterd]
storeconfigs = true
dbadapter = sqlite3
dblocation = /var/lib/puppet/storeconfigs.sqlite

  • Start the puppetmasterd daemon

  • Start the puppetd client

Now storeconfigs should be enabled and running.

Note that if you intend to use PuppetShow, you may need to include a dblocation setting under puppetmasterd, pointing to the default location. Also note that SQLite is not very happy with high volume usage (more than 5 or 10 hosts), and will report locking failures during operation.

Configuring storeconfigs for MySQL

  • Create a puppet database, and grant privileges. Replace password below with an appropriate password:

    # mysql -u root -p
mysql> create database puppet;
mysql> grant all privileges on puppet.* to puppet@localhost identified by 'password';

  • Configure your [puppetmasterd] section to reflect these settings:

    [puppetmasterd]
storeconfigs = true
dbadapter = mysql
dbuser = puppet
dbpassword = password
dbserver = localhost
dbsocket = /var/run/mysqld/mysqld.sock

  • Start the puppetmasterd daemon

  • Start the puppetd client

You can also find some information on configuring MySQL stored configurations in a Puppet manifest.

Configuring storeconfigs for PostgreSQL

By default (on RHEL/CentOS, other distros unknown), you will have to su - to the postgres user to do any administrative work such as database creation.

  • Set up the database:

    # su - postgres
$ psql template1
template1=# create database puppet;
CREATE DATABASE
create user puppet with unencrypted password 'password';
CREATE ROLE
template1=# grant create on database puppet to puppet;

Note that Puppet requires the CREATE right to do automatic table creation to initialize the database.

  • Configure your [puppetmasterd] section to reflect these settings:

    [puppetmasterd]
storeconfigs = true
dbadapter = postgresql
dbuser = puppet
dbpassword = password
dbserver = localhost
dbname = puppet

  • Start the puppetmasterd daemon

  • Start the puppetd client

  • Check /var/log/messages (or wherever you have Puppet set to log) for the message for an ''Initialized database'' message when the first client checks in.

Installing Queueing Support for Storeconfigs

To add support for queueing to an existing storeconfigs configuration, you only need to install a queue broker, configure your puppetmaster, and configure your puppetqd.

While many message queue services exist, currently Puppet only supports the Stomp protocol. We have tested using StompServer and ActiveMQ. While StompServer is easy to set up and configure, we suggest you use ActiveMQ -- if you need queueing, you need it because of scalability, and StompServer simply isn't built for scaling up to enterprise needs.

Queuing is available from release 0.25.0 onwards. It is NOT available in the 0.24.x releases.

Installing StompServer

First make sure you have the Ruby development package. On a RedHat-based system, this would be:

yum install ruby-devel

Or on a Debian-based system:

apt-get install ruby1.8-dev

StompServer is available as a gem, so you can install via:

gem install stompserver

Installing ActiveMQ

Download & install ActiveMQ: e.g.,:

$ wget http://www.apache.org/dyn/closer.cgi/activemq/apache-activemq/5.2.0/apache-activemq-5.2.0-bin.tar.gz
$ tar zxvf apache-activemq-5.2.0-bin.tar.gz

Configuring queueing storeconfigs

Configuring StompServer

Configuration can either be done on the command line or via a config file. For example, the following command-line:

stompserver -q dbm

will start the stompserver on the default port of 61613 and use a DBM file as its backing store (i.e., while messages have not yet been picked up by puppetqd, they will be stored in a DBM file).

The config file:

$ cat /etc/puppet/puppet-queue.conf
:daemon: true
:working_dir: /tmp/stompserver
:storage: .queue
:queue: file
:auth: false
:debug: false
:group:
:user:
:host: 127.0.0.1
:port: 61613

configures the stompserver to use a file backing store.

Then run the stompserver:

stompserver -C /etc/puppet/puppet-queue.conf

Consult the stompserver's README.txt for additional information on installation and configuration.

Configuring ActiveMQ

Create an activemq.xml file (based on the samples provided in the source), and enable stomp. The sample configuration uses the default backing store, AMQ, (which is a transactional journal in ActiveMQ 5) and only requires the one-line change below to enable stomp on port 61613:

<transportConnectors> <transportConnector name="stomp" uri="stomp://localhost:61613"/> </transportConnectors>

You can then start activemq:

activemq

You should consult the ActiveMQ documentation for additional information on configuring ActiveMQ.

Configuring Puppet

You need to configure your both puppetmaster and puppetqd to use queueing. In this example, we're keeping our storeconfigs information in sqlite3 -- if you're using PostgreSQL, MySQL, or others, your dbadapter and dblocation information will differ).

Please refer to the basic storeconfigs section on the wiki for details about creating the database for MySQL or PostgreSQL.

First we need to install the Stomp client. You can do this from a gem:

gem install stomp

Next, we need to configure Puppet itself. Note that the configuration of the puppetmaster and the puppetqd is the same: this is because they both need to read information out of the same database (i.e., the database is a cache). Thus, a puppet.conf would contain:

[main]
  queue_type = stomp
  queue_source = stomp://localhost:61613
  dbadapter = sqlite3
  dblocation = /var/lib/puppet/storeconfigs.sqlite
[puppetmasterd]
  async_storeconfigs = true

Note that these can all be put onto a command line:

puppetmaster --async_storeconfigs --queue_type=stomp --queue_source="stomp://localhost:61613" \
  --dbadapter sqlite3 --dblocation /var/lib/puppet/storeconfigs.sqlite

Also note that the dblocation is not needed for MySQL or PostgreSQL dbadapters.

Bootstrapping into storeconfigs land

If you have a lot of hosts, turning on storeconfigs might crush your database server. The main problem is that data for every host needs to be INSERTed into the database. Once you get past that hurdle there's much less SQL activity. Here are some notes about how we (usg.edu folks) got over the hurdle. The main bottleneck for us was CPU--IO wait wasn't an issue.

First we tuned our MySQL server (all based on masterzen's recommendations). We added these lines to the [mysqld] section of my.cnf:

innodb_buffer_pool_size=2G
innodb_log_file_size=256M
innodb_log_buffer_size=64M
innodb_additional_mem_pool_size=20M
innodb_flush_method = O_DIRECT

Next we stopped puppetd on half of our ~400 servers. This, plus the MySQL changes above, allowed our server to survive the initial storeconfigs hit of our other 200 servers. After the first 200 were in storeconfigs we ran 'puppetd --test --tags SMALLMODULE' on the second half of our hosts. Later we restarted puppetd.

Our clients check in once an hour. About 15 hosts check in per minute. puppetmasterd and mysqld run on the same server (along with a few other apps). Our server has 8 CPUs and 12G of RAM.

Here's a fine blog post by masterzen about this issue: http://www.masterzen.fr/2009/03/18/omg-storedconfigs-killed-my-database/

Attachments