PuppetDB is a Puppet data warehouse; it manages storage and retrieval of all platform-generated data, such as catalogs, facts, reports, etc.
So far, we've implemented the following features:
- Fact storage
- Full catalog storage
- Containment edges
- Dependency edges
- Catalog metadata
- Full resource list with all parameters
- Node-level tags
- Node-level classes
- REST Fact retrieval
- all facts for a given node
- REST Resource querying
- super-set of storeconfigs query API
- boolean operators
- can query for resources spanning multiple nodes and types
- Storeconfigs terminus
- drop-in replacement of stock storeconfigs code
- export resources
- collect resources
- fully asynchronous operation (compiles aren't slowed down)
- much faster storage, in much less space
- Inventory service API compatibility
- query nodes by their fact values
- drop-in replacement for Dashboard's inventory service
PuppetDB consists of several cooperating components:
REST-based command processor
PuppetDB uses a CQRS pattern for making changes to its domain objects (facts, catalogs, etc). Instead of simply submitting data to PuppetDB and having it figure out the intent, the intent needs to be explicitly codified as part of the operation. This is known as a "command" (e.g. "replace the current facts for node X").
Commands are processed asynchronously, however we try to do our best to ensure that once a command has been accepted, it will eventually be executed. Ordering is also preserved. To do this, all incoming commands are placed in a message queue which the command processing subsystem reads from in FIFO order.
Submission of commands is done via HTTP, and documented in the spec
directory. There is a specific required wire format for commands, and
failure to conform to that format will result in an HTTP error.
Storage subsystem
Currently, PuppetDB's data is stored in a relational database. There are two supported databases:
-
An embedded HSQLDB. This does not require a separate database service, and is thus trivial to setup. This database is intended for proof-of-concept use; we do not recommend it for long-term production use.
-
PostgreSQL
There is no MySQL support, as it lacks support for recursive queries (critical for future graph traversal features).
REST-based retrieval
Read-only requests (resource queries, fact queries, etc.) are done
using PuppetDB's REST APIs. Each REST endpoint is documented in the
spec directory.
Remote REPL
For debugging purposes, you can open up a remote clojure REPL and use it to modify the behavior of PuppetDB, live.
Vim support would be a welcome addition; please submit patches!
Puppet module
There is a puppet module for automated installation of PuppetDB. It uses Nginx for SSL termination and reverse proxying.
Puppet terminuses
There are a set of Puppet terminuses that act as a drop-in replacement for stock storeconfigs functionality. By asynchronously storing catalogs in PuppetDB, and by leveraging PuppetDB's fast querying, compilation times are much reduced compared to traditional storeconfigs.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the PuppetDB docs are to be interpreted as described in RFC 2119.
-
JDK 1.6 or above. The built-in Java support on recent versions of MacOS X works great, as do the OpenJDK packages available on nearly any linux distribution. Or if you like, you can download a recent JDK directly from Oracle.
-
An existing Puppet infrastructure. You must first setup a Puppetmaster installation, then setup PuppetDB, then configure the Puppetmaster to enable storeconfigs and point it at your PuppetDB installation.
-
A Puppetmaster running Puppet version 2.7.12 or better.
There are 2 currently supported backends for PuppetDB storage:
- PuppetDB's embedded database
- PostgreSQL
The embedded database works well for small deployments (say, less than 100 hosts). It requires no additional daemons or setup, and as such is very simple to get started with. It supports all PuppetDB features.
However, there is a cost: the embedded database requires a fair amount of RAM to operate correctly. We'd recommend allocating 1G to PuppetDB as a starting point. Additionally, the embedded database is somewhat opaque; unlike more off-the-shelf database daemons, there isn't much companion tooling for things like interactive SQL consoles, performance analysis, or backups.
That said, if you have a small installation and enough RAM, then the embedded database can work just fine.
For most "real" use, we recommend running an instance of
PostgreSQL. Simply install PostgreSQL using a module from the Puppet
Forge or your local package manager, create a new (empty) database for
PuppetDB, and verify that you can login via psql to this DB you
just created. Then just supply PuppetDB with the DB host, port, name,
and credentials you've just configured, and we'll take care of the
rest!
As mentioned above, if you're using the embedded database we recommend using 1G or more (to be on the safe side). If you are using an external database, then a decent rule-of-thumb is to allocate 128M base + 1M for each node in your infrastructure.
For more detailed RAM requirements, we recommend starting with the above rule-of-thumb and experimenting. The PuppetDB Web Console shows you real-time JVM memory usage; if you're constantly hovering around the maximum memory allocation, then it would be prudent to increase the memory allocation. If you rarely hit the maximum, then you could likely lower the memory allocation without incident.
In a nutshell, PuppetDB's RAM usage depends on several factors: how
many nodes you have, how many resources you're managing with Puppet,
and how often those nodes check-in (your runinterval). 1000 nodes
that check in once a day will require much less memory than if they
check in every 30 minutes.
The good news is that if you under-provision memory for PuppetDB,
you'll see OutOfMemoryError exceptions. However, nothing terrible
should happen; you can simply restart PuppetDB with a larger memory
allocation and it'll pick up where it left off (any requests
successfully queued up in PuppetDB will get processed).
So essentially, there's not a slam-dunk RAM allocation that will work for everyone. But we recommend starting with the rule-of-thumb and dialing things up or down as necessary, and based on observation of the running system.
For truly large installations, we recommend terminating SSL using Apache or Nginx instead of within PuppetDB itself. This permits much greater flexibility and control over bandwidth and clients.
If installing from a distribution maintained package, such as those listed on the Downloading Puppet Wiki Page all OS prerequisites should be handled by your package manager. See the Wiki for information on how to enable repositories for your particular OS. Usually the latest stable version is available as a package. If you would like to do puppet development or see the latest versions, however, you will want to install from source.
While we recommend using pre-built packages for production use, it is occasionally handy to have a source-based installation:
$ mkdir -p ~/git && cd ~/git
$ git clone git://github.com/puppetlabs/puppetdb
$ cd puppetdb
$ rake install DESTDIR=/opt/puppetdb
You can replace /opt/puppetdb with a target installation prefix of
your choosing.
The Puppet terminuses can be installed from source by copying them into the
same directory as your Puppet installation. Using Ruby 1.8 on Linux (without
rvm), this is probably /usr/lib/ruby/1.8/puppet.
$ cp -R puppet/lib/puppet /usr/lib/ruby/1.8/puppet
While installing from source is useful for simply running a development version for testing, for development it's better to be able to run directly from source, without any installation step. This can be accomplished using leiningen, a Clojure build tool.
# Install leiningen
# Get the code!
$ mkdir -p ~/git && cd ~/git
$ git clone git://github.com/puppetlabs/puppetdb
$ cd puppetdb
# Download the dependencies
$ lein deps
# Start the server from source. A sample config is provided in the root of
# the repo, in config.sample.ini
$ lein run services -c /path/to/config.ini # or to a conf.d directory with ini fragments
From here you can make changes to the code, and trying them out is as easy as restarting the server.
Other useful commands:
-
lein testto run the test suite -
lein docsto build docs indocs/uberdoc.html
To use the Puppet module from source, add the Ruby code to $RUBYLIB.
$ export RUBYLIB=$RUBYLIB:`pwd`/puppet/lib
Restart the Puppet master each time changes are made to the Ruby code.
While a full discussion of how to install PostgreSQL is outside the scope of this document, you can get a local copy of PostgreSQL installed for testing fairly easily.
For example, if you're on a Mac you can install PostgreSQL easily through Homebrew:
$ brew install postgresql
# Now start the database
$ postgres -D /usr/local/var/postgres &
# Create a database
$ createdb puppetdb
On a Debian box you can install PostgreSQL locally like so:
$ apt-get install postgresql
# Switch to the postgres user
$ sudo -u postgres sh
# Create a new user and database for puppetdb
$ createuser -DRSP puppetdb
$ createdb -O puppetdb puppetdb
$ exit
# You can now login to the database via TCP with the credentials
# you just established
$ psql -h localhost puppetdb puppetdb
In order to talk to PuppetDB, Puppet must be configured to use the PuppetDB
terminuses. This can be achieved by installing the module on the Puppet master
and changing the storeconfigs_backend setting to puppetdb. The location of
the server can be specified using the $confdir/puppetdb.conf file. The basic
format of this file is:
[main]
server = puppetdb.example.com
port = 8081
If no config file is specified, or a value isn't supplied, the defaults will be used:
server = puppetdb
port = 8081
Additionally, you will need to specify a "routes" file, which is located by
default at $confdir/routes.yaml. The content should be:
master:
facts:
terminus: puppetdb
cache: yaml
This configuration tells Puppet to use PuppetDB as the authoritative source of fact information, which is what is necessary for inventory search to consult it.
PuppetDB can do full, verified HTTPS communication between Puppetmaster and itself. To set this up you need to complete a few steps:
-
Generate a keypair for your PuppetDB instance
-
Create a Java truststore containing the CA cert, so we can verify client certificates
-
Create a Java keystore containing the cert to advertise for HTTPS
-
Configure PuppetDB to use the files you just created for HTTPS
The following instructions will use the Puppet CA you've already got to create the keypair, but you can use any CA as long as you have PEM format keys and certificates.
For ease-of-explanation, let's assume that:
-
you'll be running PuppetDB on a host called
puppetdb.my.net -
your puppet installation uses the standard directory structure and its
ssldiris/etc/puppet/ssl
This is pretty easy with Puppet's built-in CA. On the host acting as your CA (your puppetmaster, most likely):
# puppet cert generate puppetdb.my.net
notice: puppetdb.my.net has a waiting certificate request
notice: Signed certificate request for puppetdb.my.net
notice: Removing file Puppet::SSL::CertificateRequest puppetdb.my.net at '/etc/puppet/ssl/ca/requests/puppetdb.my.net.pem'
notice: Removing file Puppet::SSL::CertificateRequest puppetdb.my.net at '/etc/puppet/ssl/certificate_requests/puppetdb.my.net.pem'
Et voilà, you've got a keypair. Copy the following files from your CA to the machine that will be running PuppetDB:
/etc/puppet/ssl/ca/ca_crt.pem/etc/puppet/ssl/private_keys/puppetdb.my.net.pem/etc/puppet/ssl/certs/puppetdb.my.net.pem
You can do that using scp:
# scp /etc/puppet/ssl/ca/ca_crt.pem puppetdb.my.net:/tmp/certs/ca_crt.pem
# scp /etc/puppet/ssl/private_keys/puppetdb.my.net.pem puppetdb.my.net:/tmp/certs/privkey.pem
# scp /etc/puppet/ssl/certs/puppetdb.my.net.pem puppetdb.my.net:/tmp/certs/pubkey.pem
The rest of the SSL setup occurs on the PuppetDB host; once you've copied the aforementioned files over, you don't need to remain logged in to your CA.
On the PuppetDB host, you'll need to use the JDK's keytool command
to import your CA's cert into a file format that PuppetDB can
understand.
First, change into the directory where you copied the generated certs
and the CA cert from the previous section. If you copied them to, say,
/tmp/certs:
# cd /tmp/certs
Now use keytool to create a truststore file. A truststore
contains the set of CA certs to use for validation.
# keytool -import -alias "My CA" -file ca_crt.pem -keystore truststore.jks
Enter keystore password:
Re-enter new password:
.
.
.
Trust this certificate? [no]: y
Certificate was added to keystore
Note that you must supply a password; remember the password you used, as you'll need it to configure PuppetDB later. Once imported, you can view your certificate:
# keytool -list -keystore truststore.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
my ca, Mar 30, 2012, trustedCertEntry,
Certificate fingerprint (MD5): 99:D3:28:6B:37:13:7A:A2:B8:73:75:4A:31:78:0B:68
Note the MD5 fingerprint; you can use it to verify this is the correct cert:
# openssl x509 -in ca_crt.pem -fingerprint -md5
MD5 Fingerprint=99:D3:28:6B:37:13:7A:A2:B8:73:75:4A:31:78:0B:68
Now we can take the keypair you generated for puppetdb.my.net and
import it into a Java keystore. A keystore file contains
certificate to use during HTTPS. Again, on the PuppetDB host in the
same directory you were using in the previous section:
# cat privkey.pem pubkey.pem > temp.pem
# openssl pkcs12 -export -in temp.pem -out puppetdb.p12 -name puppetdb.my.net
Enter Export Password:
Verifying - Enter Export Password:
# keytool -importkeystore -destkeystore keystore.jks -srckeystore puppetdb.p12 -srcstoretype PKCS12 -alias puppetdb.my.net
Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
You can validate this was correct:
# keytool -list -keystore keystore.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
puppetdb.my.net, Mar 30, 2012, PrivateKeyEntry,
Certificate fingerprint (MD5): 7E:2A:B4:4D:1E:6D:D1:70:A9:E7:20:0D:9D:41:F3:B9
# puppet cert fingerprint puppetdb.my.net --digest=md5
MD5 Fingerprint=7E:2A:B4:4D:1E:6D:D1:70:A9:E7:20:0D:9D:41:F3:B9
Take the truststore and keystore you generated in the preceding
steps and copy them to a directory of your choice. For the sake of
instruction, let's assume you've put them into /etc/puppetdb/ssl.
First, change ownership to whatever user you plan on running PuppetDB
as and ensure that only that user can read the files. For example, if
you have a specific puppetdb user:
# chown puppetdb:puppetdb /etc/puppetdb/ssl/truststore.jks /etc/puppetdb/ssl/keystore.jks
# chmod 400 /etc/puppetdb/ssl/truststore.jks /etc/puppetdb/ssl/keystore.jks
Now you can setup PuppetDB itself. In you PuppetDB configuration
file, make the [jetty] section look like so:
[jetty]
host = <hostname you wish to bind to for HTTP>
port = <port you wish to listen on for HTTP>
ssl-host = <hostname you wish to bind to for HTTPS>
ssl-port = <port you wish to listen on for HTTPS>
keystore = /etc/puppetdb/ssl/keystore.jks
truststore = /etc/puppetdb/ssl/truststore.jks
key-password = <pw you used when creating the keystore>
trust-password = <pw you used when creating the truststore>
If you don't want to do unsecured HTTP at all, then you can just leave
out the host and port declarations. But keep in mind that anyone
that wants to connect to PuppetDB for any purpose will need to be
prepared to give a valid client certificate (even for things like
viewing dashboards and such). A reasonable compromise could be to set
host to localhost, so that unsecured traffic is only allowed from
the local box. Then tunnels could be used to gain access to
administrative consoles.
That should do it; the next time you start PuppetDB, it will be doing HTTPS and using the CA's certificate for verifying clients.
With HTTPS properly setup, the PuppetDB host no longer requires the PEM files copied over during configuration. Those can be safely deleted from the PuppetDB box (the "master copies" exist on your CA host).
When using Puppet without a master (using Puppet apply), the configuration slightly differs. You will need to configure puppetdb.conf on every node, exactly as above.
The inventory service is not currently supported when using Puppet apply, due
to a limitation in Puppet. Further, the routes file $confdir/routes.yaml will
need to be configured to entirely disable facts storage, to prevent Puppet from
retrieving its facts from PuppetDB and therefore using stale facts:
apply:
facts:
terminus: facter
cache: facter
This will cause Puppet to use only Facter for retrieving facts, and not to store them anywhere.
If you are indeed entirely masterless, and thus don't have a central CA with
client certificate, you will be unable to use the embedded SSL support in
PuppetDB. This limitation is due to the fact that PuppetDB currently requires
client authentication, which is impossible when the client has no certificate.
Whitelist authorization is also unavailable in this mode. You will need to
disable the ssl/key/trust configuration options from jetty.ini and use an SSL
proxy which can be configured for optional or no client authentication.
Once you have PuppetDB running, visit the following URL on your PuppetDB host (what port to use depends on your configuration, as does whether you need to use HTTP or HTTPS):
/dashboard/index.html?pollingInterval=1000
PuppetDB includes a simple, web-based console that displays a fixed set of key metrics around PuppetDB operations: memory use, queue depth, command processing metrics, duplication rate, and REST endpoint stats.
We display min/max/median of each metric over a configurable duration, as well as an animated SVG sparkline.
Currently the only way to change the attributes of the dashboard is via URL parameters:
- width = width of each sparkline
- height = height of each sparkline
- nHistorical = how many historical data points to use in each sparkline
- pollingInterval = how often to poll PuppetDB for updates, in milliseconds
PuppetDB is configured using an INI-style file format. The format is the same that Puppet proper uses for much of its own configuration.
Here is an example configuration file:
[global]
vardir = /var/lib/puppetdb
logging-config = /var/lib/puppetdb/log4j.properties
resource-query-limit = 20000
[database]
classname = org.postgresql.Driver
subprotocol = postgresql
subname = //localhost:5432/puppetdb
[jetty]
port = 8080
You can specify either a single configuration file or a directory of .ini files. If you specify a directory (conf.d style) we'll merge all the .ini files together in alphabetical order.
If you've installed puppetdb from a package, the default is to use
the conf.d style, directory-based approach. The default config
directory is /etc/puppetdb/conf.d (or /etc/puppetlabs/puppetdb/conf.d
for Puppet Enterprise installations). If you're running from source
you may use the "-c" command-line argument to specify your config file
or directory.
There's not much to it, as you can see. Here's a more detailed breakdown of each available section:
[global] This section is used to configure application-wide behavior.
vardir
This setting is used as the parent directory for the MQ's data directory. Also, if a database isn't specified, the default database's files will be stored in /db. The directory must exist and be writable in order for the application to run.
logging-config
Full path to a log4j.properties file. Covering all the options available for configuring log4j is outside the scope of this document, but the aforementioned link has some exhaustive information.
For an example log4j.properties file, you can look at the ext
directory for versions we include in packages.
If this setting isn't provided, we default to logging at INFO level to standard out.
You can edit the logging configuration file after you've started PuppetDB, and those changes will automatically get picked up after a few seconds.
resource-query-limit
This setting defines a maximum number of legal results that a resource query can return. If you issue a query that would result in more results than this value, the query will simply return an error. (This can be used to prevent accidental queries that would yield huge numbers of results from consuming undesirable amounts of resources on the server.)
The default value is 20000.
[database]
gc-interval
How often, in minutes, to compact the database. The compaction process reclaims space, and deletes unnecessary rows. If not supplied, the default is every 60 minutes.
node-ttl-days
Auto-deactivate nodes that haven't seen any activity (no new catalogs,
facts, etc) in the indicated number of days. Nodes will be checked for
staleness every gc-interval minutes. Manual deactivation will
continue to work as always.
If unset, auto-deactivation of nodes is disabled.
classname, subprotocol, and subname
These are specific to the type of database you're using. We currently support 2 different configurations:
An embedded database (for proof-of-concept or extremely tiny installations), and PostgreSQL.
If no database information is supplied, an HSQLDB database at /db will be used.
Embedded database
The configuration must look like this:
classname = org.hsqldb.jdbcDriver
subprotocol = hsqldb
subname = file:/path/to/db;hsqldb.tx=mvcc;sql.syntax_pgs=true
Replace /path/to/db with a filesystem location in which you'd like
to persist the database.
PostgreSQL
The classname and subprotocol must look like this:
classname = org.postgresql.Driver
subprotocol = postgresql
subname = //host:port/database
Replace host with the hostname on which the database is
running. Replace port with the port on which PostgreSQL is
listening. Replace database with the name of the database you've
created for use with PuppetDB.
It's possible to use SSL to protect connections to the database. The
PostgreSQL JDBC docs
indicate how to do this. Be sure to add ssl=true to the subname
parameter.
Other properties you can set:
username
What username to use when connecting.
password
A password to use when connecting.
log-slow-statements
The number of seconds that any individual SQL query may run before it is considered "slow" and is logged as a warning. Note that this does not actually interrupt the query in any way; it simply checks queries after they complete and logs them if their duration exceeded this setting. The default value is 10 seconds. You may set this setting to zero to disable this feature.
conn-max-age
How many minutes a database connection can be idle before we terminate the connection. We default to 60 minutes.
conn-keep-alive
How often, in minutes, should we send a keep-alive query to the database. We default to every 240 minutes.
[command-processing]
The following options relate to the command-processing subsystem. Every change to PuppetDB's data stores comes in via commands that are inserted into a message queue (MQ). Command processor threads pull items off of that queue, persisting those changes.
threads
How many command processing threads to use. Each thread can process a single command at a time. If you notice your MQ depth is rising and you've got CPU to spare, increasing the number of threads may help churn through the backlog faster.
If you are saturating your CPU, we recommend lowering the number of threads. This prevents other PuppetDB subsystems (such as the web server, or the MQ itself) from being starved of resources, and can actually increase throughput.
This setting defaults to half the number of cores in your system.
store-usage
The maximum amount of disk space (in MB) to allow for ActiveMQ permanent message
storage. This corresponds to the SystemUsage / StoreUsage setting in
ActiveMQ (for reference, see: http://activemq.apache.org/producer-flow-control.html).
ActiveMQ's default value for this setting is 102400 (100GB). Assuming that you have plenty of disk space available, you shouldn't need to modify this setting (and PuppetDB typically won't use anywhere near that much disk space); however, if you are getting warnings about available disk space you can lower this value to get rid of them.
temp-usage
The maximum amount of disk space (in MB) to allow for ActiveMQ temporary message
storage. This corresponds to the SystemUsage / TempUsage setting in
ActiveMQ (for reference, see: http://activemq.apache.org/producer-flow-control.html).
ActiveMQ's default value for this setting is 51200 (50GB). Assuming that you have plenty of disk space available, you shouldn't need to modify this setting (and PuppetDB typically won't use anywhere near that much disk space); however, if you are getting warnings about available disk space you can lower this value to get rid of them.
[jetty]
HTTP configuration options.
host
The hostname to listen on for unencrypted HTTP traffic. If not supplied, we bind to localhost.
port
What port to use for unencrypted HTTP traffic. If not supplied, we won't listen for unencrypted traffic at all.
ssl-host
The hostname to listen on for HTTPS. If not supplied, we bind to localhost.
ssl-port
What port to use for encrypted HTTPS traffic. If not supplied, we won't listen for encrypted traffic at all.
keystore
The path to a Java keystore file containing the key and certificate we should use for HTTPS.
key-password
Passphrase to use to unlock the keystore file.
truststore
The path to a Java keystore file containing the CA certificate(s) for your puppet infrastructure.
trust-password
Passphrase to use to unlock the truststore file.
certificate-whitelist
Optional. Path to a file that contains a list of hostnames, one per line. Incoming HTTPS requests will have their certificates validated against this list of hostnames, and only those with an exact, matching entry will be allowed through.
If not supplied, we'll perform standard HTTPS without any additional authorization. We'll still make sure that all HTTPS clients supply valid, verifiable SSL client certificates.
[repl]
Enabling a remote (REPL)[http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop] allows you to manipulate the behavior of PuppetDB at runtime. This should only be done for debugging purposes, and is thus disabled by default. An example configuration stanza:
[repl]
enabled = true
type = nrepl
port = 8081
enabled
Set to true to enable the REPL.
type
Either nrepl or swank.
The nrepl repl type opens up a socket you can connect to via
telnet. If you are using emacs' clojure-mode, you can choose a type of
swank and connect your editor directly to a running PuppetDB
instance by using M-x slime-connect. Using emacs is much nicer than
using telnet. :)
port
What port to use for the REPL.
A Puppet Face action is provided to "deactivate" nodes. Deactivating the node will cause it to be excluded from storeconfigs queries, and it useful if a node no longer exists. The node's data is still preserved, however, and the node will be reactivated if a new catalog or facts are received for it.
puppet node deactivate <node> [<node> ...] --mode master
This command will submit deactivation commands to PuppetDB for each of the nodes provided. It's necessary to run this in master mode so that it can be sure to find the right puppetdb.conf file.
Note that puppet node destroy can also be used to deactivate nodes,
as the current behavior of destroy in PuppetDB is to simply
deactivate. However, this behavior may change in future, and the
command is not specific to PuppetDB, so the preferred method is
puppet node deactivate.
If you'd like PuppetDB to automatically deactivate nodes, refer to the
node-ttl-days configuration variable.
If you've recently changed the certificates in use by the PuppetDB server, you'll need to update the SSL configuration for PuppetDB itself.
If you've installed PuppetDB from Puppet Labs packages, you can simply
re-run the puppetdb-ssl-setup script.
Otherwise, you'll need to perform again all the steps in the SSL Configuration section of this document.
If you have configured your PuppetDB instance to start up a remote REPL, you can connect to it and begin issuing low-level debugging commands.
For example, let's say that you'd like to perform some emergency database compaction, and you've got an nrepl type REPL configured on port 8082:
$ telnet localhost 8082
Connected to localhost.
Escape character is '^]'.
;; Clojure 1.3.0
user=> (+ 1 2 3)
6
At this point, you're at an interactive terminal that allows you to manipulate the running PuppetDB instance. This is really a developer-oriented feature; you have to know both Clojure and the PuppetDB codebase to make full use of the REPL.
To compact the database, you can just execute the function in the PuppetDB codebase that performs garbage collection:
user=> (use 'com.puppetlabs.puppetdb.cli.services)
nil
user=> (use 'com.puppetlabs.puppetdb.scf.storage)
nil
user=> (use 'clojure.java.jdbc)
nil
user=> (with-connection (:database configuration)
(garbage-collect!))
(0)
You can also manipulate the running PuppetDB instance by redefining
functions on-the-fly. Let's say that for debugging purposes, you'd
like to log every time a catalog is deleted. You can just redefine
the existing delete-catalog! function dynamically:
user=> (ns com.puppetlabs.puppetdb.scf.storage)
nil
com.puppetlabs.puppetdb.scf.storage=>
(def original-delete-catalog! delete-catalog!)
#'com.puppetlabs.puppetdb.scf.storage/original-delete-catalog!
com.puppetlabs.puppetdb.scf.storage=>
(defn delete-catalog!
[catalog-hash]
(log/info (str "Deleting catalog " catalog-hash))
(original-delete-catalog! catalog-hash))
#'com.puppetlabs.puppetdb.scf.storage/delete-catalog!
Now any time that function is called, you'll see a message logged.
Note that any changes you make to the running system are transient; they don't persist between restarts. As such, this is really meant to be used as a development aid, or as a way of introspecting a running system for troubleshooting purposes.
YourKit has given us an open source license for their profiler, greatly simplifying the profiling of PuppetDB's performance.
YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: