vRealize Automation & Orchestrator API ODATA Filters

This is following on from my recent blog as I often hear people are confused about ODATA filters and specifically, what filters are available to use in both Orchestrator and the vRealize Automation API

ODATA is a common feature used in API calls to apply filter queries to obtain a subset of information based on the query passed in to the REST API call. ODATA filters can be used in both vRA REST calls and vRO workflows. This blog explores a couple of different ways to interact with the vRA REST API and vRO workflows to help minimize the data set returned in the response. Before we look in more detail on how to construct these API calls, lets go through some basic ODATA expressions:

Select a range of values

filter=Entry_No gt 610 and Entry_No lt 615


filter=Country_Region_Code eq 'ES' and Payment_Terms_Code eq '14 DAYS'


filter= Country_Region_Code eq 'ES' or Country_Region_Code eq 'US'

Less than

filter=Entry_No lt 610




filter=startswith(Name, 'S')

These are some of the most common parameters used, but it doesn’t stop there.

Additionally, it is import to append the pagesize and limit to you ODATA filter to limit the dataset returned.


The page size is important when expecting a large dataset response. Without providing a pagesize or a limit size, the defaults will be used, which might mean the dataset does not include the information you are expecting.

It is also worth noting that the filter expressions are discussed in the API docs found on the vRealize Automation 7.1+ appliance. To access the docs, goto the following URL on a vRealize Automation 7.1+ appliance


With the above information in mind, let’s start looking at some basic ODATA filter expressions that can be used.

I am using POSTMAN as a REST client to interact with the vRA API.

Take this URI example:

I only have a few catalog resources but see the following response:

"metadata": {
    "size": 20,
    "totalElements": 29",
    "totalPages": 2,
    "number": 2,
    "offset": 20

This shows that there is a total of 29 elements in the response, 2 pages and a size of 20. This means that the JSON response has 2 pages and the 1st page list the first 20 elements. You would need to go to page 2 to get the other 9 elements. If using POSTMAN, you can see the HREF element that shows you what pagesize and limit defaults are being used.


By changing the pagesize and limit, you can control how many elements are returned, or you can search through each page to find the element that you want.

To limit the response to 40 items instead of the default 20 in this case

or to get to page 2,

However, even though it is important to understand the pagesize and limit used, it is not an efficient way to query resources using the API.  Many times, the query is set with a large limit and a pagesize of 1 to accommodate all resources, but it’s not an efficient way to find resources. What happens if you have 000’s of resources or you have more resources than what your limit is set to? Here is where you can use ODATA filters and use filter expressions to minimize the result set.

So how do we do this? One way is to read the API docs that come with vRealize Automation and it is important to read through how ODATA filters work.

A very simple example is to add a query expression that targets a particular name or ID. For example:
$filter=id eq '21da4606-ebfa-440e-9945-6ef9102c6292'

If you look at the metadata, you can see we have only one element returned:

"metadata": {

    "size": 20,
    "totalElements": 1,
    "totalPages": 1,
    "number": 1,
    "offset": 0


This is indeed the element that has the ID equal to 21da4606-ebfa-440e-9945-6ef9102c6292.

Just breaking down the filter expression, here is how it worked:

$filter=id eq '21da4606-ebfa-440e-9945-6ef9102c6292'

The $filter is used and is set to equal an expression. In this case id eq ”

You can append the pagesize and limit with a filter. For example:

?page=1&limit=1&$filter=id eq ''

Notice the ? used to append a query to the URI, the & sign to have multiple filters and the $filter value which is the beginning of the ODATA filter query.

We can do additional searches too. /catalog-service/api/consumer/resources/
?$filter=id eq '21da4606-ebfa-440e-9945-6ef9102c6292' 
or id eq 'f89877f9-7c86-43e1-9956-9cefe22f127e'

Just checking the metadata, we can see the following

  "metadata": {

    "size": 20,
    "totalElements": 2,
    "totalPages": 1,
    "number": 1,
    "offset": 0


That matches our query as we see 2 elements returned.

If we supply an invalid ID, it simply doesn’t return an element.

This is all straight forward so far, so how do we do more complex ODATA filters? Well, there are a couple of ways to see how filters are used within vRealize Automation that will help you understand how to append ODATA filters to your API calls.

The first way is to look at the access_lot.txt log file that resides in /var/log/vcac/. This log file shows you all the API calls that are being made either by an end user or the vRealize Automation system. Looking at this log file can help you understand more complex ODATA filters.

You can simply use the UI and execute a search, which logs the ODATA filter used in the UI when searching in the access_log.txt  . A good example of this is searching for catalogue items as we have a nice advanced search box that we can use to trigger a ODATA search.

Screen Shot 2017-04-09 at 19.21.30

Searching the access_log.txt log file, I can see the following API call which show how the search made via UI translates in to a more complex ODATA query.


The URL has been encoded but we can simply decode this using an online decoder to give you the format that you require:

asc&$filter=substringof('docker - coreos',tolower(name)) and 
substringof('sample',tolower(description)) and outputResourceType/id 
eq 'composition.resource.type.deployment' and 
providerBinding/provider/id eq '5a4f1ed5-7a6c-4000-8263-5cbbcc888e84'
and status eq 'PUBLISHED' and organization/tenant ne null

Some interesting filters have been used, specifically new filters we haven’t discussed, for example  substringof(‘docker – coreos’,tolower(name)).

Substringof returns name records for catalog items with names containing the string ‘docker – coreos’

We could also use this :


This returns 2 results as expected, as we have 2 OOTB catalogue items that have a name that starts with Docker.

You can also do precedence grouping which enables you to override the precedence conventions to manipulate your dataset based on the precedence grouping defined.

('docker - coreos',tolower(name)) and 

You can list all virtual machines that were created between 2 different dates:

gt '2017-01-01T00:0:00' and dateCreated le ‘17-01-02T00:0:00')

This will display all virtual machine created between ‘2017-01-01T00:0:00’ and ‘2017-01-02T00:0:00’

You can actually append a filter to this precedence and say all virtual machine that are active.

gt '2017-03-29T11:42:00' and dateCreated le '2017-03-29T11:50:00') 
and status eq 'ACTIVE'

To be safe, you could make the active status to uppercase, just to ensure any parameters programmatically set are capitalized

catalog-service/api/consumer/resources/?$filter=(dateCreated gt 
'2017-03-29T11:42:00' and dateCreated le '2017-03-29T11:50:00') 
and status eq toupper('active')

Additionally, you can use the orderby function to sort your queries too.


But how do we know what parameters can be passed in using the ODATA filter?

Well, this is a bit of trial and error. You can use the JSON response elements and using the key element and see whether you get results you are expecting that means the expression is working successfully. So take this JSON response payload:


      "@type": "CatalogResource",
      "id": "21da4606-ebfa-440e-9945-6ef9102c6292",
      "iconId": "cafe_default_icon_genericCatalogItem",
      "resourceTypeRef": {
        "id": "composition.resource.type.deployment",
        "label": "Deployment"
      "name": "vm-app-01-88805337",
      "description": null,
      "status": "ACTIVE",
      "catalogItem": {
        "id": "8e04dceb-f040-4bbc-b312-8a1b0d0a3b95",
        "label": "VM app 01"
      "requestId": "3f49e42a-5e4e-4293-b3f0-eff2a34a5108",
      "providerBinding": {
        "bindingId": "3ee7fe9f-197c-4fac-8e59-cea6ac4f2336",
        "providerRef": {
          "id": "5a4f1ed5-7a6c-4000-8263-5cbbcc888e84",
          "label": "Blueprint Service"
      "owners": [
          "tenantName": "vsphere.local",
          "ref": "",
          "type": "USER",
          "value": "Oliver Leach"
      "organization": {
        "tenantRef": "vsphere.local",
        "tenantLabel": "vsphere.local",
        "subtenantRef": "8723d842-d6f0-48c0-b0a3-e555eaeecdd5",
        "subtenantLabel": "Corp"
      "dateCreated": "2017-01-00T00:00:00.000Z",
      "lastUpdated": "2017-01-00T00:00:00.000Z",
      "hasLease": true,
      "lease": {
        "start": "2017-01-00T00:00:00.000Z"
      "leaseForDisplay": null,
      "hasCosts": true,
      "totalCost": null,
      "hasChildren": false,
      "operations": null,
      "forms": {
        "catalogResourceInfoHidden": false,
        "details": {
          "type": "external",
          "formId": "composition.deployment.details"
      "resourceData": {
        "entries": []
      "destroyDate": null,
      "pendingRequests": []

We can test setting the expression filter to use any of the above JSON payload elements, although there are ones that won’t work and other caveats you need to follow.

For example, take the request ID. This should look like ?$filter=requestId, however that doesn’t work. Trial and error lead me to use ?$filter=request as shown here:

eq '3f49e42a-5e4e-4293-b3f0-eff2a34a5108'

Additionally, in the vRA API docs, there are some caveats to be aware of. See this notes in the API docs.

Note: Notice how even though the providerRef object contains a label element, we use provider (and not providerRef) and name (and not label) to craft the filter:

$filter=provider/name eq 'theName'
      "providerBinding": {
        "bindingId": "cd1c0468-cc1a-404d-a2e0-a4e78fc06d4d",
        "providerRef": {
          "id": "2575e506-acfe-487a-b080-9898a30f519f",
          "label": "XaaS"

Therefore, in order to get all providers that are of type XaaS, we need to run the following ODATA filter

providerBinding/provider/name eq ‘XaaS’

Notice provider is missing the Ref and the label is now the name. However, you can just use id too, for example:

providerBinding/provider/id eq 

In order to get to an element that is an array, for example using the owner information in above JSON payload, you can simply do this:

eq ''

You can also use firebug or chrome dev tools to see what API calls are being made via the UI and intercept the API call and look at the filters being used. For example, let’s redo the search we did above to see catalogue item entitlements and sniff the request using the firebug add-on in Firefox.

Here you can see the output of firebug:

Screen Shot 2017-04-08 at 12.34.14

The request URL will give you the UTF-8 encoded format and you can see also the ODATA filters used.

Additionally, you can also use ODATA filters in vRealize Orchestrator and these principles still apply.

Take this code example:

service = cafeHost.createCatalogClient().getCatalogConsumerResourceService();

var filter = new Array();

filter[0] = vCACCAFEFilterParam.substringOf("providerBinding/provider/id",
var query = vCACCAFEOdataQuery.query().addFilter(filter);
var odataRequest = new vCACCAFEPageOdataRequest(1, 10000, query);
resources = service.getResourcesList(odataRequest);

You can add you filter parameters in the same was as you can using the API ODATA filters.

To add additional parameters, you can build up the filter array with your queries.

Remember to consider the caveats I have been talking about above but by using the API JSON response, you can determine what filters to use. Please add comments if you find any caveats regarding ODATA filters not mentioned above.

vRealize Automation – Changing a Virtual Machine’s Business Group in 7.2 and 6.x

Changing business groups in both vRealize Automation v6 and v7 is a common task and below demonstrates how to automate the process. Luckily, VMware provides an out of the box vRealize Orchestrator workflow solution in both vRealize 6.x and 7.2, however, the workflow you need to use in both versions is different. Let’s have a look:

vRealize Automation 7.2

As of v7.2, we have a new workflow that changes the business group of the IaaS virtual machine entity as well as the CAFE deployment entity. The virtual machine entity is still part of the IaaS model, however, the deployment is a CAFE entity and therefore there are a few things you need to be aware of before changing the business group. We’ll cover those later. However, here is a screenshot of the workflow that you need to use:

The workflow you need to run to change business group can be found in the following location, noting this is under the Library folder:


You need to provide the following inputs:

Screen Shot 2017-01-28 at 16.56.39.png

There are a few things to be considered.

1 – The owner of the virtual machine must also be a member of the target business group. The business group is determined based on the reservation you select. If the user is not a member of the target business group, the virtual machine IaaS entity business group will get updated, however, the deployment CAFE entity business group entity will not get updated, leaving you in a position where the 2 entities that are out of sync. If this is the case, your my items will be out of sync as the deployment will still appear under the original business group but the virtual machine will appear under the new business group.

2 – You may have more than one Virtual Machine within the deployment so make sure you update all virtual machine entities within the deployment.

3 – This workflow is not available prior to version vRealize Automation 7.2, although if you need it, contact VMware GSS support as they may be able to help you get your hands on a fix. Or, better still upgrade to vRealize 7.2.

Based on the above, one way I would implement this solution would be to create a day 2 operation on the deployment CAFE entity. Then you can do a couple of things as follows:

1 – Check that the owner of the deployment is a member of the target business group. If not, flag this on the day 2 XaaS form.

2 – Get all virtual machines that are part of the deployment so you update all virtual machines which are part of the deployment.

That’s it but you can enhance the custom day 2 resource operation if you need to.

vRealize Automation 6.x

In my opinion, it is slightly easier to update the virtual machine business group as we don’t have to worry about the CAFE deployment as the entity doesn’t exist in vRA 6.x. We do have MMBP, but this is part of the IaaS model which can be updated. This post just concentrates on single machine blueprints. The workflow you need to run to change business groups can be found in the following location, noting this is under the Library folder:

Screen Shot 2017-01-28 at 17.12.29.png

You can just run the import vCAC Virtual Machine and you can rerun this over an over again. It won’t register the machine multiple times but just update the business group based on the following inputs:


You can see there are few more options to supply in vRA 6.x. You can also make this a day 2 operation by wrapping this workflow around another workflow as you can see the actual code that changes the business group is this:


Where vm is a vCAC:VirtualMachine type. You can see this in the ‘start register workflow‘ scripting element.

You can see the requirements parameters using the API library, but here’s a screenshot of what you need:


That’s pretty much it. Happy Business Group Automating.

vRealize Automation 7. How to get Deployment Name and Virtual machines from a deployment

In vRealize Automation 7, we have a deployment now that contains all the resources. This is different to how it worked previously. But how do we get information about the deployment, so in a case that we need to update the owner of a deployment if we have deployed a composite blueprint using XaaS?

Here is  snippet of code I use to determine what is what in the deployment.


The key is once you have found the catalogue resource, this line determines whether it is the deployment or virtual machine.

var catResources = vCACCAFEEntitiesFinder.findCatalogResources(cafeHost);
var resourceName = "<deployment name>;
for each (resource in catReosurces){
    if (resource.resourceTypeRef.getLabel() == "Deployment")) +
    resource.getName() == resourceName;
        catalogResource = resource;

The getLabel() tells you whether it is a virtual machine or whether its the deployment. Now , once you have the catalogue resource, you can get the deployment and execute a change owner on the deployment.

var operations = catalogResource.getOperations();
var operationName = 'Change Owner';

for each(op in operations){
    System.log( + ": " + op.getName() +
    " operation is available on this catalog resource");
    if (op.getName() = operationName){
        System.log( + ": " + op.getName() +
        " operation identified and selected");
        changeOwnerOperation = op;

Now we have the operation of on the deployment, we can pass that in to the requestResourceAction in vRealize Orchestrator and submit the change owner process.

The inputs to the resource action requires the vCACCAFE:ConsumerResourceOperation and the payload for the new user. The payload is a property object. as follows:

ownerProperties = new Properties();
ownerProperties.put("provider-NewOwner", newOwner);

This helps go some way to changing the owner of a deployment. Great is you need to use XaaS to wrap deployments. In future releases, we won’t need to do this as we will have the ability to hide CBP from the catalogue. So we will be able to deploy a catalogue item using the correct owner but the owner won’t see the catalogue item as you can mark it hidden. Hence the logged in user won’t see the catalogue item, but still be entitled to request it.

Cloudstack deploying cloudstack


I was talking at the Cloudstack European Meet up the other day about getting Cloudstack to deploy Cloudstack.

Well its more than that – I was talking about the concepts rather than what I was doing, as really, Cloudstack deploying Cloudstack is a bit of a gimick.

Here is my github link to the project

However, I thought I would include some scribble on how the project works in case anyone is interested:

Cloudstack 4.0.1 deploy

This project enables a running Cloudstack instance 3.x and above to deploy Cloudstack 4.0.1 instances. This could be to demo Cloudstack 4.0.1 or used for testing purposes, by easily spinning up Cloudstack 4.0.1 instances. It could also be used to provision private clouds using Cloudstack.

How to use this project

The python script and puppet manifests in this repo deploys cloudstack 4.0.1 using Puppet and the Cloudstack API. For this to work, you need to use the python scripts and 2 puppet templates. One template is the puppet-master, and one template is the puppet-agent.

You need to configure the python parameters in the script as per your environment. This includes:

1 – your API key
2 – your secret key
3 – your service offering
4 – your Cloudstack zone ID
5 – your network ID of your user account
6 – The Cloudstack provisioning host IP and port
7 – The Cloudstack protocol used (http or https)

You will also need your 2 template IDs for the puppet master and puppet agent

These templates can be downloaded using the following link:

Puppet Agent

Puppet Master

Both templates are running CentOS 6.4 64-bit. The puppet master is configured as a puppet-ca, a puppet server and a puppet agent. The puppet agent has the puppet agent installed and runs a userdata.rb script on startup. See the rc.local file for further information.

Once the templates are unzipped and uploaded in to your provisioning Cloudstack instance, update the script with the template IDs.

To run the python scripts, use an interpreter like idle, open and run the file, or simply run from the command line where an instance of python is installed; e.g. python This has been tested against python 2.7.3.

The git repo holds the puppet manifests which are included in the puppet-master template.
They are just for reference.

This project is just for fun, could be better written and I hold no responsibility if you use the project. You are allowed to do so.

Puppet manifests for cloudstack and co.

Here are some of my puppet manifests for Cloudstack and co. You can also find more information here:

I also recently did a presentation at the Cloudstack European meetup in London on the 11th April about deploying applications using puppet and Cloudstack. The talk was well received so I thought I would jot down the manifests that I wrote as part of the talk.

Here is the Cloudstack manifest:

# Class cloudstack
# This class installs cloudstack
# Author: Oliver Leach
# Date : 18th March 2013
# Actions:
# Installs the cloudstakc package
# This includes, the cloudstack-client, mysql-server and any install commands
# Currently the clodustack class is monlithic for demo purposes.
# Requires:
# Centos - other operating systems are not supported by this manifest
# Sample Usage:
# include cloudstack

class cloudstack ($mysql_password) {

case $::operatingsystem {

'centos': {
$supported = true

if ($supported == true) {

yumrepo { 'cloudstack':
baseurl  => "",
descr    => "Cloudstack 4.0.1 yum repo",
enabled  => 1,
gpgcheck => 0,

package { "cloud-client":
ensure   => "installed",
require  => Yumrepo["cloudstack"],

exec { "cloud-setup" :
command  => "/usr/bin/cloud-setup-databases cloud:cloud@localhost --deploy-as=root:${mysql_password}; /bin/sleep 3 ; /usr/bin/cloud-setup-management ; /bin/sleep 3 ;
/usr/bin/cloud-setup-databases cloud:cloud@localhost --deploy-as=root:${mysql_password} ; /bin/sleep 3 ; /usr/bin/cloud-setup-management",

creates  => '/var/lib/mysql/cloud',
require  => Package["cloud-client"],

service { "cloud-management":
ensure   => "running",
enable   => "true",
require  => Exec["cloud-setup"],

MySQL server manifest

# Class myslqserver
# This class installs cloudstack
# Author: Oliver Leach
# Date : 20th March 2013
# Actions:
# Installs the mysql server package
# This includes, the mysql-server package
# Requires:
# Centos - other operating systems are not supported by this manifest
# Sample Usage:
# include mysql::server

class mysql::server ($mysql_password) {

case $::operatingsystem {
"centos": {
$supported = true

if ($supported == true) {

file { "flag-files" :

ensure     => directory,
path       => "/var/lib/puppet/flags/",
owner      => root,
group      => root,
mode       => "0644",

package { "mysql-server":
ensure     => installed,
require    => File["flag-files"],

file { "/etc/my.cnf" :
owner      => "root",
group      => "root",
source     => "puppet:///modules/mysql/my.cnf",
require    => Package["mysql-server"],

service { "mysqld" :
enable     => true,
ensure     => running,
require    => File["/etc/my.cnf"],

$mysql_server_exec_1 => "0000000000001"
$mysql_server_file_1 => "/var/lib/puppet/flags/mysql_server_exec_1"

exec { "restart-mysqld" :
command    => "/sbin/service mysqld restart ; echo \"$mysql_server_exec_1\" ; \"$mysql_server_file_1\"",
unless     => "/usr/bin/test \"`/bin/cat $mysql_server_file_1; /dev/null`\" = \"$mysql_server_exec_1\"",
require    => Service["mysqld"],

exec { "set-mysql-password" :
unless     => "/usr/bin/mysqladmin -uroot -p${mysql_password} status",
command    => "/usr/bin/mysqladmin -uroot password ${mysql_password}",
require    => Exec["restart-mysqld"],

$mysql_server_exec_2 => "0000000000001"
$mysql_server_file_2 => "/var/lib/puppet/flags/mysql_server_exec_2"

exec { "remove-test-dbs" :
command    => "/usr/bin/mysql -u root -p${mysql_password} -e \"DELETE FROM mysql.db WHERE Db='test' OR Db='test%';\"; echo \"$mysql_server_exec_2\" >> "$mysql_server_file_2\"",

unless     => "/usr/bin/test \"`/bin/cat $mysql_server_file_2 /dev/null`\" = \"$mysql_server_exec_2\"",
require    => Exec["set-mysql-password"],

$mysql_server_exec_3 => "0000000000003"
$mysql_server_file_3 => "/var/lib/puppet/flags/mysql_server_exec_3"

exec { "remove-remote-root-access" :
command     => "/usr/bin/mysql -u root -p${mysql_password} -e \"DELETE FROM mysql.user WHERE User='root' AND Host NOT IN ('localhost', '', '::1'); \" echo \"$mysql_server_exec_3\"; \ "$mysql_server_file_3\"",
unless      => "/usr/bin/test \"`/bin/cat $mysql_server_file_3 /dev/null`\" = \"$mysql_server_exec_3\"",
require     => Exec["set-mysql-password"],

Apache manifest

class apache {

$snat_ip   = $snat_ipaddress

case $operatingsystem {
'CentOS', 'RedHat' : {
$supported = true

if ($supported == true) {

file { 'httpd_conf_d' :
ensure     => directory,
path       => '/etc/httpd/conf.d/',
recurse    => true,
purge      => true,
force      => true,
owner      => "root",
group      => "root",
mode       => 0644,
require    => Package['httpd'],

file { 'cloudstack_conf' :

path       => '/etc/httpd/conf.d/cloudstack.conf',
owner      => root,
group      => root,
mode       => '0644',
content    => template('apache/cloudstack.conf.erb'),
require    => File['httpd_conf_d'],

package { 'httpd' :
ensure     => present,

package { 'mod_ssl' :
ensure     => present,

service { 'httpd' :
ensure     => 'running',
enable     => true,
require    => File['cloudstack_conf'],

and the ERB template:

LoadModule ssl_module modules/

NameVirtualHost *:80

DocumentRoot /var/www/html/
Redirect permanent / https:///client

Listen 443
NameVirtualHost *:443

SSLEngine On
#SSLCertificateFile /var/lib/puppet/ssl/certs/ca.pem
#SSLCertificateKeyFile /var/lib/puppet/ssl/certs/cert001.pem

SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

RewriteEngine On
ProxyRequests On
ProxyVia On

Order deny,allow
Deny from all
Allow from all

Redirect permanent / /client

ProxyPass /client http://localhost:8080/client
ProxyPassReverse /client http://localhost:8080/client

And the iptables manifest

class iptables {

$ipv4_file        => $operatingsystem ? {
"debian"          => '/etc/iptables/rules.v4',
/(RedHat|CentOS)/ => '/etc/sysconfig/iptables',

exec { "purge default firewall":
command     => "/sbin/iptables -F &amp;&amp; /sbin/iptables-save &gt;

$ipv4_file /sbin/service iptables restart",
onlyif      => "/usr/bin/test `/bin/grep \"Firewall configuration

written by\" $ipv4_file | /usr/bin/wc -l` -gt 0",
user        => 'root',

/* Make the firewall persistent */
exec { "persist-firewall":
command     => "/bin/echo \"# This file is managed by puppet. Do not modify manually.\" $ipv4_file /sbin/iptables-save $ipv4_file",
refreshonly => true,
user        => 'root',

/* purge anything not managed by puppet */
resources { 'firewall':
purge      => true,

firewall { '000 INPUT allow related and established':
state      => ['RELATED', 'ESTABLISHED'],
action     => 'accept',
proto      => 'all',

firewall { "001 accept all icmp requests":
proto      => 'icmp',
action     => 'accept',

firewall { '002 INPUT allow loopback':
iniface    => 'lo',
chain      => 'INPUT',
action     => 'accept',

firewall { '100 allow ssh':
state      => ['NEW'],
dport      => '22',
proto      => 'tcp',
action     => 'accept',
firewall { '100 allow port 80':
state      => ['NEW'],
dport      => '80',
proto      => 'tcp',
action     => 'accept',
firewall { '100 allow port 443':
state      => ['NEW'],
dport      => '443',
proto      => 'tcp',
action     => 'accept',
firewall { '100 allow 9090':
state      => ['NEW'],
dport      => '9090',
proto      => 'tcp',
action     => 'accept',

firewall { '100 allow 8250':
state      => ['NEW'],
dport      => '8250',
proto      => 'tcp',
action     => 'accept',

firewall { '100 allow 7080':
state      => ['NEW'],
dport      => '7080',
proto      => 'tcp',
action     => 'accept',

firewall { '100 allow 8080':
state      => ['NEW'],
dport      => '8080',
proto      => 'tcp',
action     => 'accept',

firewall { "998 deny all other requests":
action     => 'reject',
proto      => 'all',
reject     => 'icmp-host-prohibited',

firewall { "999 deny all other requests":
chain     => 'FORWARD',
action    => 'reject',
proto     => 'all',
reject    => 'icmp-host-prohibited',

Running multiple puppet masters


When running multiple puppet masters, you need to dedicate a single puppet master to be your CA server. If you are adding additional puppet master, there is some key config that needs to be set which disables the puppet CA functionality from running on the puppet master. Additionally, you need to set up a proxy pass to forward certificate requests to the puppet CA server. This is detailed below.

1. Install puppet agent and puppet master

Install the puppet and latest epel repo – note the epel repo below is now out of date.

rpm -Uvh
rpm -Uvh

Install the puppet agent and puppet master

yum install puppet puppet-server -y

2. Edit the puppet.conf file and turn off the puppet master from acting as a CA

vi /etc/puppet/puppet.conf

ca = false
ca_server =

3. Start the agent and create and sign the puppet master cert

• On the additional puppet master

puppet agent --test

• On puppet CA server

puppet cert sign --all

• On the puppet master

puppet agent --test

4. Configure iptables

iptables -I INPUT 5 -s -m tcp -p tcp --dport 8140 -j ACCEPT

service iptables save
service iptables restart

Note: If you are adding multiple bridges, assign the correct iptables rules to each vnic adapter and the the applicable source network address.

5. Install the required packages for the puppet master

yum install sudo mod_ssl rubygem-passenger 
mod_passenger policycoreutils-python rsync -y

6. Copy the example puppet virtual host config to /etc/httpd/conf.d/

cp /usr/share/puppet/ext/rack/files/apache2.conf 

7. Edit the puppet-master.conf file and update accordingly

# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off

Listen 8140

SSLProxyEngine On
ProxyPassMatch ^/([^/]+/certificate.*)$$1

SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1

SSLCertificateFile /var/lib/puppet/ssl/certs/
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem

# If Apache complains about invalid signatures on the CRL, you can try disabling
# CRL checking by commenting the next line, but this is not recommended.
SSLCARevocationFile /var/lib/puppet/ssl/crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1

# The `ExportCertData` option is needed for agent certificate expiration warnings
SSLOptions +StdEnvVars +ExportCertData

# This header needs to be set if using a loadbalancer or proxy
RequestHeader unset X-Forwarded-For

RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

DocumentRoot /usr/share/puppet/rack/puppetmaster/public/
RackBaseURI /

Options None
AllowOverride None
Order allow,deny
allow from all

Note: We are using the ProxyPassMatch. This is matching any inbound requests on /certificate and redirecting them to the puppet CA server. Change the config accordingly.

Note: As the puppet master you are building is not a CA server, note the SSLCertificateChainFile, SSLCACertificateFile, SSLCARevocationFile paths are different to a puppet master running the CA server. This reflects where the CA certs reside when the PM is not a CA server itself.

8. Create rack directories

mkdir -p /usr/share/puppet/rack/puppetmaster/{public,tmp}

9. Copy rack file to rack web directory

cp /usr/share/puppet/ext/rack/files/ /usr/share/puppet/rack/puppetmaster/

10. Change ownership of rack file to puppet

chown puppet:puppet /usr/share/puppet/rack/puppetmaster/

11. Set httpd to start on boot and puppet master to not start

chkconfig httpd on
chkconfig puppetmaster off

12. Set up a puppet master to connect to puppetdb

• Run the following on each of your puppet masters:

sudo puppet resource package puppetdb-terminus ensure=latest

• Add this to /etc/puppet/puppetdb.conf. Note: you may have to create this file.


server =
port = 8081

• Add this to /etc/puppet/puppet.conf


storeconfigs = true
storeconfigs_backend = puppetdb

• Add this to /etc/puppet/routes.yaml. Note: you may have to create this file.

terminus: puppetdb
cache: yaml

13. Enable puppet-dashboard reports on the puppet master

• Add to each puppet master in the puppet environment in /etc/puppet/puppet.conf:


reports = store, https, puppet_dashboard
reporturl = https:///reports/upload

14 Copy the following script to the puppet dashboard server and all the puppet masters

• Create /usr/lib/ruby/site_ruby/1.8/puppet/reports/https.rb with the following code

require 'puppet'
require 'net/http'
require 'net/https'
require 'uri'

Puppet::Reports.register_report(:https) do

desc &lt;&lt;-DESC
Send report information via HTTPS to the `reporturl`. Each host sends
its report as a YAML dump and this sends this YAML to a client via HTTPS POST.
The YAML is the `report` parameter of the request."

def process
url = URI.parse(Puppet[:reporturl].to_s)
http =, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE

req =
req.body = self.to_yaml
req.content_type = "application/x-yaml"

http.start do |http|
response = http.request(req)
unless response.code == "200"
Puppet.err "Unable to submit report to #{Puppet[:reporturl].to_s} [#{response.code}] #{response.msg}"


• Remove the following file from the puppet dashboard and puppet masters


rm -f /usr/lib/ruby/site_ruby/1.8/puppet/reports/http.rb

15 Edit the auth.conf file on each puppet master so inventory can pick up facts.

vim /etc/puppet/auth.conf

path /facts
method find
auth any
allow *

path /inventory
auth any
method search, find
allow dashboard

# this one is not strictly necessary, but it has the merit
# of showing the default policy, which is deny everything else

path /
auth any

Note: The config for /facts and /inventory must go above the config for `path /` – otherwise you may get an access forbidden 404 error message when running the inventory service on puppet-dashboard.

Installing puppetdb


It just makes sense to use the puppet database in your puppet environment. It is used to store the inventory facts about every node in your puppet environment and as this can be big, you need to have a good back end database which can handle this. Here is how I installed puppet DB

There are no special requirements for SELINUX when installing puppet DB. Keep SELINUX set to enforcing mode.

1. Install the puppet repo

rpm -Uvh
rpm -Uvh

2. Install the puppet agent and request a CSR

yum install puppet -y
puppet agent --test

3. Sign the puppet agents CSR on the puppet master

puppet cert --list
puppet cert sign --

4. Install puppet DB on the puppet DB server

puppet resource package puppetdb ensure=latest

5. Install postgres

Install instructions taken from

vi /etc/yum.repos.d/CentOS-Base.repo

To the [base] and [updates] sections in /etc/yum.repos.d/CentOS-Base.repo, you need to append a line


Download and install the rpm repos for postgres

<code>rpm -Uvh <a href=”; title=””></a&gt;

yum list postgres*
yum install postgresql91-server -y

6. Start the postgres service and run at boot

service postgresql-9.1 initdb
service postgresql-9.1 start
chkconfig postgresql-9.1 on

7. Configure postgres

vi /var/lib/pgsql/9.1/data/postgresql.conf

listen_addresses = '*'
log_line_prefix = '%t %u %d'

8. Create the puppet DB

sudo -u postgres sh
cd /var/lib/puppetdb
createuser -DRSP puppetdb
createdb -O puppetdb puppetdb

9. Allow host access to the postgresql pupetdb

vi /var/lib/pgsql/9.1/data/pg_hba.conf

local all all trust
host puppetdb puppetdb trust

• Comment out this line (near the end of pg_hba.conf)

#local all all peer

• Restart the postgres service

service postgresql-9.1 restart

• Test login

psql -h puppetdb puppetdb


psql -d puppetdb -U puppetdb -W

10. Configure puppdb conf.d/config.ini file

vi /etc/puppetdb/conf.d/config.ini

Below details an example config.ini file

# See for more thorough explanations of each section and
# option.


# Store mq/db data in a custom directory
vardir = /var/lib/puppetdb

# Use an external log4j config file

logging-config = /etc/puppetdb/conf.d/../

# Maximum number of results that a resource query may return

resource-query-limit = 20000


classname = org.postgresql.Driver
subprotocol = postgresql
subname = //localhost:5432/puppetdb
username = puppetdb
password =


port = 8080


# How many command-processing threads to use, defaults to (CPUs / 2)
# threads = 4

11. Start the puppetdb services

sudo puppet resource service puppetdb ensure=running enable=true

12. Open puppetdb port 8081 in iptables

Modify where necessary

iptables -I INPUT 5 -s -m tcp -p tcp --dport 8081 -j ACCEPT
iptables -I INPUT 6 -s -m tcp -p tcp --dport 8080 -j ACCEPT

service iptables save
service iptables restart

The 2nd firewall rule is used to access the puppetdb-dashboard

13. Set up a puppet master to connect to puppetdb

• Run the following on each of your puppet masters:

sudo puppet resource package puppetdb-terminus ensure=latest

• Add this to /etc/puppet/puppetdb.conf. Note: you may have to create this file.


server =
port = 8081

• Add this to /etc/puppet/puppet.conf


storeconfigs = true
storeconfigs_backend = puppetdb

• Add this to /etc/puppet/routes.yaml. Note: you may have to create this file.

terminus: puppetdb
cache: yaml

14. Restart the puppet service on each puppet master

service httpd restart

15. Check in an agent and monitor the puppetdb logs

On the puppetdb server, monitor the puppetdb log

tail -f /var/log/puppetdb/puppetdb.log

On an agent run the following:

puppet agent --test

16. (Optional) Open up access to the puppet DB dashboard


To access the puppetdb, you need to make some configuration changes. Edit the following file




host =

Access the puppet DB using the following URL, changing the host name accordingly

Troubleshooting issue:

If the puppet DB, port 8080 and 8081 are not listening when the puppetdb service is set to start and you see in /var/log/puppetdb/puppdb.log the following error:

2013-02-26 19:40:43,327 ERROR [main] [puppetlabs.utils] Uncaught exception /etc/puppetdb/ssl/keystore.jks (No such file or directory)

Run the following command and reboot the server

sudo /usr/sbin/puppetdb-ssl-setup

This will create a keystore and truststore in /etc/puppetdb/ssl and will print the password to both files in /etc/puppetdb/ssl/puppetdb_keystore_pw.txt.