Month: January 2013

Installing puppet dashboard

puppet-dash

Here are my notes in installing puppet dashboard

1. Install EPEL and puppet repos

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -Uvh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-6.noarch.rpm

2. Install required packages

yum install vim puppet-server puppet puppet-dashboard
sudo mod_ssl rubygem-passenger mod_passenger
policycoreutils-python mysql mysql-server -y

3. Generate a puppet client cert and sign on the puppet master

• Run the following:

puppet agent --test

• Sign the certificate on the puppet master

puppet cert --list
puppet cert sign

for example

puppet cert sign puppet-dashboard.myorg.net4.

4 Configure MYSQL and create dashboard database

4.1 Start the mysqld server and configure chkconfig

service mysqld start
chkconfig mysqld on

4.2 Run and secure mysql

/usr/bin/mysql_secure_installation

4.3 Log in to mysql and create a database and configure permissions / password

CREATE DATABASE puppetdash;
GRANT ALL PRIVILEGES ON puppetdash.* TO puppet@'%' IDENTIFIED BY '';
FLUSH PRIVILEGES;
set global max_allowed_packet = 33554432;

4.4 Update database entries for the production DB settings as follows:

vi /usr/share/puppet-dashboard/config/database.yml

production:
database: puppetdash
username: puppet
password:
encoding: utf8
adapter: mysql

4.5 Create dashboard schema

cd /usr/share/puppet-dashboard
rake RAILS_ENV=production db:migrate

5. Copy the example puppet virtual host config to /etc/httpd/conf.d/

cp /usr/share/puppet-dashboard/ext/passenger/dashboard-vhost.conf
/etc/httpd/conf.d/puppet-dashboard.conf

6. Remove the ssl.conf file /etc/httpd/conf.d/ directory.

Note, the mod_ssl.so is loaded in the custom puppet-dashboard.conf virtual host file.

rm -f /etc/httpd/conf.d/ssl.conf

7. Configure the puppet-dashboard virtual host file.

vim /etc/httpd/conf.d/puppet-dashboard.conf

# you may want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RailsAutoDetect On

RewriteEngine on
ReWriteCond %{SERVER_PORT} !^443$
RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [NC,R,L]

LoadModule ssl_module modules/mod_ssl.so

Listen 443

SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

SSLCertificateFile /usr/share/puppet-dashboard/certs/dashboard.cert.pem
SSLCertificateKeyFile /usr/share/puppet-dashboard/certs/dashboard.private_key.pem
SSLCACertificateFile /usr/share/puppet-dashboard/certs/dashboard.ca_cert.pem

# If Apache complains about invalid signatures on the CRL, you can try disabling
# CRL checking by commenting the next line, but this is not recommended.
SSLCARevocationFile /usr/share/puppet-dashboard/certs/dashboard.ca_crl.pem

SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars

ServerName ukcrlcscin0010.tcl-oob.net
DocumentRoot /usr/share/puppet-dashboard/public/

Options None
Order allow,deny
allow from all

ErrorLog /var/log/httpd/puppet-dashboard-net_error.log
LogLevel warn
CustomLog /var/log/httpd/puppet-dashboard-net_access.log combined
ServerSignature On

Order deny,allow
Allow from ALL
# Enable this to require client-side certificates for Dashboard connections
#SSLVerifyClient require

8. Ensure the following log files exist

touch /var/log/httpd/ukcrlcscin0010.tcl-oob.net_access.log
touch /var/log/httpd/ukcrlcscin0010.tcl-oob.net_error.log
touch /usr/share/puppet-dashboard/log/production.log
chmod 0666 /usr/share/puppet-dashboard/log/production.log

9. Create the puppet-dashboard SSL certs for https access

• On the puppet-dashboard server run:

$ sudo -u puppet-dashboard rake cert:create_key_pair
$ sudo -u puppet-dashboard rake cert:request

• On the puppet master, sign the certs

puppet cert sign dashboard

• On the puppet-dashboard, retrieve the SSL certs

$ sudo -u puppet-dashboard rake cert:retrieve

10. Copy the following script to the puppet dashboard server and all the puppet masters

Create /usr/lib/ruby/site_ruby/1.8/puppet/reports/https.rb with the following code

require 'puppet'
require 'net/http'
require 'net/https'
require 'uri'

Puppet::Reports.register_report(:https) do

desc <<-DESC
Send report information via HTTPS to the `reporturl`. Each host sends
its report as a YAML dump and this sends this YAML to a client via HTTPS POST.
The YAML is the `report` parameter of the request."
DESC

def process
url = URI.parse(Puppet[:reporturl].to_s)
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE

req = Net::HTTP::Post.new(url.path)
req.body = self.to_yaml
req.content_type = "application/x-yaml"

http.start do |http|
response = http.request(req)
unless response.code == "200"
Puppet.err "Unable to submit report to #{Puppet[:reporturl].to_s} [#{response.code}] #{response.msg}"
end
end

end
end

• Remove the following file from the puppet dashboard and puppe masters

/usr/lib/ruby/site_ruby/1.8/puppet/reports/http.rb
rm -f /usr/lib/ruby/site_ruby/1.8/puppet/reports/http.rb

11. Enable puppet-dashboard reports for each puppet master in the environment

Add to each puppet master in the puppet environment in /etc/puppet/puppet.conf:

[master]

reports = store, https, puppet_dashboard
reporturl = https:///reports/upload

12. Create the reports directories on each puppet master

mkdir -p $(puppet master --configprint libdir)/puppet/reports

13. Copy the puppet_dashboard.rb files to each of the puppet masters

Note: To find out where to scp the files to on your puppet master, run the following command on you puppet master:

puppet master --configprint libdir
$/var/lib/puppet/lib

The scp command needs to be run on your puppet-dashboard server

scp /usr/share/puppet-dashboard/ext/puppet/puppet_dashboard.rb root@puppet:/var/lib/puppet/lib/puppet_dashboard.rb

Change the tag accordingly

14. Update the puppet_dashboard.rb file on each puppet master

Run on your puppet server and update the parameters HOST and PORT

vim /var/lib/puppet/lib/puppet_dashboard.rb

HOST =
PORT = 443

15. Edit the auth.conf file on each puppet master so inventory can pick up facts.

Make sure the following is set on each puppet master.

path /facts
method find
auth any
allow *

path /inventory
auth any
method search, find
allow dashboard

# deny everything else; this one is not stricly necessary, but it has the merit of showing the default policy, which is deny everything else

path /
auth any

Note: The config for /facts and /inventory must go above the config for `path /` – otherwise you may get an access forbidden 404 error message when running the inventory service on puppet-dashboard.

16. Restart apache on puppet masters

service httpd restart

17. On the puppet dashboard server, turn on and configure the inventory service and cutoff time

vi /usr/share/puppet-dashboard/config/settings.yml

# The "inventory service" allows you to connect to a puppet master to retrieve and node facts

enable_inventory_service: true

# Hostname of the inventory server.

inventory_server: 'puppet'

# Port for the inventory server.

inventory_port: 8140

# Amount of time in seconds since last report before a node is considered no longer reporting

no_longer_reporting_cutoff: 43200

18. Add the puppet-dashboard delayed_job script to rc.local

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
cd /usr/share/puppet-dashboard
sudo -u puppet-dashboard env RAILS_ENV=production script/delayed_job -p dashboard -n 4 -m start

19. Configure IP tables

iptables -I INPUT 5 -s 10.33.22.0/24 -m tcp -p tcp --dport 8140 -j ACCEPT
iptables -I INPUT 6 -s 10.33.22.0/24 -m tcp -p tcp --dport 80 -j ACCEPT
iptables -I INPUT 7 -s 10.33.22.0/24 -m tcp -p tcp --dport 443 -j ACCEPT
iptables -I INPUT 8 -s 10.33.22.0/24 -m tcp -p tcp --dport 3306 -j ACCEPT

service iptables save
service iptables restart
iptables -L

20. Start apache and connect to the dashboard URL

service httpd start

Check the apache service has started and the webserver is listening on port 80 and 443.

21. Start the delayed_job script

cd /usr/share/puppet-dashboard
sudo -u puppet-dashboard env RAILS_ENV=production script/delayed_job -p dashboard -n 4 -m start

Connect to the puppet-dashboard web UI. Run a puppet agent –test on the puppet-dashboard and you should see the puppet nodes report in to the puppet dashboard.

Once you have done all of thisand checked the puppet dashboard is working correctly, I would reboot the linux box to ensure that if there was a OS failure and the box got rebooted, or if someone rebooted the box, pupp dashboard would work on a reboot – it is up to you.. but recommended.

Installing puppetdb

puppet_db

It just makes sense to use the puppet database in your puppet environment. It is used to store the inventory facts about every node in your puppet environment and as this can be big, you need to have a good back end database which can handle this. Here is how I installed puppet DB

There are no special requirements for SELINUX when installing puppet DB. Keep SELINUX set to enforcing mode.

1. Install the puppet repo

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -Uvh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-6.noarch.rpm

2. Install the puppet agent and request a CSR

yum install puppet -y
puppet agent --test

3. Sign the puppet agents CSR on the puppet master

puppet cert --list
puppet cert sign --

4. Install puppet DB on the puppet DB server

puppet resource package puppetdb ensure=latest

5. Install postgres

Install instructions taken from http://wiki.postgresql.org/wiki/YUM_Installation

vi /etc/yum.repos.d/CentOS-Base.repo

To the [base] and [updates] sections in /etc/yum.repos.d/CentOS-Base.repo, you need to append a line

exclude=postgresql*

Download and install the rpm repos for postgres

<code>rpm -Uvh <a href=”http://yum.postgresql.org/9.1/redhat/rhel-6-i386/pgdg-centos91-9.1-4.noarch.rpm&#8221; title=”http://yum.postgresql.org/9.1/redhat/rhel-6-i386/pgdg-centos91-9.1-4.noarch.rpm”>http://yum.postgresql.org/9.1/redhat/rhel-6-i386/pgdg-centos91-9.1-4.noarch.rpm</a&gt;
</code>

yum list postgres*
yum install postgresql91-server -y

6. Start the postgres service and run at boot

service postgresql-9.1 initdb
service postgresql-9.1 start
chkconfig postgresql-9.1 on

7. Configure postgres

vi /var/lib/pgsql/9.1/data/postgresql.conf

listen_addresses = '*'
log_line_prefix = '%t %u %d'

8. Create the puppet DB

sudo -u postgres sh
cd /var/lib/puppetdb
createuser -DRSP puppetdb
createdb -O puppetdb puppetdb

9. Allow host access to the postgresql pupetdb

vi /var/lib/pgsql/9.1/data/pg_hba.conf

local all all trust
host puppetdb puppetdb 10.10.10.0/24 trust

• Comment out this line (near the end of pg_hba.conf)

#local all all peer

• Restart the postgres service

service postgresql-9.1 restart

• Test login

psql -h 10.33.22.162 puppetdb puppetdb

and

psql -d puppetdb -U puppetdb -W

10. Configure puppdb conf.d/config.ini file

vi /etc/puppetdb/conf.d/config.ini

Below details an example config.ini file

# See README.md for more thorough explanations of each section and
# option.

[global]

# Store mq/db data in a custom directory
vardir = /var/lib/puppetdb

# Use an external log4j config file

logging-config = /etc/puppetdb/conf.d/../log4j.properties

# Maximum number of results that a resource query may return

resource-query-limit = 20000

[database]

classname = org.postgresql.Driver
subprotocol = postgresql
subname = //localhost:5432/puppetdb
username = puppetdb
password =

[jetty]

port = 8080

[command-processing]

# How many command-processing threads to use, defaults to (CPUs / 2)
# threads = 4

11. Start the puppetdb services

sudo puppet resource service puppetdb ensure=running enable=true

12. Open puppetdb port 8081 in iptables

Modify where necessary

iptables -I INPUT 5 -s 10.10.10.0/24 -m tcp -p tcp --dport 8081 -j ACCEPT
iptables -I INPUT 6 -s 10.10.10.0/24 -m tcp -p tcp --dport 8080 -j ACCEPT

service iptables save
service iptables restart

The 2nd firewall rule is used to access the puppetdb-dashboard

13. Set up a puppet master to connect to puppetdb

• Run the following on each of your puppet masters:

sudo puppet resource package puppetdb-terminus ensure=latest

• Add this to /etc/puppet/puppetdb.conf. Note: you may have to create this file.

[main]

server =
port = 8081

• Add this to /etc/puppet/puppet.conf

[master]

storeconfigs = true
storeconfigs_backend = puppetdb

• Add this to /etc/puppet/routes.yaml. Note: you may have to create this file.

master:
facts:
terminus: puppetdb
cache: yaml

14. Restart the puppet service on each puppet master

service httpd restart

15. Check in an agent and monitor the puppetdb logs

On the puppetdb server, monitor the puppetdb log

tail -f /var/log/puppetdb/puppetdb.log

On an agent run the following:

puppet agent --test

16. (Optional) Open up access to the puppet DB dashboard

Reference:

http://docs.puppetlabs.com/puppetdb/1/maintain_and_tune.html#monitor-the-performance-dashboard

To access the puppetdb, you need to make some configuration changes. Edit the following file

/etc/puppetdb/conf.d/config.ini

[jetty]

...

host = 0.0.0.0

Access the puppet DB using the following URL, changing the host name accordingly

http://puppetdb.example.com:8080/dashboard/index.html

Troubleshooting issue:

If the puppet DB, port 8080 and 8081 are not listening when the puppetdb service is set to start and you see in /var/log/puppetdb/puppdb.log the following error:

2013-02-26 19:40:43,327 ERROR [main] [puppetlabs.utils] Uncaught exception
java.io.FileNotFoundException: /etc/puppetdb/ssl/keystore.jks (No such file or directory)

Run the following command and reboot the server

sudo /usr/sbin/puppetdb-ssl-setup

This will create a keystore and truststore in /etc/puppetdb/ssl and will print the password to both files in /etc/puppetdb/ssl/puppetdb_keystore_pw.txt.

Installing a puppet master and ca server

puppet

Here are my install notes on how I set up my puppet server. Note, with EPEL 6.8, passenger is available via yum. This is a much better way to install passenger, rather than using ruby gems.

1 – Install EPEL and puppet repos

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-6.noarch.rpm

2- Install required packages

yum install puppet-server puppet sudo mod_ssl rubygem-passenger
mod_passenger policycoreutils-python vim rsync -y

3. Create a puppet CA cert

puppet master --no-daemonize --verbose

ctrl+c to break out of puppet master deamon once the puppet daemon is running when you see the following message

Notice: Starting Puppet master version 3.x.x

4. Copy the example puppet virtual host config to /etc/httpd/conf.d/

cp /usr/share/puppet/ext/rack/files/apache2.conf
/etc/httpd/conf.d/puppet-master.conf

5. Edit the puppet-master.conf file and update accordingly

# you probably want to tune these settings

PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500

PassengerMaxRequests 1000
PassengerStatThrottleRate 120
RackAutoDetect Off
RailsAutoDetect Off

Listen 8140

SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/puppet-server.com.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/puppet-server.com.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem

# If Apache complains about invalid signatures on the CRL, you can try disabling
# CRL checking by commenting the next line, but this is not recommended.

SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1

# The `ExportCertData` option is needed for agent certificate expiration warnings

SSLOptions +StdEnvVars +ExportCertData

# This header needs to be set if using a loadbalancer or proxy
RequestHeader unset X-Forwarded-For
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

DocumentRoot /usr/share/puppet/rack/puppetmaster/public/
RackBaseURI /

Options None
AllowOverride None
Order allow,deny
allow from all

6. Create rack directories

mkdir -p /usr/share/puppet/rack/puppetmaster/{public,tmp}

7. Copy config.ru rack file to rack web directory

cp /usr/share/puppet/ext/rack/files/config.ru
/usr/share/puppet/rack/puppetmaster/

8. Change ownership of config.ru rack file to puppet

chown puppet:puppet /usr/share/puppet/rack/puppetmaster/config.ru

9. Set httpd to start on boot and puppet master to not start

chkconfig httpd on
chkconfig puppetmaster off

10. IPTABLES configuration

This is environment specific depending on the bridges you need to create to let the puppet master communicate to its various networks. Port 8140 needs to be open on the puppet master for each interface you have added, specifying the source networks and interface adapter as applicable. Below is an example taken from the UK Cressex LAB puppet master deployment. All IPTABLES information is covered later in the document under the configured puppet environments section.

Here is the following content of the /etc/sysconfig/iptables

# Generated by iptables-save v1.4.

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [24:2304]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 10.10.10.0/24 -i eth0 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -s 10.10.10.0/24 -i eth0 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.10.11.0/24 -i eth1 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.10.12.0/24 -i eth2 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.10.13.0/24 -i eth3 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.10.14.0/24 -i eth4 -p tcp -m tcp --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT

You can add these rules following the example command below

iptables -I INPUT 6 -i eth1 -p tcp -s 10.10.10.0/24 --dport 8140 -m state --state NEW,ESTABLISHED -j ACCEPT

service iptables save

or by editing the /etc/sysconfig/iptables and running iptables-save and service iptables restart

11. Start apache

service httpd start

12. check that httpd is running and test connectivity

https://puppet-server1.tcl-oob.net:8140/

If everything is working correctly, you should see

The environment must be purely alphanumeric, not ”

Hope this helps some people. There are some more blogs I have written concerning how to install puppet DB, puppet dashboard and adding additional puppet masters.

Gitlab, Jenkins and puppet playing together

jenkinsLogo1

Jenkins in one of my favourite open source tools. It must also be one of the most flexible

You can set up Jenkins to monitor your Gitlab project for any changes. If Jenkins sees any commits to the master repo, Jenkins will execute a build script and run any customised scripts. Here is how to set this up.

  • Set up a project in Gitlab. This will be your git repo.
  • Create a your git repo using a git client on a linux box of your choice.
  • Configure Jenkins job to poll the git repo and to run a build script if there are any changes.

Here is how to configure the Jenkins server to poll the git repo.

Install the gitlab plugin to Jenkins

https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin

Set up a Jenkins job and specify the following under source code management

jenkins_1

note: The repository URL can be obtained from gitlab as shown below

jenkins_2

Setting up build triggers

Under build triggers, select the Poll SCM and set a Schedule. The schedule is in cron format and be set as desired. Here we have set Jenkins to poll the git repo every 5 minutes for changes.

Note: Once the job is setup, you can click on the jenkins_4

Here you will see the git polling working as shown below:

jenkins_5

SSH keys and sudo access

We will need to set up ssh keys and sudo access for the jenkins_admin user which is used to log on to the puppet masters to sycnronise the manifests and modules. To set up the keys, run the following commands on the jenkins box.

1 – Log in as jenkins_admin@jenkins on the jenkins server

2 – Run the following command to drop into the jenkins user

sudo - s -H -u jenkins

3 – Run the following to set up the ssh key pair

$ cd ~/.ssh/
$ ssh-sopy-id -i id_rsa.pub jenkins_admin@puppet-server

note: # where puppet-server is the puppet master you need to sync the manifests on

4- Configure sudo on the puppet-master so Jenkins_admin is not prompted for the password. Note you can use a manifest to configure sudo via puppet.

on the puppet-server (as root)

sudo visudo

...

jenkins_admin ALL= NOPASSWD:/usr/bin/rsync

Make sure you do this process for each puppet-server you wish to configure sudo and ssh keys for

Configure the Jenkins build job

Configure the Jenkins build job as follows:

# Specifies a AD user who has sudo rights on the puppet server to run the rsync
commands without requiring a password

USER=jenkins_admin

# List of puppet masters that will receive manifests and modules

for HOST in 'puppet-server1t' 'puppet-server2' 'puppet-server3'

do

# rsync command to sync changes from gitlab repo

rsync -e "ssh -t -l jenkins_admin" --rsync-path='sudo rsync' -avz --stats
--progress --human-readable --exclude .git --exclude README $WORKSPACE/
$USER@$HOST:/etc/puppet/

done

This completes the setup.