Python

Creating multiple log files using Python logging library

I was looking at the Python logging library as I needed to create 2 log files for different logging purposes. I ended up creating the following Python log functions.

#!/usr/bin/env python
import logging

LOG_FILE_ONE = "/var/log/one.log"
LOG_FILE_TWO = "/var/log/two.log"

def main():

    setup_logger('log_one', LOG_FILE_ONE)
    setup_logger('log_two', LOG_FILE_TWO)

    logger('Logging out to log one...', 'info', 'one')
    logger('Logging out to log two...', 'warning', 'two')

def setup_logger(logger_name, log_file, level=logging.INFO):

    log_setup = logging.getLogger(logger_name)
    formatter = logging.Formatter('%(levelname)s: %(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')
    fileHandler = logging.FileHandler(log_file, mode='a')
    fileHandler.setFormatter(formatter)
    streamHandler = logging.StreamHandler()
    streamHandler.setFormatter(formatter)
    log_setup.setLevel(level)
    log_setup.addHandler(fileHandler)
    log_setup.addHandler(streamHandler)

def logger(msg, level, logfile):
 
    if logfile == 'one'   : log = logging.getLogger('log_one')
    if logfile == 'two'   : log = logging.getLogger('log_two') 
    if level == 'info'    : log.info(msg) 
    if level == 'warning' : log.warning(msg)
    if level == 'error'   : log.error(msg)

if __name__ == "__main__":

    main()

In the main() function, we set up a logging instance. We do this for both LOG_FILE_ONE and LOG_FILE_TWO.

    setup_logger('log_one', LOG_FILE_ONE)
    setup_logger('log_two', LOG_FILE_TWO)

This uses standard logging python library. The formatter is important as that is how the log entries look in the log file (below). Here is mine:

formatter = logging.Formatter('%(levelname)s: %(asctime)s %(message)s', 
            datefmt='%m/%d/%Y %I:%M:%S %p')

Have a look at this online doc from Python.org for more info on the formatter method.

https://docs.python.org/2/library/logging.html#formatter-objects

Once we set up the logging instance, including the file streamer and handler, we can then use those instances,  in this case log_one or log_two.

Both have different log file paths /var/log/{one.log,two.log}.

We then call the logging function, telling it what logging instance to use (log_one or log_two) and we also pass in what log level we want (info, warn, debug).

    logger('Logging out to log one...', 'info', 'one')
    logger('Logging out to log two...', 'warn', 'two')

Interesting, we get this output when we run this script.

INFO: 01/01/2016 01:01:01 AM Logging out to log one…

Notice we do not get WARNING being logged.

That is because of this:

def setup_logger(logger_name, log_file, level=logging.INFO):

Change level=logging.WARN, and this happens:

INFO: 01/01/2016 01:01:01 AM Logging out to log one...
WARNING: 01/01/2016 01:01:01 PM Logging out to log two...

So, pretty handy and you can create as many log files as you need…

Using VIRTUALENV in python

virtualenv

I wanted to blog about virtualenv which is really an important tool to use when installing Python 2.x or 3.x. Centos, for exmaple, comes with its own version of Python. Install a newer Python version and you have broken yum. This is not good, but this is where virtualenv comes in. In essence, virtualenv allows you to run multiple python environments on one server which in turn allows you to run different libraries that will not conflict with each python environment. Say I have a mysql library that needs to be at a specific version – I have installed the mysql python package. If I use virtualenv, that mysql python package will only apply to the virtualenv python environment. It therefore is a way to keep your libraries separate. Now you can still use your main python library for common libraries, but virtualenv just helps separate each python environment – good also when mulitple developers are working on the one server.

Here is how to set up a virtualenv for Python (pre-requisite is that you have your version of python installed)

1 – Make sure you have virtualenv installed.

yum search python-virtualenv

2 – Create a directory to install your python environment

mkdir ~/my_python_app/

3 -Run the virtualenv –<directory from step 2>

virtualenv ~/my_python_app/

Here’s the output for  steps 2 & 3

[root@centos ~]# mkdir my_python_app
[root@centos ~]# cd my_python_app/
[root@centos my_python_app]# virtualenv ~/my_python_app/
Running virtualenv with interpreter /usr/local/bin/python2.7
New python executable in /root/my_python_app/bin/python2.7
Also creating executable in /root/my_python_app/bin/python
Installing Setuptools............
Installing Pip...................
done.

And there you have it – an independent python environment. There is more that you should do to really customise the environment, but that’s all you need to do to initially get yourself up and running.

I do however want to highlight a useful option, which can be found using virtualenv –help:

-p PYTHON_EXE, --python=PYTHON_EXE
        The Python interpreter to use, e.g.,
        --python=python2.5 will use the python2.5 interpreter
        to create the new environment.  The default is the
        interpreter that virtualenv was installed with
        (/usr/local/bin/python2.7)

This option allows you to create a python environment using a different python version. This is really good if you want your python environment to be independent but use different version of python, 2.6, 3.x etc.

Activating your environment.

So once you have configured your python environment, you need to activate it so you can work on it. Here is how to do this:

[root@centos my_python_app]# source /root/my_python_app/bin/activate

Once you have done this, you will see the prompt change as follows:

(my_python_app)[root@v28388581sl0001 my_python_app]#

The (my_python_app) just shows you what virtualenv is activated.

Once you are activated, you can start using pip to install your libraries, code in python, etc. Once you are finished, you just need to deactivate the virtualenv, as follows:

(my_python_app)[root@centos my_python_app]# deactivate
[root@centos my_python_app]#

Now you can see your prompt has gone back to normal and the active virtual environment has been removed.

So, how do you run a python script using a specific python environment? Well, you do not need to activate the python virtualenv, you just specify the full path to the python interpreter in the bin folder within the virtualenv.

We installed a virtualenv in /root/my_python_app, so the python exe or interpreter (which ever you fancy calling it) will be found in the bin folder, as follows:

[root@centos /]# /root/my_python_app/bin/python
Python 2.7.5 (default, Oct 24 2013, 15:46:16)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Notice that the version is Python 2.7.5 – if I had specified a different python version, that would reflect when calling the python interpreter in the bin folder. All you need to do is specify the full path followed by the script you wish to run.

TIP:  However, I can give you a easy tip here. Rather than specify the full path, update your bash profile or better still, /etc/bashrc as follows:

alias python_v1=/root/my_python_app/bin/python

Happy Python coding – and please get to use virtualenv. It will save you hassle.

Deploying a VM to vCloud Director using API and Python

pyton_api_1

Further to my other post about getting started with vCloud Director API using Python, I thought I would write down how I deployed a VM to a vApp.

Once you have got a login function working by obtaining the x-vcloud-authorization token, you need to use the recomposeVApp REST API call, which as outlined by the documentation is done by using appending this URI, /vApp/{id}/action/recomposeVApp, to your endpoint. NOte, your endpoint is your API URL.

Here is my example code, which can be found on my github account, https://github.com/oliverleach/vcloud-autodeploy/blob/master/vCloudDeployVm.py


def recompose_vapp(self):

    post_headers = self.headers
    post_headers['Content-Type']='application/vnd.vmware.vcloud.recomposeVAppParams+xml'

    xml = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
             <RecomposeVAppParams
              xmlns="http://www.vmware.com/vcloud/v1.5"
              xmlns:ns2="http://schemas.dmtf.org/ovf/envelope/1"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1"
              xmlns:environment_1="http://schemas.dmtf.org/ovf/environment/1">
              <Description> "api deployed vm" </Description>
                  <SourcedItem sourceDelete="false">
                      <Source href="%s"/>
                  </SourcedItem>
              <AllEULAsAccepted>true</AllEULAsAccepted>
              </RecomposeVAppParams>""" % (self.vapp_template)

    post = requests.post(self.vapp + '/action/recomposeVApp', data=xml, headers=post_headers)
    result = ET.fromstring(post.text)

Firstly, you can see I am setting the post_headers value to self.headers – these headers include the x-vcloud-authorization token. You also need to add the content type which is outlined in the documentation. It just tells the API what XML to expect, which in this case is recomposeVAppParams+xml.

Next, I have specified the XML file and I’ve cut down what I have specified and kept it simple. The only thing I have provided is the vapp_template value, which is something I have obtain earlier in my API script.

Finally, I then post this request as shown again below:

post = requests.post(self.vapp + '/action/recomposeVApp', data=xml, headers=post_headers)

Here I am using the python requests library and adding the required parameers, which in cludes the self.vapp URL, again obtained earlier in my API script, which appends /action/recomposeVApp. I then provide the XML which is contained in the function and add the post_headers.

You need to look at the whole API script to understand how I have built up the post request, including how I have obtained the various requirements such as vapp href and the vapp_template href.

Please do post a comment if you would like more info or use the contact me link on my blog.

Happy scripting. 🙂

Getting started with the vCloud Director API with Python

api

I have been working closely with vCloud Director from a front end point of view as one of our customers wanted to leverage our public cloud offering. I wanted to show them the power of the API and I decided to use Python as the scripting language of my choice to script some API calls. Now there is one problem I faced straight away which is there is no vCloud SDK for Python. What does this mean? Well, an SDK provides a library that you can leverage which makes vCloud API calls easier. So was the only other option was to use the REST API provided by vCloud. This is fine, but it just means more work rather then use something that someone else has written to make your life easier.

Just for the record, there are SDK’s for vCloud for DOT NET, Java and PHP.

So the first thing to understand is the documentation – you need to understand how it’s laid out and how to find what you want to do. Easier said than done. There are loads of goods resources here but I have listed a few to start with:

vCloud Dirtector API reference documentation
https://www.vmware.com/support/pubs/vcd_pubs.html

Exploring the vCloud Director API
http://blogs.vmware.com/vsphere/2012/03/exploring-the-vcloud-rest-api-part-2.html

Once you have muddled through the API documentation, you need to use a Python library that will interact with HTTP. I used the great Python Requests library rather than HTTPLIB2. I find it much easier to use, more simple and cleaner to understand.

The next thing to do is to build a login function. Here is my snippet of Python code:

class vCloud_Deploy(object):

    def __init__(self):

        self.login = None
        self.headers = None
        self.endpoint = None
        self.org = None
        self.root = None

    def sessions(self, username, org, password, key, secret, endpoint):

        self.endpoint = endpoint

        self.login = {'Accept':'application/*+xml;version=5.1', \
            'Authorization':'Basic '+ base64.b64encode(username + &quot;@&quot; + org + &quot;:&quot; + password), \
            'x-id-sec':base64.b64encode(key + &quot;:&quot; + secret)}

        p = requests.post(self.endpoint + 'sessions', headers = self.login)

        self.headers = {'Accept':'application/*+xml;version=5.1'}

        for k,v in p.headers.iteritems():
            if k == 'x-json':
                access_token_value = 'Bearer %s' % v[21:57]
                self.headers[&quot;Authorization:&quot;]=access_token_value
                if k == &quot;x-vcloud-authorization&quot; : self.headers[k]=v

The public cloud I was working with needed 2 forms of authentication. The first authentication mechanism was to send an api and secret key in a base64 encoded URL. This bit doesn’t really matter as it’s not part of the vCloud Director authentication mechanism but merely an added layer of authentication provided by the company I work for. What is important is logging in to the vCloud Director API and for this, you need to get the x-vcloud-authorization header value. This is then passed in when ever you call other vCloud APIs. This is your token.

You can see above that I am requesting a login and once I have passed in all the headers I need to for the inital login request, if it is successful I then look at the headers returned. I iterate through all the headers and find the x-vcloud-authorization header and its value and then add this information to my self.headers. Whenever I call another API, I just use the self.header which contains a valid x-vcloud-authorization token. The point here, is once you have the login function set up and working, you’ve gone a long way to then starting to use the API. Get this sorted and your in basically.

So now you can start with some simple get requests. Here is an example function that gets the org URL. This tacks on to the script snippet above.

    def org_url(self):

        g = requests.get(self.endpoint + 'org', data=None, headers = self.headers)
        root = ET.fromstring(g.content)

        for child in root:

            self.org = child.get(&quot;href&quot;)

            g = requests.get(self.org, data=None, headers = self.headers)
            self.root = ET.fromstring(g.content)

            return self.org

In my python class constructor, I have set up various parameters I need when using the API. This includes my endpoint which is something you need to know before you use the API.

    def __init__(self):

        ....
        self.endpoint = None
        ....

So back to the how I get the org URL, you will notice that I have passed the endpoint in my get request, followed by the ‘org’ appended. This essentially gives me the REST API call which gets me the org URL and how this is done can be found in the VMware documentation, see here – http://pubs.vmware.com/vcd-55/topic/com.vmware.vcloud.api.reference.doc_55/doc/operations/GET-Organizations.html. Once I collect the information, I set the self.org value to the HREF org_url. I use the self.org value to specify in future API calls that its this org I want to query or provision to.

This is just a blog, not a how to step by step article. The full version of the script can be found on my github account here – https://github.com/oliverleach/vcloud-autodeploy/blob/master/vCloudDeployVm.py. Get in touch if you have any questions or comments.

Installing Python and Django

django

Here are my notes on how to set up Python and Django

UPDATE: These are old notes and I would definitely use virtenv when setting up Python. I will try to do a blog about this

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# Depending on whether you run as root, you may need to use sudo.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

sudo yum install gcc tcl tk sqlite-devel readline-devel gdbm-devel -y
sudo yum install tkinter ncurses-devel libdbi-devel tk-devel zlib-devel -y
sudo yum install openssl-devel bzip2-devel -y
sudo yum install httpd httpd-devel -y
sudo yum install mysql mysql-devel -y
wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz
tar zxvf Python-2.7.3.tgz
cd Python-2.7.3
sudo ./configure --prefix=/opt/python2.7 --with-threads --enable-shared
make
sudo make install
ln -s /opt/python2.7/bin/python /usr/bin/python2.7
echo '/opt/python2.7/lib'>> /etc/ld.so.conf.d/opt-python2.7.conf
ldconfig
echo "alias python='/opt/python2.7/bin/python'" >> /etc/bashrc
echo "alias python2.7='/opt/python2.7/bin/python'" >> /etc/bashrc

***log out and log back in at this point. This is to ensure your bash rpofie is updated with the new python location***

wget http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.3.tar.gz
tar zxvf mod_wsgi-3.3.tar.gz
cd mod_wsgi-3.3
./configure --with-python=/opt/python2.7/bin/python
make
make install
curl http://python-distribute.org/distribute_setup.py | python
curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python
pip install Django