Saturday 9 March 2013

Backup All VMs on a remote Xen host

Hi Everyone,

Continued digging of my old scripts has turned up another which I think may have some use. It's a script to backup all Xen VMs running on a remote host. I should explain that this ran between a backup host with plenty of space, and the Xen host, which contained a lab environment of several VMs.
I had also configured SSH keypair authentication between the two systems, as it largely relies on a string of SSH commands.

The script works as follows:
  1. Pause all guests running on the Xen host. (This helps to keep things between the hosts in sync)
  2. Takes each guest, and saves it in it's entirety on the local file system.
  3. Un-pause all the guests and show their status.
  4. Working through each of the guest backup files, it zips, transfers, and deletes the not needed copy.
After it's complete you have a backup for all your guests at a given time. 

As an aside, I would like to say the Xen project has really matured recently (last two/three years as I write this). I can remember a time when it was very buggy, VMs would randomly freeze and it was a complete pain to set-up. I still feel the pain of regularly compiling custom Kernels for it on RHEL machines. They even tried to push it on me during my RHCE study and it was not well behaved then! It's not always my virtualisation tool of choice, but it does have some advantages today like speed of install, opensource freedom and scriptability. If you didn't like it before, it might surprise you!

The Script:


#!/bin/bash
#
#  backup_lab.sh will log into the Dom0 machine containing the lab and 
#  instruct it to backup and zip every image it finds.

XENIP="10.1.1.1"
GUESTS=$(ssh root@$XENIP "xm list | grep -Ev '(Name|Domain-0)'" | awk '{ print $1 }')
DATE=$(date +%Y-%m-%d)

# First Get the list of machines, pause them all then save them, and restore, in order

echo "Working on Guests:  $GUESTS"
echo


# Pause all the guests before we start. (Not sure how needed this is, just helps not to let them get out of sync)
echo
for g in $GUESTS
do 
    echo Pausing $g 
    ssh root@$XENIP "xm pause $g" 
done

echo "All Guests should now be paused"
echo 
echo "Current Xen State:"
ssh root@$XENIP "xm list" 
echo 
echo

# Now unpause, then backup the guest, restore from its backup, and pause!
for g in $GUESTS
do
    echo "Backing up guest $g"
    echo
    ssh root@$XENIP "xm unpause $g"
    echo "Guest Un-paused for backup"
    echo
    echo "Saving Guest $g to /root/$g.bak"
    echo
    ssh root@$XENIP "xm save $g /root/$g.bak"
    echo "Guest $g Saved"
    echo 
    echo "Restarting Guest"
    echo
    ssh root@$XENIP "xm restore /root/$g.bak"
    echo "$g has been restored, pausing"
    ssh root@$XENIP "xm pause $g"
    echo "$g has been paused"
    echo
    echo "Guest $g has been backed up."
    echo
done


# Un-pause all the guests
echo
for g in $GUESTS
do 
    echo Un-pausing $g 
    ssh root@$XENIP "xm unpause $g" 
done

# Display the VM list in case anyone is watching
ssh root@$XENIP "xm list" 

# Now all the guests have a backed up file, we can zip and move these
for g in $GUESTS
do
    echo 

    # Tell the remote end to compress
    echo "Compressing /root/$g.bak"
    ssh root@$XENIP "gzip /root/$g.bak"

    # get the remote file back to this directory
    echo "Collecting /root/$g.bak.gz"
    scp root@$XENIP:/root/$g.bak.gz /backup/lab/$g.$DATE.gz

    # Delete the remote backup
    echo "Deleting remote backup /root/$g.bak.gz"
    ssh root@$XENIP "rm -fv /root/$g.bak.gz"
done

echo "All Done!"

Quickest Python Emailer

Hi Everyone,

Over the years I've written many scripts which end up sending an email report. Here is a short and sweet Python version, which has served me well a few times.
  1. This needs to run on a server which can send mail. (Many Unix or Linux will relay localhost with sendmail / postfix etc.)
  2. If you are sending to an external mail system, check your Spam / Junk folders. It may look suspicious, especially if you use a from address which is not the actual domain of the sending machine.
It's not wrapped in a class, or given any bells or whistles. It's just an easy way to send mail!

Enjoy.

'''
Created on 9 Mar 2013

@author: HRH

Probably the quickest way to send an email in python!

'''
import smtplib

SERVER  = "localhost"
FROM    = "Test_Script@myhost.co.uk"
TO      = ["joe_butler99@hotmail.com"] # must be a list
SUBJECT = "Test Mail from RoyalScripts"
TEXT    = "This message was sent with Python's smtplib."

# Prepare actual message
message = """\
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)


# Send the mail
server = smtplib.SMTP(SERVER)
server.sendmail(FROM, TO, message)
server.quit()


Friday 8 March 2013

Checking for the most recent database update with Python and Nagios

Hi Everyone,

Another update to kick-start this blog! I've found another script which I worked on a little over a year ago. This time its purpose was to check that we had a constant stream of new records appearing in a database table. The scripts output was designed to feed our Nagios monitoring system, but it will work just as well as a standalone or called from another script.

For those that don't know, Nagios works with exit codes and standard output from the checks it runs. It's also easy to make new, custom checks. For this to be considered a 're-usable nagios check' I should have made the script accept arguments to fill the variables I've hard coded at the top. It could also parse them to check what's been passed in is suitable. I don't have time tonight, but if someone asks nicely....

Anyway I hope you will see that writing these kind of checks can be very simple.


#!/usr/bin/python
'''
@author: HRH

A Nagios Check to ensure recent database records
'''

import sys
import datetime
import MySQLdb

# Script Variables
db_host    = "127.0.0.1"
db_user    = "root"
db_pass    = "password"
db_name    = "mydatabase"
db_table   = "mytable"
warn_secs  = 3600
error_secs = 7200
date_field = 2 # What column number contains the date we want to check (Zero indexed) 

# Make sure we can connect, if not something is wrong.
try:
    db = MySQLdb.connect(db_host, db_user, db_pass, db_name)
    db_c = db.cursor()
except:
    print "Unable to connect to database."
    sys.exit(2)

# Try and run the query for the latest data
try:    
    query = "SELECT * FROM `%s`.`%s` ORDER BY id DESC LIMIT 1 " % (db_name,db_table)
    db_c.execute(query)
    result = db_c.fetchall()[0]
except:
    print "Query Failed to select latest record from `%s`.`%s`" % (db_name,db_table)
    sys.exit(2)

latest_cdr  = result[date_field]
time_now    = datetime.datetime.now()
time_diff   = time_now - latest_cdr

# Check how long since last record and now 
if time_diff.seconds < warn_secs:
    print "New records within %d seconds" % warn_secs
    sys.exit(0)
elif time_diff.seconds < error_secs:
    print "Between %d and %d seconds since last record in DB" % (warn_secs,error_secs)
    sys.exit(1)
else:
    print "Over %d seconds since last record in DB" % error_secs
    sys.exit(2)


    

Log file class with Minimal Python Libraries

Hi Everyone,

Its been a long time since I was here, but I have worked on many scripts in the mean time and I think it would be nice to share a few!

I thought I would start with a small script I wrote about a year ago, the output of which was intended to be consumed by Splunk. Now I look at this, it could also be useful straight away for any work which wanted to keep a simple log file.

The reason I wrote this with 'time' as the only python import is because it was due to run on a hardware system with an old, cut back python installation. This may be an advantage for some, it should at least run on nearly any version of python out there.

The output is a nice timestamped line with all the key=value pairs displayed as strings. (quoted where necessary)


'''
A script to write simple timestamped logs to a file. 
@author: HRH

Example test usage below. Built with limited libraries
due to the tiny python installation where this script is
first intended to be used. 

'''
import time

class SplunkLog:
    
    # Writes a line to the given file with the current timestamp
    # the data value should be a python dictionary where the key => val is 
    # used for logtype=logtext 
    def write_local_log(self,fh,data):
        t = time.localtime()
        logtime = str(t[0])+"-"+str(t[1]).zfill(2)+"-"+str(t[2]).zfill(2)+" "+str(t[3]).zfill(2)+":"+str(t[4]).zfill(2)+":"+str(t[5]).zfill(2)
        log_line = logtime + "\t"
        for row in data:
            for k , v in row.iteritems():
                if " " in v:
                    log_line += k + "=" + '"' + v + '"' + ",\t"
                else:
                    log_line += k + "=" + v + ",\t"
            
        # Write the line, minus the last spacing and with added newline
        log_line = log_line.rstrip(",\t") + "\n"
        fh.write(log_line)


'''
#
#    A usage example - remember to include or add the main class somewhere
#
#
#
#        Main Loop
#
from sys import exit
dummy_data = []
dummy_data.append({"severity" : "Warning","EventID": "Bad Login", "user" : "Test"})
dummy_data.append({"severity" : "Major","EventID": "Possible DoS", "user" : "Remote"})
s = SplunkLog()

try:
    fh = open("splunk-test.txt",'a')
except:
    print "Unable to open log file"
    exit(1)
    
# If we are still here, then we can try write something into the file
for log in dummy_data:
    s.write_local_log(fh, log)

fh.close()
exit(0)
'''