Showing posts with label BASH. Show all posts
Showing posts with label BASH. Show all posts

Saturday, 9 March 2013

Backup All VMs on a remote Xen host

Hi Everyone,

Continued digging of my old scripts has turned up another which I think may have some use. It's a script to backup all Xen VMs running on a remote host. I should explain that this ran between a backup host with plenty of space, and the Xen host, which contained a lab environment of several VMs.
I had also configured SSH keypair authentication between the two systems, as it largely relies on a string of SSH commands.

The script works as follows:
  1. Pause all guests running on the Xen host. (This helps to keep things between the hosts in sync)
  2. Takes each guest, and saves it in it's entirety on the local file system.
  3. Un-pause all the guests and show their status.
  4. Working through each of the guest backup files, it zips, transfers, and deletes the not needed copy.
After it's complete you have a backup for all your guests at a given time. 

As an aside, I would like to say the Xen project has really matured recently (last two/three years as I write this). I can remember a time when it was very buggy, VMs would randomly freeze and it was a complete pain to set-up. I still feel the pain of regularly compiling custom Kernels for it on RHEL machines. They even tried to push it on me during my RHCE study and it was not well behaved then! It's not always my virtualisation tool of choice, but it does have some advantages today like speed of install, opensource freedom and scriptability. If you didn't like it before, it might surprise you!

The Script:


#!/bin/bash
#
#  backup_lab.sh will log into the Dom0 machine containing the lab and 
#  instruct it to backup and zip every image it finds.

XENIP="10.1.1.1"
GUESTS=$(ssh root@$XENIP "xm list | grep -Ev '(Name|Domain-0)'" | awk '{ print $1 }')
DATE=$(date +%Y-%m-%d)

# First Get the list of machines, pause them all then save them, and restore, in order

echo "Working on Guests:  $GUESTS"
echo


# Pause all the guests before we start. (Not sure how needed this is, just helps not to let them get out of sync)
echo
for g in $GUESTS
do 
    echo Pausing $g 
    ssh root@$XENIP "xm pause $g" 
done

echo "All Guests should now be paused"
echo 
echo "Current Xen State:"
ssh root@$XENIP "xm list" 
echo 
echo

# Now unpause, then backup the guest, restore from its backup, and pause!
for g in $GUESTS
do
    echo "Backing up guest $g"
    echo
    ssh root@$XENIP "xm unpause $g"
    echo "Guest Un-paused for backup"
    echo
    echo "Saving Guest $g to /root/$g.bak"
    echo
    ssh root@$XENIP "xm save $g /root/$g.bak"
    echo "Guest $g Saved"
    echo 
    echo "Restarting Guest"
    echo
    ssh root@$XENIP "xm restore /root/$g.bak"
    echo "$g has been restored, pausing"
    ssh root@$XENIP "xm pause $g"
    echo "$g has been paused"
    echo
    echo "Guest $g has been backed up."
    echo
done


# Un-pause all the guests
echo
for g in $GUESTS
do 
    echo Un-pausing $g 
    ssh root@$XENIP "xm unpause $g" 
done

# Display the VM list in case anyone is watching
ssh root@$XENIP "xm list" 

# Now all the guests have a backed up file, we can zip and move these
for g in $GUESTS
do
    echo 

    # Tell the remote end to compress
    echo "Compressing /root/$g.bak"
    ssh root@$XENIP "gzip /root/$g.bak"

    # get the remote file back to this directory
    echo "Collecting /root/$g.bak.gz"
    scp root@$XENIP:/root/$g.bak.gz /backup/lab/$g.$DATE.gz

    # Delete the remote backup
    echo "Deleting remote backup /root/$g.bak.gz"
    ssh root@$XENIP "rm -fv /root/$g.bak.gz"
done

echo "All Done!"

Monday, 29 November 2010

Converting all video files within a directory with Bash & ffmpeg

I recently found myself needing to convert around 45 .flv videos into a format which my PS3 would find acceptable. Not being a fan of doing anything manually I decided to do this from within a small script. It requires that you have ffmpeg installed, and the 'rename' program, if you need to remove spaces from the  original filenames. (If not, you may want to comment out the line beginning rename)

You can also tweak the inputformat and outputformat variables and the ffmpeg settings to make it suitable for almost any type of conversion. If you start hitting errors in the conversion, its probably because you have some codec’s or packages missing on your system, so you can also check the extra codec’s section below.


#!/bin/bash
# A script to convert videos files within a directory
# An HRH Script!! 


# Set video types
inputformat=flv
outputformat=mp4
outputdir=converted

# Remove any spaces from filenames
rename s'/ //' *.$inputformat

# check if we have an output directory
if [ -d $outputdir ]
then
echo "Directory Exists, ready to convert files"
else
mkdir $outputdir
echo "Created directory $outputdir/"
fi

# read in the files and start the conversion
for file in `ls *.$inputformat | sed "s/\.$inputformat//"`
do
    ffmpeg -i "$file"."$inputformat" -ab 56k -ar 22050 -b 600k -s 640x480 $outputdir/"$file"."$outputformat" 
done




Getting more codec’s to work in ffmpeg  -  (This was what worked for me, ffmpeg can seem like a dark art at times!)

By going through the process below, I was able to get ffmpeg converting from and to a wide range of video formats. I don't think all the packages were necessary, but it would take me forever to go back and find out which ones made the difference, so I'm just going to show you all of them!

You may also need to enable the multiverse software source, if you do not currently have this enabled.
System -> Administration -> Software Sources


# Install some extra packages which will help ffmpeg handle more formats
# This should be done before installing a fresh version of ffmpeg
sudo apt-get install ubuntu-restricted-extras gstreamer0.10-plugins-ugly-multiverse 
sudo apt-get install gstreamer0.10-plugins-ugly gstreamer0.10-ffmpeg ffmpeg2theora 
sudo apt-get install mencoder libogg0 libogg-dev libvorbis0a libvorbis-dev vorbis-tools 
sudo apt-get install imagemagick youtube-dl poppler-util yasm


# Install the Trunk version of ffmpeg
sudo apt-get purge ffmpeg          # Make sure packaged version is removed
sudo apt-get install subversion    # Skip if subversion is already installed
cd ~
svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg
cd ffmpeg
./configure
make
sudo make install 


Assuming the sudo make install completed without any errors, you should now be running with the latest ffmpeg installed on your system. You can check this with

ffmpeg -version

Tuesday, 2 November 2010

Preparing tagged code for use with syntax highlighter

In my last two posts, this has helped me upload my code. I have just been copying the output directly into the <pre> tags and it seems to work well. In the example below, the cisco_router_checker.php is the name of the file I wish to convert
 cat cisco_router_checker.php | sed s'/</\&lt;/g'

I could be missing something really obvious here, so if there is a better way of doing it please let me know!

Monday, 1 November 2010

BASH & PHP Football Score Checker and Emailer

In short, this is a system to check the BBC website for the football results, and send out automated email to people who have seen their team lose, complete with a link to the match report. This one is a bit of a mishmash of Bash & PHP, as it was written with speed in mind as opposed to elegance and readability. That said, it should be easy to modify to suit your purposes -  (A find and replace on the teams you are interested in should do). I also need to give some thanks to the BBC here, as the consistent way their site is setup has meant this has been running flawlessly for me for close to a year. Originally written in a hurry before a trip to India, this meant that despite my absense, I was still able to taunt colleagues at work when their football teams lost.

HOW IT WORKS

  • The Bash script football_check.sh, is the parent, which is configured in cron to run each Monday morning. Its job is to collect the source code from the BBC football results site, and grep for the first line matching the team we are looking for. It then dumps this into another file in the same directory which we will use later. Once it has completed this, it calls create_report.php
  • create_report.php opens the files created above and checks if the team has lost. If they have won, or drawn, it will not send out the email. If an email needs to be sent, it contains a template to the content of the email, and creates the text contained in the mail, including a link to the match report.

THE SCRIPTS

  • football_check.sh
  • #!/bin/bash
    #
    # A football results checker script. Checks if Portsmouth or
    # Palace have lost in the last week, and if so, sends out a
    # 'Commiseration' email
    #
    #
    #  An HRH Script!!!!
    results_site="http://news.bbc.co.uk/sport1/hi/football/results/default.stm"
    palace_file="/var/www/hrhscript/footiecheck/palace_res.txt"
    pompy_file="/var/www/hrhscript/footiecheck/pompy_res.txt"
    palace_email="/var/www/hrhscript/footiecheck/palace_email.txt"
    pompy_email="/var/www/hrhscript/footiecheck/pompy_email.txt"
    
    
    # collect the data from the bbc site
    #wget --proxy=off $results_site -o results.txt
    wget $results_site -o results.txt
    # Check for the last time they played, and create a file with that information.
    cat default.stm | grep -m 1 "Crystal Palace" > $palace_file
    cat default.stm | grep -m 1 "Portsmouth" > $pompy_file
    
    # Run the php script to check these files
    ./create_report.php 2> /dev/null
    
    # Send out the emails if there is one to send.
    if [ -f $palace_email ] 
    then
    ./paul_email.php
    rm -f $palace_email
    fi
    
    if [ -f $pompy_email ] 
    then
    ./mark_email.php
    rm -f $pompy_email
    fi
    
    
    # Remove the used files to start fresh next monday
    rm -f default.stm  # remove the BBC website information
    rm -f p*res.txt    # remove the results text files
    rm -f results.txt
    



  • create_report.php


  • #!/usr/bin/php
    # This is a bit of code to parse the output from the football result files. If there are files to check, it
    # will have a look at them, work out whats happened, and send out the good news!!!
    
    # Assign variables
    $palace_email = "your.victim@theirdomain.co.uk";
    $pompy_email = "your.victim@theirdomain.co.uk";
    $palace_str = "Crystal Palace";
    $pompy_str = "Portsmouth";
    $palace_draw = false;
    $pompy_draw = false;
    
    # Collect data which has come from bash and wget
    $palace_text = `cat /var/www/hrhscript/footiecheck/palace_res.txt`;
    $pompy_text = `cat /var/www/hrhscript/footiecheck/pompy_res.txt`;
    $palace_len = strlen($palace_text);
    $pompy_len = strlen($pompy_text);
    
    # A function to clean up the html text passed in to a point where we can use it.
    # This is not done via a strip tags function as we can use this to our advantage later 
    function clean_up_html($text_in) {
        $out = str_replace('<div class="mvb"><a href="', "", $text_in);
        $out = str_replace('</div>', "", $out);
        $out = str_replace('" class="stats">', " % ", $out);
        $out = str_replace(' | <a href="', "", $out);
        $out = str_replace('">Report', "", $out);
        $out = str_replace('<a href="', "", $out);
        return $out;
    }
    
    # A function to create the email text which will be sent out
    function create_email($team,$lost_to,$notify_name,$notify,$report_url,$file_name) {
    
        # Variables
        $start = "Dear $notify_name , \n\nIn my absence I would like to let you know that $team lost to $lost_to this weekend. \n\nI cannot personally tell you how rubbish their performance was, however, if it would make you any more miserable, I would like you to read the following match report: $report_url  \n\nKind regards,\nLord Butler";
    
        # Dump it into the file
        `echo "$start" > $file_name`;
    }
    
    # Check what we have to work with, and create the email text.
    if ( $palace_len > 200 ) {
        ######### If the script is here, there are palace resutls to analyse:
        # Split up the data into lots of little bits
        $palace_clean = clean_up_html($palace_text);
        $palace_parts = explode("</a>", $palace_clean);
        $home_data = explode(" % ", $palace_parts[0]);
        $away_data = explode(" % ", $palace_parts[1]);
        $result_data = explode(" ", $away_data[0]);
        $result = $result_data[1];
        $scores = explode ("-", $result);
        $home_score = $scores[0];
        $away_score = $scores[1];
    
        #########  Start with the logic, find out if anything is worth sending.
        # Check if it was a draw.
        if ( $home_score == $away_score ) { $palace_draw = true; echo "The game was a draw \n";}
        if ( !$palace_draw ) { 
            # Check if Palace where home or away
            if ( $home_data[1] == $palace_str ) { $palace_home = true; $palace_away = false;$opposing = $away_data[1];} 
            if ( $away_data[1] == $palace_str ) { $palace_away = true; $palace_home = false; $opposing = $home_data[1];}
            # Check who won, home or away
            if ( $home_score > $away_score ) { $home_victory = true; } else { $home_victory = false; }
     # Check if palace lost
     if (( $palace_away && $home_victory ) || ( $palace_home && !$home_victory )) { 
                create_email($palace_str,$opposing,"Paul",$palace_email,$palace_parts[2],"palace_email.txt");
            } else { 
                echo " $palace_str won \n"; 
            }     
        }
    } else {
        echo "Palace Results Not found";
    }
    
    #################################
    #  Now check the pompy results  #
    #################################
    
    # Check what we have to work with, and create the email text.
    if ( $pompy_len > 200 ) {
        ######### If the script is here, there are pompy resutls to analyse:
        # Split up the data into lots of little bits
        $pompy_clean = clean_up_html($pompy_text);
        $pompy_parts = explode("</a>", $pompy_clean);
        $home_data = explode(" % ", $pompy_parts[0]);
        $away_data = explode(" % ", $pompy_parts[1]);
        $result_data = explode(" ", $away_data[0]);
        $result = $result_data[1];
        $scores = explode ("-", $result);
        $home_score = $scores[0];
        $away_score = $scores[1];
    
        #########  Start with the logic, find out if anything is worth sending.
        # Check if it was a draw.
        if ( $home_score == $away_score ) { $pompy_draw = true; echo "The $pompy_str game was a draw \n";}
        if ( !$pompy_draw ) { 
            # Check if pompy where home or away
            if ( $home_data[1] == $pompy_str ) { $pompy_home = true; $pompy_away = false; $opposing = $away_data[1];} 
            if ( $away_data[1] == $pompy_str ) { $pompy_away = true; $pompy_home = false; $opposing = $home_data[1];}
            # Check who won, home or away
            if ( $home_score > $away_score ) { $home_victory = true; } else { $home_victory = false; }
     # Check if pompy lost
     if (( $pompy_away && $home_victory ) || ( $pompy_home && !$home_victory )) { 
                create_email($pompy_str,$opposing,"Mark",$pompy_email,$pompy_parts[2],"pompy_email.txt");
            } else { 
                echo " $pompy_str won \n"; 
            }
        }
    } else {
        echo "pompy Results Not found";
    }
    
    
    


  • victim_name_email.php



  • #!/usr/bin/php
    /*
    * The PHP mailer programs - HRH
    */
    $to = "notify.victim1@theirdommain.co.uk, notify.victim1@theirdommain.co.uk";
    $subject = "This Weekends results";
    $msg = `cat /var/www/hrhscript/footiecheck/pompy_email.txt`;
    $header = 'From: Lord Butler';
    mail($to,$subject,$msg,$header);
    

    NOTES

    • This script assumes you are able to send / relay mail from the host.
    • Yes I know it could have all been handled by a single php script. It was written in a big hurry!
    • I have added the following line to my crontab, so that this runs at half eight on Monday morning:
    "30 8 * * 1 cd /var/www/hrhscript/footiecheck/ ; ./football_check.sh #Football check report"