esync.sh - Script to easily synchronize two directories

Doomicide · 2011-07-16 14:13:04

Hey, I had some time on my hands and wrote myself a Script to easily sync my backup-directory with the original directory. So after I worked on it for some days (I'm somewhat new to bash), I decided to post it here to, hopefully, contribute something to the community. While writing this script, I learned a lot about bash, but I'm sure there's still a bunch of stuff in there, that can be done better or more elegant.
(And yes, I know about rsync, but this script is faster and easier to handle, though of course by far not as powerful and flexible as rsync.)
So here you go: (Constructive criticism is always appreciated. smile )

#!/bin/bash

#get script name
script=`basename $0`

#define quiet as `n¸
quiet=n

#define usage
usage ()
{
cat  << EOF
usage: $script [options] DIR1 DIR2 [LISTDIR]

This script synchronizes DIR2 with DIR1.

options:
    -h    Show this message
    -q    Quiet (Exeption: If -y or -n are not given, ask before removing extraneous files in DIR2)
    -c    Create DIR2 if it doesn't exist
    -y    Don't ask before removing extraneous files in DIR2, always remove them
    -n    Don't ask before removing extraneous files in DIR2, never remove them

DIR1:      Source-directory (e.g.: /home/user/)
DIR2:      Destination-directory / directory to synchronize with DIR1 (e.g.: /media/windows/backup/)
LISTDIR: Directory where the lists will be stored. If missing, list will be stored in: $HOME/.sync-list-tmp/1/1-1 and removed afterwards
EOF
}

#get options
while getopts ":ynqch" OPTION ${wambo[@]}  ; do 
case $OPTION in 
        y)
           rmfiles="y"
           ;;
        n) 
           rmfiles="n"        
           ;;
        q) 
           quiet="y"
           ;;
        c)
           create="y"
           ;;
        h)
            usage
            exit 1    
            ;;
        ?)
           usage
           exit
           ;;
esac;  
done


#reset `$@¸ and `$#¸
shift $(($OPTIND-1))


#check positional parameters
if [ ! $# == 2 ] && [ ! $# == 3 ]; then
    if [ ! "$quiet" = "y" ]; then
        usage
    fi
    exit

else
    
    #set directories & the date etc.
    
    number=1
    
    dir1="$1"   

    dir2="$2"
    
    if [ ! -z $3 ]; then

        dat=`date +%y%m%d`
    
        hour=`date +%H%M`
    
        bs=`basename $dir1`
    
        listdir="$3/$dat"
    
        rmld="n"

    else 
        
        dat=1
        
        hour=1
        
        bs=1
        
        listdir="/$HOME/.sync-list-tmp/"
        
        rmld="y" #remove listdir afterwards
    
    fi
fi



#check if destination-directory doesn't exists & if -c , if both is true create it
if [ ! -d $dir2 ]; then 
    if [ "$create" = "y" ]; then 
        mkdir -p $dir2
    else
        echo "ERROR - destination-directory: $dir2 was not found & -c flag is not set."
        echo "Quitting..."
        exit
    fi
fi

if [ ! -d $dir1 ]; then
    if [ ! "$quiet" = "y" ]; then
        echo "ERROR - source-directory: $dir1 was not found"
        echo "Quitting..."
    fi
    exit
fi    

#copy new(er) files to $dir2
if [ ! "$quiet" = "y" ]; then
    echo "Starting to synchronize $dir2 with $dir1..."
fi
cd $dir1
cp -ru * $dir2

cp -ru .??* $dir2   #copy hidden files

if [ ! "$quiet" = "y" ]; then
    echo "Finished Copying - Searching for differences..."
fi


#create dirlists & log-directorys 

#create listdir if it doesn't already exist
if [ ! -d $listdir ]; then
    mkdir -p $listdir
fi

#create sub-directory in listdir for this execution
while [ -d $listdir/$bs-$hour-$number ]; do
    number=$((number+1))
done    

mkdir $listdir/$bs-$hour-$number

#create dirlists of dir1 and dir2 

cd $dir1
find . -name '*' | cut -n -b 3- > $listdir/$bs-$hour-$number/dirlist1

cd $dir2
find . -name '*' | cut -n -b 3- > $listdir/$bs-$hour-$number/dirlist2




#compare files & create list of unique files in dir2

cd $listdir/$bs-$hour-$number

cat $listdir/$bs-$hour-$number/dirlist1 $listdir/$bs-$hour-$number/dirlist2 >> $listdir/$bs-$hour-$number/dirlistall

sort $listdir/$bs-$hour-$number/dirlistall | uniq -u > $listdir/$bs-$hour-$number/dirlistrm-tmp

cd $dir2

while read line;
do
    if [[ -a $line ]]; then
        echo $line >> $listdir/$bs-$hour-$number/dirlistrm
    fi
done < $listdir/$bs-$hour-$number/dirlistrm-tmp

if [[ -f $listdir/$bs-$hour-$number/dirlistrm-tmp ]]; then
    rm $listdir/$bs-$hour-$number/dirlistrm-tmp
fi




#ask for permission to rm unique files in dir2 & do it if permission is given

cd $dir2

if [[ -s $listdir/$bs-$hour-$number/dirlistrm ]]; then
    
    if [ ! "$quiet" = "y" ]; then
        cat $listdir/$bs-$hour-$number/dirlistrm
    fi
    if [ -z $rmfiles ]; then
            echo "Delete the files listed above, in $dir2? (They don't exist in $dir1.)"
            echo "Type "y" for yes or "n" or no!"
            read answer
    fi    
    if [ "$answer" == "y" ] || [ "$rmfiles" == "y" ]; then

        if [ ! "$quiet" = "y" ]; then
            echo "Deleting files..."
        fi    
            cd $dir2
            
            while read line; 
                do rm -rf "$dir2/$line"; 
            done < $listdir/$bs-$hour-$number/dirlistrm
            
        if [ $? == 0 ]; then 
                
                if [ ! "$quiet" = "y" ]; then
                    echo "Files deleted"
                fi
        else
                if [ ! "$quiet" = "y" ]; then
                    echo "ERROR - See the list in $listdir/$bs-$hour-$number for more information."
                    echo "Saving list..."
                fi
        fi
    
    elif [ "$answer" == "n" ] || [ "$rmfiles" = "n" ]; then
        if [ ! "$quiet" = "y" ]; then
            echo "Files not deleted - Saving list..."
        fi
            cd $dir2
            find . -name '*' | cut -n -b 3- > $listdir/$bs-$hour-$number/dirlist2-after
            
            
    
            cat $listdir/$bs-$hour-$number/dirlist1 $listdir/$bs-$hour-$number/dirlist2-after >> $listdir/$bs-$hour-$number/dirlistall-after
            
            sort $listdir/$bs-$hour-$number/dirlistall-after | uniq -u > $listdir/$bs-$hour-$number/check

            
        if [ ! "$quiet" = "y" ]; then
            echo "Quitting..."
        fi
        
        #remove list if listdir is not given
        if [ $rmld = "y" ]; then
            rm -rf $listdir
        fi
        exit    

    else
        if [ ! "$quiet" = "y" ]; then
            echo "Incorrect Input - Quitting..."
        fi

        #remove list if listdir is not given
        if [ $rmld = "y" ]; then
            rm -rf $listdir
        fi
        exit

    fi
else 
    if [ ! "$quiet" = "y" ]; then
        echo "No differences found!"
    fi
fi


        

#create dirlist of dir2 after rm and cp & check if contents  of dir1 and dir2 are identical


cd $dir2
find . -name '*' | cut -n -b 3- > $listdir/$bs-$hour-$number/dirlist2-after



cat $listdir/$bs-$hour-$number/dirlist1 $listdir/$bs-$hour-$number/dirlist2-after >> $listdir/$bs-$hour-$number/dirlistall-after

sort $listdir/$bs-$hour-$number/dirlistall-after | uniq -u > $listdir/$bs-$hour-$number/check


#remove the listdir entries from the check-log & determine dir of unique files


cd $dir2

#create list of files, only existing in $dir2

if [ -s $listdir/$bs-$hour-$number/check ]; then
    while read line; do  
        if [[ -f $line ]]; then
            echo $line >> $listdir/$bs-$hour-$number/check-dir2
        fi
    done < $listdir/$bs-$hour-$number/check


    
    cd $dir1

    #create list of files, only existing in $dir1
    while read line; do 
        if [[ -f $line ]]; then
            echo $line >> $listdir/$bs-$hour-$number/check-dir1-tmp
        fi
    done < $listdir/$bs-$hour-$number/check

    

    #go to $listdir
    cd $listdir/$bs-$hour-$number/

    #remove obsolete "check" log & clean up
    rm check  

    if [[ -s check-dir1 ]]; then
        while read line; do
            echo $line | grep $listdir/$bs-$hour-$number >> $listdir/$bs-$hour-$number/log-files-check
        done < $listdir/$bs-$hour-$number/check-dir1
    
        if [[ -s log-files-check ]]; then
            cat check-dir1-tmp log-files-check >> $listdir/$bs-$hour-$number/check-dir1-tmp2
            sort check-dir1-tmp2 | uniq -u > $listdir/$bs-$hour-$number/check-dir1
            rm check-dir1-tmp check-dir1-tmp2
        else 
            mv check-dir1-tmp check-dir1
        fi
            
    fi
fi


#tell user if successfull or not
cd $listdir/$bs-$hour-$number

if [ ! -f check-dir1 ]  || [ ! -s check-dir1 ] && [  ! -f check-dir2 ] || [ ! -s check-dir2 ]; then
    if [ ! "$quiet" = "y" ]; then
        echo  "Synchronization successful - Quitting..."
    fi
else
    if [ ! "$quiet" = "y" ]; then
        echo "ERROR"
        if [ -s check-dir1 ] && [ -s check-dir2 ]; then
            echo "$dir1 & $dir2 are not identical: Unique files in both."
        elif [ -s check-dir1 ] && [ ! -s check-dir2 ]; then
            echo "$dir1 & $dir2 are not identical: Unique files in $dir1."
        elif [ ! -s check-dir1 ] && [ -s check-dir2 ]; then
            echo "$dir1 & $dir2 are not identical: Unique files in $dir2."
        fi
        if [ $rmld = "n"            
        echo "See the lists in $listdir/$bs-$hour-$numberH for more information."
        fi
    fi
fi

#remove lists if listdir is not given
if [ $rmld = "y" ]; then
    rm -rf  $listdir
fi

#exit 

exit

For people to lazy to copy and paste wink : Download

jmbarnes · 2011-07-16 15:23:05

Interesting...

Do you know about rsync? If so, is there a reason you've chosen your method over a couple of rsync commands (with a --delete flag) perhaps?

johnraff · 2011-07-16 17:29:02

Very impressive, but is it really faster than rsync?
rsync has this protocol which only sends the changed parts of files, so I'd have thought it would be hard for cp to compete.

Doomicide · 2011-07-16 22:03:54

jmbarnes wrote:

Interesting...
Do you know about rsync? If so, is there a reason you've chosen your method over a couple of rsync commands (with a --delete flag) perhaps?

Yes as stated above I know about rsync and also about the --delete flag wink And the reason to write this Script was, that I didn't know about rsync when I actually started writing it big_smile

johnraff wrote:

Very impressive, but is it really faster than rsync?
rsync has this protocol which only sends the changed parts of files, so I'd have thought it would be hard for cp to compete.

Mhm, I let both run at the same time to synchronize two different directories with the same content with my home-directory and even though I started rsync earlier, my Script was finished before rsync... I think the tests rsync runs take longer, but there's also the possibility, that I ran rsync with obsolete options or so...

johnraff · 2011-07-18 17:02:57

...also the advantage of rsync's protocol might come out more when you're on a slow data path - like over an ssh connection, or to a slow usb stick...

Doomicide · 2011-07-18 18:35:51

johnraff wrote:

...also the advantage of rsync's protocol might come out more when you're on a slow data path - like over an ssh connection, or to a slow usb stick...

As stated above I know that rsync is more powerful and flexible than my script.... This was written solely for the purpose of creating a backup on your hardrive or second hardrive or whatever or just synchronize 2 directories on you pc (e.g. for Dropbox) without having to read the whole rsync man page etc... And of course I wrote it to improve my bash knowledge and get some feedback wink

CrunchBang Linux

SEARCH

#1 2011-07-16 14:13:04

esync.sh - Script to easily synchronize two directories

#2 2011-07-16 15:23:05

Re: esync.sh - Script to easily synchronize two directories

#3 2011-07-16 17:29:02

Re: esync.sh - Script to easily synchronize two directories

#4 2011-07-16 22:03:54

Re: esync.sh - Script to easily synchronize two directories

#5 2011-07-18 17:02:57

Re: esync.sh - Script to easily synchronize two directories

#6 2011-07-18 18:35:51

Re: esync.sh - Script to easily synchronize two directories

Board footer