SEARCH

Enter your search query in the box above ^, or use the forum search tool.

You are not logged in.

#1 2014-12-02 11:23:26

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Copying and syncing files [SOLVED]

I have two external USB drives Drive A and Drive B (ext4) hooked up to my computer. I want to copy the files from one drive to another. At the moment I am using this code and it seems to be taking a long time to copy the files over. Is there a better more efficient and speediy way to do this. I want USB drive B to be identical to USB drive A. The files are FLAC files of my digitized LP's. I want to make sure I have copies of them just in case. I am still in the process of digitizing LP's so I would need to continuously backup/sync to Drive B.

A straight copy/paste goes faster than what I am doing with rsync.


* * * * * /usr/bin/rsync -a --delete /media/masterlps/lps/H/ /media/lpsbackup/lps/H

I tried using this but it was really taking a long time (days) so I thought I would break it down and it's still taking a long time

* * * * * /usr/bin/rsync -a --delete /media/masterlps/lps/ /media/lpsbackup/lps

Last edited by rmcellig (2014-12-02 15:55:20)


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

Be excellent to each other!

#2 2014-12-02 11:53:53

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2012-03-22
Posts: 1,026
Website

Re: Copying and syncing files [SOLVED]

rmcellig wrote:

I have two external USB drives Drive A and Drive B (ext4) hooked up to my computer. I want to copy the files from one drive to another. At the moment I am using this code and it seems to be taking a long time to copy the files over. Is there a better more efficient and speediy way to do this.

Define "taking a long time to copy the files" smile AKA, how large USB drives, i.e. how much data you have. And how long does it take?

Note: it is more-less normal that file operations over USB are slow ... In particular, when multiple programs/tasks need to access USB. For example, it is possible that your computer USB ports cannot read from Drive A and Drive B concurrently. Instead they do it sequentially: first read chunk of Drive A, then write this chunk to DriveB. Perhaps you can change USB ports ... maybe two of them can work concurrently.

rmcellig wrote:

I am still in the process of digitizing LP's so I would need to continuously backup/sync to Drive B.

In my experience, first backup is the longest one. Afterwards, 'rsync' will do sort of incremental backup, i.e. it will only copy new files, and delete missing files. Files which are same as before (on Drive A) won't be copied again. In effect, subsequent 'rsync' is expected to be quite faster.


Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

Offline

#3 2014-12-02 12:12:16

luc
#! Die Hard
From: Munich, Germany
Registered: 2010-03-21
Posts: 597

Re: Copying and syncing files [SOLVED]

Questions

  1. What exactly do you mean by "A straight copy/paste goes faster than what I am doing with rsync"? Which command is faster then which?

  2. Do you want the actual hard disks to be identical or rather (which sounds more reasonable to me) some directory structure to be identical between the two disks?

  3. Do you want to have snapshot like directories or only one "mirror" on the backup disk?

Tips/Hints/Answers

One of the simplest but already interesting (i.e different from plain cp) use cases for rsync is this command:

rsync --archive --update --delete /the/original/location /the/backup/location

The --update will only copy files that changed in the original location to the backup location (new files and modified files). Files that did not change will not be copied so the backup will take shorter the second time you run it.

The --delete option you used will delete all the files that are not present in the original location any more and delete them from the backup location.

You can use the --verbose and --stats options to get some more information on the files that are processed and the efficiency.

For the backup of my home folder and my system settings I use something similar to

new=$(date +%F)
rsync --archive --update --verbose --stats --exclude={some,files} --delete-during --delete-excluded --one-file-system --link-dest=../$(readlink /the/backup/location/latest) $HOME/ /the/backup/location/$new
ln --symbolic --force --no-dereference $new /the/backup/location/latest

This will create folders like this:

/the
└── backup
    └── location
        ├── 2014-10-01
	│   └── ...
        ├── 2014-11-01
	│   └── ...
        ├── 2014-12-01
	│   └── ...
	└── latest -> 2014-12-01

Insted of the dots there is the full directory structure of my home folder exactly as it was on that date.  But all files that did not change between two backups are hardlinked so the backups do not take up as much space and do not take as long to create, as they would if I make them with cp.  (For your use case the options --exclude, --delete-excluded and --one-file-system might not be very interesting.)

You can read about the exact semantics of all these options in the man page and you can also have a look at an interesting wiki articles about rsync.

Best
Luc

Offline

#4 2014-12-02 13:09:28

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Re: Copying and syncing files [SOLVED]

Both external USB drives are 1TB in size and are plugged directly into the USB ports on the Dell 3000. I'm not going through a USB hub.

By a straight copy/paste, I mean selecting a few FLAC files (say 8), and just copying/pasting them to Drive B. That may take 10 minutes to do, if that.

By taking a long time, I mean days to sync the files on Drive a to Drive B. What I am going to do is, as a test, choose one of my folders that contain about 10 FLAC files and see how long it takes to sync them to drive B. My largest folders so far are A, B, C, D. I am now going through the letter E in my record LP collection.

I'll try the test I mentioned above and post back with the results. I agree that the first time can take a long time.

Thanks for the advice!!


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

#5 2014-12-02 13:21:39

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2012-03-22
Posts: 1,026
Website

Re: Copying and syncing files [SOLVED]

^ You're welcome.

Just a side note: USB 2.0 standard transfer speed is about 60 MBytes/sec, in theory. Often smaller (depends on various factors). In ten minutes, you can copy max 36 GBytes of data.

If you have USB3.0, max data rate is roughly 10-12 times faster .... in theory  wink


Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

Offline

#6 2014-12-02 13:26:37

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Re: Copying and syncing files [SOLVED]

It took around two and a half minutes to copy 2.3GB of data (FLAC files) from drive A to drive B. When I used rsync, after over 35 minutes it was still syncing one file.

I think I will just copy the folders from Drive A to drive B and then implement rsync to keep both drives in sync unless there is another option. I can post the verbose output when trying to sync if that will be helpful.


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

#7 2014-12-02 13:40:10

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Re: Copying and syncing files [SOLVED]

I am running the rsync code verbose directly from a terminal window instead of using Cron. This way I can see if there are any hickups happening. I was successful syncing the letter K folder (2.1GB), in just under 3 minutes so that's good.

What I was seeing before when I had show hidden files on, were a few . files that were created and they were all dealing with the same file. Now I only see one . file at a time that corresponds to the FLAC file being processed if that makes any sense.


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

#8 2014-12-02 15:36:47

luc
#! Die Hard
From: Munich, Germany
Registered: 2010-03-21
Posts: 597

Re: Copying and syncing files [SOLVED]

If I understand your last post correctly there are some hidden files in the original location that you do not want to appear in the backup (presumably they are temporarily created by the program that produces the flac files from your LPs). If that is the case you can add the --exclude option with a suitable pattern to your rsync command line. Read the man page for the syntax of the pattern and be careful to quote it for your shell.

The other interpretation of your last post can be that you have hard linked files in the original location. In this case the --hard-links option might be of interest to you.

Offline

#9 2014-12-02 15:54:36

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Re: Copying and syncing files [SOLVED]

Thanks Luc. What I decided to do was run the rsync code I have been using, from the terminal instead of a crontab. So far it is working great and I am almost finished doing the initial syncing. Once this is done than I will run rsync from a crontab set to run every hour or something like that.

Thanks to everyone who submitted posts to this thread as it will greatly help me in the future.


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

#10 2014-12-02 16:13:26

Head_on_a_Stick
CatMod
From: A world of pure imagination
Registered: 2014-01-21
Posts: 4,797

Re: Copying and syncing files [SOLVED]

Randy -- you should consider always doing the backups manually: it's too important an operation to be left to a cronjob (IMO).
If there are any problems with the cronjob, the first you'll know of it is when your backup fails to load...

Offline

#11 2014-12-02 16:19:36

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Re: Copying and syncing files [SOLVED]

Thanks Head_on_a_Stick!

I will manually run my regular rsync code in the terminal instead of a cronjob My sync issues that I had mentioned in my initial post were probably because I was running the code in a crontab. I just thought that automating the sync's would require less of me to get involved but I will do it manually from now on.


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

#12 2014-12-02 16:22:36

damo
#! gimpbanger
From: N51.5 W002.8 (mostly)
Registered: 2011-11-24
Posts: 5,434

Re: Copying and syncing files [SOLVED]

^^ +1

@Randy   You could put your rsync command in a script, and run that with a keyboard shortcut. I do something similar, and my script also has a test to make sure the usb drive is connected/mounted.

Last edited by damo (2014-12-02 16:22:57)


BunsenLabs Group on deviantArt
damo's gallery on deviantArt
Openbox themes
Forum Moderator smile

Offline

#13 2014-12-02 16:26:03

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Re: Copying and syncing files [SOLVED]

Good idea damo!

How would I write the code to check the drive like you mentioned in your previous post?

Last edited by rmcellig (2014-12-02 16:43:31)


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

#14 2014-12-02 16:29:01

luc
#! Die Hard
From: Munich, Germany
Registered: 2010-03-21
Posts: 597

Re: Copying and syncing files [SOLVED]

I partly disagree with Head_on_a_Stick: It is true that one has to take extra care when running a backup script/command with cron (or similar). But I still think it can be a good solution to run such a command (especially such an important command) automatically/periodically because the user is not required to remember to do it, or maybe is to lazy to execute the command every hour. But in this case it is important to thoroughly test the command before leaving it to cron (and maybe check the integrety of the backups after first doing so).

But now I start to go off topic ... Sorry.

Good that you found a solution that works.

Offline

#15 2014-12-02 16:46:52

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Re: Copying and syncing files [SOLVED]

i do small tests all the time so I will try this as well just to see what it does and how it behaves. Instead of having the cron run every minute I will try something like every 5 hours or something like that on a test folder.


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

#16 2014-12-02 17:29:58

damo
#! gimpbanger
From: N51.5 W002.8 (mostly)
Registered: 2011-11-24
Posts: 5,434

Re: Copying and syncing files [SOLVED]

rmcellig wrote:

Good idea damo!

How would I write the code to check the drive like you mentioned in your previous post?

I use a function which is called at  the appropriate point in the script.
I have hard-coded paths, so the script isn't portable as it stands, but assuming the drive is "/dev/sdc2", and the mountpoint is "/media/usb_data"

CheckMount(){
	if grep -qs '/media/usb_data' /proc/mounts; then
		echo "ExternalHDD partition '/media/usb_data' is mounted..."
		echo
	else
		echo "ExternalHDD partition is not mounted..."
		read -sn 1 -p "Connect drive, then press any key to continue:" 
		echo
		sudo mount /dev/sdc2 /media/usb_data
		sleep 0.5
		echo "Mounted on '/media/usb_data'"
	fi
}

And in anticipation of your next request ]:D
Bash scripting tutorial
Bash Functions


BunsenLabs Group on deviantArt
damo's gallery on deviantArt
Openbox themes
Forum Moderator smile

Offline

#17 2014-12-02 20:50:45

rmcellig
#! Die Hard
From: Ottawa, Canada
Registered: 2012-11-15
Posts: 624
Website

Re: Copying and syncing files [SOLVED]

Thanks dano!

I'll try this out.


Cheers Randy
www.mcran.com - my web site
www.chuo.fm - My radio show Sundays  noon-2pm EST or 89.1 fM

Offline

#18 2014-12-03 03:28:16

johnraff
nullglob
From: Nagoya, Japan
Registered: 2009-01-07
Posts: 4,148
Website

Re: Copying and syncing files [SOLVED]

Luc, don't want to nitpick, but:

luc wrote:

The --update will only copy files that changed in the original location to the backup location (new files and modified files). Files that did not change will not be copied so the backup will take shorter the second time you run it.

Just for the record, it's my understanding that this is what rsync does anyway, even without that option. smile

My reading of man rsync is that the --update option will cause rsync to skip files at the destination that are newer than the equivalent in the source, even if the source file is different.

I have been known to be wrong on occasions though...


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
#! forum moderator    BunsenLabs

Offline

#19 2014-12-03 12:17:23

luc
#! Die Hard
From: Munich, Germany
Registered: 2010-03-21
Posts: 597

Re: Copying and syncing files [SOLVED]

@john I have been known to forget to double check the man page on occasions though ...

Offline

Board footer

Powered by FluxBB

Copyright © 2012 CrunchBang Linux.
Proudly powered by Debian. Hosted by Linode.
Debian is a registered trademark of Software in the Public Interest, Inc.
Server: acrobat

Debian Logo