SEARCH

Enter your search query in the box above ^, or use the forum search tool.

You are not logged in.

#1 2014-01-12 13:18:41

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Backup Solution - your thoughts?

member rebornmechanics started off a thread about building a NAS or doing a home build, to which I commented:

http://crunchbang.org/forums/viewtopic. … 43#p353943

This got me thinking about building a homebrew backup solution, in addition to my server.

Initial plan is to use an existing PC box (probably a Shuttle with sata ports), load it up with two or three 3tb HDDs, install a copy of Ubuntu server or Crunchbang, and once all the data for backup is in place, set it up to poll the server and specific desktops around the house, using an rsync command, to pull in all new files on those machines, maybe even get it to only switch on at a certain time of day to do this, and then switch off again when finished. I'll have to do some thinking around avoiding duplicates, and whether I need to make two copies on separate drives for additional safety (especially for the unique, unrecoverable, personal stuff).

Does this sound like a plan ?

Offline

Help fund CrunchBang, donate to the project!

#2 2014-01-12 15:06:45

damo
#! gimpbanger
From: N51.5 W002.8 (mostly)
Registered: 2011-11-24
Posts: 5,434

Re: Backup Solution - your thoughts?

Raspberry Pi homeserver of course smile  You could run a headless server install, and leave it on permanently - power consumption is tiny
RasPi home server

There is a crunchbang raspi server edition - Pibang Server - but I've not tried that yet


BunsenLabs Group on deviantArt
damo's gallery on deviantArt
Openbox themes
Forum Moderator smile

Offline

#3 2014-01-12 18:57:29

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

Thanks gimpbanger. Bought a raspberry pi played with it for a week, then sold it (that should say all). So it will be a "proper" PC wink

Offline

#4 2014-01-12 19:32:34

rebornmechanics
#! CrunchBanger
Registered: 2012-11-05
Posts: 176

Re: Backup Solution - your thoughts?

It _does_ sound like a plan.
Some food for thought however:
Why even consider using Ubuntu? With a solution like this I'd opt for anything featuring rolling releases.
With rsync you won't have to worry about duplicates. If you have a folder with especially precious files you can just add another rsync job with a different location for that folder alone.

Offline

#5 2014-01-12 19:47:41

porkpiehat
#! Die Hard
Registered: 2012-10-02
Posts: 1,007

Re: Backup Solution - your thoughts?

This is the exact opposite of where I would use a rolling release. I have a similar setup (an old P4 I use as an NFS server) and it runs a Debian stable netinstall. It runs headless, so it has no X and no GUI. I don't want to have to worry about updating it, so Debian stable is perfect; a rolling release would require much more frequent updates. Even #! is overkill for a server -- the server does not need X, Openbox, or any of the GUI apps like Gimp, Geany, Iceweasel, etc, that are installed with #!.

Last edited by porkpiehat (2014-01-12 19:50:13)

Offline

#6 2014-01-12 20:05:09

retf
#! CrunchBanger
From: On top of spaghetti
Registered: 2013-12-25
Posts: 200

Re: Backup Solution - your thoughts?

Agree with @porkpiehat...

Install Debian stable - uncheck everything except ssh server and basic tools... and then only install exactly what you need... also, once the backup system is setup, I'd remove the internet routing for that machine and configure your perimeter firewall to drop all outgoing packets for that host...

Offline

#7 2014-01-12 21:36:06

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

All sound advice smile I'll keep you informed...

Offline

#8 2014-01-14 20:17:05

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

Hmmm seems to be cheaper (UK) to buy external HDDs. Best price I can find is dabs for a Toshiba 3TB Ext HDD for £77 closely followed by a Samsung 3TB Ext HDD for £79. Would mean breaking into the cases to get at the HDDs and could possible put something smaller (capacity) in the enclosures to make use of?

OS wise, am planning on doing a #! install, stripping it back a bit and fixing it to boot to cli, with X as an option (would help if I "ssh -X" in if needed). The PC will have plenty of grunt, so might do some server stuff with it too. Planning to use crontab with some rsync scripting, maybe even using rsnapshot ?

Still waiting for daughter to finish copying all her photos off the PC before a trial build wink

Offline

#9 2014-01-14 20:33:14

damo
#! gimpbanger
From: N51.5 W002.8 (mostly)
Registered: 2011-11-24
Posts: 5,434

Re: Backup Solution - your thoughts?

Joe90 wrote:

....

Still waiting for daughter to finish copying all her photos off the PC before a trial build wink

big_smile My daughter wants a macbook for her 18th (it's a style thing apparently). I've said OK as long as I can have the Acer I got her for her 16th! Then my old laptop can be another element in my own backup solution smile

Last edited by damo (2014-01-14 20:33:44)


BunsenLabs Group on deviantArt
damo's gallery on deviantArt
Openbox themes
Forum Moderator smile

Offline

#10 2014-01-14 20:58:56

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

: smile

Offline

#11 2014-03-23 15:57:41

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

Update (only led to more issues though!)

In a moment of weakness I purchased a Samsung D3 3TB external HDD, seemingly good value at £80 (couldn't buy a 3TB HDD on it's own for that price!

In the process of getting it to work by plugging it straight into my BT Home Hub 3 (router) via USB. The router won't work with ntfs so have to reformat to FAT32. Trying to get rsync to work with this but it is throwing up permissions issues all over the place.

Currently mounting with this:

sudo mount.cifs //192.168.1.254/usb_disk/Samsung_04E80A49_1/ \
/media/Homehub -o iocharset=utf8,uid=1000,gid=1000,umask=022

Which works but when rsyncing with this:

rsync -abuv  /media/FILES/files /media/Homehub/files

I get loads of these:

rsync: mkstemp "/media/Homehub/files/.image1076.png.KQKgAu" failed: Permission denied (13)
rsync: mkstemp "/media/Homehub/files/.image1077.png.QQ91BQ" failed: Permission denied (13)
rsync: mkstemp "/media/Homehub/files/.image1078.png.AXxQHc" failed: Permission denied (13)
rsync: mkstemp "/media/Homehub/files/.image1079.png.ChGzRy" failed: Permission denied (13)
rsync: mkstemp "/media/Homehub/files/.mydesktop.mkv.vlWQ5U" failed: Permission denied (13)

Have scoured google for correct settings for permissions but none I have tried worked. The Homehub has its own samba server with workgroup BT and no password for user LANMAN - perhaps this is it?

~$ smbclient -L 192.168.1.254
Enter joe's password: 
Domain=[HOME] OS=[Unix] Server=[Samba 3.0.37]

	Sharename       Type      Comment
	---------       ----      -------
	USB_Disk        Disk      linux
	IPC$            IPC       IPC Service (Samba Server)
Domain=[HOME] OS=[Unix] Server=[Samba 3.0.37]

	Server             Comment
	---------            -------
	BTHUB3          Samba Server

        Workgroup       Master
	---------            -------
	HOME              BTHUB3

Offline

#12 2014-03-23 16:36:50

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2012-03-22
Posts: 1,026
Website

Re: Backup Solution - your thoughts?

^See this. (I googled 'mkstemp fat32'.)

As I suspected, problem is with FAT32 ... and mkstemp. Pity your router handle FAT32 only.

Perhaps you should give 'fsarchiver' a try ... people say it can overcome some problems related to different file systems. Not sure if this is true, as I never used it so far.


Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

Offline

#13 2014-03-23 16:43:32

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

Great find IMBeCil, I didn't get that back in search sad  I'll give it a try and come back.

Offline

#14 2014-03-23 18:38:41

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

Hmmm, that didn't work completely, getting timestamp errors still. May just be easier to reformat to ntfs and plug it into the server, but I'll try partitioning the 3TB first, as a 500gb EXT HDD in ntfs shows up in the Homehub.

Offline

#15 2014-03-23 18:59:50

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2012-03-22
Posts: 1,026
Website

Re: Backup Solution - your thoughts?

^You are aware that the best backup you gonna get is if your source and target file systems are the same, aren't you?. So, if you are trying to backup files from ext4 file system, the question is of your router recognizes ext4 as a file system ... you may try to format spare USB stick with ext4 for test ...


Postpone all your duties; if you die, you won't have to do them ..
--> The very new BL forum! <--

Offline

#16 2014-03-24 03:33:08

slacktechjoe
Member
Registered: 2014-03-06
Posts: 15

Re: Backup Solution - your thoughts?

Your idea of building a backup point within your house works quite nicely. I did something similar but also wanted to be able to backup files from other OS's as well that don't have rsync ability. the solution for me was crashplan. It's free to download and install the client on any computer you want, they only charge if you want to use the web backup services that they offer which I of course do not. Install the software on all your pc's and run your backups smile Crashplan is cross platform so works on pretty well any nix, windows or mac computer.


I think computer viruses should count as life. I think it says something about human nature that the only form of life we have created so far is purely destructive. We've created life in our own image.
Stephen Hawking

Offline

#17 2014-03-24 10:34:18

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

iMBeCil wrote:

^You are aware that the best backup you gonna get is if your source and target file systems are the same, aren't you?. So, if you are trying to backup files from ext4 file system, the question is of your router recognizes ext4 as a file system ... you may try to format spare USB stick with ext4 for test ...

I'll try it, but don't hold out much hope, given how Windows centric BT are (e.g. BT Sport on a linux machine - yes I know it can be achieved through wine). I think it has something to do with the size of the drive too, as a 500gb NTFS EHDD is recognised.

Offline

#18 2014-03-24 11:02:29

xor
#! Die Hard
Registered: 2012-01-11
Posts: 775

Re: Backup Solution - your thoughts?

i'd go with an external USB drive as a backup solution. found these on amazon a few days ago for myself:

http://www.amazon.de/Elements-Portable- … _1?ie=UTF8

high mobility factor since it does not have/need an external power supply. pricing is acceptable.

Offline

#19 2014-03-24 11:02:57

mynis01
#! Die Hard
From: 127.0.0.1
Registered: 2010-07-02
Posts: 2,005

Re: Backup Solution - your thoughts?

Why not just use FreeNAS?

Offline

#20 2014-03-24 11:47:36

Snap
#! Die Hard
Registered: 2013-10-22
Posts: 567

Re: Backup Solution - your thoughts?

I use an external USB HDD formated in FAT32. I do the backups using Luckybackup. I don't know if it's the best method ever, but works great for me. I want external drives because you can plug them to any PC out there and the FAT32 format can be read by almost any current OS. I also favor using cross-platform apps (email, web browsing, etc...) so my data can be opened/used/rescued on any other OS available.

I've lost data in the past due to the HDD formats and single platform apps. I learned my lesson the hard way.  Never again. My backup disk mounts on Linux, Mac or Windows and I can run my Firefox and Thunderbird profiles from it on the fly.

IMO, NAS is great for shared storage, but never thought of it as a backup solution. I'll check the FreeNAS link anyway.... just in case.  cool

Thanks, mynis01.

Offline

#21 2014-03-24 13:58:50

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

My aim is purely for backup, not shared storage, I have my full server setup for all that so FreeNAS is too much for this job wink

BT Home Hub didn't like an ext4 partition at all, neither did the SAMSUNGD3, grumbled away it did.

So, reformatted to NTFS, single 3tb partition, plugged it into the back of my PC, and rsyncing away quite happily as I speak. It'll be going out into the garage to plug into the server, then it will be automated with some little bash scripts running rsync commands smile

Found NTFS to be very stable, and of course read/writeable by Windows and Linux

Offline

#22 2014-03-24 17:41:20

Snap
#! Die Hard
Registered: 2013-10-22
Posts: 567

Re: Backup Solution - your thoughts?

That's it. Better make the backups as compatible and widely readable as possible if you want to have real backups. Windows HDD formats? Yes, they're universal.   cool

Last edited by Snap (2014-03-24 17:43:58)

Offline

#23 2014-03-24 18:13:50

Different55
#! CrunchBanger
Registered: 2014-03-12
Posts: 216

Re: Backup Solution - your thoughts?

It really doesn't matter as long as it's stable, readable, and writable by all his computers as he says it is.


Sorry about that. I accidentally posted my password in IRC and I had the same password set here so they changed my signature.

Offline

#24 2014-03-30 00:12:08

Joe90
#! Junkie
Registered: 2013-10-10
Posts: 337

Re: Backup Solution - your thoughts?

Mini update: taken perhaps the best part of the last 24 hours to rsync everything off my server and my desktop onto the External HDD, through USB 2.0 and over 10/100 LAN. Used the following rsync command:

rsync -abuvzhP --log-file="/home/user/rsync/rsync.$(date +%Y%m%d%H%M%S).backup.log" /source/directory/ /media/SAMSUNGD3/target/directory/

with @ 8 different sources and targets. rsync reported a few errors, so by adding "grep | failed" to the end of the above command I could pin point the errors. Mostly due to permissions (root/root) or invalid encoding (umlauts/egraves, carets etc.) Edited those things and rerun until all clear.

Because I am rsyncing from ext4 to ntfs the output from rsync is completely verbose it throws up every directory it covers. To get at the files that are actually rsync'd I wrote a little script to run on the log files afterwards:

#!/bin/bash

# for each file in directory that starts with rsync
# grep all the entries of actual file transfers
# create new file or append to existing file all 
# the entries, then delete/backup original file

#run script on each file
for i in rsync*; do

#grepping
#create / append output

newname=${i:21}
grep -w ">f+++++++++" $i >> $newname

#backup original file

mv $i .~$i

#finish when all files are done
done

Now the initial rsync is over, I can enjoy much shorter backup times. I have 1.2 tb of space left to use, that'll take a while!

2 Issues still remain:

1. Still feeling a little uncomfortable about relying on a HDD instead of a DVD-R as a back up medium....

2. Although the External HDD is "automounted" by #!, it needs to be woken up by clicking on it in thunar before running my rsync scripts. How to wake it up through the script ?

Answer: make entries in fstab to mount on startup, then

put this at the front of any script:

ls /media/SAMSUNGD3 > /dev/null
sleep 10
other stuff here

Last edited by Joe90 (2014-03-30 09:43:12)

Offline

Be excellent to each other!

#25 2014-03-30 06:44:08

Alad
Software Satan
Registered: 2014-02-20
Posts: 1,512

Re: Backup Solution - your thoughts?

Although the External HDD is "automounted" by #!, it needs to be woken up by clicking on it in thunar before running my rsync scripts. How to wake it up through the script ?

ls /media/SAMSUNGD3 > /dev/null

How does ls "wake it up" when the drive isn't mounted? Isn't that what clicking it in Thunar does?

Last edited by Alad (2014-03-30 06:45:12)

Offline

Board footer

Powered by FluxBB

Copyright © 2012 CrunchBang Linux.
Proudly powered by Debian. Hosted by Linode.
Debian is a registered trademark of Software in the Public Interest, Inc.
Server: acrobat

Debian Logo