CrunchBang Linux

cchhrriiss121212

#! Junkie

Registered: 2010-03-26

Posts: 357

File transfer to external NTFS/fat32 USB devices not reliable

This is a problem I have had for a while, so I don't think it is related to any recent changes I have made. Have seen it on my main desktop as well as a couple of laptops I have used #! with. USB devices used have been a 8GB flash drive and my Android phone.

When copying files to any external USB device I find that the progress dialogue is inaccurate. In other words, when Thunar (or PCmanfm) says that copying is complete, it is actually still transferring data, which is verified by trying to unmount the device.

Needless to say, unmounting the device before transfer is complete gives me incomplete or corrupted files.

My current workaround is to just copy my files, then wait ~30mins or so for things to finish, but this gets kind of annoying after a while.

How can I actually tell when files are finished transferring? Sometimes iotop shows flush-8:32 at 99% IO but maybe that is not related?

Does anyone else have the same issues? If you want to try it out all you need is a USB storage device and a few GBs of data...

Last edited by cchhrriiss121212 (2011-12-14 22:08:13)

kowloonboy

#! Junkie

Registered: 2011-10-13

Posts: 443

Re: File transfer to external NTFS/fat32 USB devices not reliable

Yeah, I normally do umount, to be safe. Sometime I actually open up gparted, and use it to unmount the USB.

I am using Thunar 1.2.3 and it seems to be much better than before. So no more gparted or umount ever since.

Last edited by kowloonboy (2011-12-13 20:00:36)

---

"To me, the extraordinary aspect of martial arts lies in its simplicity. The easy way is also the right way, and martial arts is nothing at all special; the closer to the true way of martial arts, the less wastage of expression there is." - Bruce Lee

pvsage

Internal Affairs

From: North Carolina

Registered: 2009-10-18

Posts: 13,970

Re: File transfer to external NTFS/fat32 USB devices not reliable

After the copy dialog disappears, go to a terminal window and run `sync`.  When the next command prompt pops up, it's safe to unmount the USB device.

cchhrriiss121212

#! Junkie

Registered: 2010-03-26

Posts: 357

Re: File transfer to external NTFS/fat32 USB devices not reliable

Thanks pvsage, that works nicely.
But I'm still wondering why something like this should be necessary, I mean if the transfer is not complete, then why does my file manager think it is?
Anyone know any more about how the whole process works? Google has not been my friend on this one.

pvsage

Internal Affairs

From: North Carolina

Registered: 2009-10-18

Posts: 13,970

Re: File transfer to external NTFS/fat32 USB devices not reliable

It's because USB devices are automatically mounted in asynchronous mode.  In other words, when files are copied, they're first copied to cache, then dumped from cache to the drive.  Apparently, under certain conditions, some flash drives have shorter life spans if mounted synchronously.   I haven't figured out yet how to change the auto-mount preferences to synchronous...

rhowaldt

#!*$%:)

Registered: 2011-03-09

Posts: 4,396

Re: File transfer to external NTFS/fat32 USB devices not reliable

it is a thing i have learned to live with, but i'd love to see a usable solution for this. never really dove into it myself, might do so in the future, who knows

cchhrriiss121212

#! Junkie

Registered: 2010-03-26

Posts: 357

Re: File transfer to external NTFS/fat32 USB devices not reliable

mynis01

#! Die Hard

From: 127.0.0.1

Registered: 2010-07-02

Posts: 2,005

Re: File transfer to external NTFS/fat32 USB devices not reliable

xaos52

The Good Doctor

From: Planet of the @s

Registered: 2011-06-24

Posts: 4,602

Re: File transfer to external NTFS/fat32 USB devices not reliable

@ OP:
I just copied 10GB of random data from my HD to an external USB disk.
Had no problem at all.
The job ran in the background and takes a bit more than 40seconds per Gig.
Here are the timings:

0.06user 3.59system 0:46.46elapsed 7%CPU (0avgtext+0avgdata 3120maxresident)k
2028208inputs+2053496outputs (0major+248minor)pagefaults 0swaps
0.02user 4.05system 0:43.07elapsed 9%CPU (0avgtext+0avgdata 3136maxresident)k
2028256inputs+2053496outputs (2major+247minor)pagefaults 0swaps
0.04user 3.72system 0:42.43elapsed 8%CPU (0avgtext+0avgdata 3136maxresident)k
2028392inputs+2053600outputs (3major+246minor)pagefaults 0swaps
0.05user 3.89system 0:42.62elapsed 9%CPU (0avgtext+0avgdata 3120maxresident)k
2028008inputs+2053376outputs (3major+246minor)pagefaults 0swaps
0.03user 3.72system 0:42.36elapsed 8%CPU (0avgtext+0avgdata 3136maxresident)k
2028864inputs+2053640outputs (3major+246minor)pagefaults 0swaps
0.02user 3.96system 0:42.67elapsed 9%CPU (0avgtext+0avgdata 3136maxresident)k
2028032inputs+2053384outputs (3major+246minor)pagefaults 0swaps
0.02user 3.76system 0:42.61elapsed 8%CPU (0avgtext+0avgdata 3136maxresident)k
2028200inputs+2053608outputs (3major+247minor)pagefaults 0swaps
0.04user 3.92system 0:44.25elapsed 8%CPU (0avgtext+0avgdata 3136maxresident)k
2028048inputs+2053344outputs (3major+246minor)pagefaults 0swaps
0.01user 3.66system 0:41.94elapsed 8%CPU (0avgtext+0avgdata 3136maxresident)k
2027400inputs+2053520outputs (3major+246minor)pagefaults 0swaps
0.05user 3.76system 0:42.83elapsed 8%CPU (0avgtext+0avgdata 3120maxresident)k
2027936inputs+2053648outputs (3major+245minor)pagefaults 0swaps

All that time I was able to browse the net normally.

My system only has 512 MB of RAM.

BUT: the external drive is used for backups and is formatted ext2.
Here is the output of fdisk -l /dev/sdd

┌─[16:37:02]──[xaos52]──[crunchie]:~/work/free$
└──(6 files, 1001Mb)>>sudo /sbin/fdisk -l /dev/sdd

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

There is no partition table on the disk. It is used as /dev/sdd

What filesystem are you using on the external USB drive?

ivanovnegro

Ivan #000000

From: unstable madness

Registered: 2011-06-02

Posts: 5,432

Re: File transfer to external NTFS/fat32 USB devices not reliable

If the external USB drive is NTFS, it will be slow as hell like on my system with the problems already described like no possibility to browse the web flawlessly and a high CPU usage, at least this is my experience.

Formatting the USB drive to a native Linux file system improves the process.

pvsage

Internal Affairs

From: North Carolina

Registered: 2009-10-18

Posts: 13,970

Re: File transfer to external NTFS/fat32 USB devices not reliable

^ This is certainly an option for thumb drives...not so much for media players and flash memory cards used in external devices that expect/require either NTFS or FAT32.

ivanovnegro

Ivan #000000

From: unstable madness

Registered: 2011-06-02

Posts: 5,432

Re: File transfer to external NTFS/fat32 USB devices not reliable

^ I know, just saying, e.g. if it is an external USB hard drive and you do not need Windows support for it.

cchhrriiss121212

#! Junkie

Registered: 2010-03-26

Posts: 357

Re: File transfer to external NTFS/fat32 USB devices not reliable

Thanks for the suggestions everyone.

Yeah the devices in question are fat32 and ntfs, from what I have seen so far mounting with sync still has the same problems.

I think I will try out using ext4 for my phone's sd card, as I'm only going to be plugging it in to my main machine, and Android 2.3 should support it...

Edit: it appears that the phone will only play nice with fat32, probably needs a custom ROM to work

Last edited by cchhrriiss121212 (2011-12-14 19:08:14)

xaos52

The Good Doctor

From: Planet of the @s

Registered: 2011-06-24

Posts: 4,602

Re: File transfer to external NTFS/fat32 USB devices not reliable

@OP:
I thought it was.
I just mentioned my tests to show that your statement "File transfer to external USB not reliable" needs to be qualified.
Remember this site is indexed for search engines.
If this thread comes up for somebody who is ignorant about linux, it is probably enough to scare him off ever.
So why dont you change the title to reflect the fact that the problem occurs for FAT32 or NTFS formatted partitions?
cheers

cchhrriiss121212

#! Junkie

Registered: 2010-03-26

Posts: 357

Re: File transfer to external NTFS/fat32 USB devices not reliable

^ Done, although I think it looks just as scary to a new user anyway