Find the largest files on your system from the command line.

xaos52 · 2011-10-07 13:14:57

I have been using this perl script for years.
Found it on the internet.
Can not remember where though:

#!/usr/bin/perl
@queue = @ARGV;
while ($current = shift (@queue) ) {
       opendir DH,$current;
       @content = readdir DH;
       foreach $item (@content) {
               next if ($item =~ /^\.{1,2}$/);
               $fullname = "$current/$item";
               if (-l $fullname) { next; }
               if (-d $fullname) {
                       push @queue,$fullname;
                       next;
                       }
               if (-s $fullname > 100000) {
                       $big{$fullname} = -s $fullname;
                       }
               }
       }
foreach $item (sort {$big{$b}<=>$big{$a}} keys %big) {
       printf ("%8d %s\n",$big{$item}, $item);
       }

Save this script in a directory on your PATH (e.g. find-biggest-files.pl )and make it executable.
There is a hard limit in there of 100000. Smaller files are ignored.
To find the biggest files in your root dir:

find-biggest-files.pl /

PS:
1. it will search all mounted partions. If you dont want that, umount them first.
2. you can specify multiple directories to be searched as arguments on the command line.

hth

luc · 2011-10-07 13:48:13

on my machine this runs much faster:

find -x some/dir/foo ! -type d -exec du -d 0 {} + | sort -n

or if you do not care to skip the directories:

du -xa some/dir/foo | sort -n

Note: the -x option will case both find and du to stay on the partition so you do not have to unmount something first..

ErSandro · 2011-10-07 14:01:40

Very cool.

Try ncdu too, it's in the repositories.

Awebb · 2011-10-07 14:18:58

@luc: What du implementation is this? I'm @work, sitting in front of a recent Ubuntu box, seing no -d switch in the manpage. Doh!

luc · 2011-10-07 15:41:40

@awebb: thats bsd du. you have to use "--max-depth=0" or "--summarize". Sorry didn't think about that wink

johnraff · 2011-10-07 16:01:38

^me neither on Squeeze. Could '-d 0' be an alias for '--max-depth?=0' ?

edit oops, sorry Luc, you got in with the answer while I was posting that.

Last edited by johnraff (2011-10-08 04:53:22)

xaos52 · 2011-10-07 16:46:50

OK,
replaced perl script by

du -xa some/dir/foo | sort -n

thanks luc

luc · 2011-10-07 18:00:16

I apologize for the confusion about the -d option. I just retested my code and I think it is not needed at all, as directories are not picked up by find due to "! -type d". But you where right john. It is the BSD version of the GNU option "--max-depth".
The new code is now:

 find -x some/dir/foo ! -type d -exec du {} + | sort -n

xaos52 · 2011-10-07 18:29:35

On debian:

find some/dir/foo -xdev ! -type d -exec du {} + | sort -n

so I propose:

find some/dir/foo -xdev ! -type d -exec du {} + | sort -nr | head

for top 10 biggest files.

johnraff · 2011-10-08 05:03:48

Maybe add du's -h option for human-readable sizes?

find some/dir/foo -xdev ! -type d -exec du -h {} + | sort -nr | head

luc · 2011-10-08 06:51:23

but can "sort" sort that in the right way? I get

...
9K  file0
10M  file1
10K  file2
...

if I use "sort -n"

johnraff · 2011-10-11 17:39:20

^good point. I can't think of any way round that.

Awebb · 2011-10-11 17:43:08

AWK! That's not only the sound the creature on your userpic makes ;-)

johnraff · 2011-10-12 05:01:24

^...true... any suggestions as to how to get 2M to display above 10k with awk?

...or maybe better to sort first, then convert 20000 to 20K etc?

Awebb · 2011-10-12 07:40:18

Yes. Sort first, then convert the numbers. I must admit I'd write an ugly chain of sed+regex, because my AWK-fu is still in the sign of a very low Kyū...

aiBo · 2011-10-12 09:03:00

luc wrote:

but can "sort" sort that in the right way? I get
...
9K  file0
10M  file1
10K  file2
...
if I use "sort -n"

This can be achieved with

$ sort -h

sort man page wrote:

-h, --human-numeric-sort
compare human readable numbers (e.g., 2K 1G)

Awebb · 2011-10-12 10:07:47

Haha, sometimes a man page is a man page is a man page.

johnraff · 2011-10-12 11:09:31

aiBo wrote:

This can be achieved with
$ sort -h

WOW vYTUxZQ

luc · 2011-10-12 18:49:29

blame on me sad sorry guys for making a fuss after not rtfm.

johnraff · 2011-10-13 16:56:03

^we're all equally guilty. neutral

CrunchBang Linux

SEARCH

#1 2011-10-07 13:14:57

Find the largest files on your system from the command line.

#2 2011-10-07 13:48:13

Re: Find the largest files on your system from the command line.

#3 2011-10-07 14:01:40

Re: Find the largest files on your system from the command line.

#4 2011-10-07 14:18:58

Re: Find the largest files on your system from the command line.

#5 2011-10-07 15:41:40

Re: Find the largest files on your system from the command line.

#6 2011-10-07 16:01:38

Re: Find the largest files on your system from the command line.

#7 2011-10-07 16:46:50

Re: Find the largest files on your system from the command line.

#8 2011-10-07 18:00:16

Re: Find the largest files on your system from the command line.

#9 2011-10-07 18:29:35

Re: Find the largest files on your system from the command line.

#10 2011-10-08 05:03:48

Re: Find the largest files on your system from the command line.

#11 2011-10-08 06:51:23

Re: Find the largest files on your system from the command line.

#12 2011-10-11 17:39:20

Re: Find the largest files on your system from the command line.

#13 2011-10-11 17:43:08

Re: Find the largest files on your system from the command line.

#14 2011-10-12 05:01:24

Re: Find the largest files on your system from the command line.

#15 2011-10-12 07:40:18

Re: Find the largest files on your system from the command line.

#16 2011-10-12 09:03:00

Re: Find the largest files on your system from the command line.

#17 2011-10-12 10:07:47

Re: Find the largest files on your system from the command line.

#18 2011-10-12 11:09:31

Re: Find the largest files on your system from the command line.

#19 2011-10-12 18:49:29

Re: Find the largest files on your system from the command line.

#20 2011-10-13 16:56:03

Re: Find the largest files on your system from the command line.

Board footer