The following shell script finds duplicate (2 or more identical) files and outputs a new shell script containing commented-out rm statements for deleting them (copy-paste from here):
::: updated on 02 May 20121, seems like wordpress did not like it so well so reformatting the code :::::::
#!/bin/bash - #=============================================================================== # # FILE: a.sh # # USAGE: ./a.sh # # DESCRIPTION: # # OPTIONS: --- # REQUIREMENTS: --- # BUGS: --- # NOTES: --- # AUTHOR: Amit Agarwal (aka), amit.agarwal@roamware.com # COMPANY: blog.amit-agarwal.co.in # CREATED: 02/05/12 06:52:08 IST # Last modified: Wed May 02, 2012 07:03AM # REVISION: --- #=============================================================================== OUTF=rem-duplicates.sh; echo "#!/bin/sh" >$OUTF; find "$@" -type f -exec md5sum {} \; 2>/dev/null | sort --key=1,32 | uniq -w 32 -d |cut -b 1-32 --complement |sed 's/^/rm -f/' >>$OUTF
Pretty good one line, I must say 🙂
Related articles
- Bash shell-scripting libraries ” Striving for greatness (dberkholz.com)
- Release Engineering 101 – Build Script (shafiqissani.wordpress.com)
- Shell scripting EC2 for fun and profit (r-bloggers.com)
- Book Review: Linux Shell Scripting Cookbook (books.slashdot.org)
Link to this post!
a: line 2: syntax error near unexpected token `|’
a: line 2: `OUTF=rem-duplicates.sh; echo “#! /bin/sh” > $OUTF; find “$@” -type f -exec md5sum {} ; |sort –key=1,32 |uniq -w 32 -d’
Looks like wordpress messed up the script, here it is again :
OUTF=rem-duplicates.sh;
echo “#!/bin/sh” >$OUTF;
find “$@” -type f -exec md5sum {} ; 2>/dev/null | sort –key=1,32 | uniq -w 32 -d |cut -b 1-32 –complement |sed ‘s/^/rm -f/’ >>$OUTF
You need to be a part of a contest for one of the highest quality blogs on the internet.
I will recommend this blog!