The following shell script finds duplicate (2 or more identical) files and outputs a new shell script containing commented-out rm statements for deleting them (copy-paste from here):
::: updated on 02 May 20121, seems like wordpress did not like it so well so reformatting the code :::::::
#!/bin/bash -
#===============================================================================
#
# FILE: a.sh
#
# USAGE: ./a.sh
#
# DESCRIPTION:
#
# OPTIONS: ---
# REQUIREMENTS: ---
# BUGS: ---
# NOTES: ---
# AUTHOR: Amit Agarwal (aka), amit.agarwal@roamware.com
# COMPANY: blog.amit-agarwal.co.in
# CREATED: 02/05/12 06:52:08 IST
# Last modified: Wed May 02, 2012 07:03AM
# REVISION: ---
#===============================================================================
OUTF=rem-duplicates.sh;
echo "#!/bin/sh" >$OUTF;
find "$@" -type f -exec md5sum {} \; 2>/dev/null | sort --key=1,32 | uniq -w 32 -d |cut -b 1-32 --complement |sed 's/^/rm -f/' >>$OUTF
Pretty good one line, I must say 🙂
Related articles
- Bash shell-scripting libraries ” Striving for greatness (dberkholz.com)
- Release Engineering 101 – Build Script (shafiqissani.wordpress.com)
- Shell scripting EC2 for fun and profit (r-bloggers.com)
- Book Review: Linux Shell Scripting Cookbook (books.slashdot.org)
a: line 2: syntax error near unexpected token `|’
a: line 2: `OUTF=rem-duplicates.sh; echo “#! /bin/sh” > $OUTF; find “$@” -type f -exec md5sum {} ; |sort –key=1,32 |uniq -w 32 -d’
Looks like wordpress messed up the script, here it is again :
OUTF=rem-duplicates.sh;
echo “#!/bin/sh” >$OUTF;
find “$@” -type f -exec md5sum {} ; 2>/dev/null | sort –key=1,32 | uniq -w 32 -d |cut -b 1-32 –complement |sed ‘s/^/rm -f/’ >>$OUTF
You need to be a part of a contest for one of the highest quality blogs on the internet.
I will recommend this blog!