The following shell script finds duplicate (2 or more identical) files and outputs a new shell script containing commented-out rm statements for deleting them (copy-paste from here):

::: updated on 02 May 20121, seems like wordpress did not like it so well so reformatting the code :::::::

#!/bin/bash -
#===============================================================================
#
#          FILE:  a.sh
#
#         USAGE:  ./a.sh
#
#   DESCRIPTION:
#
#       OPTIONS:  ---
#  REQUIREMENTS:  ---
#          BUGS:  ---
#         NOTES:  ---
#        AUTHOR: Amit Agarwal (aka), amit.agarwal@roamware.com
#       COMPANY: blog.amit-agarwal.co.in
#       CREATED: 02/05/12 06:52:08 IST
# Last modified: Wed May 02, 2012  07:03AM
#      REVISION:  ---
#===============================================================================

OUTF=rem-duplicates.sh;
echo "#!/bin/sh" >$OUTF;
find "$@" -type f -exec md5sum {} \; 2>/dev/null | sort --key=1,32 | uniq -w 32 -d |cut -b 1-32 --complement |sed 's/^/rm -f/' >>$OUTF

Pretty good one line, I must say 🙂

Related articles

• Bash shell-scripting libraries ” Striving for greatness (dberkholz.com)
• Release Engineering 101 – Build Script (shafiqissani.wordpress.com)
• Shell scripting EC2 for fun and profit (r-bloggers.com)
• Book Review: Linux Shell Scripting Cookbook (books.slashdot.org)

Related Articles:

• 2011/01/25   shopt causes bash completion to stop working.
• 2011/01/05   Shell Scripting
• 2011/03/14   cd across parallel directories
• 2010/12/23   Bash Script Beautifier
• 2010/12/26   Empty a file