find duplicate entry in a list in bash with sed

Here I will take an example of rss2email list, but I guess I will be able to pass on the concept.

Here is example of the output of the r2e list command:

1: http://blog.amit-agarwal.co.in/feed (default: amitag@localhost)
2: http://feeds2.feedburner.com/AllAboutLinux (default: amitag@localhost)
3: http://feeds2.feedburner.com/Command-line-fu (default: amitag@localhost)
4: http://blogs.members.freewebs.com/Members/Blogs/viewBlogRSS.jsp?userid=29731143 (default: amitag@localhost)

Target here is to get the list of all duplicate entries if any. So, first we need to remove the numbers from the begining and the email ID from the end.

We will use sed to remove the email and the numbers. Heres what we can use for doing this

sed \’s/^[0-9]*: //\’

and

sed \’s/ (.*//\’

So, let\’s try now with

r2e list |sed \’s/^[0-9]*: //\’ |sed \’s/ (.*//\’

If you see just the lines we are interested in then it is time to use the uniq command.

r2e list |sed \’s/^[0-9]*: //\’ |sed \’s/ (.*//\’ |uniq -d

\"Reblog