Mirror or download a website with a tool much better than wget

If you want to mirror a website, the simplest tool I know if is wget. Once you have used wget, you know the troubles associated with it. You also get added couple of files as search html files, which are useless and wastage of bandwidth for you as well as the webserver.

lftp -e \”mirror -c\”  <URL>

This will mirror the website for you and will not leave with all those un-necessary files.

\"Reblog

Download with browse with bash and wget

Been pretty busy for last couple of days. Will be busy for another few days, but here\’s something to munch in the meantime (Script may need changes depending on the type of file you want to download or the site you are browsing, but the chnages should be minimal):

file=\”/tmp/temp.dir\”
url=\”URL Here\”
IFS=\’
\’
cont=\”y\”
while [ $cont != \”n\” ]
do
name=\”\”
wget \”$url\” -O $file -o /dev/null
for i in $(grep href $file |grep -v Parent)
do
name=${i##*href=\”}
name=${name%%\”>*}
echo $name
if [[ $name == *gz ]]
then
cont=\”n\”
fi
done
if [ ! $name ]
then
echo \”No files here.. Exiting\”
exit -1
fi
echo
if [ $cont == \”n\” ]
then
echo \”Enter the filename for download :\”
read file
fi
echo \”Select one of the options:\”
read product
url=\”$url/$product\”
echo \”About to get $url\”
done
wget \”$url\” -O $file -o /dev/null

I have used the script on Solaris 8/9 and Fedora 9/10.