Apache server stats with small and useful bash script.

Just copy this script to your web-server cgi-bin directory and enjoy.

The script with show the common errors like 404 Error, Internal Server Error and others. It will show the User agent distribution using simple commands like grep, uniq, awk and so on.

You would need to change the tfile – which is temporary file and also the access.log path in the next line.

Just re-direct the output to some file with html extenstion. You could even put this in the cron which re-directs the output to some html in server document root.

cvs add files recursively – not already in repository

When you have a lot of files in some repository and you have added a couple of new, in CVS there is no command to add just the new ones to the repository, so here is a workaround for that.

cvs status 2>/dev/null | awk '{if ($1=="?")print "cvs add -kb " $2}'

Well, if you are adding text files then you might want to remove the “-kB” in the cvs command above.

BASH Script Performace

Today we will look at some bash code snippests and the performance issues. Lets first look at the problem and the implemented solution:

Problem: We needed to log the output of the ps command for all the process’s. This was required to be done on per minute basis and the output was required in comma separated files. So, here is what was implemented:

pslog=`ps -e -opid,ppid,user,nlwp,pmem,vsz,rss,s,time,stime,pri,nice,pcp:u,args|grep -v PID|sort -r -k 13,13`
        OLD_IFS=$IFS
        IFS=$'\n'
        logarr=( $pslog )
        for LOGLINE in ${logarr[@]}
        do
                LOGLINE=`echo $LOGLINE|awk '{OFS=",";print $1,$2,$3,$4,$5,$6,$7,:$8,$9,$10,$11,$12,$13,$14}'`
                echo $LOGLINE >> output
        done
        IFS=$OLD_IFS

This was working well and there were no issues. But suddenly we started seeing issues with the reported CPU usages. We would see that whenever this script was running the CPU usage was high, specially if there were too many process’s/thread’s on the system during that time.  This code was definitely part of a very large code base, and at this point of time we did not know what was causing the issues.

Rekursive Grep on Solaris or AIX Systems without GNU egrep -r funcionality

If you work regularly on a Solaris or systems which do not have the “-r” (recursive grep) for grep, then you know what a lifesaver this command can be.

Here is one from command line fu:

find . -type f -exec awk '/linux/ { printf "%s %s: %s\n", FILENAME, NR, $0; }' {} \;

The benefit of using awk here is that you can print the line number also 🙂

Linux hardware details.

Here is one of the scripts that I found on the net while searching for something … Note the URL for the script in the Description.

#!/bin/bash -
#===============================================================================
#
#          FILE:  linux_hw.sh
#
#         USAGE:  ./linux_hw.sh
#
#   DESCRIPTION:  http://www.howtogeek.com/howto/solaris/get-the-processor-type-on-solaris/
#
#       OPTIONS:  ---
#  REQUIREMENTS:  ---
#          BUGS:  ---
#         NOTES:  ---
#        AUTHOR: Amit Agarwal (aka), amit.agarwal@roamware.com
#       COMPANY: Roamware India Pvt Ltd
#       CREATED: 09/13/2011 03:57:34 PM IST
# Last modified: Sun Oct 30, 2011  04:59PM
#      REVISION:  ---
#===============================================================================

function linux_hw_CPU {
	typeset num=0
	typeset name=""
	typeset cores=""
	name="$( cat /proc/cpuinfo | awk -F: '
/vendor_id/ { vendor=$2 }
/model name/ { model=$2 }
/cpu MHz/ {
if( model ~ "Hz" ) {speed=""} else { speed=$2? MHz" };
print vendor, model, speed; }
		' | tail -1
	)"

        num=$(if [ -r /proc/vmware/cpuinfo ]; then awk '/pcpu/ { print NF-1 }' /proc/vmware/cpuinfo; else cat /proc/cpuinfo | grep processor| wc -l; fi)

	# ESX: mas info sobre logical/cores/packages
	if [ -r /proc/vmware/sched/ncpus ]
	then
		cores=$( echo $( cat /proc/vmware/sched/ncpus ) )
	fi

	echo $num $( echo "$name ($cores)" | enclose )
}

function enclose {
	tr -s " " | sed -e "s/^/\"/; s/$/\"/; s/\"\ /\"/; s/\ \"/\"/"
}

function linux_hw_CPU {

	typeset num=0
	typeset name=""
	typeset cores=""

	name="$(
		cat /proc/cpuinfo | awk -F: '
/vendor_id/ { vendor=$2 }
/model name/ { model=$2 }
/cpu MHz/ {
if( model ~ "Hz" ) {speed=""} else { speed=$2" MHz" };
print vendor, model, speed; }
		' | tail -1
	)"

	num=$(
		if [ -r /proc/vmware/cpuinfo ]
		then
			awk '/pcpu/ { print NF-1 }' /proc/vmware/cpuinfo
		else
			cat /proc/cpuinfo | grep processor| wc -l
		fi

	)

	if grep -q "physical id" /proc/cpuinfo || grep "siblings" /proc/cpuinfo
	then
		chip_count=$( grep "physical id" /proc/cpuinfo | sort -u | wc -l )
		chip_core=$( grep "siblings" /proc/cpuinfo | tail -1 | cut -d: -f2 )
		cores="($chip_count chips x $chip_core cores)"
	fi

	# Blades HP con
	if [ -x /sbin/hpasmcli ]
	then
		chip_name=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Name" | head -1 | cut -d: -f2 )
		chip_speed=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Speed" | head -1 | cut -d: -f2 )
		chip_core=$( /sbin/hpasmcli -s "SHOW SERVER" | grep "Core" | head -1 | cut -d: -f2 )
	fi

	# ESX: mas info sobre logical/cores/packages
	if [ -r /proc/vmware/sched/ncpus ]
	then
		cores="($( echo $( cat /proc/vmware/sched/ncpus ) ))"
	fi

	# Linux Itanium IA64
	if grep -q -i itanium /proc/cpuinfo
	then
		name="$(
		grep "vendor" /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "arch " /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "family" /proc/cpuinfo | cut -d: -f2- | tail -1 ) $(
		grep "cpu MHz" /proc/cpuinfo | cut -d: -f2- | cut -d. -f1 | tail -1 ) Mhz"

		chip_count=$( grep "physical id" /proc/cpuinfo | sort -u | wc -l )
		chip_core=$( grep "siblings" /proc/cpuinfo | tail -1 | cut -d: -f2 )
		cores="($chip_count chips x $chip_core cores)"
	fi

	echo $num $( echo "$name $cores" | enclose )
}

linux_hw_CPU

Related articles

• linux?????? (weijianhui16.wordpress.com)
• How to check the uptime of a process (or an application) running in Linux machine using shell script? (singztechmusings.wordpress.com)
• Some examples of grep command in unix (javarevisited.wordpress.com)

Total upload and download on any interface.

Something that I have been searching for a long time, finally some look inside the <a class=“zem_slink” title=“Procfs” rel=“wikipedia” href=“http://en.wikipedia.org/wiki/Procfs" _mce_href=“http://en.wikipedia.org/wiki/Procfs">/proc got me what I wanted. Hope this will save someone’s day. So far, the only option I could think of was to run conky which becomes a little heave on some of the very old laptops I have to use sometimes 🙁

  <td>
    <div class="text codecolorer">
      &nbsp;cat /proc/net/dev|grep eth2|awk '{print $2/1024/1024"&nbsp; "$10/1024/1024}'
    </div>
  </td>
</tr>

Related articles

• CPUSPEED Slowing you down? (jarneil.wordpress.com)
• Interface Curmudgeons versus Gestures (lingpipe-blog.com)
• Alias in .bashrc (michaeldaranto.wordpress.com)

Ranking of the most frequently used commands

Lets take a quick look at how to get the most frequently used commands on you shell. So what we need to do is this:

  <td>
    <div class="text codecolorer">
      history | awk '{print $2}' | awk 'BEGIN {FS="|"}{print $1}' | sort | uniq -c | sort -n | tail | sort -nr
    </div>
  </td>
</tr>

So, how did we arrive at this and will this always work? No it might not always work. A typical example is where HISTTIMEFORMAT variable is set. In that case, if you check history, you will see that after the number column we have time and date in the specified format, in which case, you will get wrong information from the above command. Anyways, forgetting these special cases, lets go to how we got this command: