Node health (hardware/software) is very critical in cluster environment. Whether you run a lot of web servers in load balancer or having compute node in High Performance Computing setup, each node is critical which is been used for any type of work. It has to be healthy to run processes we need to run. This script created to check node health before using as a compute node or any service in load balancer. If all checks run fine then node will be mark good and will available to run in production.
1. License
GPL. You can add/edit/delete script to fit your need. Please do share your edit to enhance the script.
2. Functions
Script is created using functions. Every check is based on function, so to perform any check or not is depends on whether you call the function or not. Just make to call function or not in the Main Section of the script and it will run as per your need. Even you can add/edit/delete as per your need to make changes in script.
3. Checks
Following checks are made as per my need.
- Loadavg
- Memory / Swap
- CPU
- Ethernet Speed
- Infiniband
- NFS Mounts
- NIS Server
- MCELog (Machine Check Event, hardware error reporting)
You can create your own checks and add to the script for enhance checking.
4. Script
Here it is
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
#!/bin/bash # # Script: Node Health Script # Created By: Sohail Riaz (sohaileo@gmail.com) www.sohailriaz.com # Created On: 9th April 2013 # Detail: To Check Single Node Health. # HowTo Run: chmod +x nodecheck.sh; ./nodecheck.sh # Checks: loadavg, memory, cpu, ehternet, infiniband, infiniband ipath test, cpu test, nfs mounts, nis, mcelog # Functions: You can load/unload any function you need to run or not to run. # License: GPL # ## LoadAvg Function loadavg() { LoadAvg=`uptime | awk -F "load average:" '{print $2}' | cut -f 1 -d,` echo "LoadAvg = $LoadAvg" >> $Report } ## Memory Function memory() { TOTAL_MEM=`grep "MemTotal:" /proc/meminfo | awk '{msum+=($2/1024)/1024} END {printf "%.0f",msum}'` FREE_MEM=`grep "MemFree:" /proc/meminfo | awk '{mfree+=($2/1024)/1024} END {printf "%.0f",mfree}'` TOTAL_SWAP=`grep "SwapTotal:" /proc/meminfo | awk '{ssum+=($2/1024)/1024} END {printf "%.0f",ssum}'` FREE_SWAP=`grep "SwapFree:" /proc/meminfo | awk '{sfree+=($2/1024)/1024} END {printf "%.0f",sfree}'` echo "TotalMemory = $TOTAL_MEM GB ($FREE_MEM GB Free)" >> $Report echo "TotalSwap = $TOTAL_SWAP GB ($FREE_SWAP GB Free)" >> $Report } ## CPU Function cpu() { PROCESSOR=`grep processor /proc/cpuinfo | wc -l` CPU_MODEL=`grep "model name" /proc/cpuinfo | head -n 1 | awk '{print $7 " " $8 " " $9}'` echo "Processors = $PROCESSOR" >> $Report echo "ProcessorModel = $CPU_MODEL" >> $Report } ## Ethernet Function (eth1 for me, you can edit for yours) ethernet() { ETHER_SPEED=`ethtool eth1 | grep "Speed:" | awk '{print $2}'` echo "EthernetSpeed = $ETHER_SPEED" >> $Report } ## IB Function ib() { IB_STATE=`cat /sys/class/infiniband/*/ports/1/state | awk -F ":" '{print $2}'` IB_PHYS_STATE=`cat /sys/class/infiniband/*/ports/1/phys_state | awk -F ":" '{print $2}'` IB_RATE=`cat /sys/class/infiniband/*/ports/1/rate` echo "IB_STATE = $IB_STATE" >> $Report echo "IBLink = $IB_PHYS_STATE" >> $Report echo "IBRate = $IB_RATE" >> $Report } ## IB Test Function ibtest() { IB_TEST=`ipath_pkt_test -B | awk -F ":" '{print $2}'` echo "IPathTest = $IB_TEST" >> $Report } ## NFS Mounts Function nfs() { NFS_MOUNTS=`mount -t nfs,panfs,gpfs | wc -l` echo "NFS_MOUNTS = $NFS_MOUNTS" >> $Report } ## NIS Function nis() { NIS_TEST=`ypwhich` echo "NIS_SERVER = $NIS_TEST" >> $Report } ## MCELog Test Function mcelog() { MCELog=`if [ -s /var/log/mcelog ]; then echo "Check MCELog"; else echo "No MCELog"; fi` echo "MCE Log = $MCELog" >> $Report } ### MAIN SCRIPT ## Get Node Name Hostname=`hostname -s` touch ./$Hostname-checks.txt Report=./$Hostname-checks.txt echo " " > $Report echo "Node = ${Hostname}" >> $Report echo "----------------" >> $Report ## Get Cluster Name Cluster=`echo $Hostname | cut -c1-4` ## Call Function loadavg memory cpu ethernet ib ibtest nfs nis mcelog ## Generate Report echo " " >> $Report cat $Report |
or press to download. [button link=”https://www.sohailriaz.com/downloads/nodecheck.sh” bg_color=”orange”]Download Script[/button]
To run the script run following command.
|
1 2 |
chmod +x nodecheck.sh ./nodecheck.sh |
5. Enhancement
You can add/edit/delete any function inside script to meet your need but do share your edits to let us improve the script for maximum checks.
