Few servers I maintain totally confused me. The loadavg is steadily increasing every round hour. With top command I can’t see any relevant process which can produce high load.
top - 15:07:17 up 41 days, 3:52, 1 user, load average: 4.22, 1.61, 0.76
Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.7%sy, 0.0%ni, 85.5%id, 13.5%wa, 0.1%hi, 0.2%si, 0.0%st
Mem: 1025084k total, 1016732k used, 8352k free, 24472k buffers
Swap: 2064376k total, 116k used, 2064260k free, 133380k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6082 root 15 0 126m 9632 5008 S 0.3 0.9 0:00.50 php
7363 root 15 0 12736 1112 808 R 0.3 0.1 0:00.03 top
27418 root 15 0 347m 3860 1096 S 0.3 0.4 0:22.80 radiusd
1 root 15 0 10344 680 568 S 0.0 0.1 0:01.88 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.54 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:15.33 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:01.81 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:39.01 migration/2
... |
The server is CentOS 5.5 64b, quad core Intel processor. After some digging I found out that 4 servers are affected and they are CentOS 5.x 64b. 32bit systems are not affected…
So, the first step is to check cron settings because it is obvious that something is triggered by cron (hourly). Here it is: mcelog.cron. After Googleing about this problem I found this LINK. Or here LINK.
The bug is “closed” but I wouldn’t say so… I had latest mcelog installed and it causes the same problem which is described above.