debugging high load averages

[tiff2342]

Ive been running high and spiking well into 10.0 at times.

Running CloudLinux with MYSQL Governor and on kernel:

2.6.32-379.1.1.lve1.1.7.6.el6.x86_64 #1 SMP Sat Aug 4 09:56:37 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Ideas?

MySQL Gov are default settings.

CL Settings
[url]http://i48.tinypic.com/33yl7ch.png[/url]

Ive been watching MySQL stats but dont seem to be seeing anything major....

[Tue Oct 30 05:27:21 2012] USER1 LIMIT_ENFORCED period current value, field read value 1293386735/limit 1000000000 restrict level 1 loadavg(0.08 0.18 0.27 2/391 1039879) vmstat( 2 1 22932 51052 726680 4894784 0 0 261 94 10 18 4 1 88 7 0 ) cpu = 0.059913, read = 1293386735, write = 211944051
[Tue Oct 30 05:35:37 2012] USER2 LIMIT_ENFORCED period current value, field read value 1246583757/limit 1000000000 restrict level 1 loadavg(0.35 0.26 0.27 1/395 1041210) vmstat( 0 1 22932 87608 725112 4964776 0 0 261 93 10 18 4 1 88 7 0 ) cpu = 0.159669, read = 1246583757, write = 350923065
[Tue Oct 30 06:28:21 2012] Penalty period is over USER1 0 0

[Tue Oct 30 06:30:08 2012] USER2 LIMIT_ENFORCED period current value, field read value 1219056766/limit 1000000000 restrict level 2 loadavg(1.07 0.45 0.33 1/397 2357) vmstat( 0 5 23444 51396 691560 5327440 0 0 261 93 11 20 4 1 88 7 0 ) cpu = 0.089844, read = 1219056766, write = 253732663
[Tue Oct 30 06:32:27 2012] USER3 LIMIT_ENFORCED period current value, field read value 1042943342/limit 1000000000 restrict level 1 loadavg(3.54 1.34 0.66 1/399 2944) vmstat( 2 11 23700 64668 594456 5418376 0 0 261 93 11 20 4 1 88 7 0 ) cpu = 0.000000, read = 1042943342, write = 304145648
[Tue Oct 30 06:33:03 2012] USER4 LIMIT_ENFORCED period current value, field read value 1521629551/limit 1000000000 restrict level 1 loadavg(7.22 2.47 1.06 1/405 3026) vmstat( 0 14 23700 53680 586436 5404000 0 0 262 93 11 20 4 1 88 7 0 ) cpu = 0.000000, read = 1521629551, write = 578295641
[Tue Oct 30 06:33:05 2012] USER5 LIMIT_ENFORCED period current value, field read value 1133351848/limit 1000000000 restrict level 1 loadavg(7.22 2.47 1.06 2/405 3032) vmstat( 0 10 23700 57216 586532 5410796 0 0 262 93 11 20 4 1 88 7 0 ) cpu = 0.000000, read = 1133351848, write = 351443103
[Tue Oct 30 06:33:37 2012] USER3 LIMIT_ENFORCED period current value, field read value 1227512249/limit 1000000000 restrict level 2 loadavg(8.63 3.38 1.41 1/401 3079) vmstat( 0 12 23700 50096 575384 5383116 0 0 262 93 12 20 4 1 88 7 0 ) cpu = 0.000000, read = 1227512249, write = 455282059
[Tue Oct 30 06:34:09 2012] USER6 LIMIT_ENFORCED period current value, field read value 1726125945/limit 1000000000 restrict level 1 loadavg(10.14 4.27 1.77 2/398 3135) vmstat( 0 5 23696 77072 570860 5365076 0 0 262 93 12 20 4 1 88 7 0 ) cpu = 0.000000, read = 1726125945, write = 2039432679
[Tue Oct 30 06:46:57 2012] USER2 LIMIT_ENFORCED period current value, field read value 1602247117/limit 1000000000 restrict level 3 loadavg(0.75 0.75 1.12 5/387 5132) vmstat( 1 6 23740 54912 547288 5585412 0 0 262 93 12 20 4 1 88 7 0 ) cpu = 0.119730, read = 1602247117, write = 579189425
[Tue Oct 30 06:52:42 2012] USER7 LIMIT_ENFORCED period current value, field read value 1286770012/limit 1000000000 restrict level 1 loadavg(0.83 0.62 0.93 1/388 6186) vmstat( 0 4 23740 56364 538108 5590756 0 0 262 93 12 21 4 1 88 7 0 ) cpu = 0.000000, read = 1286770012, write = 258317866
[Tue Oct 30 07:34:02 2012] Penalty period is over USER4 0 0

[Tue Oct 30 07:34:05 2012] Penalty period is over USER5 0 0

[Tue Oct 30 07:35:09 2012] Penalty period is over USER6 0 0

[Tue Oct 30 07:40:31 2012] USER3 LIMIT_ENFORCED period current value, field read value 1319877809/limit 1000000000 restrict level 3 loadavg(2.00 1.10 0.88 1/396 14239) vmstat( 1 4 22976 67084 368844 5687184 0 0 266 92 13 22 4 1 88 7 0 ) cpu = 0.000000, read = 1319877809, write = 259563887
[Tue Oct 30 07:53:42 2012] Penalty period is over USER7 0 0

[Tue Oct 30 08:20:54 2012] USER2 LIMIT_ENFORCED period current value, field read value 1390051069/limit 1000000000 restrict level 4 loadavg(0.17 0.26 0.40 2/392 20992) vmstat( 0 1 23232 53332 379856 5922204 0 0 267 92 14 23 4 1 88 7 0 ) cpu = 0.299469, read = 1390051069, write = 263020870
[Tue Oct 30 08:55:37 2012] USER5 LIMIT_ENFORCED period current value, field read value 1245294960/limit 1000000000 restrict level 1 loadavg(0.61 0.49 0.52 1/399 26775) vmstat( 1 4 23236 62200 296792 5778400 0 0 267 92 15 24 4 1 88 7 0 ) cpu = 0.000000, read = 1245294960, write = 363029864
[Tue Oct 30 09:20:54 2012] Penalty period is over USER2 0 0

[Tue Oct 30 09:40:30 2012] Penalty period is over USER3 0 0

[Tue Oct 30 09:56:37 2012] Penalty period is over USER5 0 0

[Tue Oct 30 10:20:16 2012] USER2 LIMIT_ENFORCED period current value, field read value 1330879493/limit 1000000000 restrict level 1 loadavg(0.83 0.44 0.32 1/397 41206) vmstat( 0 1 23572 54516 278032 5851188 0 0 267 91 17 4 4 1 88 6 0 ) cpu = 0.359223, read = 1330879493, write = 329853757
[Tue Oct 30 11:21:16 2012] Penalty period is over USER2 0 0

[Tue Oct 30 12:50:39 2012] USER2 LIMIT_ENFORCED period current value, field read value 1363008446/limit 1000000000 restrict level 1 loadavg(0.49 0.54 0.43 1/387 66759) vmstat( 1 2 24068 69256 284168 6029272 0 0 266 90 20 8 4 1 88 6 0 ) cpu = 0.089812, read = 1363008446, write = 413302263
[Tue Oct 30 12:55:55 2012] USER2 LIMIT_ENFORCED period current value, field read value 1358200072/limit 1000000000 restrict level 2 loadavg(1.57 0.86 0.56 2/399 67517) vmstat( 1 4 24068 50440 263520 5861820 0 0 266 90 21 8 4 1 88 6 0 ) cpu = 0.009988, read = 1358200072, write = 337364348
[Tue Oct 30 12:56:53 2012] USER7 LIMIT_ENFORCED period current value, field read value 1365504545/limit 1000000000 restrict level 1 loadavg(1.73 1.05 0.64 2/398 67825) vmstat( 3 2 24068 61904 261104 5779940 0 0 266 90 21 8 4 1 88 6 0 ) cpu = 0.049938, read = 1365504545, write = 415023971
[Tue Oct 30 13:12:50 2012] USER2 LIMIT_ENFORCED period current value, field read value 1005241456/limit 1000000000 restrict level 3 loadavg(0.66 0.58 0.64 3/375 71269) vmstat( 0 2 19896 120828 294968 6131528 0 0 267 90 21 9 4 1 88 6 0 ) cpu = 0.069837, read = 1005241456, write = 367887978
[Tue Oct 30 13:14:00 2012] USER3 LIMIT_ENFORCED period current value, field read value 1367982475/limit 1000000000 restrict level 1 loadavg(0.48 0.54 0.62 3/382 71394) vmstat( 0 2 19896 54968 287008 6106464 0 0 267 90 21 9 4 1 88 6 0 ) cpu = 0.000000, read = 1367982475, write = 417486561
[Tue Oct 30 13:14:33 2012] USER8 LIMIT_ENFORCED period current value, field read value 1102924547/limit 1000000000 restrict level 1 loadavg(4.42 1.46 0.92 1/388 71465) vmstat( 0 8 19896 56940 284260 6014464 0 0 267 90 21 9 4 1 88 6 0 ) cpu = 0.000000, read = 1102924547, write = 762715994

is this a MySQL I/O issue?

could this be a PHP issue?

[pratkal]

Hi

i was just wondering if you could output of iostat and iotop -o

thanks