🚀 go-pugleaf

On Sat, Sep 17, 2005 at 07:26:11AM +0300, Al Boldi wrote:
> Monitoring disk access using gkrellm, I noticed that a command like
>
> cat /dev/hda > /dev/null
>
> shows eradic disk reads ranging from 0 to 80MB/s on an otherwise idle system.
>
> 1. Is this a hardware or software problem?

Difficult to tell without more info. Can be a broken IDE disk or defective
ribbon.

> 2. Is there a lightweight perf-mon tool (cmd-line) that would log this
> behaviour graphically?

You can do " readspeed </dev/hda | tr '\r' '\n' > log " with the readspeed
tool from there :
   http://w.ods.org/tools/readspeed

Then you just have to graph $6 (kB/s) versus $1 (bytes read). There may
be other tools which do all this automatically though.

Regards,
Willy

Willy Tarreau wrote:
> On Sat, Sep 17, 2005 at 07:26:11AM +0300, Al Boldi wrote:
> > Monitoring disk access using gkrellm, I noticed that a command like
> >
> > cat /dev/hda > /dev/null
> >
> > shows eradic disk reads ranging from 0 to 80MB/s on an otherwise idle
> > system.
> >
> > 1. Is this a hardware or software problem?
>
> Difficult to tell without more info. Can be a broken IDE disk or defective
> ribbon.

Tried the same with 2.4.31 which shows steady behaviour with occasional dips
and pops in the msec range.

>
> > 2. Is there a lightweight perf-mon tool (cmd-line) that would log this
> > behaviour graphically?
>
> You can do " readspeed </dev/hda | tr '\r' '\n' > log " with the readspeed
> tool from there :
>    http://w.ods.org/tools/readspeed

Does it have msec resolution?

Thanks!

--
Al

--Boundary-00=_AO/KDDsXIk/KkWh
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

> > > Monitoring disk access using gkrellm, I noticed that a command like
> > >
> > > cat /dev/hda > /dev/null
> > >
> > > shows eradic disk reads ranging from 0 to 80MB/s on an otherwise idle
> > > system.
> > >
> > > 1. Is this a hardware or software problem?
> >
> > Difficult to tell without more info. Can be a broken IDE disk or defective
> > ribbon.
>
> Tried the same with 2.4.31 which shows steady behaviour with occasional dips
> and pops in the msec range.
>
> > > 2. Is there a lightweight perf-mon tool (cmd-line) that would log this
> > > behaviour graphically?
> >
> > You can do " readspeed </dev/hda | tr '\r' '\n' > log " with the readspeed
> > tool from there :
> >    http://w.ods.org/tools/readspeed

Try attached one.

# dd if=/dev/hda of=/dev/null bsM

On another console:

# nmeter t6 b d1
...
13:42:28.352037 bio    0    0
13:42:28.354675 bio    0    0
13:42:28.357703 bio    0    0
13:42:28.360671 bio    0    0
13:42:28.364712 bio    0    0
13:42:28.367671 bio 128k    0
13:42:28.370673 bio    0    0
13:42:28.373656 bio    0    0
13:42:28.376653 bio    0    0
13:42:28.379680 bio 128k    0
13:42:28.382656 bio    0    0
13:42:28.385819 bio    0    0
13:42:28.388651 bio    0    0
13:42:28.391736 bio 128k    0
13:42:28.394673 bio    0    0
13:42:28.397656 bio    0    0
...
--
vda

--Boundary-00=_AO/KDDsXIk/KkWh
Content-Type: application/x-tbz;
  name="d.tar.bz2"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename="d.tar.bz2"

QlpoOTFBWSZTWWF3g44ANf7//////////////////////////////3//////////////4ErWx9UD
eCoHsyPLbLtm7O7WBvPu+4D0o9Hxz31772jp6hzjAL73umax63WryweHN7VotajkMmtG91avD3Hv
Z17XsB3rpzalcre3V23c9xrAZ0jJ3Ly1E6dzVREoI3brenetTmdCrYO4MZFdtHK07DTpoFt3GPL1
dqCqKoqiV7u11t3apve3nomy3vce92boPIUp0Vh00HkCjoPp5B9fI4b6dCFADs74aIQEAEyZDQAa
AAJgTTEwmhpPTRiTBoGingAAI0xMJpplPJoxU/INGhhJhNNPJNNNHkm0BT2pmKeim09KP0EabVG0
0BplBoICAEMgCaBMTTTIaamJpT9IyZT01Bsp6j09KNlNPaKfpQ9T1G1D9TSP1T1NDQAeoHlA/VNN
BoabUNGgANB6gaANAAPKBoAAAEpoIRBNNEpvSap+mKT8mpPyp+Kn5Keyh6mo/VNpPaiPSep6j1PI
Tag0AeoGnqBoekZMgGTRmoDQaAeoGmT0gA00A0A0NAAAADQA0BJpRICAjQ0CaCYJiZTGlM0zU1P1
RmFNPaoekxowiNkRpiYGgJ6mE9JgAmmDKDD1TyaIMARmiMACaaaaPSZqYATBMTAIKigamGSaemgn
oEaaMaNAJpgGiMJhDJhGCZoAI2p6aCMIwTEZoARkyMCaaNMEMBMhk00xMJibSMjEegGpgTNIwSJE
EyACNJgJoaZAaBMAmmmhPU9E9TKeEwTU9TR5E8k1PZTQ0nptUZGNT9KZP1NR5Tyhp4mobUaGjaZT
ym1HqeUAaDQGgDQ0MgNqD0jQ/+ITyABwcIRYAVz0IijCNhj0WTNRf7GhZ4ZMQ1v5aZnHjlkbT6Y8
fAo1R7FoxwlG55P/ycDY2KMk+wQ3LjyFyRASbhl0bFCwUT1fDs7oYWnYoad9b46/zWj2G23vD8nz
g7UZNGLeIiSBiepZ6mlqelqKmqmKu1ngWqMgHm1IIfGRFtTEno9rs8gtNC+MI151RkCDkhDQaSAN
A1nC/H4m3w9LvLOtydnecyTDwSw3wQA9HhDhidV+dh8frxy/zIEd8Xli9qSP/H/3gnmH5Ixghh8U
+5Nc6p1QWHLDwfODe11XXqIiugY2wMvhBlugd/ioPsiRGKGNDA7ApR3R9uYD0j8I5RpFzTOaf4T3
xNGm0t0YhHidY+mcyaF0j6Imxvg4AcEWWx920JQyHs5DOsCtMwNSifMzAvhDHkZCH3/gdzqdXR97
rOMxIVWcezPTZntDhusDW3ibl3ujCTTMengSOlyXL7lvXnFmN7CeBZxufMxf8lvWg012pvwkTTSh
3hgs2OUikk5acUAFE70yQFn1CkmIqNocrtT7MjX0tYjzAezz3dCQUd+gaA/8909z3P6U/6kiuGjj
SmYQSYN6i67KAXU/kx6Xv/Wv6TaNVuq77FPyAMzvUbDp8R4mw6eK/N+Hym6H2ur+fJS96x0KH3vt
uZET1nCahrDLA8SlDKyEA2bfJ0uMy26yQbkMNxBuvbqPaZ3PK5jUfcrxVcGHT+vYoO0TMbT+LeqZ
M/OZ9tY7h624Zh1xlkyGDAIbAwEtgbe3UVbEiFTA4ZhkHBREQREkEEgggJ2SlHQKFjnSBUwSUfMj
Cc7NQk1MsImgglJoYKJMFClKUiREYXNMCT6pmRPIMhgT0uZDIzJtbJDYJDCBtE1hksJwjSjlhDXE
EmZRDN2aLpgkuPCYDUgwXcJlA0ZAqQtpAULTJtLnQ3Ypmy4j0641oazlqdiYqCLucbF6nTi5Fmyf
LHo7FLSgSEnedGM5mxk6Tl1KBISkCYtKJaJKgGdTTw86ZbTFngPLokssVI1kOmridKiAkeq8D7eK
8RWE2ZhzxBaBXp2tMwu+12tix5aHk1J+fw3es/h5y9I4FUpjgRkKBMU/f9hrrZk9l9SmsmhB9QUQ
Q6xlDRJCggDHF3VGl4i13cpwLINxbJrX4o9qZgJX33xfrnjVqmAqmbUlTTVGIxQKzZ7VFMbByU2X
Vdbh242Tfw3anOhzdLv2VKY266V1od5ajtToOE4HX3GmzMqa/Z3rc42atLstOtmVc2nPpcNpd9rw
8VJm8hnU0WRW/N4VjgzOZw+F87oyZhJjSmSy0fdRpkq0Kzy/Il7PvuotMgQhOtcUplwZXHM9KhnH
c4JMM9Yd78oY9kajOuy+r48FZhJVRZUK1AWFGFYsFUmUqAajuQ7rqYIoTifuD7j/SuClP9B8h4S+
WtWtX4r2n/D2i7vuXdvsgvoXyEiwtKiQb5rSD/FC1GNpICgZzD6hCEqR1MNRo1GfFOCbPZbY4JHP
JEiAZBQAFacScqH9QQgHzgkmnLIaYnaGSZc/UIsWsFIYlMDCX/3/vnCApAfu/eheGZt4b0pEQIRM
lwplXMCAn56gFHowRTBFMG8MCW+IRjemfbD5/ATUhD+TbYYSQh4yHOwywWKBFIiTukKhIsCdjt7L
9r3b7D0WlP7VgdMQm3p2EMGxvI8DUrI0rkT96QofQ+tu8H5b+/ly3zpUujmK9S0D8dYMj+W4SiIm
LWZYB9xjBhAWXAMbK4ga2YRBmJM8ZE8jPa2kOriiusuLS0a/UdcxDsfQaYFFnffMlkwYT4sqYiPX
6ckuvrQig785xIsnB1XLIIttn0NWmQbzuMeGu5uSVBsa2OSm0l27kH4LSrlCenCisOVwFsSotjbX
0F4x3JDzd7pMXHx4PWEw9Bobt+oRZKF7hi0MkNDTuEc6EGONdYJLX93zbhJhYfs+Kw1hAGFvQs/v
q6S2xNcvAd/hjOwhlxw1eaWPey2kwqvQgltReyUypcVWb9tNnaoM1NFCr4FKQKiyFrPC+Y+6drvZ
o1nJ6Tk8jSAk9rTl5RWKRCIIKnLpKZQ6d6zxna7jV2/Ht6BjBKd1tbYmJtqLbzsLwWCuh9sq5aVg
QeYAqOVlx+ZRrLTfaQjG685CEBQwVlcBhYsFRrac8OhFmKx2uk2gZzEnZ+buVeC2h8ajTS0Ln2lA
j7RAA8pjxmszkBmpAs23vvP3MDXc/T3wthR1SQL4Ah4rokNGTLIHr2YQRkmWThcp/lScHgWBt8Wm
3y9IG4TwkkydXZg9MycWlBSeGljBfCQqDwFgVkEYsiP4tob/h9MGRESKPxVon9v2+gHgsMg7klZ8
TcOAXy6VIYtDCcgw9jmjP6dkr8v/8+MzwAHqJ+rrygjypwJhkOFCs4Y4mqX4RZSEr52WrFKfuP2r
kENEx9lh/lSfinBpqtc1wAcTKp/R8ba5nMIX3fKG+FuLAfKNhaDmjyRyQY98cAYoYYmxnBhjxR+s
d4OePUHDHbCTHOGtHGEamojnaiPH1sneV+tLzSPYdyY0Wc1U53RIrWOol/KefGQwgcC9Ra2QaMAZ
i7JE9F2iZ+THGhPfnYbMWHAcx43nBgAEAAbCemT2rJPqWSepYAoaQRD2FDrB8sH+M+fPsxy5VmYO
Ejzfpz+X8b0x82WMHuA40hkhzMBCYDAMGCIfcJ5DImeCuojWHpjEjphscHy2Q0rWF5h+R0F345aN
TrkCp2wRuaRBDT5IOUkUmIYgJIUmo+RML9A5aYVQglhyB5Ycj3lPsd442kT1A49YGJ82QC4ovBsx
CE2JkVQ148YVg1orA6wBkQgDDG4FcAn0zIfxt6S07mY4w0vTt+Xn++kGaEpgPSmPGrBL2tqB5N0N
sGgJDq++nOB6uE7g/ncyoHh1IoReC4BhYIBi6HujEiMNWMIfSBwR1BHFmMZeDzwwV6P9CP8/O9Vc
RgQMGCBiJ5WzhCKMrkP15GMNtvLecAv/RhjQ83rdio+lqv8KJ/g+x/P0bj0P0t3ZgBwdUNfmkjEm
PNwAXiebm8ApAAfeEBC9xAORCkSZofCCaR2+ew+pRs1s9rWGQRQwd3YePonRLAaIxwCPdvRrBpfO
XV8JlIriF8cnAkgMWMWgcS94KdDg5aw3Gmn+d0cHG+7xvL9OXIs8AR0gIYSIBx6wXMsogX+t1G5/
PCg7R55IkJdfgH6cvtiI98ym//6kTdRRRR2DraDq0H7QzpIrRbjE1ClGPMB+ZY7xKAuEFVfCMHnV
y2p8hHz9iYgSoMLXIgDkSqIUAeubuZaDkw2DkPBR3sUIUgXwXChhJH4xwht9J3ajASkzQkwEpcu2
tPr6nUSyaQD5dE0ITiXlOFjHgmIg0CZIgPtMkVhF93b659P6fP5vWzZLwQ5nb9KvHE1Eh2Zy2Jxm
iR0L65Ubx2r66PwIhrvJHYIOiDf/rUXjot9WHVfv9nq/dRUdBx/9/q+H/XSFwLYAC3RavkHvUXqa
hYZMJFuSyAQuh0vjgc/Nj3j2570esZ8zapXACj7BhKV2vZQA5MnY8p20hwsA6NlCEGLtejdOg7NQ
Bhdj63KSDglnmd5Nw9Xz3ysbNX/i3PhuSyeoN0KHfp9iuNrzFLBgjarzaNoftelY10aFDbHVXfI4
otxLRInvn+eoyV0+rWM0AJDFyAOw1Cr3q4amVXkrIykIggpNQaoErMUUIBCs2z25r8kE100NnaXq
5HAaDjkNKBY5+B9glFuiNB5xmxgnvsebTMeRl5kNssYQ5NNShU6NEhUPLpVI1i7wThjDipgfqsYw
bOMz6LJsDPpfm+lLTGEsoiGYOjYbyaE/ocu7mj0XUHNaP4w/Tw3geFLfLHzBclE10p3uwnTj077X
jmoGQ9b6/9rZlZPn77HD66gbJyxiD88O3Zz+SaZvtZ8VcB4euDZyuNuGnjW48nAW5RySQpYICw3P
TnkGsFBohsTwGQ3JqASNdjnfjepJ+pl8DxOtO9rWAe76ZhXur4MB7FHPYD0DzKeRss4xJ+SzlPEG
C/upu7/X9Ez2y6Bi+QA6lFqpqEEBO6oFuSkcUO2UEw44xbods/M+r1I8Lwag/uBWgOx7L2dBbV9O
5yPo9XfeQG2XttajjvHjw4i3uJvC1fubt515NgiKjOMAZFjEXISV+DDvQ1tn6X6yvaXmaetqO29p
CM/b+Jo/W87RuI4gaRAgsJEJaPOfekpcOgxzWWrSlzx0FD8dyPmPQZwCwXVGkLZ+ZBQYj8v2JBoZ
vJKhfXau82A/Ze7v/XnaiW6SSUhjGkMRmS53ogMjzRYgpbXc86gP3hEb47N3qbujuyJIG/yjZ/GK
/y6Bd66C/CxJDNhr6wxeiM9w7usQpDX6Ay80fzDlkX/lERUQGBiBjPj7sS/EM1syY9IYfkHY73EA
9B6Kn5QO1n637B6nEyvJT+gmFYvoLUMcHZGlLYIUNIwM/SmaEg0WhDYUksY5VDWRiycKScqTpo5v
iepKBhk3nlf7ntfaZCbPdVH/lKSf2etS6FkmWFYSoVAWBPgmYFzqx6tBHOMYXKyJPHYTRgCwUgsP
HsoH/QOYTyV+5KcJ+qU1TyCmwwRWPGkD8ljhBAp0GCWBoBgZGoQwDBFJKykK4XFsVFGsd7vl+t7W
Yu5/k9v+Xvo9wCpCvAXP19eBGfQYnLEShhfRR0YM0maTSaESKh+QyCDUshkiOX7KWmk1yH8cgqZQ
Qzbh3qZm8r7N7dHmP4OORHd3k9fkHSC0sPW3um3KGTJhUtpAxqhuxU26Vtny72EySRpAQ7LWkkh0
zYJ2QpD0ETl28G5q04JlA81+xfv5DNCrS0SZsKYtFYDEQRFE+9S/H+JiadXT8zHbbeTTMIrFJiyQ
7bHNtkqC/bIQ+ZxYR1X4hozomeg79MJiY693dW4Fkftk1I6yyLmOtPT26aU9zPbbLg2KGpKjE1Ul
V6DJYIjA7MMobsGUgsd1DvO74tmrl2azdku9U5LVwhUUUqQqI114wImJDLJsjIZcsQUGDN8qWqTB
bIibGZxvwGzBUSakvqLhWBxM2oTuHbsvCgir5KYs8EG1oITEKUizkrKQv++p0/wx9OiQIGRmaqZq
I+RearQ70p9qeQCbAqiATwqYbG4/VY8bjEuGDND8uydNFKSU3ociJobSSKGtzwLGbmKKLcVdzQ3q
VVr5kYSIycLN02+9753PXF8AMpvwrAMgG6MLgRpKqXkAZPpWXGanmZXgZl4GIinY0B8d8rbm0+jA
Rs82zaefLM1Y8x057Vb+8xghx2y82ZB4uRACcMzEiZB+ZAwcSBm8GM67F3PWAie9hCV2Acr3m1hg
e+IAFiK4amW9bidRzMf8IH7GtSxou937G+BjLBAMSAMJMguIkAws8vMSGGnp621cDUl7n3jnx8CI
Uf1IlyKYJfYqn2j3CPZOrRdYT6RQV3nldJ76A92WmO/Iro8sGjB+m9k3X7mPVS5pTnW9ViSfPeSc
IN0PRZSSCBHknK4Dzv3xwiR1yjaNUIILVd+zz7X7h1fGt9faP730qbxpB+KHZFM4P1AkAg6sAhFu
vQE9HjiGAHQWp8Q6uFBGgxaROrjBvYsxy8p8LtPguNvu20kPuP1/vR4s7oQ31D6jUUXdP7De7/sp
ed+YOEV6db3g40bdVThWmoKQX9V2Gqdr4oxDed7nbRQDk6Hf+56Cez5vo/qZIhVy4fe/LhbABit6
ePuj92m0p577PzOZl8e1fiWgbbvOEvkp67V6dkPWZv3zJ/Y7P2LrjwSSpI4JjLm8M3myTPpj8l2M
2s+YLDM1gED0zDU/WePmLVYEaZ6YxjI2yBQi/5/idG02qG+3HjSnX/taSQWIwI0Gwkgdw04fHjH9
mMZgjCZJo9YetNBLWawFJkIJExQMXEggZodSK/QL26X7stjvc1pQf5/5PyJ+4vDpJeCsGvSNGThj
+dPl6Zhcw/OORkygZ00/aK9L9U4awqZPIkpN7l9CddU8Cn6dN0lzmuW9Zhd7Wv3LdUZdXwt9tTJ4
+BKo9b5Z8YSgCgQzy5P6RZzApl3lzbpXo+h6O40Ach6CZ0pwjIHWoePXXqbToy+eY+mdOQk5gOT3
h9jHvBQ0xtlUpDJQxFRoBm4cfw1tXsXbPFUckatCSh+6hmeBCI/zr9p2dI6Tf6tBAHjgO999JnuJ
e6Gb77/GJ67vr6DrPQ+diUMoFKETflS0o+1rTHBGx7p1WZmbuPkqt5kax1hmU5fjS6mDT9iryWan
EDsyPKLtwJB0PY5V0+r9em9j6hLH1IIGi17jSk3LoQjnT6Hj5aPEfSYd08x0PrZz25d7p93cih4r
2rat8UBndOtTS/HihcC2APitLx31e5hu1bng8K76BYg/BQjT76AD5f3MIAytIXzWgQtBhJoPksAE
fJaQg2LYARuDF3PRJkkkgDzENCQJHsJHkNttHsZ/d9Vy5kNIJCLFEYIgiCputRQWKAx9OkCiApJF
iwFQRjFBjIpJKkKkhFAlBAKYbCKRWMIYQrJfU+y8j7HOm7yfYbs+ztJDUgUN9gHUmk+2kgWeswH2
Hh61jBS9BZWE8yxABm+qrtA9gHUUp7fch57vNIZu+QRB8of1O2Bxq1b2CEIxACQhCvJubbpk3IgD
R391SdLh+I5Mk1Q42SIanXnNz8fWOZrq4p5vmVlpMS+SWgTuEEMzsLSIdLef7uSfU7hxZnMJghhb
p6xIlq33PopXytIIJjmRgSUdw6K0wxKpAMMAzP81C/GIwxKlmbfdzelw5DO2l8vwrjPxo49vV8Ut
lpg+fEEWqM0STNDIQuJoFNZt0EhWF7kQA3b+02qjaM+QQFk2rpGyBnWz3W2cjlcvmk8xpxXDkgMJ
m0qEjgtJsGAZaULQLg1QECbRlmpSt6EN+FQ5gzOtzEMKm0YA6uiZCBGYBgzz684IaGlaY5NsomSg
wh2xAGm+XKo4ZBbHzzKI2yFdGgXyLE94bmb7Y2DccqIs4QhFuZjcHFXrvHHE/XyYzx72bipWczDk
M+fi3TSrGK4lNpUbBICUQNkJEmCidbz/BvxauRbOcI5GZRRa0ETOW/oO8XaZGJAzRvWE8zSHWTEX
QzErlV4wYCm9cMQX7ztxOstqS8WotkOalj2T2JWJouMBU36eBrQ0XimivElG6zptEBtaHexlWgS0
CW6SgMB9rTBIylEUjzBSclIJfC3EFSGMb2o1d8NUZdsGhNBTHpC6SmRhbJDp9kXN5erxqdTDLVau
Mw0aAgp6uxGADw7nXkmBu9QaKDL73EFsqcOgNtIWPFoK8GKJEET2hvVh26u6PEwFXOCMYYBiioGa
A9PLZYHv26dm3oe3dTx+jwa+6OpxdOQOJkOl5qnOvFVVVVVVXbqnGY4NXTMF0kLI6wtsttqtYHuB
uyGCPRdd0GhVcom04drImBm3mwCFp4KNu1hkR8LZnBvR33yB4Wd4lmOqzNrOl1BsxpsaKvvhdE6u
PILCtd0nhLbyaDGevW0nZ0XDBlzkXphQ0gpFlon1hpTU0oMz9vDBssg0IU56hEhkM1p7gpwvSTVB
UVdyQn24PI4JWPYMdt5fTH2h9LYGpJyLLjOYa2Q5xLiVnQKTb1p6ei23bHFXL697avxENhwGRr8T
dL3IKTh4duc8y0cvCmXNqqHZw8FhSPROZDZLGlYlvJVLDJpog7KozM5AwueJRe0vCMYJlCkch8SW
8YZtfZGzfIaoEM8wzNlnzh6MCB16NYY4WNDDZoQNMC4rnyAxtR2PMM5Z3kRt4FcPIY24HhpkzvIa
VZDWIigI/WUg7cgKaoLZs0qwBQkxzA2Fuprtls1LYqIjKojfMb3CsvzyoqYek3lMHz6SDguB6Wjd
v4iZSsQXYIiGfO1z0hRW2/3EjNzNVm0iFyyXoozIkmbKt4jtm4RJdnxXgq9vetJ152O81fWRpzTm
cqoorrLCEasTwE4QviRwkmElgkgHUNFRzGB2gldzTOmYHDHM6ubG94dMYoDGYcbe3INwKLw4TqEg
jEVkFORgcz0WBDEETr+JjzPjTympM6MKMwK4YrGTwMUhlUyCCToFD4aUjZpM1sptkBscjNOC0oxk
fC60F02i04v0Q4uhTJHM+95dnmM9ZoXomNeXwl2O/p5wxqbshNR6SioIaykz50aaZKcGlQ4FxVhA
BVDq0Mg+Jy1SY+wWStdts2StlMDZaDI6WXKoFgy/T8KaNRgxnvx+QhNS24rBGCJ6ue3YMKoIkeOp
c+lJ3gk70QpYK/cPoSNphMYoYkecIoaBNoueh6FNJuHs/pHgjxiakyMXwuRO8cMSMQ7iYwYxEtD1
WOumpMlSFdyBHhdMwx1ZC7Kz2gHoE4ld1JEsai8TKdnu7l3Uw8Oi7upRRBdFWcSMrMrReXQKzkw0
pEFlaYgLIaC8vCJe5eIaahOiSLlUEP4CdND2EQYeWuXsQTkavTdI2SC2AgzWAwmVb+ISV9asg2Q/
R930G2HKfSMl/0Dr6Y4/kUOH0/V7+yzWt3JoG0TfHYQZik92MXxjBwrIcx7vLN+w57Om+xp91UPi
TKeNzrcno/8dN28fIVe3CYef84gO6AGxP5NL/kfM/AJzyiprDFHkItiC2GNT8038SN92Ox4uvyqB
cjlQgmUoRi0aLeqxF6sCYu2NJodVXUXG4zZDPbD4M5BM1iDt40/WW00niAnZyoeaS9qFgyo0njuG
bLdozu/F8B+V9edvKO3+PvF7KgvNe0SUh+2DoB7xqiaxbkqtuIK6ZE2E6SYSTcDhtEZ3wrkLxn4P
oF3T8st5F5nkkEuLyC48gyXQbwBP4Px9Ie0eEQ5ZhS8dL0gDCvWzePfSX1vE7albgGnn7NOQvMwa
tqhNFrTxSC8zXkqDETHWRfyvDmKisst6sipvqpmmhNezWkmcVB34/FJ4VybS+ShkU1L3GWp6EaFh
aLDjIgw4XkCUMlxqQAnNgvTXnZGOPAqRBuK5jbuDG/GojsVrsYpMeMq9Ew8QdXKmlHOxkcp8Y0sS
J/bJNKkJ9dcPj49HTkKUNScUaR7LoIRUno5CjSAJfGhRAEea8HokSHV5TxRDrSF097gn9OSat+ow
G+K0+VAiaQmfXTPmDL4JsqiaMqw4vXSsk482/lg7d8hN85A3ktpCgNmVmim6WRM8XwwLJgY+MAnM
AuMpajA1GYWjAMVjyIV1WTskzeM6rhhflvzFDrdfVnXC4FTl1LphXiM5BORVAJHm3rzeXfERPLl4
kFebLmq7vBfMcTar76Vmqgk009KwQHKPf5H9eOMCSWiebnrP8oOoutK6TRNXghkEjXNnlQzOXPGU
yAVwFscGvQLRKKb8LyG2WByGc0MVLQyD0qBMI5hq3n6M5EVAQ/oQeofgcvqdKPGD9t8T6GH7Et6o
6J4Gu3fFqNEIOHM35sTWyykRXZjD+92ccHfadsGmzB6wZ2R8FhKrEWIpBQJja5/hd3eoGee/dS6+
p/OsXSXGhHTklm4z8NLj6Lx86vVm5Qs1KYsZIWbQplZwQckCdAi2Npb0U7E+hubtXhVctLe+HN2e
BfBFjXHKaO5qa+94e94f7kd3H0wu2b/fsvtBhp/L6XNra3zTGygDJwh8s6xW23AjrHDW688kUh95
1zvL9zLeig5L8DQ+9bi7FFnxoay6Azi4kx3NvLdUGUDbF54yO3JgspOMYFSAKYShr8eBgiGCPslq
GV8iVhJ9epFYCDpBYzG6cRdgKD5SpVhQHbt5NtIF4h8zJY45KZLzzC50qspk50KsTyc7RkYAp9Wg
iEWWrZnC8E6dQtXhwaoVZfOv0AbKSNkTQs40iHDBAnyMEUgUtkmALbKzc4M/EmyYEd4T0LHKUcow
UKs/jAl6eD3+XsxjhfVAM3oMj0zyh55O3fqSXaaEczKbPN2+bNHug2jQlX3FYBeA6PdcGQc5SAiD
pNM+1aWG7u0Lj7kM6rntuQOnOUp4LYBNDKgRg1gOTNCmpOJaCEaXEQxUHZkCq6TIWbNfqc8WmiD+
GShIRnYJtIoeiM33D1PFxs4CiAbSPdojPW3CYtqF34wJilcUttuMGMFlPDtAhFM5mLiWS4w4tHsy
HS7ztb9RxSSQ5EC8evSSNAqL2MrGaGqb+1Rc2gZcS6QyLg0X7d1ttVbauAOfyl5jdtmBNxx8pmbg
uhg5M+l4QQKx47M8IWYt6E4MJIJqPbBJEQmKcMOwo9JVVqbQwWJvUCrGx8mIW5st0epdtd3Q6EiP
Psc5CTTRo8t25iiIRuasm1vrkKuj53FJpQieWAvf7pJPVQZqFGJJpe6913fxvxqEtEY0Jg6l7WA7
x2gUTDfOEHNvU4utdrkeU694NmmM51Z3BobZoi1EYpBZAryoy9bR/VyoQViuOEMGj0zub/8XQ+/2
lHAeAf33lyjwKcUHgZgWQqakzqRUhUnRLpqA9cGRYsS0laNzC+b2OTBQ5smmwAuaJYMyNOIO5p5w
834vocu24YzfqDGGnfWt36FUxuh8VyTwIt0ZYHiu2SR+6taCCtcmEd9ZpNTLaPeLjphMkIW+n1Ju
jpyko6Wx8/drwZS3Q1b+fkYvYekMKixEAAjkA2QKrMEFDEM0O857/8QaFrpN3+btMeoznE1B6dnq
pFi71mct0QybJwY6lbCFPxfm/O81zQxyRBiIrqDd7CN1dbF9e1GKHb4gDQ0C7g3O85QROJ3Zc5eA
E4cxAhiPCgA8AimHYYwnDtWpQkFBH2iH1aGVru5wPl4betVoJjy1WXYsTkzddxkx4+s+uUaHwRby
DCxfB7HwDJ8Y1SWPHC3chu3XgMjzSgsyEAajR2AZTo3uzlnBIoGIPbYi+a9RhsuYbLETzhF1CmFX
RQO/8XacjQilooT8U+1qNa/l5EDDERUSLFCJMyhIWKSUgjQCGjUHU9RYvC5Vw2iknQrS8Xq4rEFd
hVRoRTfdd4ZFkxFuUAneVtin5oc7c5MlZTv4/e38BvPkINT2mC68xJ2d6HhBcMb3HClsTn3aMjQA
ENuRGzMny5AXsfAtoPq1zyCxrvEqZMRhwWxMJ0u79VHMBwn4sXDzxniyzFmry2TzueKB48Mj5pwg
ABqYiFvMguuzYzTSkg/bon28hjLOJWxEjEUhRSwUjtLznQsvwQU0iVitEGQaEEEc4ABVz87Bzupe
+d+V/Sq3x8OC3MVVYrtvD2wtwMtxv6piDhEYlibZUIAeTJyBx6fjtXeszXdbe8nttPbDGMZQAXsa
ojVaBxvEkIFm5UzWMwQyqiRzhXtzKR58qyYySaDg3JEy1UHfHB3exEjYfg/TGs5RfKRK2pAzuUf5
18FiHrGAz/DQdX8t5CGVoGhAA2eM7gLpsNTWTZ6q0qjpDQZ7FtzlvbA5oEkDBhISZCHHA6nqL7Us
bYPlIg+D5x6HvWytvVkyn0b5E9rFNO5agroUHjw3iIQDLiJV18U68glCN06XFdXeZR8McMZjijMj
ihyPByeYP35XHCOERyAKKNlMmRQ0HKXMr3yJtdm7QpZE0sJZwhdDFp8pPYau4jwbHRYjYggR0tDg
QwSq095y9tbt2dQnaMnZI14Z2czDQoNlA5u0alp8S8DHP3sc7Q5ovwn2PzqDUQY38b3SvoWCn6CB
Mm26X1NKvvEs5ceXjgwUQZcZIpuxNFnhS1uQipG7iVWVqRBDBmQEw+b76iuaL54T4zLPysdAzYDh
gz4bnIZUIZYADJVipWIEaaMBpUDtub1uxOHTqRXw7m3u3Jvd47ObrYO6kWiYhI2vHfaNui8G5vZ7
aI128yQEeNVZjHBARkp1PLBK1xEYJRdyGzKqs4AjTpw5SoV225pQKltt4OMPYItrWZg54VZ15jZY
9GtsbP0qSo8DusdZN6LHuOc3xfEC4gZwrtPn48LAgnaBGjMraLLTtCO3AXFkXcTgnjUxR7d2yTu2
gb+SgLunIsTKOYLOhsrxVEi0+11qYQki7cEhcthJps+8aQR3IXqs9QiA+XQNE0dbAjIkFhrTLPHB
gZYTsJKgKsMITMNBirJ75q1zPND/0c+vQMmrBEkVo786YMbDwI2NqkSSPc40JInnns5/WwZTH7xk
cZhumjDbOJg8iQmjFsBEIXguA9tpPm0oyRnKOcjIuQtTGtdhUvTF5JIXS8TOTNUGuQ+T4hR9T5pZ
721/napsBzDu/0RHdjQWmQgAfrOc5o+DgmxV4KBK6Gscn3iziiBiFxT1C3jVj7dSVTIuAYZPPIAh
y+K8cFkgKe+Ih5OmA33bAvLCQRZQiIWYncreQX8+IARMGgoRg88AgTgHQLkGB0FguOSC4VY3jsPG
3mBtW2OehB6acxAq4bzD3fgVMyvFKLpTsL34fdpVHnI+es4ZymhNIx9zS0vCp+xz9or/U4Rwl+iz
9L+HT2TDpaaWrD1tUTRqEoXOcmoc3M/7e0haBVDI3NeKtzbHmnwTQ9Qz5J/ERYRFM9Dl0c59yk/i
/Bxhz1QsxtlSuSOyJ9jEXjAedKTbGn4Wv7nvJ0d3xzrIghyOTSUFDSgOvl6vxA+lzCDmI0RsOvPF
9mcAej9R6Dxu0HZ7/znV8r1Si5aCIvatGloqoQg59CAPZZirigWXpQokEJfuwLBOgsTB95c91RVL
YtVhNOw4phj4IteCgVZTkE6QkUCzmt97VQEyY+KAGRQQIITBuqgXOs1X13m2xPxOKbp7XkaJQckP
QOWgk21rBBBMYzxTvto9QZ0TyeoHNCRUAitFxg2i1O88q3hqUT0PkpVen3Ezksr9PNmv5Uzi5ogy
BRZAAjSoOVKfYOZyoT7+CTXA2QXvT996/TviwL7IR2/M7ALcD2t0iCQICDDIn5FnoZFNHwpBPflv
zGxdzawIOZADOksCtki1esIhw45Bvr+w2/BQOIUQ+pyTpm1yfb3oqWMyNet2INtnprXEQeUxBxFu
EaQe6hBokw4A0qSZpYof9rHo9Gfssyt6obgBd7sNwvZSlb8y0xQmhcbH01+Ds/rc/ay7TZg+VpSC
Rl614/F8CvcWXt8EWxXz6RHNxyj8pHEhIW9ZBa2QWdWuHMwidvwsTlrySjUEbAQESrlFRgphuXCT
ktbwFQd9yEz5uDUdOq7WVEzinCuaYy9VhQDCYlFtvoSDCbYNCWA6XEnSfxusux/B3MHv2JYCuGOr
DQCn8mnpcIr2NNr9dtjfPHaYetZqXKTR0dMoebRq6nEsPV8MWiJcEJ+nqkr6+j82EXj2UQ9RJGdl
gt7sqDt0xnSH6uaqieSCrkzNZYAZsw6rFISQEn3278/uFIhfB03Sxiyr7vDGGvSfuBgcNnPfvY9v
Ioba9zOKJ+/kjuM54ziJI+t6/qexrJI8deUG9wtsoQfkQG58Fjwa3dPuaJkPwIhjG30edTKvfOkY
z2/edOFBy8JcmIkSCVhX0N7lRA2EP7KWx61/D1YIEwc/B/XmtuiGPN5/5HJzHG3btyGsjfD0NEfl
b7azrmh3j0nJgQwhocEKrvfbSQSaw38P5zwTPP9G/n0sqki7qYsmysjkYIzv5hASHpj+wOa3Jhwg
zMIQ5khcC2iO0SWx+9wPPPT2Yi2aoBecf70/bsfmVDaQyxg6rI/N/2s6WoZ4ehax6I0EwU5iiDaJ
c45qWNFS/cshZcN3axLY42V9dt6fRFhMB3vYqAKuNFX1yztIDaH5WsMBbyf8xPYzyX5XKY9G6T45
LDFyzTVZlpRwwzcwNUPNlwMhpljqKGIe1iJmkOQ7odPPDKMHlzlBDCnXkja3L3gZijnbyLOxZUMN
Hxl+pOCSnnL9TeyMOuivfiWEiAJEGlGcIhEMkDJSQeifN2ADUopo/t0UyQvyd7NixedoZI8a1jZK
hodWrFCVIKq8V22SMgEmQpwQhypGgAzDZUeyXxosVu+DpHJ9QQN4vIDtQTIH8FQekIFDqPkF66Pr
KPTFer8KFc2M8XudKClmCw9Z4QDxdKgh7ckKE6Ik9HyTQ0zhQBBpYTy5Dc61EMhIQyMn4SgLhMZx
g2RRoytcLiWB2eRFnve8TZoEXGIjkdzo5vj/ee2XOFy3ckziTz7NDWxZlzl+f7/PMQddn9LQI7Rz
lGKrlfwd3RfdVWtFby+e138Tm9AeD6L4X1h63YSfIhZ78fOqqqqqip4BpYl5nu5JIP3n0OksHbCX
aPFvvtOrwqwW+pLgSBMJW+DWx3KzC4NWbO2i/YJCmrXIizZD4Gdrg0fgIkV/YovHQYy7SkiUJBAq
WLFYXkDQTtuhZfbFBq2Xkw8TmNTBp7adgSD3hocxE9YKxtZkAma5V76wbAVIRjQy1M5juhtSzCN8
yDSy0b1VlMpNglnJrQC/507qTEiOkhFYxkjJIQQgQcGvc+EkZIxjGMYxi+NpXrxPBRFYA6MoWMSI
gHGQsQgItmGBjMFIjHkFXvSyZlTdDTCjgOOU+c95ya3GvGHSdof+C97PD+9oR5H1BPpnxHStt0sz
6JCenCgVQUgS+ADaAj8e9h17LCskk7B10hZe6hAqm87ond90Na60Fxm0/Iv3KEqUNCv4gLTgXaBC
vhUXzBRwNt1/WC4JOku3gpKCeBpBPiMFAqFJYkClBGDxsJGjEDnZXDho0qEi7cKUFiDCCJGVMV6D
eaEjHUO4NKomlkK1KsV281jFcwkQ0SsKUOEnQk4mkEo0AZA5iB0HZvSgGk36HPRdGMGtSIaZ+HEM
V4qPui+ijAWX0ZBP+BXGRWVyzLDZt62264whrnA3k0shJyEkDoGBELFAQY0AGUgFe8rg8AeWyUlF
Bgsm7D7vpn8n6HCVUPX+AtBmJaejMX5vxoIpAcnJ7EKHSjh/SveKkUChCDRRBITzrDfmblBQYCDc
uBdBYUTDLl4gXq6yEUoNr6tktfRWuwC8mwM8FAw2wcySJRKUh2EokpKIkASrgG0gmQD4IE6JJdF+
zu7LLyyljksDFDSf8Xw/j8rDRyXiFaDRqEiyAkF1BSyhEsCMIKTBzEZBZxDopAzjc2WL53hJuBHK
48Q6G6SLEYCxBkdfDNr1pCc+qBmUHgMAb9XAQN3HyC0SgqUYsnntcMHKKGvdDedblwYiVqLIYvWa
zGRILgHXA3cJxRDjScEi6bd7v4pDTuaeDwHeBO22YUWKKKKKKKKKKbSc14JOTRFhOXMxgMOrJqUR
WyYpbWckm4qVF/Ix7ZkymzgBMcoRCQjIhhaXr0BdV2c5wTv5pG3KdSl+HxNfT9f+nxz4WIMg9U6X
QvcLQOlSQafLESGlI4FjIY0Fo4lEB9VyakiaHQ0fAJr0LGGaIcFY0u9nK8h5H9L7K567v+p1EsqW
lxtIm06md9cOmHeytkFHN5TsG812BwTJZBgGQ9YxavF49Hq+dwBHBAbTSetEDQ2hsB9HcICSPQYb
ussIGJKWnpzMAPikDShmIC4cVbeR1a11zSpRdOyIHCHvoDEpQ5K8VNGiwVZZVvXhLtOhpp0QD5gx
kHE2CSVtYsrDai7s7dWtNmTUZA4dS15oowbMrBmwlVVdhDv2JLucFIUJBwcX17ubzTlQSFw6V560
hTScw9QjXWPkMucrtkGTzEE+BD5Ww3DdOJ9V3MjQQscR1lNgWb568GqxrgzbLEIvjUvJXJp2wjRa
IGqGl0vwu9bsFZRRTbZdqIpxlF+moxfbaCMS4TqLac77dmjEmoDPQTtya7M0UyTBQnTMWNI4Uh1K
ZbkQoW1RfO02tI9kqZQiVtRZTZn0m6ZoK13vJeIp3U6iSm4hI9at12IxiyF2BYIMkBYodk17zQQD
hkBgTTdoU4QM8kEFY7qlo1SoqFhQ6PAboWYrO7N0MmY0odMzBRnRtWI2GuSXKTRll861xtKdI4JT
NF3rrp0Q7E7e8km41htQ2sRdDxJGkTpEX0L7cYkUyN3SumiCLWNpLB99pX+r0vDp4Box/1fIOcQU
5YQ4jpJzBSsVKPpSlsiFQ6KSXgsDqVZ0Ruiyhx55wMHGTRzyk230aoJpF6eUEJyAGiNWi5RVmRos
Cb6YyshpDxA6xjWw1aSx5HKwmCuhcCtl7G79uaCezCjWIo5BeqJjoKFjLxlVBGVEslsJVXAypiLE
NaZxrKwKcKVJcASylwgqTY6oBQZMMkrcWbQmhYLEMaN9Wi0RzztPiY9wG+IV9CYpsHgGGspBJQkl
84MDijoAh0ENEMiEtsCxCApPUaCGBCTXhhsRPWIEhQ0gYSM5ag2dEz4GQcYbMBEIi+lZYiFailQd
yhFOco5rE7hN345yqntbWqg7IJHTVrtE7owVzZsvxS4O3tpgoljXSi4VQ8JvwUsRqwK4rzUyqLTQ
Gm5BSgm4iQ23NrISgKDQUx3zxOes4adZWNXPMxKvirtUXDIZSFYIwvbDKV4kYWQZdnD1dJyTiBhu
CoqbVVurU2Qy4btJOejDIqWPQv0kxFBf5crw5JKj883ikDc/o+8w7PqeLDsfgRxRLmDGoXHgpZOK
VBdS8OV4VfmS2+FBRqWK200VANgJS8dzmrRmmlIkcTUN2VAqRhcEVDt9JqnnoCj1hjMm3WWX0rag
gJfxKFNE99b33j8DDPpo9bOvhua8eOroLh0FC2TIl3STSWmk+bs7PSSEc5IJmcNfkcucQkM513YC
toLtaS2FA2nhuo1hkSf0ZrJ5fmg+NcXDisZqoUMWWIaAhBKdN00M0wnAmh99ovyTTsINMnJOnx5k
grZRJc3aDGVMBgDQlLIHdnJrRpJIVggrhaJ5wAsFEhWikiMIOTfhvJlGqxbixBVYgtIokArA6vFI
ckx5vqe5Yz1zfuRRRRXuLRUUUUXSzpJ/XGcC6dy7DlptkjFwMC9pWmutCggUSpdYmqUuqSYloiTO
vtzsGLuKKtM1YbMRIbuzo2YIbbBbRq/cFzOYS2kpqpdQzSSJGkm1jIyV5zZtQYXURMO9W+uZIG4y
JAbQsaUiAMs0riBoRUXC1YniRCVQl3L/5/b2JFKSpoFJre5Tp8jR5eyzV2xiABTT9y9skmddXQDB
taZhA6aQhwrSJRNHI73F8zFMvWqV9V4i6Nz+ODssm355gExuNpM7XYoAkDudpGhh8YnxUMa1M8B1
LfyulOxo7Fsb8fY2CMD6vU6PftwfqNTT6FBCdwlKBtPWYhMEaWD3K0UJkIWKx3JkrlqFlEdumwVA
5kosqbUwJbH0/m/dXy74T31s7zyMp8Y2a5TIIxZkNJtcMVJDKwJA4eg8x4Qepv1tZgaHdhsoybOH
zdNPP5sggZ0bW6ZPKeY1TUrDVQnqIBxwQ1fKcDFrdo2jXWhVzoGY/DxgCuA8eGVmCmTl24lNjM5w
TEYGbIVQUrCtZWKWA5QYEIgLtOHECFUUFAi8U7QTsarn54/J9ojOwxHhtF60weRaX6+MLaTUww4a
MYwtMgg3jNIbCkGQw0AQpEICSGLFUm1pZbEieLJfTUxVK8nXhrl35oZ8KLLW3bgcgZD1FK7xAjzC
4UoxlokWXsV0AmhFCDyrUiEF+8np47rRPLmv5i/eOSdaA5oM4GcIhys44CObSz0nwK5hlkYd33fc
B51qG3swWKsJzm8UY0Rpekjbz3JhvrquHhZEUhbiEUFEYKRFZpbIOw4OTQwHFKZncZFpSUtsDBY7
ErLYMB6I1oySKy24kPzFoXALtZAeTTWkF8uBo05sEMV5AxCHRa0FBrhubBTTZvep6mU9TIXri0YJ
KXvXLVEQ0IJkL5EEy7QSxkEOt1VGOhnvPx2M1OXvXzCPiaaTYhuXGEE73vVt0E9q3Y4HANYNkNS6
rhB9vbr8h8yV7Ysx3ValdSPHwTRthz6LZqTgNMhDBLNJoTLGSCiijFleAVpGPSKQxl9UDxHMTbx0
grWm0wbG2xNjuirr9qX1QkBQlShEQRClAnT6tUX8+fHhvlQ3Bhx1zePASE/tIp9IZhLWlhFVCEsS
JsSQsAMgwkxilREid9v35DT32l3w45HbkLhQRYL9DmeYb44bd/8Pf9iwyjXIqK4onAe9qjPVhcp0
1ubJZLkRge4EPO9Ww64JGEIwBggM4yNMosWdLRQqytnTCFidvNQuMXrnoYwpRbNaqYkNNAwLuAQl
euHhSMhIYX2VvDJSlDK5NMpS4pSwKqgDIhF6oVxBWhL7NpTpCoR0sdfg2mrjLnub1+rNgsBMUUKC
CAu1h0hkFlBnAZDp84oLyyw4Q1j1CGox1vN0KSS9MbTDBM6B+A1pUBha0FhCgw5yoySGzNoiDDBn
DFidMYsvwNPOUAvmDnp1Do74PCGJNSdvZiwbWOVNUzuQIkPVd031IUNqIG7dvoOZJEzbYMs7Wy3H
HjNUnjrHBcu61NpRF0w5LjnBnu6d3VjRm7sDR0OfiVAboXvaaxGKqwRiG5jLqArsWwKAaVQGy9RB
L8UO5L8TBAis3V6DRKPubFGEQqFXaNF/dqMG5KByhJRXmgxIbn6LkUua6JA1wzgUV0UICUd9s7MM
BXLm2yW22wtmrwrx9hPGE+RE8odOE7HGAIt4GSC2CCGiGoGpjcOhGnTwae67W7ugdXqs/MhNi8i2
AnLgPXfoa1Gu3zdEi9kebbaLv0aaCDCMiAGRMvCJcyliBQSsxjGIKWxClSnfGMCEbK7jmvtJpm2k
y2XRyeXTOhANnF5Hdz5GB98xI8i7coK6pbBEWJKFAHTFSQIILIllRbCXttzJ4HdNw0sJjFtusOF3
sA3atXpGpi05sMcYfCgn0RKCijbANguGIR85NjCy+y5DNx1oB7XZSwSDqoBaAwrVsrB36BxGqAYG
kseZlCQoNJKhAUDCgIIS0AQqmDMKrSSXMeE1iy+jz0VGDudyoDd3b2AawDhiGwMIgcszG6xJACZO
JA3eIDXwlqjFkhT7A5nff0xtKw0MATaRCVVzoQo3c/PB7RZTNgPyVarVFQ5tX16l0b9LM3NhQa5k
x6jNGp3IfDAiJGsLG1ChIgQQBq25B0CqYFIm8MNt3apkbIH0Y0xl5Ug1Snt0G7hnTDRCh+SnTsGq
6btWTk2mhr6Gy0UO7QXCOG4YkVaRMKIiTEsp0ShjK0zijJRlNUNx0znnayhpYb7M0R8q1YbMXkiJ
5ytd7pw56jqyEZlD1kSaIIUXL0x8VVtDVIFBk1Casn9KTVM+5mesYCNwM5zpdYYcR2ZUjDoTq4Co
smTeYTYWWNLbZoPTxr0ES6y4Zyx5xp1PNPkcXayBxmrdyzLsMtXNozM2dKWTgEimZ0aDVFJKhlZV
blWi6nWN0jZ361MHTrgoUAUp6eJuaY9Bl1+MaG+JO5QbTOMhp6uDrtlEDXbUat10yykFtYZrT6Ww
6SIRZnGkOCHAkgskgsh3MvfAc+edSt2HNDdjejF0nIJr5+pmZa3q+5XEREX/BpgxxjyHLG4KGsjO
XCxqFLhTTo2tqonhibhoMpd3x2byNjJQKBkI2zzd8Iq0TgvtZ8cssrvXmtKqCainSrm2ZQ1kpDOj
xfBvT8CQrTd0qDgcRmkqQw+J5IQoFS1VVjpxf7z0vD4fNkYQPpx7yPuaPqJvydfYM5k11xDGAQUD
XJ2MDfEBmQBmYiHeKw5csZNmX0XedHZRIaLIfqphmiGGswIwmxy5goydmugq7mFjAOsrkBDDZjs0
w7IWEkPCrOTOzU5oJGjVhrxemL+n0Ze/Hr+GM6b29CTMhIMQYKsh6G2MVnoxnP7yc03cHUOIOOUC
RJp44oaykJDMuKiaJnW7ytHF6VKEUzoUEhFmspZKMe9UODcHs/U4tnSKbQgQiFi6QwYV5LZeK7o9
mGZbTBhk6oRoqIrzS1VEVYc+ZNxETORURURW6MWTGFXDKqvFy8sqMiIaTSSWffPhdjNbpoC/cE0C
FoGC4xAGBYBfZ2YHNYyPdE0FaKN1xzctZhTZm6bOcO9kZaOtYDvwVgNW9brRqPKMpDMFF0M2k5t0
rs2TkmTIcafeSlpjsVNw5aWjIk17Nxr+mZSeHQxxprWQx6gSmlVdprR21oQBkKkFWujsUaVCEXLw
zFisM46ziQFqV3Pk99sF4ohNkNgNlxQlW0AZsBmSqG+9NBABiuccGa2PfGWeu+1Oct+hWEMziWA/
lE9hbgXWHghLPTDhUIxgnPsoFTSI9/OYgSApQjEZKrGmgBuUF/+x6+zuXp4Lx6g6B9QtIc2nCe7w
3nHn9oqobhAVBBUxZhRNmoj8CvTm1EGZz8Xp1ZOjOBqJG4DQZDopAl8gbaFjvwFwg4/tpUOEiRAS
nIhc9F4YNFwTYFlq/EtvI6KVy6qTy59W/SikKYPM7whXyhdBJTASkTPNoIWMa6WSwUjJdjzTjwZl
OOTHRFo5PjopMGm3pEJwIhpA0IA6cIQTc/lkbzy/7+0Xav/w5FVVzjGJsowmDwXmZtbCrVofa+NS
qzYZAmXIdPl6pFcw37QuvL7uVHh+kkpIVRSDaOR1CpoFvUIPdspfhqofDIVExpj7dLfMpW9SNyqA
tEsR0revQbAMjMGimcBJMtVrpaZ7/TB6I28lWBhy9uTgPZpXx+b2fDvYUJ8WLYcHYy45hV3ovvxR
ssOAUFFPeN7pTJSGm6fOEpopFaQHGZQMmCnTs1jVFiPF8KkuY249PDwvN4NVqH9n4W578cGTGd7j
ye6v0OyQ0VDAXwYzvGEomPVFFDFc779pd9kT9fn9q4U8zy5rncPtyUDodMNFqvkmpxKJ7WcNll1J
SE2Kw7vcpvdX8KqTBtKuhQiGSa4rUpEIhi7jI0kIaxCHsVfP5GFCBeAwVG++4ur2FR3827QWHMlR
W9waTafMfgfYflE2nUH+N5OMfot30Ph2cPMhDkAdBoF0CU0IZDTon/pok1bzuryr1auVWNCSAaMw
B4W5VtY2UEkoOTB/7EgKwQnOs/Ay/zWA4VlZxQq3nVnNk52bxD8UCPi4qhByL4ZmQ+ia4qPsrn5/
1mnSe84P1Rh8s+fpH7RwH67bCt/MdNKzxhJel5Mgld/jx9soFukRBHe8pSdOSNnlKviQ8nqanK0O
Bg0IkIuWBxX0on5E1X2fJww7VAkJNCdZ9uJ7ir9h7IfVC307G/9hS0Ecny7UfpxbBNmDfGi+z2RP
seJ16JxhmHYKeWEd4nCX9h4ZDpvPke9UTqTLOPXCteDxyzKeY1xdytk5AzuPg+9IvLDXcqdqxHMk
Gg+GJEXW2sxKh/SDQRFdMIo6GbxmFLc7KdImpl+9imt93ssK0jip/eP5joCuFAzNF8w4HsBM+VBA
7fnav6EzWeH2fPuSB/K/q9QqDw/6EB231q1ptGxb1kIHK46I/vHv5+BRErSoBVx8562rxU9x/Zh/
aNosRwFuQmRix8t/7BWfYTlr1kds66GFZ5rGRfrsAHgE03IUwYCgMvX2wSVvsB7AlQ4a8DqpAchU
O2dUmbC5RaX+Tr8ueM/6z/MzNYAT03xpkQvb82PknrS1HOOXLj/Nbulgbr22Ap65QL3khEqHfJic
ofJyfyWXroW6Of6bN9JiLDdlCf/2gQrB92XTErm2LvGWgcRfNCGoY354imknklEELzS+mq4NU/uu
5k5HGDGVym9jOSr3WTgG4m1GjMEMcY5GcChO6T+Hi+jWw3UJplSkcs3eVqeS7mZut0P8tjD2UHmN
4Mr12A9uudnfFlT7QlSLMOfV9e978kF/kq7WMQKI8jh+64cn0goS8c8oeIaOQ7MSDqvoX3Y7ZoPp
LcY3GUMWZphLU01ZR8kzNzNBUib3Nttu1aSEo8soXWA3Zrp7UmLOrjt932rP463pULFRV2xuhT8P
a3TVt7wz6dRjr7HKemU3AvlIFHp8bCoUUDZENTeZwKExi2hwkYOutS/PwbMJg0dA5pqSBKG6nUVa
LGRdrKqdg+V7H/VC7U7j1Y+Ndb9D2zxPC3qN9Ag5Xu9B3dbrmNHuu+fzAqZ24TpZEInbbF4gXHfy
/nXOnr31lR36ctXwmdK5oHokYR0QcZf8MqrSNwFyUzLQ26FuUJhGIZW/orSe0zi2M3Gi6R1oOEuH
ufcCknuMlzqXcvKSgcScrN0LitihCcIPm+q4jDrdiF8viY+9gmTOg2jMHdSEPPp6kHHulRU9/nHJ
DSbdjQuFm59OoEaciSFgor+3hpHwnkHP5OEIaDg+ATvsNYswg/xBItMBmjsT2LK9TNpGv1PPOOoS
JuwmZQDMTdkgzJhtCDpOk5lX0xtmbLxOVJ1kDaMIn9Zl55SwO9PZNj9w/fQsCrOU3g8sT6g4bQKT
9KjPP5PxhivtIT1z+Xz3HS8ynY2pOvhfDeTnIKEiOtEH9c9eDQQSDWMBBiCro3Z6cEUnp+zWdviA
LqrLEFZu3ZWZbhAEPE/dN1oZGEPlMFqNXk1f4QrBW/NFdFRPAkh594/6GRH9BBjLVBtG/4WZ3aPo
WXMbHKfg0H7VLDG/SjVm1L3QSwkXqVskAC1guFRFuIbIxeZ6kkyv9+Nvc19cvYP2ku0TG8cxw7/8
nUI4nT22PuB6vnSLweV0QK/MIDFavWQLuIMiKiIzJLDwR/8YJqUwjWr3Y9w2gah33Qw1Ctf2l8o3
NYsoL4NPOdGAHrHO/jpBkAwQv7Jxa+XAIAEGEJOagRF5lTOCiJWn9xhzQkNtUjgr0M3pfNKmxz4x
P395XL7afz9B1JhGn7nRSKvIBurgFYpsbYZSXrnwwWMzOxf+iv+apbsJD+eaE0DyCXeCpB3BgQeY
zK4qQbm22rZARn3h7NCwdgRATc3SLi+48+5D8Z6f1d/bOgY3oXv76VFJwUMImfy8QegeVtsMrPaO
8mXtLh+0oLzPYxhLnwTlVoI01r/8KiWxWe/m9H1RqbRboHC/u0JJg8IKvFJ0/lahZfLC30eg92l3
jpyNE6QTk8UbxZcKeBSFkLMPF1krIXdB44QZm6du0vDWDl607BX2Phwpzf9qgydvR7KYGbiUGoQk
BEcISfaNKyFgsYPXZSnDQmqMZ+ojwqPl38Sa5VJMC3MAnVtJIen19wnRWQAJFlIRFDlNuwxZIWSO
OaY4QY55yXyLV6gF6g5VJWIjSdGArat+IkNRYcRI641X+VEalD+HVMdEMMEEmAhAHDMJLyoib7MT
6oZU9Dwf/s/v0pJp/GnR7O47fqdH23CmGRSG01gxGs5pnMsNWLp3eBzdntQYnX+mTA2Eu3QvE2c1
n27abcqTno6UJCJ7roMNh8LOe/ghfMSd+YjJHU7dgleHvM5YqTkSm+7WoCw6LUpplpbxTmcTEjq7
mUyGI3qkuIAfxGAQA42svywTOB9S9kpwdchoxCqNau/GXXl2N36DoMuznF+z83ozmQyfe2weiO5R
AMhSGCk1C2YkUj0Eh94khcnNMLL4WwUFGAYMwCu+P1vIppxcBhLYiZNRTcjSpnzxJhk41+go7cpK
/3QLE5c2BqM5ZGZX3RX6RlTuQFHOUwoaO84Adm92jtnoRCYdQLtQV3Oa8R5R8I+2NmgyV4Kis+fc
LhTNSn6A8zykBEfdEWcOKHt+f0LuTuFa4U/f5i9UTvVdaBc6dS1c+uNC2hnwBhQ+7fekhrsgyDs/
2PKapOJ/ogwcPh+a0o7xqENjGetGCSWU1tfRtfw/pexxqEYz94fs4H4vnKsZ/xDPgnhHr0Y1l1xt
XfecgbyGArsEO2OJhlaBh1oTWHzydVLlE3HenFyXDHjW+ayCxPJfmccbQazlyN9CLD/O2TZZScE8
t1axXoEpfZTBpmO53Wq3lDeEAMPPlCjmPJQBgvO72842TB7wYOriYmbAi3KgYxXio2xNH476DEey
TMrCZN/AEklyYWMMQQdTm9p6Udauv4Bb3OdmpmG8TOPXEjMQy8817vP5/A+MWO57GcdqMVkWUNzK
2KIvHRuST36mHxUJeMztZr07mytm67BVevyMkOSrj0qyxx0eQzYbhOudtrEwnExeOWb62GKPUh+H
Fsnf4Dvxuy7/BwNi1JVMYCsHgzOvvRoyeWvh5ZRVu/rXqOrRp7ZPD4NRbDnzV3tVnurcUA7IM8YN
MWhcR6GpruVcyKHTBVaQuvT31hfZKozmjgZres8r8fG0MvD4W2WlYOE9TM4a1uMXFX39u0v15XUt
ixu44YmBmV3dDrk4SsHHTKyt5oExKGALKoRFS0LlP019H1V5nYobfvKA7B4hwkWba++QFSaov4uU
lcmzGPvbyfunE5OgCvB14MIBIAi24YtEGEB3mGm6PBGgMUaCQZ3wzArhlfzf8/tz2dyuadNHPm9G
K8zxcTDWkWtGNCFfymB2q9N+dUpwotZhAc4HU9Hy+cGeNX3dHclRKHkQH5XgTsOQH7enV+JWJS4Y
V+i3+Bw7lH0J/aoRynve59GhH3+N8ftb7aK87mP8gDxPkSV6fdrzdKKJCA1RmSO1/O+5izhmWrm+
fY6z/Xc6m4uaQF+5IAwxQrLIM0T71PqbnodT0vL/BEyXdTk5dST/fPXOc2pbdWm6YHg+bbXyltvB
zPiDph+p4DP1UcBPBgASBltz93p7RhF5qd9/xe6+WD0yDuhBhAv0ICD7dYKSJjrdf9md/jwqqV8+
0cCf1Bu/DMllvslZHGzza4y5wj2ZmZixBgKDmKZlaIg9FoAnuYAWacDGJgh1tk3s7mdrCaprqdnu
BmXUH6jHtv/cxGyzK8Ml4qNno8Cslc1NwxLtD9SkH96Jd271eZo8ZQ87v8rZ/BjPqf/lx5APx0o1
Ffq1Zr+Fius9XwzlW9U0XXMWzGBTtKll0sKfKCj+LkIDJqMf1MgVyYCTA+ww5ekFN8NSK5aqGgFy
V/vjWH/zlm94+rcnahbP/hxLib6PASy3UoN2geC+moMZnRLM922nlr6eLtGfHgZiRl1gxOoqTIAp
sGsZEVe1xGg4zAhzaSqiEy3FikeTQikd2ZArbCBGyJf1V8Yx3Rl634P3d3uFixPnFN8fn8jp+u99
bApqU5lP1ZFdTRSPp6JYK30mRELHeBM0XmxQ4zSNbVJMcIh5000qlRoUsVZMbXbTTg6TLdwXKMq2
OnhJMHV6HZf8/Gf3f6/R7+leU9b4v/CcPxLKuNH1yH3u2SwysOUM40+qDFkVweWEAYAG+LuSKcKE
gwu8HHA
--Boundary-00=_AO/KDDsXIk/KkWh--

Denis Vlasenko wrote:
> > > > Monitoring disk access using gkrellm, I noticed that a command like
> > > >
> > > > cat /dev/hda > /dev/null
> > > >
> > > > shows eradic disk reads ranging from 0 to 80MB/s on an otherwise
> > > > idle system.
> > > >
> > > > 1. Is this a hardware or software problem?
> > >
> > > Difficult to tell without more info. Can be a broken IDE disk or
> > > defective ribbon.
> >
> > Tried the same with 2.4.31 which shows steady behaviour with occasional
> > dips and pops in the msec range.
> >
> > > > 2. Is there a lightweight perf-mon tool (cmd-line) that would log
> > > > this behaviour graphically?
>
> Try attached one.

Nice!

> # dd if=/dev/hda of=/dev/null bsM

This is what I got; what do you get?

Thanks!

2.4.31 # nmeter t6 c x i b d100
18:56:36.009981 cpu [SSSSSSSS..] ctx  145 int   86 bio 4.7M    0
18:56:36.110327 cpu [SSSSSSSS..] ctx  145 int   87 bio 4.8M    0
18:56:36.210735 cpu [SSSSSSSS..] ctx  139 int   88 bio 4.7M    0
18:56:36.310315 cpu [SSSSSSSS..] ctx  142 int   85 bio 4.7M    0
18:56:36.409979 cpu [SSSSSS....] ctx  147 int   86 bio 4.7M    0
18:56:36.510764 cpu [SSSSSSSSS.] ctx  161 int   88 bio 4.8M    0
18:56:36.610213 cpu [SSSSSSSS..] ctx  165 int   85 bio 4.7M    0
18:56:36.710194 cpu [SSSSSSSSS.] ctx  152 int   89 bio 4.7M 4096
18:56:36.809977 cpu [SSSSSSS...] ctx  135 int   86 bio 4.7M    0
18:56:36.909970 cpu [SSSSSSS...] ctx  142 int   89 bio 4.7M  16k
18:56:37.009972 cpu [SSSSS.....] ctx  130 int   86 bio 4.7M    0
18:56:37.109979 cpu [SSSSS.....] ctx  142 int   86 bio 4.7M    0
18:56:37.210335 cpu [SSSSSSSS..] ctx  135 int   87 bio 4.8M    0
18:56:37.309973 cpu [SSSSS.....] ctx  100 int   68 bio 3.5M    0
18:56:37.410011 cpu [SSSSSSS...] ctx  146 int   87 bio 4.7M    0
18:56:37.509980 cpu [SSSSSSS...] ctx  145 int   86 bio 4.8M    0
18:56:37.610117 cpu [SSSSSSS...] ctx  137 int   87 bio 4.7M    0
18:56:37.709978 cpu [SSSSSSSSS.] ctx  133 int   87 bio 4.7M    0
18:56:37.809981 cpu [SSSSSSSS..] ctx  146 int   87 bio 4.8M    0
18:56:37.909972 cpu [SSSSS.....] ctx  139 int   86 bio 4.7M    0
18:56:38.010504 cpu [SSSSSS....] ctx  149 int   87 bio 4.7M    0
18:56:38.110025 cpu [SSSSSSS...] ctx  136 int   87 bio 4.7M    0
18:56:38.210122 cpu [SSSSSSS...] ctx  137 int   85 bio 4.7M    0
18:56:38.309975 cpu [SSSSSSS...] ctx  141 int   87 bio 4.7M    0
18:56:38.410095 cpu [SSSSSSSS..] ctx  138 int   86 bio 4.8M    0
18:56:38.509978 cpu [SSSSS.....] ctx  145 int   87 bio 4.7M    0
18:56:38.610115 cpu [SSSSS.....] ctx  145 int   86 bio 4.7M    0
18:56:38.710204 cpu [SSSSSSSS..] ctx  141 int   87 bio 4.7M    0
18:56:38.811411 cpu [SSSSSSS...] ctx  128 int   88 bio 4.8M    0
18:56:38.911221 cpu [SSSSSSSS..] ctx  131 int   86 bio 4.7M    0
18:56:39.010779 cpu [SSSSSSSS..] ctx  127 int   86 bio 4.7M    0
18:56:39.110211 cpu [SSSSS.....] ctx  127 int   86 bio 4.7M    0
18:56:39.210086 cpu [SSSSSSSSS.] ctx  131 int   86 bio 4.7M    0
18:56:39.313572 cpu [SSSSSSS...] ctx  137 int   90 bio 5.0M    0
18:56:39.410307 cpu [SSSSSSS...] ctx  139 int   84 bio 4.6M    0
18:56:39.510618 cpu [SSSSSSSSS.] ctx  131 int   85 bio 4.7M    0
18:56:39.610558 cpu [SSSSSSSS..] ctx  144 int   87 bio 4.7M    0
18:56:39.709981 cpu [SSSS......] ctx  141 int   86 bio 4.7M    0
18:56:39.810109 cpu [SSSSSSS...] ctx  139 int   86 bio 4.7M    0
18:56:39.910581 cpu [SSSSSSS...] ctx  144 int   87 bio 4.8M    0
18:56:40.010554 cpu [SSSSSSSSS.] ctx  144 int   87 bio 4.7M    0
18:56:40.110382 cpu [SSSSSSSS..] ctx  143 int   86 bio 4.7M    0
18:56:40.210709 cpu [SSSSSSSSS.] ctx  138 int   87 bio 4.8M    0
18:56:40.310512 cpu [SSSS......] ctx  144 int   86 bio 4.7M    0
18:56:40.410830 cpu [SSSSSSS...] ctx  147 int   87 bio 4.7M    0
18:56:40.510371 cpu [SSSSSSSS..] ctx  160 int   86 bio 4.7M    0
18:56:40.610664 cpu [SSSSSSSS..] ctx  170 int   87 bio 4.8M    0
18:56:40.710106 cpu [SSSSSSS...] ctx  149 int   87 bio 4.7M    0
18:56:40.810056 cpu [SSSSSSSS..] ctx  141 int   85 bio 4.6M    0
18:56:40.910416 cpu [SSSSSSS...] ctx  138 int   86 bio 4.8M    0
18:56:41.009974 cpu [SSSSSS....] ctx  142 int   87 bio 4.7M    0
18:56:41.110094 cpu [SSSSSS....] ctx  142 int   86 bio 4.7M    0
18:56:41.209972 cpu [SSSSSSSS..] ctx  139 int   87 bio 4.7M    0
18:56:41.310140 cpu [SSSSSSSS..] ctx  135 int   88 bio 4.8M    0

2.6.13 # nmeter t6 c x i b d100
18:09:22.117959 cpu [SSSSSSSSSD] ctx   80 int   47 bio 4.7M    0
18:09:22.217932 cpu [SSSSSSDDDD] ctx   83 int   48 bio 4.8M    0
18:09:22.319200 cpu [SSSSDDDDDI] ctx   81 int   56 bio 4.7M  28k
18:09:22.407979 cpu [SSSSSSSSSD] ctx   60 int   38 bio 3.8M    0
18:09:22.517960 cpu [SSSSSSSSDI] ctx   95 int   61 bio 5.2M  52k
18:09:22.617942 cpu [SSSSSSSSSD] ctx   77 int   47 bio 4.7M    0
18:09:22.718043 cpu [SSSSSDDDDD] ctx   81 int   48 bio 4.8M    0
18:09:22.817976 cpu [SSSSDDDDDD] ctx   80 int   48 bio 4.6M    0
18:09:22.918080 cpu [SSSSSSSDDI] ctx   80 int   48 bio 4.8M    0
18:09:23.017975 cpu [SSSSSSSSDD] ctx   80 int   47 bio 4.7M    0
18:09:23.117912 cpu [SSSSDDDDDD] ctx   36 int   25 bio 3.1M    0
18:09:23.217943 cpu [SSSSDDDDDD] ctx   44 int   29 bio 2.7M    0
18:09:23.317949 cpu [SSSSSSSDDD] ctx   80 int   47 bio 4.8M    0
18:09:23.417944 cpu [SSSSSSSSDI] ctx   84 int   49 bio 4.7M    0
18:09:23.517944 cpu [SSSSSSSSDD] ctx   78 int   47 bio 4.7M    0
18:09:23.617948 cpu [SSSSDDDDDD] ctx   67 int   42 bio 4.5M 8192
18:09:23.718238 cpu [SSSSDDDDDD] ctx   80 int   48 bio 4.7M    0
18:09:23.818039 cpu [SSSSSSSDDI] ctx   79 int   48 bio 4.7M    0
18:09:23.917961 cpu [SSSSSSSSDD] ctx   80 int   49 bio 4.7M    0
18:09:24.019101 cpu [SSSSSSSDDD] ctx   80 int   48 bio 4.8M    0
18:09:24.107934 cpu [SSSSSSSDII] ctx   71 int   42 bio 4.2M    0
18:09:24.217946 cpu [SSSSDDDDDD] ctx   90 int   53 bio 5.2M    0
18:09:24.317965 cpu [SSSSSSDDDI] ctx   83 int   49 bio 4.7M    0
18:09:24.417974 cpu [SSSSSSSSDD] ctx   78 int   46 bio 4.6M    0
18:09:24.517938 cpu [SSSSDDDDDD] ctx   48 int   30 bio 3.7M    0
18:09:24.617940 cpu [SSSDDDDDDD] ctx   80 int   48 bio 4.7M    0
18:09:24.717948 cpu [SSSSSSSDDI] ctx   79 int   48 bio 4.7M    0
18:09:24.817955 cpu [SSSSSSSSDI] ctx   82 int   49 bio 4.7M    0
18:09:24.917914 cpu [SSSSDDDDDD] ctx   31 int   21 bio 2.7M    0
18:09:25.018051 cpu [SSSDDDDDDD] ctx   31 int   23 bio 2.7M    0
18:09:25.117988 cpu [SSSSSSDDII] ctx   81 int   49 bio 4.8M    0
18:09:25.217970 cpu [SSSSSDDDDD] ctx   37 int   24 bio 3.1M    0
18:09:25.318115 cpu [SSSSSSSSDD] ctx   81 int   48 bio 4.8M    0
18:09:25.417959 cpu [SSSSSSSSDD] ctx   81 int   47 bio 4.7M    0
18:09:25.517943 cpu [SSSSSSDDDD] ctx   65 int   41 bio 4.3M    0
18:09:25.617959 cpu [SSSSDDDDDD] ctx   80 int   48 bio 4.7M    0
18:09:25.717945 cpu [SSSSSSSDDD] ctx   81 int   47 bio 4.8M    0
18:09:25.817979 cpu [SSSSSSSSDI] ctx   79 int   49 bio 4.6M    0
18:09:25.917913 cpu [SSSSDDDDDD] ctx   30 int   21 bio 2.7M    0
18:09:26.017940 cpu [SSSSDDDDDD] ctx   64 int   39 bio 3.7M    0
18:09:26.118932 cpu [SSSSDDDDDI] ctx   79 int   49 bio 4.7M    0
18:09:26.217959 cpu [SSSSSDDDDD] ctx   43 int   26 bio 3.3M    0
18:09:26.317948 cpu [SSSSSDDDDD] ctx   64 int   38 bio 4.1M    0
18:09:26.417962 cpu [SSSSSSSSDD] ctx   78 int   47 bio 4.7M    0
18:09:26.517958 cpu [SSSSDDDDDD] ctx   27 int   20 bio 2.2M    0
18:09:26.617978 cpu [SSSDDDDDDD] ctx   24 int   18 bio 1.6M    0
18:09:26.717966 cpu [SSSSSDDDDI] ctx   76 int   45 bio 4.5M    0
18:09:26.817963 cpu [SSSSSSDDDI] ctx   62 int   37 bio 4.0M    0
18:09:26.917972 cpu [SSSSSDDDDD] ctx   69 int   40 bio 4.3M    0
18:09:27.017971 cpu [SSSSSSSDDD] ctx   78 int   47 bio 4.7M    0
18:09:27.117942 cpu [SDDDDDDDDD] ctx   40 int   25 bio 3.3M    0
18:09:27.217949 cpu [SSSSSSSSDI] ctx   69 int   42 bio 4.2M    0
18:09:27.318990 cpu [SSSSSSSSDI] ctx   84 int   48 bio 4.8M    0
18:09:27.417950 cpu [SSSSSSSSSI] ctx   79 int   47 bio 4.6M    0
18:09:27.518161 cpu [SSSSSSSSDI] ctx   79 int   47 bio 4.8M 8192
18:09:27.617941 cpu [SSSSSSSDDD] ctx   87 int   51 bio 4.7M  24k

On Sat, Sep 17, 2005 at 01:32:53PM +0300, Al Boldi wrote:
> Willy Tarreau wrote:
> > On Sat, Sep 17, 2005 at 07:26:11AM +0300, Al Boldi wrote:
> > > Monitoring disk access using gkrellm, I noticed that a command like
> > >
> > > cat /dev/hda > /dev/null
> > >
> > > shows eradic disk reads ranging from 0 to 80MB/s on an otherwise idle
> > > system.
> > >
> > > 1. Is this a hardware or software problem?
> >
> > Difficult to tell without more info. Can be a broken IDE disk or defective
> > ribbon.
>
> Tried the same with 2.4.31 which shows steady behaviour with occasional dips
> and pops in the msec range.
>
> >
> > > 2. Is there a lightweight perf-mon tool (cmd-line) that would log this
> > > behaviour graphically?
> >
> > You can do " readspeed </dev/hda | tr '\r' '\n' > log " with the readspeed
> > tool from there :
> >    http://w.ods.org/tools/readspeed
>
> Does it have msec resolution?

no, it does not. It will only show you the average transfer rate during
the past second each second. It often helps me tune network, nfs, raid, ...

Denis' tool seems clearly more suited to analyse your problem.

Regards,
Willy

Willy Tarreau wrote:
> On Sat, Sep 17, 2005 at 01:32:53PM +0300, Al Boldi wrote:
> > Willy Tarreau wrote:
> > > On Sat, Sep 17, 2005 at 07:26:11AM +0300, Al Boldi wrote:
> > > > Monitoring disk access using gkrellm, I noticed that a command like
> > > >
> > > > cat /dev/hda > /dev/null
> > > >
> > > > shows eradic disk reads ranging from 0 to 80MB/s on an otherwise
> > > > idle system.

New synonym: eradic=erratic :)

> Denis' tool seems clearly more suited to analyse your problem.

The problem seems to be a multi-access collision in the queue, which forces a
~50% reduction of thruput, which recovers with another multi-access
collision.  Maybe?!

--
Al

On Sun, Sep 18, 2005 at 07:16:46AM +0300, Al Boldi wrote:
> Willy Tarreau wrote:
> > On Sat, Sep 17, 2005 at 01:32:53PM +0300, Al Boldi wrote:
> > > Willy Tarreau wrote:
> > > > On Sat, Sep 17, 2005 at 07:26:11AM +0300, Al Boldi wrote:
> > > > > Monitoring disk access using gkrellm, I noticed that a command like
> > > > >
> > > > > cat /dev/hda > /dev/null
> > > > >
> > > > > shows eradic disk reads ranging from 0 to 80MB/s on an otherwise
> > > > > idle system.
>
> New synonym: eradic=erratic :)

I've noticed from the first mail too :-)

> > Denis' tool seems clearly more suited to analyse your problem.
>
> The problem seems to be a multi-access collision in the queue, which forces a
> ~50% reduction of thruput, which recovers with another multi-access
> collision.  Maybe?!

do you have anything else connected to the same controller ? I've encountered
a problem a long time ago with SATA drives (it was around 2.4.25, with the
beginning of SATA support), where I could not use RAID on my drives because
simultaneous accesses to the drives was not possible. I then realized that
every time I accessed a drive during a read from or a write to another one
would stall. I don't remember if CDROM access caused the same trouble. This
was with a SATA-patched ata-piix driver, but fortunately this is fixed by now.

Regards,
Willy

Willy Tarreau wrote:
> On Sun, Sep 18, 2005 at 07:16:46AM +0300, Al Boldi wrote:
> >
> > New synonym: eradic=erratic :)
> >
> > The problem seems to be a multi-access collision in the queue, which
> > forces a ~50% reduction of thruput, which recovers with another
> > multi-access collision.  Maybe?!
>
> do you have anything else connected to the same controller ?

Nothing, but hda.

This problem only shows in 2.6.13, 2.4.31 works ok!

Maybe this is also accociated to the block-dev read-ahead problem?

Thanks!

--
Al

> > > You can do " readspeed </dev/hda | tr '\r' '\n' > log " with the readspeed
> > > tool from there :
> > >    http://w.ods.org/tools/readspeed
> >
> > Does it have msec resolution?
>
> no, it does not. It will only show you the average transfer rate during
> the past second each second. It often helps me tune network, nfs, raid, ...
>
> Denis' tool seems clearly more suited to analyse your problem.

Denis' tool just got a homepage :)

http://195.66.192.167/linux/vdautil-0.5/src/
--
vda

On Saturday 17 September 2005 19:38, Al Boldi wrote:
> > > > > Monitoring disk access using gkrellm, I noticed that a command like
> > > > >
> > > > > cat /dev/hda > /dev/null
> > > > >
> > > > > shows eradic disk reads ranging from 0 to 80MB/s on an otherwise
> > > > > idle system.
> > > > >
> > > > > 1. Is this a hardware or software problem?
> > > >
> > > > Difficult to tell without more info. Can be a broken IDE disk or
> > > > defective ribbon.
> > >
> > > Tried the same with 2.4.31 which shows steady behaviour with occasional
> > > dips and pops in the msec range.
> > >
> > > > > 2. Is there a lightweight perf-mon tool (cmd-line) that would log
> > > > > this behaviour graphically?
> >
> > Try attached one.
>
> Nice!
>
> > # dd if=/dev/hda of=/dev/null bsM
>
> This is what I got; what do you get?
>
> Thanks!
>
> 2.4.31 # nmeter t6 c x i b d100
> 18:56:36.009981 cpu [SSSSSSSS..] ctx  145 int   86 bio 4.7M    0
> 18:56:36.110327 cpu [SSSSSSSS..] ctx  145 int   87 bio 4.8M    0
> 18:56:36.210735 cpu [SSSSSSSS..] ctx  139 int   88 bio 4.7M    0
> 18:56:36.310315 cpu [SSSSSSSS..] ctx  142 int   85 bio 4.7M    0
>
> 2.6.13 # nmeter t6 c x i b d100
> 18:09:22.117959 cpu [SSSSSSSSSD] ctx   80 int   47 bio 4.7M    0
> 18:09:22.217932 cpu [SSSSSSDDDD] ctx   83 int   48 bio 4.8M    0
> 18:09:22.319200 cpu [SSSSDDDDDI] ctx   81 int   56 bio 4.7M  28k
> 18:09:22.407979 cpu [SSSSSSSSSD] ctx   60 int   38 bio 3.8M    0
> 18:09:22.517960 cpu [SSSSSSSSDI] ctx   95 int   61 bio 5.2M  52k

Well, I do not see any bursts you mention...
Anyway, my data:

# uname -a
Linux firebird 2.6.12-r4 #1 SMP Sun Jul 17 13:51:47 EEST 2005 i686 unknown unknown GNU/Linux

hda:

13:52:05.401922 cpu [..........] ctx    4 int   64 bio    0    0
13:52:05.500900 cpu [..........] ctx    6 int  100 bio    0    0
13:52:05.600879 cpu [..........] ctx    2 int  102 bio    0    0
13:52:05.700857 cpu [..........] ctx    5 int  101 bio    0    0
13:52:05.800836 cpu [..........] ctx    6 int  103 bio    0    0
13:52:05.900815 cpu [..........] ctx    2 int  100 bio    0    0
13:52:06.000795 cpu [..........] ctx    2 int  102 bio    0    0
13:52:06.100776 cpu [..........] ctx    3 int  100 bio    0    0
13:52:06.200752 cpu [..........] ctx    2 int  100 bio    0    0
13:52:06.300731 cpu [..........] ctx    2 int  100 bio    0    0
13:52:06.401724 cpu [SSDDDD....] ctx   36 int  133 bio 857k    0
13:52:06.501749 cpu [SSSSSSDDDI] ctx   44 int  134 bio 3.2M    0
13:52:06.601721 cpu [SSSSSSDDII] ctx   38 int  161 bio 4.0M    0
13:52:06.701674 cpu [SSSSSSDDDI] ctx   43 int  141 bio 3.7M    0
13:52:06.802877 cpu [SSSSSSDDDI] ctx   33 int  148 bio 3.6M    0
13:52:06.901690 cpu [SSSSSSSDDI] ctx   40 int  140 bio 3.5M    0
13:52:07.001663 cpu [SSSSSSDDDI] ctx   37 int  139 bio 4.1M    0
13:52:07.102638 cpu [SSSSSSUDDI] ctx   38 int  150 bio 4.1M    0
13:52:07.201653 cpu [SSSSSSSDDI] ctx   45 int  148 bio 4.1M    0
13:52:07.301605 cpu [SSSSSSSDII] ctx   38 int  142 bio 4.0M    0
13:52:07.401556 cpu [SSSSSSDDDI] ctx   42 int  143 bio 4.0M    0
13:52:07.501513 cpu [SSSSSSSDDI] ctx   35 int  145 bio 4.1M    0
13:52:07.601485 cpu [SSSSSSSDDI] ctx   29 int  142 bio 3.6M    0
13:52:07.701839 cpu [SSSSSSSSDI] ctx   30 int  141 bio 4.0M    0
13:52:07.801519 cpu [SSSSSSDDII] ctx   34 int  147 bio 4.0M    0
13:52:07.901527 cpu [SSSSSSSDDI] ctx   32 int  141 bio 4.1M    0
13:52:08.001448 cpu [SSSSSSSDDI] ctx   53 int  150 bio 3.7M    0
13:52:08.101378 cpu [SSSSSSDDDI] ctx   47 int  138 bio 3.7M    0
13:52:08.201395 cpu [SSSSSSSSDI] ctx   44 int  138 bio 3.8M    0
13:52:08.301555 cpu [SSSSSDDDII] ctx   33 int  136 bio 3.3M    0
13:52:08.402275 cpu [SSSSSSSDDD] ctx   43 int  150 bio 3.7M  16k
13:52:08.501617 cpu [SSSSSSSSDI] ctx   35 int  142 bio 4.0M    0
13:52:08.601614 cpu [SSSSSSSDDI] ctx   46 int  148 bio 4.0M    0
13:52:08.701250 cpu [SSSSSDDDDI] ctx   36 int  129 bio 2.7M    0
13:52:08.801316 cpu [SSSSSSSDDI] ctx   41 int  154 bio 3.8M    0
13:52:08.902231 cpu [SSSSSSSDDI] ctx   42 int  140 bio 3.8M    0
13:52:09.001419 cpu [SSSSSSSDDI] ctx   36 int  142 bio 4.1M    0
13:52:09.102196 cpu [SSSSSSSSDI] ctx   43 int  140 bio 3.8M    0
13:52:09.201450 cpu [SSSSSSDDII] ctx   34 int  142 bio 4.1M    0
13:52:09.301176 cpu [SSSSSSSDDI] ctx   39 int  138 bio 3.8M    0
13:52:09.401806 cpu [SSSSSSSSDI] ctx   55 int  157 bio 4.3M    0
13:52:09.501077 cpu [SSSSDDDDDI] ctx   32 int  121 bio 2.2M    0
13:52:09.601067 cpu [SSSDDDDDDD] ctx   24 int  127 bio 1.7M    0
13:52:09.701108 cpu [SSSSSSSDDI] ctx   50 int  153 bio 4.1M    0
13:52:09.802156 cpu [SSSSDDDDII] ctx   34 int  139 bio 2.7M    0
13:52:09.902026 cpu [SSSSSSSSDD] ctx   44 int  152 bio 4.0M    0
13:52:10.002041 cpu [SSSSDDDDDI] ctx   39 int  126 bio 2.2M    0
13:52:10.100997 cpu [SSSSDDDDDI] ctx   46 int  129 bio 2.7M    0
13:52:10.201964 cpu [SSSSSSSDDI] ctx   57 int  139 bio 3.6M    0
13:52:10.303468 cpu [SSSSDDDDDI] ctx   34 int  133 bio 2.2M    0
13:52:10.401523 cpu [SSSSSSSDDI] ctx   50 int  145 bio 3.6M 4096
13:52:10.500934 cpu [SSSSSSSSII] ctx   66 int  139 bio 4.1M    0
13:52:10.601854 cpu [SSSSSSSSDI] ctx   59 int  147 bio 4.0M    0
13:52:10.701938 cpu [SSSSSSSDDI] ctx   46 int  151 bio 3.7M    0
13:52:10.801827 cpu [SSSSSSSDDI] ctx   47 int  153 bio 4.0M    0
13:52:10.900829 cpu [SSSSSSSDII] ctx   58 int  142 bio 4.1M 100k
13:52:11.000810 cpu [SSSSSSSDII] ctx   62 int  153 bio 4.0M    0
13:52:11.100836 cpu [SSSSSSSSDI] ctx   57 int  143 bio 4.1M    0
13:52:11.201740 cpu [SSSSSSSDII] ctx   54 int  154 bio 4.1M    0
13:52:11.300827 cpu [SSSSSSSSDI] ctx   66 int  138 bio 4.1M    0
13:52:11.402139 cpu [SSSSSSSDII] ctx   62 int  149 bio 4.0M 4096
13:52:11.501748 cpu [SSSSSSSSDI] ctx   55 int  142 bio 4.1M    0
13:52:11.601638 cpu [SSSSSSSSDI] ctx   58 int  144 bio 4.0M    0
13:52:11.701665 cpu [SSSSSSSDII] ctx   59 int  143 bio 4.1M    0
13:52:11.801702 cpu [SSSSSSSSDI] ctx   57 int  142 bio 4.1M    0
13:52:11.900620 cpu [SSSSSSDDII] ctx   60 int  147 bio 3.9M  76k
13:52:12.001602 cpu [SSSSSSSSDI] ctx   59 int  141 bio 4.1M    0
13:52:12.102003 cpu [SSSSSSSSDI] ctx   60 int  141 bio 4.1M    0
13:52:12.202031 cpu [SSSSSSSDII] ctx   46 int  154 bio 4.1M    0
13:52:12.300878 cpu [SSSSSSSSDI] ctx   56 int  138 bio 4.0M    0
13:52:12.401747 cpu [SSSSSSSDDI] ctx   56 int  149 bio 4.1M    0
13:52:12.501482 cpu [SSSSSSSSDI] ctx   62 int  140 bio 4.0M    0
13:52:12.600455 cpu [SSSSSSSDII] ctx   57 int  151 bio 4.0M    0
13:52:12.700400 cpu [SSSSSUDDDI] ctx   35 int  138 bio 2.6M    0
13:52:12.801719 cpu [SSSSSSSDDI] ctx   48 int  140 bio 3.7M    0
13:52:12.901651 cpu [SSSSSSSSDI] ctx   50 int  148 bio 4.0M    0
13:52:13.001352 cpu [SSSSSSDDII] ctx   43 int  138 bio 4.1M    0
13:52:13.103070 cpu [SSSSSSSDDI] ctx   47 int  151 bio 4.0M    0
13:52:13.201283 cpu [SSSSSSD...] ctx   41 int  123 bio 2.3M    0
13:52:13.301254 cpu [..........] ctx    2 int  101 bio    0    0
13:52:13.401230 cpu [..........] ctx    2 int  100 bio    0    0
13:52:13.501211 cpu [..........] ctx    4 int  100 bio    0    0
13:52:13.601189 cpu [..........] ctx    2 int  102 bio    0    0
13:52:13.701168 cpu [..........] ctx    2 int  100 bio    0    0
13:52:13.801146 cpu [..........] ctx    2 int  101 bio    0    0
13:52:13.901125 cpu [..........] ctx    2 int  100 bio    0    0

/dev/hda:

 Model=SAMSUNG SV0411N, FwRev=UA100-07, SerialNo=S01RJ10X122455
 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
 RawCHS383/16/63, TrkSize4902, SectSizeU4, ECCbytes=4
 BuffType=DualPortCache, BuffSize 48kB, MaxMultSect, MultSect
 CurCHS383/16/63, CurSects514064, LBA=yes, LBAsectsx242976
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: (null):

 * signifies the current active mode

Note: CPU is mostly busy in system (S), not in disk wait (D).
My CPU is not that new:

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 11
model name      : Intel(R) Celeron(TM) CPU                1200MHz
stepping        : 1
cpu MHz         : 1196.195
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips        : 2359.29


hdc (an old disk):

13:52:38.701903 cpu [..........] ctx    3 int  101 bio    0    0
13:52:38.801869 cpu [..........] ctx    2 int  100 bio    0    0
13:52:38.901849 cpu [..........] ctx    2 int  101 bio    0    0
13:52:39.001830 cpu [..........] ctx    4 int  101 bio    0    0
13:52:39.101107 cpu [SUDDDDDDI.] ctx   24 int  108 bio 1.0M    0
13:52:39.202024 cpu [SSDDDDDDDD] ctx   21 int  111 bio 1.2M    0
13:52:39.301787 cpu [SSDDDDDDDD] ctx   21 int  109 bio 1.2M    0
13:52:39.401764 cpu [SSDDDDDDDI] ctx   27 int  112 bio 1.2M    0
13:52:39.501744 cpu [SSDDDDDDDD] ctx   22 int  110 bio 1.2M    0
13:52:39.601725 cpu [SSDDDDDDDI] ctx   24 int  111 bio 1.2M    0
13:52:39.701702 cpu [SSDDDDDDDD] ctx   27 int  110 bio 1.2M    0
13:52:39.801677 cpu [SSDDDDDDDD] ctx   28 int  111 bio 1.2M    0
13:52:39.901909 cpu [SSDDDDDDDI] ctx   26 int  110 bio 1.2M    0
13:52:40.001772 cpu [SSDDDDDDDD] ctx   25 int  110 bio 1.1M    0
13:52:40.100723 cpu [SSDDDDDDDI] ctx   25 int  111 bio 1.2M    0
13:52:40.201718 cpu [SSSDDDDDDD] ctx   26 int  112 bio 1.2M    0
13:52:40.301569 cpu [SSDDDDDDDD] ctx   28 int  109 bio 1.2M    0
13:52:40.402552 cpu [SSDDDDDDDI] ctx   23 int  111 bio 1.2M    0
13:52:40.502535 cpu [SSDDDDDDDD] ctx   22 int  110 bio 1.2M    0
13:52:40.601507 cpu [SSDDDDDDDI] ctx   26 int  111 bio 1.2M    0
13:52:40.701493 cpu [SSDDDDDDDD] ctx   23 int  110 bio 1.2M    0
13:52:40.801470 cpu [SSDDDDDDDI] ctx   24 int  110 bio 1.2M    0
13:52:40.901451 cpu [SSDDDDDDDD] ctx   23 int  111 bio 1.2M    0
13:52:41.001448 cpu [SSDDDDDDDI] ctx   26 int  110 bio 1.2M    0
13:52:41.101449 cpu [SSDDDDDDDD] ctx   24 int  111 bio 1.1M    0
13:52:41.201394 cpu [SSDDDDDDDD] ctx   24 int  110 bio 1.2M    0
13:52:41.302404 cpu [SSDDDDDDDI] ctx   23 int  112 bio 1.2M    0
13:52:41.402345 cpu [SSDDDDDDDD] ctx   23 int  109 bio 1.2M    0
13:52:41.501318 cpu [SSDDDDDDDI] ctx   23 int  109 bio 1.2M    0
13:52:41.602298 cpu [SSDDDDDDDD] ctx   26 int  115 bio 1.2M    0
13:52:41.702281 cpu [SSDDDDDDDI] ctx   23 int  112 bio 1.2M    0
13:52:41.802260 cpu [SSDDDDDDDD] ctx   24 int  110 bio 1.2M    0
13:52:41.902239 cpu [SSSDDDDDDI] ctx   35 int  116 bio 1.2M 124k
13:52:42.002220 cpu [SSDDDDDDDD] ctx   24 int  132 bio 1.2M  92k
13:52:42.102199 cpu [SSDDDDDDDI] ctx   25 int  113 bio 1.2M    0
13:52:42.203194 cpu [SSDDDDDDDD] ctx   23 int  111 bio 1.2M    0
13:52:42.302944 cpu [SSDDDDDDDD] ctx   22 int  111 bio 1.2M    0
13:52:42.402144 cpu [SSDDDDDDDI] ctx   23 int  109 bio 1.1M    0
13:52:42.502114 cpu [SSDDDDDDDD] ctx   22 int  110 bio 1.2M    0
13:52:42.601088 cpu [SSDDDDDDDI] ctx   27 int  110 bio 1.2M    0
13:52:42.702070 cpu [SSDDDDDDDD] ctx   23 int  113 bio 1.2M    0
13:52:42.802048 cpu [SSSUDDDDDD] ctx   23 int  110 bio 1.2M    0
13:52:42.902029 cpu [SSDDDDDDDI] ctx   23 int  111 bio 1.2M    0
13:52:43.002007 cpu [SSDDDDDDDD] ctx   25 int  111 bio 1.2M    0
13:52:43.101981 cpu [SSDDDDDDDD] ctx   25 int  111 bio 1.2M    0
13:52:43.201966 cpu [SSDDDDDDDI] ctx   23 int  111 bio 1.2M    0
13:52:43.301945 cpu [SSDDDDDDDD] ctx   24 int  110 bio 1.2M    0
13:52:43.402606 cpu [SSDDDDDDDI] ctx   27 int  111 bio 1.2M    0
13:52:43.501933 cpu [SSDDDDDDDD] ctx   23 int  109 bio 1.1M    0
13:52:43.601882 cpu [SSDDDDDDDI] ctx   24 int  112 bio 1.2M    0
13:52:43.701860 cpu [SSDDDDDDDD] ctx   23 int  112 bio 1.2M    0
13:52:43.801839 cpu [SSDDDDDDDI] ctx   23 int  110 bio 1.2M    0
13:52:43.901820 cpu [SSDDDDDDDD] ctx   24 int  111 bio 1.2M    0
13:52:44.001792 cpu [SSDDDDDDDD] ctx   24 int  111 bio 1.2M    0
13:52:44.101809 cpu [SSSDDDDDDI] ctx   23 int  110 bio 996k    0
13:52:44.201740 cpu [SSSSSSSS..] ctx   12 int  100 bio    0    0
13:52:44.301706 cpu [..........] ctx    2 int  100 bio    0    0
13:52:44.402686 cpu [..........] ctx    2 int  102 bio    0    0
13:52:44.502664 cpu [..........] ctx    2 int  100 bio    0    0
13:52:44.602643 cpu [..........] ctx    4 int  100 bio    0    0
13:52:44.702621 cpu [..........] ctx    4 int  104 bio    0    0

/dev/hdc:

 Model=Maxtor 90640D4, FwRev=PAS24F15, SerialNo=V41Y3ABA
 Config={ Fixed }
 RawCHS328/15/63, TrkSize=0, SectSize=0, ECCbytes)
 BuffType=DualPortCache, BuffSize%6kB, MaxMultSect, MultSect
 CurCHS328/15/63, CurSects594960, LBA=yes, LBAsects594960
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 *udma2
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: ATA/ATAPI-4 T13 1153D revision 17:  1 2 3 4

 * signifies the current active mode
--
vda

Denis Vlasenko wrote:
> On Saturday 17 September 2005 19:38, Al Boldi wrote:
> > > # dd if=/dev/hda of=/dev/null bsM
> > 2.4.31 # nmeter t6 c x i b d100
> > 18:56:36.009981 cpu [SSSSSSSS..] ctx  145 int   86 bio 4.7M    0
> > 18:56:36.110327 cpu [SSSSSSSS..] ctx  145 int   87 bio 4.8M    0
> > 18:56:36.210735 cpu [SSSSSSSS..] ctx  139 int   88 bio 4.7M    0
> > 18:56:36.310315 cpu [SSSSSSSS..] ctx  142 int   85 bio 4.7M    0
> >
> > 2.6.13 # nmeter t6 c x i b d100
> > 18:09:22.117959 cpu [SSSSSSSSSD] ctx   80 int   47 bio 4.7M    0
> > 18:09:22.217932 cpu [SSSSSSDDDD] ctx   83 int   48 bio 4.8M    0
> > 18:09:22.319200 cpu [SSSSDDDDDI] ctx   81 int   56 bio 4.7M  28k
> > 18:09:22.407979 cpu [SSSSSSSSSD] ctx   60 int   38 bio 3.8M    0
> > 18:09:22.517960 cpu [SSSSSSSSDI] ctx   95 int   61 bio 5.2M  52k
>
> Well, I do not see any bursts you mention...

Can you try 2.4.31?

> Anyway, my data:
> 13:52:09.201450 cpu [SSSSSSDDII] ctx   34 int  142 bio 4.1M    0
> 13:52:09.301176 cpu [SSSSSSSDDI] ctx   39 int  138 bio 3.8M    0
> 13:52:09.401806 cpu [SSSSSSSSDI] ctx   55 int  157 bio 4.3M    0
> 13:52:09.501077 cpu [SSSSDDDDDI] ctx   32 int  121 bio 2.2M    0
> 13:52:09.601067 cpu [SSSDDDDDDD] ctx   24 int  127 bio 1.7M    0
> 13:52:09.701108 cpu [SSSSSSSDDI] ctx   50 int  153 bio 4.1M    0
> 13:52:09.802156 cpu [SSSSDDDDII] ctx   34 int  139 bio 2.7M    0
> 13:52:09.902026 cpu [SSSSSSSSDD] ctx   44 int  152 bio 4.0M    0
> 13:52:10.002041 cpu [SSSSDDDDDI] ctx   39 int  126 bio 2.2M    0
> 13:52:10.100997 cpu [SSSSDDDDDI] ctx   46 int  129 bio 2.7M    0
> 13:52:10.201964 cpu [SSSSSSSDDI] ctx   57 int  139 bio 3.6M    0
> 13:52:10.303468 cpu [SSSSDDDDDI] ctx   34 int  133 bio 2.2M    0
> 13:52:10.401523 cpu [SSSSSSSDDI] ctx   50 int  145 bio 3.6M 4096
> 13:52:10.500934 cpu [SSSSSSSSII] ctx   66 int  139 bio 4.1M    0

Does not look smooth.

> /dev/hda:
>  UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5

same here.

> My CPU is not that new:

PII - 400Mhz here.

> hdc (an old disk):
> 13:52:40.201718 cpu [SSSDDDDDDD] ctx   26 int  112 bio 1.2M    0
> 13:52:40.301569 cpu [SSDDDDDDDD] ctx   28 int  109 bio 1.2M    0
> 13:52:40.402552 cpu [SSDDDDDDDI] ctx   23 int  111 bio 1.2M    0
> 13:52:40.502535 cpu [SSDDDDDDDD] ctx   22 int  110 bio 1.2M    0
> 13:52:40.601507 cpu [SSDDDDDDDI] ctx   26 int  111 bio 1.2M    0
> 13:52:40.701493 cpu [SSDDDDDDDD] ctx   23 int  110 bio 1.2M    0

Looks smooth.

Also, great meter!  Best of all does not hog the CPU!
Could you add a top3 procs display?

Thanks!

--
Al

On Sun, Sep 18, 2005 at 07:02:17PM +0300, Denis Vlasenko wrote:
> > > My CPU is not that new:
> >
> > PII - 400Mhz here.
>
> I meant that kernel seem to eat too much CPU here. This
> is not expected. I expected CPU bar to be all D.

This is often caused by disks running in PIO instead of DMA.

> > Also, great meter!  Best of all does not hog the CPU!
> > Could you add a top3 procs display?
>
> What is a "top3 procs display"?

probably something which will turn your tool into sort of a complex and
unusable one when another session running 'top' could do the trick.

Oh, BTW, the first reason I wrote my tool was to avoid copying into
/dev/null which consumes a small amount of CPU. Thus, I made it a pure
data eater. It might be interesting to run it instead of 'dd' while your
tool is running, to see if system usage decreases a bit.

Regards,
Willy

> > > > # dd if=/dev/hda of=/dev/null bsM
> > > 2.4.31 # nmeter t6 c x i b d100
> > > 18:56:36.009981 cpu [SSSSSSSS..] ctx  145 int   86 bio 4.7M    0
> > > 18:56:36.110327 cpu [SSSSSSSS..] ctx  145 int   87 bio 4.8M    0
> > > 18:56:36.210735 cpu [SSSSSSSS..] ctx  139 int   88 bio 4.7M    0
> > > 18:56:36.310315 cpu [SSSSSSSS..] ctx  142 int   85 bio 4.7M    0
> > >
> > > 2.6.13 # nmeter t6 c x i b d100
> > > 18:09:22.117959 cpu [SSSSSSSSSD] ctx   80 int   47 bio 4.7M    0
> > > 18:09:22.217932 cpu [SSSSSSDDDD] ctx   83 int   48 bio 4.8M    0
> > > 18:09:22.319200 cpu [SSSSDDDDDI] ctx   81 int   56 bio 4.7M  28k
> > > 18:09:22.407979 cpu [SSSSSSSSSD] ctx   60 int   38 bio 3.8M    0
> > > 18:09:22.517960 cpu [SSSSSSSSDI] ctx   95 int   61 bio 5.2M  52k
> >
> > Well, I do not see any bursts you mention...
>
> Can you try 2.4.31?

Not easy. I guess I have quite a few 2.6-only things here...

> same here.
>
> > My CPU is not that new:
>
> PII - 400Mhz here.

I meant that kernel seem to eat too much CPU here. This
is not expected. I expected CPU bar to be all D.

> > hdc (an old disk):
> > 13:52:40.201718 cpu [SSSDDDDDDD] ctx   26 int  112 bio 1.2M    0
> > 13:52:40.301569 cpu [SSDDDDDDDD] ctx   28 int  109 bio 1.2M    0
> > 13:52:40.402552 cpu [SSDDDDDDDI] ctx   23 int  111 bio 1.2M    0
> > 13:52:40.502535 cpu [SSDDDDDDDD] ctx   22 int  110 bio 1.2M    0
> > 13:52:40.601507 cpu [SSDDDDDDDI] ctx   26 int  111 bio 1.2M    0
> > 13:52:40.701493 cpu [SSDDDDDDDD] ctx   23 int  110 bio 1.2M    0
>
> Looks smooth.
>
> Also, great meter!  Best of all does not hog the CPU!
> Could you add a top3 procs display?

What is a "top3 procs display"?
--
vda

Willy Tarreau wrote:
> On Sun, Sep 18, 2005 at 07:02:17PM +0300, Denis Vlasenko wrote:
> > > Also, great meter!  Best of all does not hog the CPU!
> > > Could you add a top3 procs display?
> >
> > What is a "top3 procs display"?

top 3 procs that eat most of the CPU.

> probably something which will turn your tool into sort of a complex and
> unusable one when another session running 'top' could do the trick.

Top is super-expensive.

> Oh, BTW, the first reason I wrote my tool was to avoid copying into
> /dev/null which consumes a small amount of CPU. Thus, I made it a pure
> data eater. It might be interesting to run it instead of 'dd' while your
> tool is running, to see if system usage decreases a bit.

Good idea!

Thanks!

--
Al