use vmstat command to know about disk access

I love to use vmstat command to measure load of servers. But, to check disk-intensive overload, usual vmstat output is not device-specific.

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 25924 11324 3524 135644 0 0 3 20 1 4 0 0 100 0 0
0 1 25924 11324 3528 135640 0 0 0 6 19 33 0 0 100 0 0
0 0 25924 11324 3532 135644 0 0 0 1 18 34 0 0 100 0 0


So, I looked into /proc/diskstats , but it is cryptic.

$ cat /proc/diskstats
1 0 ram0 0 0 0 0 0 0 0 0 0 0 0
1 1 ram1 0 0 0 0 0 0 0 0 0 0 0
1 2 ram2 0 0 0 0 0 0 0 0 0 0 0
1 3 ram3 0 0 0 0 0 0 0 0 0 0 0
1 4 ram4 0 0 0 0 0 0 0 0 0 0 0
1 5 ram5 0 0 0 0 0 0 0 0 0 0 0
1 6 ram6 0 0 0 0 0 0 0 0 0 0 0
1 7 ram7 0 0 0 0 0 0 0 0 0 0 0
1 8 ram8 0 0 0 0 0 0 0 0 0 0 0
1 9 ram9 0 0 0 0 0 0 0 0 0 0 0
1 10 ram10 0 0 0 0 0 0 0 0 0 0 0
1 11 ram11 0 0 0 0 0 0 0 0 0 0 0
1 12 ram12 0 0 0 0 0 0 0 0 0 0 0
1 13 ram13 0 0 0 0 0 0 0 0 0 0 0
1 14 ram14 0 0 0 0 0 0 0 0 0 0 0
1 15 ram15 0 0 0 0 0 0 0 0 0 0 0
202 0 xvda 1900702 111759 57624068 22798740 21260401 21742422 344022684 149429332 0 25033876 172228060
202 1 xvda1 1178 2366 2 4
202 2 xvda2 2011295 57621462 43002835 344022680
253 0 dm-0 2002226 0 57550986 23818188 42992837 0 343942696 261736148 0 25033368 285554312
253 1 dm-1 8752 0 70016 102772 9998 0 79984 347408 0 16320 450180
9 0 md0 0 0 0 0 0 0 0 0 0 0 0

Then, I come back to vmstat with "-d" option. Just to see formatted information of /proc/diskstats.

$ vmstat -d
disk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec
ram0 0 0 0 0 0 0 0 0 0 0
ram1 0 0 0 0 0 0 0 0 0 0
ram2 0 0 0 0 0 0 0 0 0 0
ram3 0 0 0 0 0 0 0 0 0 0
ram4 0 0 0 0 0 0 0 0 0 0
ram5 0 0 0 0 0 0 0 0 0 0
ram6 0 0 0 0 0 0 0 0 0 0
ram7 0 0 0 0 0 0 0 0 0 0
ram8 0 0 0 0 0 0 0 0 0 0
ram9 0 0 0 0 0 0 0 0 0 0
ram10 0 0 0 0 0 0 0 0 0 0
ram11 0 0 0 0 0 0 0 0 0 0
ram12 0 0 0 0 0 0 0 0 0 0
ram13 0 0 0 0 0 0 0 0 0 0
ram14 0 0 0 0 0 0 0 0 0 0
ram15 0 0 0 0 0 0 0 0 0 0
xvda 1900690 111759 57623940 22798620 21260398 21742409 344022556 149429324 0 25033
dm-0 2002214 0 57550858 23818068 42992821 0 343942568 261736088 0 25033
dm-1 8752 0 70016 102772 9998 0 79984 347408 0 16
md0 0 0 0 0 0 0 0 0 0 0

Why linux filesystems (ext2, ext3) get slow when >1K files in a directory

I know that, but don't know in detail.
(this article is English version of my Japanese blog)

Especially, ls command is very slow, so I thought readdir system call is too slow. But, I was WRONG.
Overhead is in ls command's algorithm.
Mailing List article in linux-users ( http://his.luky.org/ML/linux-users.6/msg08919.html ) taught me that.
ls command collects information not only the list of filenames but also each file's attributes (size, permission.. etc.).
So, ls command checks attributes for each files. That takes long time.

In my application, only the list of filenames is required. So, readdir system call just works fine.
Here is the sample code (almost the same as manpage of readdir!)

#include <dirent.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[])
{
  DIR *dirp;
  struct dirent *dp;

  if (argc != 2 ) {
      printf("coundn't open dir\n");
      return;
  }


  if ( (dirp = opendir(argv[1]) ) == NULL) {
      printf("coundn't open dir\n");
      return;
  }

  do {
      errno = 0;
      if ( (dp = readdir(dirp) ) != NULL) {
          (void) printf("%s\n", dp->d_name);
      }
  } while (dp != NULL);

  if (errno != 0)
      perror("error reading directory ");

  (void) closedir(dirp);
  return(0);
}

Now, I checked the performance of readdir itself.
In the case of over 10K files in a directory, it takes 300msec (Kernel 2.6.9-42.ELsmp: Cent OS4.4 4800 bogomips)
Here is the result of 'strace -c'

# strace -c readdir . > /dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
80.44    0.442220          14     30821           getdents64
19.48    0.107076           8     13018           write
0.03    0.000149         149         1           execve
0.01    0.000054          11         5           old_mmap
0.01    0.000038          13         3           open
0.01    0.000035          35         1           read
0.01    0.000031           8         4           fstat64
0.01    0.000030           8         4           brk
0.00    0.000022          11         2           mprotect
0.00    0.000014           5         3           close
0.00    0.000013          13         1           munmap
0.00    0.000011          11         1         1 access
0.00    0.000008           8         1           mmap2
0.00    0.000008           8         1           fcntl64
0.00    0.000007           7         1         1 ioctl
0.00    0.000007           7         1           uname
0.00    0.000003           3         1           set_thread_area
------ ----------- ----------- --------- --------- ----------------
100.00    0.549726                 43869         2 total

The most time-consuming system-call is getdents64. System(Disk) cache speed up the systemcall.
If cache is full-hit, getdents64 takes only 10usecs for 1M files in a directory.

If you tried this on NFS-mounted directory, cache-effect maybe small.