4.6. Red Hat Enterprise Linux-Specific Information

Due to the inherent complexity of being a demand-paged virtual memory operating system, monitoring memory-related resources under Red Hat Enterprise Linux can be confusing. Therefore, it is best to start with the more straightforward tools, and work from there.

Using free, it is possible to get a concise (if somewhat simplistic) overview of memory and swap utilization. Here is an example:

             total       used       free     shared    buffers     cached
Mem:       1288720     361448     927272          0      27844     187632
-/+ buffers/cache:     145972    1142748
Swap:       522104          0     522104
      

We note that this system has 1.2GB of RAM, of which only about 350MB is actually in use. As expected for a system with this much free RAM, none of the 500MB swap partition is in use.

Contrast that example with this one:

             total       used       free     shared    buffers     cached
Mem:        255088     246604       8484          0       6492     111320
-/+ buffers/cache:     128792     126296
Swap:       530136     111308     418828
      

This system has about 256MB of RAM, the majority of which is in use, leaving only about 8MB free. Over 100MB of the 512MB swap partition is in use. Although this system is certainly more limited in terms of memory than the first system, to determine if this memory limitation is causing performance problems we must dig a bit deeper.

Although more cryptic than free, vmstat has the benefit of displaying more than memory utilization statistics. Here is the output from vmstat 1 10:

   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 2  0  0 111304   9728   7036 107204   0   0     6    10  120    24  10   2  89
 2  0  0 111304   9728   7036 107204   0   0     0     0  526  1653  96   4   0
 1  0  0 111304   9616   7036 107204   0   0     0     0  552  2219  94   5   1
 1  0  0 111304   9616   7036 107204   0   0     0     0  624   699  98   2   0
 2  0  0 111304   9616   7052 107204   0   0     0    48  603  1466  95   5   0
 3  0  0 111304   9620   7052 107204   0   0     0     0  768   932  90   4   6
 3  0  0 111304   9440   7076 107360  92   0   244     0  820  1230  85   9   6
 2  0  0 111304   9276   7076 107368   0   0     0     0  832  1060  87   6   7
 3  0  0 111304   9624   7092 107372   0   0    16     0  813  1655  93   5   2
 2  0  2 111304   9624   7108 107372   0   0     0   972 1189  1165  68   9  23
      

During this 10-second sample, the amount of free memory (the free field) varies somewhat, and there is a bit of swap-related I/O (the si and so fields), but overall this system is running well. It is doubtful, however, how much additional workload it could handle, given the current memory utilization.

When researching memory-related issues, it is often necessary to determine how the Red Hat Enterprise Linux virtual memory subsystem is making use of system memory. By using sar, it is possible to examine this aspect of system performance in much more detail.

By reviewing the sar -r report, we can examine memory and swap utilization more closely:

Linux 2.4.20-1.1931.2.231.2.10.ent (pigdog.example.com)      07/22/2003

12:00:01 AM kbmemfree kbmemused  %memused kbmemshrd kbbuffers  kbcached
12:10:00 AM    240468   1048252     81.34         0    133724    485772
12:20:00 AM    240508   1048212     81.34         0    134172    485600
…
08:40:00 PM    934132    354588     27.51         0     26080    185364
Average:       324346    964374     74.83         0     96072    467559
      

The kbmemfree and kbmemused fields show the typical free and used memory statistics, with the percentage of memory used displayed in the %memused field. The kbbuffers and kbcached fields show how many kilobytes of memory are allocated to buffers and the system-wide data cache.

The kbmemshrd field is always zero for systems (such as Red Hat Enterprise Linux) using the 2.4 Linux kernel.

The lines for this report have been truncated to fit on the page. Here is the remainder of each line, with the timestamp added to the left to make reading easier:

12:00:01 AM   kbswpfree kbswpused  %swpused
12:10:00 AM      522104         0      0.00
12:20:00 AM      522104         0      0.00
…
08:40:00 PM      522104         0      0.00
Average:         522104         0      0.00
      

For swap utilization, the kbswpfree and kbswpused fields show the amount of free and used swap space, in kilobytes, with the %swpused field showing the swap space used as a percentage.

To learn more about the swapping activity taking place, use the sar -W report. Here is an example:

Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com)      07/22/2003

12:00:01 AM  pswpin/s pswpout/s
12:10:01 AM      0.15      2.56
12:20:00 AM      0.00      0.00
…
03:30:01 PM      0.42      2.56
Average:         0.11      0.37
      

Here we notice that, on average, there were three times fewer pages being brought in from swap (pswpin/s) as there were going out to swap (pswpout/s).

To better understand how pages are being used, refer to the sar -B report:

Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com)      07/22/2003

12:00:01 AM  pgpgin/s pgpgout/s  activepg  inadtypg  inaclnpg  inatarpg
12:10:00 AM      0.03      8.61    195393     20654     30352     49279
12:20:00 AM      0.01      7.51    195385     20655     30336     49275
…
08:40:00 PM      0.00      7.79     71236      1371      6760     15873
Average:       201.54    201.54    169367     18999     35146     44702
      

Here we can determine how many blocks per second are paged in from disk (pgpgin/s) and paged out to disk (pgpgout/s). These statistics serve as a barometer of overall virtual memory activity.

However, more knowledge can be gained by examining the other fields in this report. The Red Hat Enterprise Linux kernel marks all pages as either active or inactive. As the names imply, active pages are currently in use in some manner (as process or buffer pages, for example), while inactive pages are not. This example report shows that the list of active pages (the activepg field) averages approximately 660MB[1].

The remainder of the fields in this report concentrate on the inactive list — pages that, for one reason or another, have not recently been used. The inadtypg field shows how many inactive pages are dirty (modified) and may need to be written to disk. The inaclnpg field, on the other hand, shows how many inactive pages are clean (unmodified) and do not need to be written to disk.

The inatarpg field represents the desired size of the inactive list. This value is calculated by the Linux kernel and is sized such that the inactive list remains large enough to act as a pool for page replacement purposes.

For additional insight into page status (specifically, how often pages change status), use the sar -R report. Here is a sample report:

Linux 2.4.20-1.1931.2.231.2.10.entsmp (raptor.example.com)      07/22/2003

12:00:01 AM   frmpg/s   shmpg/s   bufpg/s   campg/s
12:10:00 AM     -0.10      0.00      0.12     -0.07
12:20:00 AM      0.02      0.00      0.19     -0.07
…
08:50:01 PM     -3.19      0.00      0.46      0.81
Average:         0.01      0.00     -0.00     -0.00
      

The statistics in this particular sar report are unique, in that they may be positive, negative, or zero. When positive, the value indicates the rate at which pages of this type are increasing. When negative, the value indicates the rate at which pages of this type are decreasing. A value of zero indicates that pages of this type are neither increasing or decreasing.

In this example, the last sample shows slightly over three pages per second being allocated from the list of free pages (the frmpg/s field) and nearly 1 page per second added to the page cache (the campg/s field). The list of pages used as buffers (the bufpg/s field) gained approximately one page every two seconds, while the shared memory page list (the shmpg/s field) neither gained nor lost any pages.