Archive for July, 2009

10
Jul

Linux memory management

Yesterday I had a request for memory usage report on Oracle servers in my company. As we are using Centreon, Nagios frontend which makes good use of performance data reported by Nagios plugins and makes nice graphs out of it, it was a matter of pasting the images into the mail and sending it. But than interesting question was raised: how come on a server with 32GB of RAM and with 30+ databases running, only 5GB of RAM is reported as used? Strange indeed.

I quickly logged in to server and checked memory usage:

loreto:/tmp # free
             total       used       free    shared   buffers     cached
Mem:      33274944   32931032     343912         0        20   27013200
-/+ buffers/cache:    5917812   27357132
Swap:     16779884    5603256   11176628

Really only 5GB, check_memory plugin was not wrong. Next this I checked were shared memory segments – Oracle uses shared memory in huge quantities, so this is also very important parameter.

loreto:/tmp # a=0; for i in $(ipcs -m|grep ^0x|awk ‘{print $5}’); do let a+=$i; done; echo $a
20443037885

Ugh, 20GB allocated for shared memory, while system reports only 5GB. Something is very wrong here. Confused, I took out the artillery.

loreto:/tmp # cat /proc/meminfo
MemTotal:     33274944 kB
MemFree:        198580 kB
Buffers:            20 kB
Cached:       27439580 kB
SwapCached:     223880 kB
Active:       16333936 kB
Inactive:     15428724 kB
HighTotal:    32634140 kB
HighFree:        33516 kB
LowTotal:       640804 kB
LowFree:        165064 kB
SwapTotal:    16779884 kB
SwapFree:     10856700 kB
Dirty:            1668 kB
Writeback:           0 kB
AnonPages:     4089416 kB
Mapped:       10222968 kB
Slab:           427584 kB
CommitLimit:  33417356 kB
Committed_AS: 49751960 kB
PageTables:     826016 kB
VmallocTotal:   112632 kB
VmallocUsed:     22228 kB
VmallocChunk:    90180 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Usually, used memory on Linux is calculated as (Total Memory – (Unused Memory + Buffers + Page Cache)). Why buffers and caches are not counted into memory usage? Simply because it contains data that is not really critical for operating system and applications running. It contains data that can be flushed and removed from the memory at any time.

So in my case that was:

33274944 – (198580 + 20 + 27439580) = 5636764

OK, this matches output from free command. But what about those 20GB of allocated shared memory?

Next few hours I spent in searching and reading Linux documentation on memory management and found few interesting things.

Linux uses principle of memory overcommitment. Basically, what this means is that when application requests memory to be allocated, kernel will always “give: the memory hoping that application will not really use it, or at least not the whole size allocated. Only when application tries to write the data into the memory, kernel will mark the memory as used. This can lead to situation where the size of allocated memory is actually higher than the size of physical memory inside the machine. But as long as there is no demand for allocated memory, system is running without problems.

And this is the core of my dilemma. Shared memory is allocated, but since there is no data in it, it is not counted into used memory.

Memory overcommit can be configured via two parameters:

loreto:/tmp # sysctl -a|grep overcommit
vm.overcommit_ratio = 50
vm.overcommit_memory = 0

From Red Hat manual:

  • overcommit_memory — Configures the conditions under which a large memory request is accepted or denied. The following three modes are available:
    • 0 — The kernel performs heuristic memory over commit handling by estimating the amount of memory available and failing requests that are blatantly invalid. Unfortunately, since memory is allocated using a heuristic rather than a precise algorithm, this setting can sometimes allow available memory on the system to be overloaded. This is the default setting.
    • 1 — The kernel performs no memory over commit handling. Under this setting, the potential for memory overload is increased, but so is performance for memory intensive tasks (such as those executed by some scientific software).
    • 2 — The kernel fails requests for memory that add up to all of swap plus the percent of physical RAM specified in /proc/sys/vm/overcommit_ratio. This setting is best for those who desire less risk of memory overcommitment.
      Note This setting is only recommended for systems with swap areas larger than physical memory.
  • overcommit_ratio — Specifies the percentage of physical RAM considered when /proc/sys/vm/overcommit_memory is set to 2. The default value is 50.