Linux memory tweaks for improved responsiveness

Memory management techniques in the linux kernel has changed over the years.  Sometimes a kernel upgrade, or a hardware upgrade, where you expect better performance, can lead to disappointment.

For a while now, I've been struggling with our ubuntu 10.04 application server (managing 35 desktop sessions) where we experience "morning sluggishness" with the system.  Between 6am and noonish, the system was just not as responsive as it could be, but in the afternoons it became quite zippy.  I've been explaining this away with things like "everyone is just logging in morning, system is seeing a heavy load early in the day".. or "everyone does more work in the morning, and take it easy after lunch".

I decided to analyze memory usage patterns combined with I/O wait times.  Between 8am and noon, there was an average I/O wait time of between 2% - 3%, and between noon and 4pm an average of < 1%.  When I graphed out memory usage patterns, I noticed that our nightly backups were consuming a huge amount of buffered memory (understandably).. and that this buffer was slowly reclaimed overtime and around noonish every day it got close to cache size.  This couldn't be coincidence.. performance steadily increased as the buffers from previous nights backup were free'd up.  Makes total sense.. but question was.. how do I modify this behaviour?

Googling around, I came across this link

I already knew about vm.swappiness - already have this set to 10.

But vm.vfs_cache_pressure and vm.drop_caches were new to me.  So I thought, if we can drop the caches right after the backup, and give a little more priority to the inode/dentry caches - this should reduce our I/O wait.

So, I set vm.vfs_cache_pressure to 50 ( sysctl -w vm.vfs_cache_pressure=50 ), and in my backup script, added "echo 3 > /proc/sys/vm/drop_caches" to the end.  The results are quite dramatic.  So far this morning, I have an average I/O wait of  0.19%, down from 2-3%!! (and system is very zippy.)  This graph gives a good visual on the memory usage...

Notice: Everyday at 00:00.. this is backup kicking in.. Tues/Wed shows the slow reclaiming of buffers.  Thurs is where I drop the caches right after backup.. now caches reach previous days peak by 8am!!

Could this be the end of my users complaining about a slow system??   Doubt it... They never notice when its fast.. but always and only notice when its slow.

Update:  After two days now.. amazing.. average daytime I/O wait is now < 0.10%

ext4 performance and the barrier option

We recently implemented a 6-disk SAS RAID 1+0 /home ext4 filesystem on our application server to improve performance, and found that we still had a disk I/O bottleneck, with processor I/O wait averaging in the 3 to 4% range.

I came across this LWN article http://lwn.net/Articles/283161/ that talks about a 30% performance drop when the barrier option is enabled on an ext4 filesystem.

As of the time of the LWN article, barrier was not enabled by default on an ext4 filesystem.  But somewhere along the kernel release line, barrier=1 (on) became the default for newly created ext4 filesystems.

Doing a "cat /proc/mounts" on my ubuntu 10.04 distro, I found that my ext4 filesystems indeed had the option barrier=1. So on April 3rd, I added barrier=0 to my fstab, and rebooted.  The performance boost was quite dramatic.

Using measures of "average wait time", "% utilization" and "%IO wait", here is the before and after stats, measured and averaged on a typical work week 8am to 5pm...


With barrier=1:

Average wait time:   90 ms
Average Disk utilization %:   40%
Average CPU IO wait %:  3.5%

With barrier=0:

Average wait time:  2 ms
Average Disk utilization %:   2%
Average CPU IO wait %:  0.40%


Graphs depict the change more dramatically (comparing April 1st to April 4th):

Average Wait Time

Average % Utilization

Load Average

NOTE: This is not a "very busy" April 1st compared to a "very light" April 4th.  Both are with a 30 user load, approx. 2300 processes.  And every other day of the week in the comparison show the same dramatic improvement.

Of course one can argue that we are sacrificing data integrity for performance - but we do have a UPS backed system, and crashes are (thus far) non-existent.

Linux performance tuning - I/O wait

Have a 64-bit ubuntu linux (hardy) system running 30 KDE desktop sessions (using nomachine's NX) on an IBM bladeserver HS21 with 2 quad-core Xeon's and 16G of RAM.  Many times a day, users would experience "freezes" within either firefox or evolution that would last up to about 15 seconds.. window completely unresponsive - then would resume execution.  This using the stock 2.6.24-24-generic ubuntu kernel with voluntary prempting.

After much performance monitoring with sar, and associating the timing of the freezes to sar data, we determined that I/O wait seemed to be the culprit.  Real memory usage {used - (cached+buffered)} only ever peaked around 12G, and CPUs were largely idle.

Disk is 10k rpm 300GB SAS 2.5" drive.. in RAID 1 (mirrored) using HS21 onboard LSI controller.

I/O Wait is primarily disk I/O but could be network I/O as well, so we decided to tune disk and network.

Filesystems is using ext3 and mounted with noatime.

Finding this link , I tried adding the mount options: noatime,nobh,data=writeback,commit=90

NOTE:  If on root filesystem, you must add "rootflags=nobh,data=writeback,commit=90" in grub.

After a reboot, and once full load was back on system, I realized a substantial reduction in average IO wait time (from ~50ms to ~25ms), and overall average % disk utilization (from ~15% to ~8% - statistics from iostat).

This was great improvement but there were still slight 1-2s freezing incidents at this point, so I additionally tweaked the network...

For network performance tuning I ran across this link and added the following to /etc/sysctl.conf:

net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=1800
net.core.wmem_max=8388608
net.core.rmem_max=8388608
net.ipv4.tcp_rmem="4096 87380 8388608"
net.ipv4.tcp_wmem="4096 87380 8388608"

Together, these have stopped the "freezing" altogether.  There are still periods of slowness due to load, but now they are few and far between.