Tag Archive: Linux


If you’re not familiar with HAProxy, you’re missing out! HAProxy is a very intelligent high-availability reverse proxy that operates all the way up to Layer 7. Unlike Nginx (another good choice), HAProxy is not a webserver, it is designed with only high availability and load balancing in mind. Another great feature is that it’s a lot cheaper than your typical Layer 7 Load Balancer, free :).

There are plenty of good articles out there on HAProxy, but I couldn’t find any Debian/Ubuntu packages for the latest version (1.4.15). The Ubuntu 11.04 repo’s newest version is 1.4.8, but if you want some of the cool new features of 1.4.9-1.4.15, you have to build it from scratch. This process isn’t too painful, but I’ve got a lot of servers to put HAProxy on and I don’t want dpkg / apt-get to freak out and install the old HAProxy in order to satisfy a dependency.

For this reason, I’ve created an Ubuntu 11.04 x64 HAProxy package with TCP Splicing, Full Transparent Proxies and PCRE enabled:

HA-Proxy version 1.4.15 2011/04/08
Copyright 2000-2010 Willy Tarreau <w@1wt.eu>

Build options :
  TARGET  = linux26
  CPU     = native
  CC      = gcc
  CFLAGS  = -O2 -march=native -g -fno-strict-aliasing
  OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Available polling systems :
     sepoll : pref=400,  test result OK
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.

Feel free to grab haproxy_1.4.15_amd64.deb (502k SHA1: 8b2ecf05544e0f6531e50bf40261e0b112db61e9)
You can also grab the sources from here: http://www.stevekamerman.com/haproxy/

Even though RAM is cheap these days, there are some conditions in which your Linux server could run out of it completely. Just the other day, I noticed my main hosting server went down – I could still ping it, but DNS, SSH and Apache2 were not responding, so I had to call the datacenter to have them reboot my system. After analyzing the system, I realized that some unknown process ate up all the memory and all the swap space! I used Cacti to monitor my server’s performance, and so I could see that it took a nose-dive after getting hit with a few million requests in a couple days (these were raw mobile device detection requests).  After the system ran out of memory, it started swapping.  This lasted for about 2 weeks before depleting the swap space, at which point it struggled on for another 18 hours.  At this point, it was critically starved of memory and oom_killer (Out of Memory Killer) was invoked to start killing processes in a vain attempt to free up memory.  The oom_killer seems to have very little intelligence as to which processes to kill first, as sshd and named were early victims.  After this episode, I decided to create a script that adjusts the order in which key processes were killed, to make sure I have access to the server in the event of a memory leak or OOM condition.

oom_adjust.sh to the rescue!

I’ve created oom_adjust.sh to adjust (and periodically readjust via cron) the order in which the processes may be killed by oom_killer.  The script uses a config file called oom_adjust.conf (by default it looks for it in /etc) in which you can list processes and the oom_adj value that you want to give them.  The possible values are from -17 (never kill) to 15 (kill first).

oom_adjust.conf

# Adjust process oom_adj values so they are more or less likely to be killed in an oom event
# procname oom_adj

# Keep sshd ALIVE
sshd -17

# DNS is very important to me too
named -8

# I'd prefer that MySQL stays alive, but it's not required
mysqld -1

# Apache2 is a memory hog, but I'll give it a fighting chance
# I'm giving it 0 since the workers will respawn at 0 anyway
apache2 0

# Sphinx search is cool, but I can live without it if an oom occurs
searchd 3

# Memcache is in the same boat as Sphinx search
memcached 3

# I only use mongodb for testing on this server
mongod 5

# It would be nice if smtpd stayed up, so I still get alerts
smtpd 5

# These services can be killed first
pure-ftpd 10
pure-ftpd-mysql 10
snmpd 10
fail2ban-server 10
ntpd 10
authdaemond 10
saslauthd 10
qmgr 10
pickup 10

Here is the main script, which I’ve symlink’d into /usr/sbin for convinience.
oom_adjust.sh

#!/bin/sh

# oom_adjust.sh Out of Memory Killer (oom_killer) Priority Adjustment Script
# by Steve Kamerman <stevekamerman@gmail.com>, Jan 2011
# http://www.stevekamerman.com

OOM_ADJ_FILE=/etc/oom_adjust.conf

if [ ! -f $OOM_ADJ_FILE ]; then
        echo "oom_adjust.sh: config file $OOM_ADJ_FILE was not found" >&2
        exit 1
fi

echo "oom_adjust.sh is setting oom_killer priorities"
for LINE in `cat $OOM_ADJ_FILE | sed -e '/^[# \t].*/d' | sed -e '/^$/d' | sed -e 's/ /:/'`; do
        NAME=`echo $LINE | cut -d":" -f1`
        ADJ=`echo $LINE | cut -d":" -f2`
        echo "  Setting $NAME to $ADJ"
        for PID in `pidof $NAME`; do
                echo $ADJ > /proc/$PID/oom_adj
        done
done
exit 0

If your distro uses /etc/rc.local, you can put call this script there to apply the adjustments on startup.  I also call it on my servers via crontab every night to keep the processes in check, in case they have respawned/restarted with a different PID.