Category: Linux

If you’re not familiar with HAProxy, you’re missing out! HAProxy is a very intelligent high-availability reverse proxy that operates all the way up to Layer 7. Unlike Nginx (another good choice), HAProxy is not a webserver, it is designed with only high availability and load balancing in mind. Another great feature is that it’s a lot cheaper than your typical Layer 7 Load Balancer, free :).

There are plenty of good articles out there on HAProxy, but I couldn’t find any Debian/Ubuntu packages for the latest version (1.4.15). The Ubuntu 11.04 repo’s newest version is 1.4.8, but if you want some of the cool new features of 1.4.9-1.4.15, you have to build it from scratch. This process isn’t too painful, but I’ve got a lot of servers to put HAProxy on and I don’t want dpkg / apt-get to freak out and install the old HAProxy in order to satisfy a dependency.

For this reason, I’ve created an Ubuntu 11.04 x64 HAProxy package with TCP Splicing, Full Transparent Proxies and PCRE enabled:

HA-Proxy version 1.4.15 2011/04/08
Copyright 2000-2010 Willy Tarreau <>

Build options :
  TARGET  = linux26
  CPU     = native
  CC      = gcc
  CFLAGS  = -O2 -march=native -g -fno-strict-aliasing

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Available polling systems :
     sepoll : pref=400,  test result OK
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.

Feel free to grab haproxy_1.4.15_amd64.deb (502k SHA1: 8b2ecf05544e0f6531e50bf40261e0b112db61e9)
You can also grab the sources from here:

After a lot of development and tons of regression testing, Tera-WURFL 2.1.4 is finally ready for release. There has finally been a break in commits on the GitHub repo for more that a week, and no bug reports on the development branch.

Changes from Tera-WURFL 2.1.3

  • Switched to better XML Parser (old parser, SimpleXML will still be used if XMLReader is missing)
  • Improved matching for Apple, Android, Nokia, BlackBerry / RIM, DoCoMo, KDDI, LG and OperaMini
  • Allowed preloading a custom TeraWurflConfig so the default is ignored
  • Updated source documentation
  • Updated instructions
  • Greatly improved cmd_line_admin.php
  • Converted line endings to \n
  • Various bugfixes
  • Improved Remote Client
  • Added builtin regression testing (thanks digitalronin!). Tip: from the shell, go to the test/ dir and type “rake sanity” to make sure your patches are compatible with the loaded WURFL
  • Moved default config to “TeraWurflConfig.php.example” so your config isn’t overwritten
  • Fixed MSSQL DB Connector bug

If you rely on Android or BlackBerry detection, you will really want to get version 2.1.4 – it will significantly improve detection of these devices.

Even though RAM is cheap these days, there are some conditions in which your Linux server could run out of it completely. Just the other day, I noticed my main hosting server went down – I could still ping it, but DNS, SSH and Apache2 were not responding, so I had to call the datacenter to have them reboot my system. After analyzing the system, I realized that some unknown process ate up all the memory and all the swap space! I used Cacti to monitor my server’s performance, and so I could see that it took a nose-dive after getting hit with a few million requests in a couple days (these were raw mobile device detection requests).  After the system ran out of memory, it started swapping.  This lasted for about 2 weeks before depleting the swap space, at which point it struggled on for another 18 hours.  At this point, it was critically starved of memory and oom_killer (Out of Memory Killer) was invoked to start killing processes in a vain attempt to free up memory.  The oom_killer seems to have very little intelligence as to which processes to kill first, as sshd and named were early victims.  After this episode, I decided to create a script that adjusts the order in which key processes were killed, to make sure I have access to the server in the event of a memory leak or OOM condition. to the rescue!

I’ve created to adjust (and periodically readjust via cron) the order in which the processes may be killed by oom_killer.  The script uses a config file called oom_adjust.conf (by default it looks for it in /etc) in which you can list processes and the oom_adj value that you want to give them.  The possible values are from -17 (never kill) to 15 (kill first).


# Adjust process oom_adj values so they are more or less likely to be killed in an oom event
# procname oom_adj

# Keep sshd ALIVE
sshd -17

# DNS is very important to me too
named -8

# I'd prefer that MySQL stays alive, but it's not required
mysqld -1

# Apache2 is a memory hog, but I'll give it a fighting chance
# I'm giving it 0 since the workers will respawn at 0 anyway
apache2 0

# Sphinx search is cool, but I can live without it if an oom occurs
searchd 3

# Memcache is in the same boat as Sphinx search
memcached 3

# I only use mongodb for testing on this server
mongod 5

# It would be nice if smtpd stayed up, so I still get alerts
smtpd 5

# These services can be killed first
pure-ftpd 10
pure-ftpd-mysql 10
snmpd 10
fail2ban-server 10
ntpd 10
authdaemond 10
saslauthd 10
qmgr 10
pickup 10

Here is the main script, which I’ve symlink’d into /usr/sbin for convinience.


# Out of Memory Killer (oom_killer) Priority Adjustment Script
# by Steve Kamerman <>, Jan 2011


if [ ! -f $OOM_ADJ_FILE ]; then
        echo " config file $OOM_ADJ_FILE was not found" >&2
        exit 1

echo " is setting oom_killer priorities"
for LINE in `cat $OOM_ADJ_FILE | sed -e '/^[# \t].*/d' | sed -e '/^$/d' | sed -e 's/ /:/'`; do
        NAME=`echo $LINE | cut -d":" -f1`
        ADJ=`echo $LINE | cut -d":" -f2`
        echo "  Setting $NAME to $ADJ"
        for PID in `pidof $NAME`; do
                echo $ADJ > /proc/$PID/oom_adj
exit 0

If your distro uses /etc/rc.local, you can put call this script there to apply the adjustments on startup.  I also call it on my servers via crontab every night to keep the processes in check, in case they have respawned/restarted with a different PID.

I’ve been delaying the Tera-WURFL 2.1.3 release since mid-July because of the large number of feature requests and improvements coming in, but the time has come to release it. I’m still waiting for the unit test code to stabilize a bit before I put it into the tree, but if you are interested in it, follow me on GitHub and you should see it soon.

Here’s a short list of features and improvements in Tera-WURFL 2.1.3:

  • Native support for MongoDB
  • Better support and performance for Microsoft SQL Server
  • Command line administration utility
  • Better webservice performance using JSON
  • Better Python client with JSON support
  • Improved performance in MySQL5
  • Device Images are available via the PHP webservice client
  • Loaded WURFL version is now tracked in the DB
  • MySQL connector allows for alternate port
  • Better overall detection

If you’re a command line junky like me, I think you’ll really like the CLI administration tool.  To use it, just go to your Tera-WURFL/admin/ directory and type php cmd_line_admin.php

# php cmd_line_admin.php

Tera-WURFL Stable 2.1.3
The command line WURFL updater for Tera-WURFL
Loaded WURFL: - 2010-09-09 04:08:06
Last Updated: Wed, 15 Sep 2010 20:38:59 -0500
Usage: php cmd_line_admin.php [OPTIONS]

Option                     Meaning
--help                    Show this message
--update=   The source of the WURFL file:
Update from your local wurfl.xml file:
Update from
--clearCache              Clear the device cache
--rebuildCache            Rebuild the device cache by redetecting all
cached devices using the current WURFL
--stats                   Show statistics about the Tera-WURFL Database

You can update the WURFL by passing the —-update=local or –update=remote parameter (crontab will like this):

# php cmd_line_admin.php --update=remote
Downloading WURFL from ...

done (/testtw/2.1.3/mongo/data/wurfl.xml: 14.80 MB [897.30 KB compressed])
Downloaded in 4.6609511375427 sec @ 1.58 Mbps

Database Update OK
Total Time: 22.146492004395
Parse Time: 3.8051941394806 (TeraWurflXMLParser_SimpleXML)
Validate Time: 0.024373054504395
Sort Time: 1.2780990600586
Patch Time: 0.50953578948975
Database Time: 8.9265999794006
Cache Rebuild Time: 7.6026899814606
Number of Queries: 2456
PHP Memory Usage: 54.49 MB
WURFL Version: - 2010-09-17 15:50:54 (Fri Sep 17 15:56:32 -0500 2010)
WURFL Devices: 13687
PATCH New Devices: 39
PATCH Merged Devices: 1

You can also see stats on the Tera-WURFL database:

# php cmd_line_admin.php --stats
Tera-WURFL Stable 2.1.3
Database Type: MongoDB (ver MongoDB 1.6.2)
Loaded WURFL: - 2010-09-17 15:50:54
Last Updated: Sat, 18 Sep 2010 15:28:05 -0500
Config File: /testtw/2.1.3/mongo/TeraWurflConfig.php
---------- Table Stats -----------

Rows:    13726
Devices: 6229
Size:    30.00 MB

Rows:    522
Size:    1.23 MB

Head over to and give version 2.1.3 a try!

I’ve been working hard on Tera-WURFL 2.1.3 lately.  I’ve been meaning to release it for over a month now, the problem is that I just keep finding improvements to make.  Since Tera-WURFL’s development is now hosted on GitHub, you can always see what I’m doing and grab a copy for yourself.  New in version 2.1.3 is a MongoDB Database Connector.  I keep getting pounded by people about NoSQL and I haven’t given it a fair chance until now.  Simon Harris from was nice enough to contribute the initial MongoDB connector, and I’ve spent a few days working on it and testing it against MySQL5.  At first the performance seems outstanding – almost double that of MySQL5 – but I noticed some inconsistencies with my tests, so I built a benchmarking script and an comparison tool that automatically tests one installation against another and checks for consistency.  I’ve tested three different database connectors against each other and my memcached-based Tera-WURFL Enterprise.

Testing Setup

In order to test the Database Connectors, I brought up a virtual machine on an under-utilized ESXi server (dual 6-core Opterons and 32GB FB-DDR3).

Virtual Machine Specs:

  • 4x Opteron 2.2GHz cores
  • 12GB FB-DDR3 RAM
  • 20GB of storage on a SAS array over MPIO iSCSI
  • Ubuntu Server 10.04.1
  • MySQL 5.1
  • MongoDB 1.7.0

This machine doesn’t have extremely fast CPUs, just a lot of them.  At any rate, the specs were consistent so I was able to get a reasonable test result.  In order to test the performance of the DBs, I ran about 66,000 unique user agents (these are UAs that are not easily matched and create a lot of DB load) through the connectors and measured the rate at which they detected them.

The results

The following results are the average speed at which the different DB Connectors detected the test user agents.  The two graphs show the difference between uncached and cached detections.  For each database, I ran the test with the all the WURFL capabilities loaded (blue) and with only [‘product_info’][‘is_wireless_device’] loaded (red) to get an idea of the performance benefit of using Tera-WURFL’s CAPABILITY_FILTER option.

Uncached Device Detection Performance in Tera-WURFL 2.1.3

* note: memcached is not capable of serving uncached detections, so it is left out of this test.

Cached Device Detection Performance in Tera-WURFL 2.1.3


I was very surprised by these results so I tested them repeatedly just to be sure.  I was so excited that MongoDB would be the clear winner here, but it just isn’t the case for uncached detections.  Mongo did fare well when serving the cached full capabilities – 63% higher than MySQL5.  MS SQL Server 2005 was particularly painful to test.  By comparison, MongoDB took about 8 minutes to churn though all 66,000 user agents, but my initial test with MS SQL Server was 3 hours and 8 minutes.  Since I tested all the other connectors on Ubuntu Server, and the MS SQL Server connector requires Microsoft’s PHP SQL Server Driver, I had to run the tests on a different VM.  On this VM I was getting between 1-3 uncached detections per second, but I moved the testing to my development laptop (2.8GHz Core 2 Duo, 4GB RAM and a Samsung 250GB SSD) and was able to achieve better results.  MySQL fared very well overall and remains at the top of the list.  I have not mentioned the MySQL Nested Set connector before, but it’s been included in Tera-WURFL for a couple versions now.  It just extends the MySQL5 connector and adds right and left values to all of the devices in the WURFL so it can return the complete fallback tree for a given WURFL ID in a single indexed query.  Unfortunately, this seems to provide a negligible improvement in speed.  I would still recommend it for environments where the webserver is on a different host than the database.

Look for Tera-WURFL 2.1.3 to be released within a couple weeks.  I’m trying to get some unit-testing code integrated before I release it.

*** UPDATE ***

I’ve retested MongoDB and MySQL5 again on my production server against a typical distribution of 44887 user agents.  The performance was much better, but the MySQL <-> MongoDB performance was proportional to the first results:

Uncached Cached
All Capabilities Min Capabilities All Capabilities Min Capabilities
MySQL 303 510 2735 7740
MongoDB 195 255 3182 6154

I recently created Tera-WURFL Explorer to allow people to browse through the WURFL, search for devices and upload images to the WURFL images collection. I originally used MySQL’s FULLTEXT index to let people search for devices, but quickly realized that it did not suit my needs. The main problem was that it does not index words smaller than what is specified in my.cnf (ft_min_word_len), and if you want to change it, you need to change it server-wide. This was not a good option for a large virtual host setup since it would affect all the FULLTEXT indices on the server; also, if you do change it, you need to reindex every FULLTEXT column in every database to prevent data corruption.

I did some research on search engines and eventually settled on Sphinx – mainly because it has a cool name, but also because there are some big-name success stories from companies like Craigslist who switched to it and never looked back.

Here’s how I installed it on Ubuntu 9.10:

First, you need to install the dependencies and download sphinx, then extract the archive and make it:

apt-get install g++ libmysql++-dev
cd /tmp
tar -zxvf sphinx-0.9.9.tar.gz
cd sphinx-0.9.9
./configure --prefix=/usr/local/sphinx
make install

Now all the sphinx-related files are in /usr/local/sphinx.
Next, I created a system user and group called “sphinx”:

adduser --system --group sphinx

Note: on RedHat-like systems, you can use “adduser -r -n sphinx”

Now, I created an init script for it. I would recommend downloading my init.d script.

mv searchd /etc/init.d/
chown root:root /etc/init.d/searchd
chmod 755 /etc/init.d/searchd

This script adds the following functionality:

# Start the Sphinx service
service searchd start
# Stop Sphinx
service searchd stop
# Check if Sphinx is running
service searchd status
# Reindex every Sphinx index (works while started or stopped)
service searchd reindex

Now we’ll add sphinx to the startup and use the config option to setup sphinx to run as the sphinx user:

update-rc.d searchd defaults
service searchd config

Note: on RedHat-like systems you can use “chkconfig –add searchd”

Lastly, you need to configure sphinx. I would copy the default config file and edit that one:

cp /usr/local/sphinx/sphinx.conf.dist /usr/local/sphinx/sphinx.conf

You can follow along with the comments in the file, or jump on the documentation site and figure out what all the settings do.

Now everything is setup and should work properly!
If you followed my directions and put the tarball in /tmp, the sphinx PHP and Python APIs and some examples are in /tmp/sphinx-0.9.9/api/. You should put a copy of the PHP or Python API somewhere else on the system so you can use it from your applications.

To see my use of the Sphinx search engine, take a look at this site:

Background of my problem

A few days ago my Dell XPS 720 decided to die on me. When I push the power button it turns on for about 1/2 second and then turns off. After much debate, I bought a new power supply for it (non-standard of course [24pin AND 20pin power connectors]). Today I received the power supply and low-and-behold that wasn't the problem. Now I need to buy either a motherboard or a new CPU (the existing one is a Core 2 Quad Extreme 3.0). I decided that buying from Dell was a bad choice and I'll just build myself a Phenom II system. Meanwhile, the XPS 720 shipped with 2x 120GB 10k Raptors in a RAID-0 using the sub-par onboard SATA RAID controller. I would just love to get my data off the system, but I can't get it up and running and I'm not going to sink anymore money into it. I decided to figure out exactly how the data was stored on the array members and try to deinterlace it directly into a raw image that I could then mount via loopback device in Linux and copy all my data back out 🙂 It sounds easy, huh? I found this GREAT little python code snippet that does the heavy lifting for me – thanks Sim

I’ve used UFS2 snapshots on FreeBSD by means of the dump command to make stateful filesystem backups in the past, so I recently wanted to take a similar approach to backup some Linux servers. Most GNU/Linux distributions use ext3 as their root filesystem by default these days which does not natively support snapshots, so I needed to find another way to do it. As it turns out, I used LVM on my main Linux servers to add a layer of virtualization to the storage subsystem. LVM does natively support snapshots, and here’s how to do it.

1. Make room for the snapshot

I ran into a major problem right off the bat – the server I was testing LVM snapshots on had one hard drive with 2 partitions, 1 for /boot and one for LVM. The LVM partition contained one Volume Group called “TERA-VIRTUAL” and this VG had 2 Logical Volumes – “root” (mounted on /) and swap_1 (swap). The root LV had all the PE allocated to it and the filesystem beneath it was using all of the available space. The problem is that since there were no available Physical Extents in the VG (or in my PVs), I couldn’t create a snapshot volume. This is important: when you create a snapshot on LVM you are really creating another Logical Volume that is used to hold the changes to the volume. This volume needs to me in the same Volume Group as the volume you are taking a snapshot of – so you need some extra space in that VG! In my case, I needed to resize the root fs, which was on a Logical Volume – this to me an hour plus to achieve since most LiveCDs either have no support for LVM (gparted-live) or the support is broken (backtrack 3 final usb). I booted ubuntu server 7.10 64bit into rescue mode and started a shell in the installer environment and once at a prompt typed vgmknodes to make the /dev/ entries for my LVs, then used resize2fs to shrink my FS and lvreduce to shrink my LV, leaving me with 23G of available Physical Extents in my VG (seriously, if you don’t know what I’m talking about, don’t try it – you will trash your hard drive!).

2. Create the snapshot and run backup

To create a snapshot you first need to figure out how much space you need – it is critical that you have enough space to hold all the changes to your volume until the snapshot is removed!. For me 10% of my 200G is fine – 20G.
The process is to create a snapshot LV, mount it, do your backup, unmount it, and remove the snapshot:

root@TERA-VIRTUAL:~# lvcreate --size 20G --snapshot --name snap /dev/TERA-VIRTUAL/root
Logical volume "snap" created
root@TERA-VIRTUAL:~# lvs
LV     VG           Attr   LSize   Origin Snap%  Move Log Copy%
root   TERA-VIRTUAL owi-ao 200.00G
snap   TERA-VIRTUAL swi-a-  20.00G root     0.00
swap_1 TERA-VIRTUAL -wi-ao   9.46G
root@TERA-VIRTUAL:~# lvdisplay /dev/TERA-VIRTUAL/snap
--- Logical volume ---
LV Name                /dev/TERA-VIRTUAL/snap
VG Name                TERA-VIRTUAL
LV UUID                fFbaH4-22Hq-s7a2-e1mo-DPov-8wff-dvuAiI
LV Write Access        read/write
LV snapshot status     active destination for /dev/TERA-VIRTUAL/root
LV Status              available
# open                 0
LV Size                200.00 GB
Current LE             51200
COW-table size         20.00 GB
COW-table LE           5120
Allocated to snapshot  0.01%
Snapshot chunk size    8.00 KB
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           254:2
root@TERA-VIRTUAL:~# mount /dev/TERA-VIRTUAL/snap /backup_snapshot/
root@TERA-VIRTUAL:~# cp -Rf /backup_snapshot/VirtualMachines/TERA-WEBSERVER /backup/
root@TERA-VIRTUAL:~# lvdisplay /dev/TERA-VIRTUAL/snap
--- Logical volume ---
LV Name                /dev/TERA-VIRTUAL/snap
VG Name                TERA-VIRTUAL
LV UUID                fFbaH4-22Hq-s7a2-e1mo-DPov-8wff-dvuAiI
LV Write Access        read/write
LV snapshot status     active destination for /dev/TERA-VIRTUAL/root
LV Status              available
# open                 1
LV Size                200.00 GB
Current LE             51200
COW-table size         20.00 GB
COW-table LE           5120
Allocated to snapshot  1.96%
Snapshot chunk size    8.00 KB
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           254:2
root@TERA-VIRTUAL:~# umount /backup_snapshot/
root@TERA-VIRTUAL:~# lvremove /dev/TERA-VIRTUAL/snap
Do you really want to remove active logical volume "snap"? [y/n]: y
Logical volume "snap" successfully removed

As you can see from the transcript above, you can keep an eye on your snapshot size by using lvdisplay and looking for “Allocated to snapshot”. As seen above, before the backup the snapshot volume was 0.01% full, but by the time my backup finished, it was up to 1.96%.

Now I’ve got a sane copy of my data without having to dismount my filesystem!

Having done Linux administration for over 10 years now, I use the watch command almost every day. On Linux, watch will run a command every couple seconds. It’s particularly nice for watching files change:

# watch ls -lah

This example will keep showing the output of ls -lah every 2 seconds.

About a year ago I switched a couple servers over to FreeBSD 6.1 because of it’s security features and more stable development model. For the last year I’ve been occasionally using the watch command on FreeBSD and getting an annoying error like this:

[root@devel /backup]# watch ls -lah
watch: fatal: bad device name

Today it finally occured to me to look at the manpage:

WATCH(8)                FreeBSD System Manager's Manual               WATCH(8)

watch -- snoop on another tty line

watch  -cinotW -f snpdev tty

The watch utility allows the user to examine all data coming through a
specified tty using the snp(4) device.  If the snp(4) device is not
available, watch will attempt to load the module (snp).  The watch utility writes to standard output.

It hardly looks like the watch I know! It turns out that FreeBSD has a different use for watch – snooping on other consoles. On FreeBSD, the GNU watch (like on Linux) is called gnu-watch. If you don’t want to install it, you can easily whip up a poor-man’s replacement:

while [ 1 -lt 2 ]
echo ------------------------------------
eval $@
sleep 2

TIP: To replace the default ‘watch’ with this script, save the above script as /root/watch, then make it executable, chmod +x /root/watch and alias it in your shell, alias watch=’/root/watch’. You can save this alias permenantly with the bash shell by adding it to ~/.bash_profile .

Here’s a sample output from the poor-man’s watch:

Thu Jul 10 22:09:27 EDT 2008
total 17113832
drwxr-xr-x   3 root  wheel   512B Jul 10 21:03 .
drwxr-xr-x  21 root  wheel   512B Jul 10 18:25 ..
-rw-r--r--   1 root  wheel    97M Jul 10 15:41 archive1.tar
-rw-r--r--   1 root  wheel    40M Jul 10 15:41 archive1.tar.bz2
-rw-r--r--   1 root  wheel   308M Jul 10 17:19 archive2.tar
-rw-r--r--   1 root  wheel    93M Jul 10 17:19 archive2.tar.bz2
-rw-r--r--   1 root  wheel   600M Jul 10 17:13 archive2a.tar
-rw-r--r--   1 root  wheel   192M Jul 10 17:13 archive2a.tar.bz2
-rw-r--r--   1 root  wheel   3.3G Jul 10 19:09 archive2b.tar
-rw-r--r--   1 root  wheel   1.4G Jul 10 19:09 archive2b.tar.bz2
-rw-r--r--   1 root  wheel   5.4G Jul 10 19:33 archive2c.tar
-rw-------   1 root  wheel   1.9G Jul 10 22:09 archive2c.tar.bz2
-rw-r--r--   1 root  wheel   2.1G Jul 10 17:44 archive2d.tar
-rw-r--r--   1 root  wheel   886M Jul 10 16:08 archive3.tar
drwxr-xr-x   2 root  wheel   512B Jul 10 19:44 scripts

Quick Linux Tip:

If you’re trying to delete a very large number of files at one time (I deleted a directory with 485,000+ today), you will probably run into this error:

/bin/rm: Argument list too long.

The problem is that when you type something like “rm -rf *”, the “*” is replaced with a list of every matching file, like “rm -rf file1 file2 file3 file4” and so on. There is a reletively small buffer of memory allocated to storing this list of arguments and if it is filled up, the shell will not execute the program.

To get around this problem, a lot of people will use the find command to find every file and pass them one-by-one to the “rm” command like this:

find . -type f -exec rm -v {} \;

My problem is that I needed to delete 500,000 files and it was taking way too long.

I stumbled upon a much faster way of deleting files – the “find” command has a “-delete” flag built right in! Here’s what I ended up using:

find . -type f -delete

Using this method, I was deleting files at a rate of about 2000 files/second – much faster!

You can also show the filenames as you’re deleting them:

find . -type f -print -delete

…or even show how many files will be deleted, then time how long it takes to delete them:

root@devel# ls -1 | wc -l && time find . -type f -delete
real    0m3.660s
user    0m0.036s
sys     0m0.552s