Tag Archive: LinkedIn

There’s been a lot of talk about WURFL lately and with good reason. On June 6, 2011 I launched ScientiaMobile along with my business partners, Luca Passani and Krishna Guda.  If you’re reading my blog you probably know that I am the author of the popular mobile device detection library Tera-WURFL.  I wrote Tera-WURFL back in 2006, when the world of mobile devices was completely scattered in every direction.  At the time, the vast majority of mobile devices did not support a common set of capabilities, like MP3, MMS or HTML.  Instead, your best bet was to write a terrible-looking, but fairly well behaved XHTML-MP site using very conservative tags.  WURFL solved this problem nicely by accurately matching a visiting user to a device profile that allowed you to serve a WAP page to a Nokia 3360, an XHTML-MP page to a Motorola RAZR and an HTML page to a 1st-gen iPhone.

Since 2006 there have been a lot of changes, particularly in the capabilities of mobile browsers.  Thanks to Apple Webkit, even lower-end phone are able to browse sites without catastrophic rendering issues (still a terrible UX, just not a catastrophe).  With all these changes, however, one thing has remained: mobile device fragmentation.  Device fragmentation means that the capabilities and methods of interacting with mobile devices are constantly spreading in different directions, none compatible with the other.  Technically this problem is driven by consumers, since we always want the next cool feature.  What happens is that manufacturers battle it out in an attempt to be the first to release a device that has an edge over the competition, like Near Field Communications and glasses-free 3D Displays.  The problem is that manufacturers release devices with these features before the industry has settled on a standard for interacting with them.

So what does this have to do with WURFL?  Luca Passani and I have seen the writing on the wall and we realize that this problem of device fragmentation has had some temporary relief by the massive market share by Apple and Android devices, but things are starting to get worse fast.  Every day we are seeing new devices and incomplete software features that make mobile web programming a nightmare!  For example, everyone is very excited about HTML5, but how do you know if the device you’re targeting supports the HTML5 features that you need?  One option is to use a high-level JavaScript abstraction framework like jQuery Mobile.  This is very slick on my super fast Snapdragon-powered HTC Inspire 4G, but how will it work on a typical off the shelf phone?  What about BlackBerry OS 6/7, MeeGo, Symbian^3, WebOS and WP7?  Good luck.

What Luca and I decided to do is to join forces and refocus our full attention on the problem.  In June we publicly announced the merger of the WURFL project and the Tera-WURFL project into a single company called ScientiaMobile (original WMLProgramming message).  Now that we’re on the same team, Luca and I are able to work more closely on the problem of device fragmentation and detection.  In addition to better APIs and more consistent data, we’re bringing some awesome new products to market that will make it easier for organizations of all sizes to get started with mobile device detection.  Right now, we are polishing up a product that has always been near the top of my feature requests:  WURFL Cloud.  WURFL Cloud is something that we’ve been secretly working on for some time and it is now in it’s Beta-testing phase.  We are spending a lot of time to make sure this service is very fast and accurate from day 1 (by the way, you can drop us a line at wurfl-cloud-beta@scientiamobile.com to get your Beta license).  Unlike the standard WURFL APIs, WURFL Cloud is a hosted device detection service that is always updated and maintained by the WURFL Team.

Throughout this process, we have chosen to follow our roots.  WURFL has always been a Free and Open Source project and it still is.  We value our community and are excited to bring WURFL to the next level while keeping it in the public domain.  At ScientiaMobile, in addition to the FOSS APIs, we are now selling commercial licenses and support for our APIs as well.  Under this new dual-licensing scheme we are able to help organizations that are not comfortable with the restrictions of the open source license.

As we say at ScientiaMobile: WURFL has Graduated!

If you’re not familiar with HAProxy, you’re missing out! HAProxy is a very intelligent high-availability reverse proxy that operates all the way up to Layer 7. Unlike Nginx (another good choice), HAProxy is not a webserver, it is designed with only high availability and load balancing in mind. Another great feature is that it’s a lot cheaper than your typical Layer 7 Load Balancer, free :).

There are plenty of good articles out there on HAProxy, but I couldn’t find any Debian/Ubuntu packages for the latest version (1.4.15). The Ubuntu 11.04 repo’s newest version is 1.4.8, but if you want some of the cool new features of 1.4.9-1.4.15, you have to build it from scratch. This process isn’t too painful, but I’ve got a lot of servers to put HAProxy on and I don’t want dpkg / apt-get to freak out and install the old HAProxy in order to satisfy a dependency.

For this reason, I’ve created an Ubuntu 11.04 x64 HAProxy package with TCP Splicing, Full Transparent Proxies and PCRE enabled:

HA-Proxy version 1.4.15 2011/04/08
Copyright 2000-2010 Willy Tarreau <w@1wt.eu>

Build options :
  TARGET  = linux26
  CPU     = native
  CC      = gcc
  CFLAGS  = -O2 -march=native -g -fno-strict-aliasing

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Available polling systems :
     sepoll : pref=400,  test result OK
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.

Feel free to grab haproxy_1.4.15_amd64.deb (502k SHA1: 8b2ecf05544e0f6531e50bf40261e0b112db61e9)
You can also grab the sources from here: http://www.stevekamerman.com/haproxy/

Recently the battery in my aging laptop (Sony Vaio VGN-NR498E) took a nose dive.  This is to be expected after 4 years or so, so I was not too upset about it.  What I am upset about, is the fact that Sony doesn’t seem to sell a genuine replacement for the battery.  The original is a VGP-BPS9/B and the only “Sony” batteries I found online were from unreputable sources and cost around $150.  I decided to take the battery apart and see if I could replace the cells directly, but in the process I had to break most of the battery connections and it did not seem trivial to replace them.  After whining about it for a week I opted to buy a third party battery off Buy.com for $59.

Battery Shipment Contents

I received the shipment within 5 days and inside I found the battery, a couple pages of instructions and a mini CD.  There was also a written warning in poorly translated English stating that I needed to update the BIOS in order to use the battery.  It was so poorly translated that I couldn’t even get a loose interpretation of some of the text.  For example, “You’re system may isn’t suitable for this software or has already not renewed at first the bios file of factory”  (side note: is it really that hard to get a fluent English-speaker to proof read your instructions?).  Anyway, the instructions suggested that I may damage the battery and/or system if I didn’t update my BIOS before plugging in the battery, so I popped in the CD which presumably contained the update.  The CD was blank.  I ended up finding the necessary software on a site mentioned in their instructions, but the software gave me a vague warning that my BIOS was not supported and even after I forced the update the software crashed.  At this point I gave up on bricking my laptop via a failed BIOS update and opted to plug it in and hope for the best.  The battery showed about a 40% charge but the laptop said it was not charging it (I don’t haveSony’s anti third-party battery software installed).  After much frustration and trying different combinations of removing and replacing the battery at different times, I decided to take the Steve approach and hack it :).

Original Sony battery (top) and replacement (bottom)

I had already taken my original battery apart and thrown the dead cells away, so I don’t have any pictures of it, but it looked almost exactly the same as the replacement: 6x 3.6V battery cells and a controller card.  Batteries are nothing but a means of storing power, so the controller must be responsible for communicating things like battery capacity and charging status.  I decided to transplant the new battery cells onto the old Sony controller and give it a shot; in short, it worked!  Here’s the process in pictures.

Step 1: Take batteries apart – remove 4 screws and pry top cover off, then gently pry out the batteries and controller card.

Replacement Battery

Step 2: Remove the controller card from the batteries by desoldering the four metal tabs that connect to the different battery junctions.  I used desoldering wick, flux and a hot soldering iron.

A closeup of the new controller (attached), and the original Sony controller next to it.

Step 3: Solder the battery leads to the new controller.

Solder Sony Controller

Finished Controller

Step 4: Reassemble the battery and secure it with some hot glue.

Glue the Batteries and Controller

Finished Battery

Step 5: Once the top cover is replaced, put the battery back in the computer – you’re done!

For best results you will also need to recalibrate the battery by letting it charge completely, then unplugging the power cable and let the battery discharge until it’s completely dead.  I let my new battery charge overnight, then discharged it in the morning.  Incredibly, the new battery kept the laptop alive for over 3 hours – a significant improvement over the 5-10 minutes I was getting before!

If you are going to attempt this, you should know that there is a very real possibility that things could go horribly wrong if, for example, you wired the battery pack in backwards.  You could destroy the charging circuit in your laptop or even melt it down into a heap of plastic and potentially burn down your house in the process.  That having been said, if you know what you’re doing, this is a great way to recover an investment in a low-grade battery!

After a lot of development and tons of regression testing, Tera-WURFL 2.1.4 is finally ready for release. There has finally been a break in commits on the GitHub repo for more that a week, and no bug reports on the development branch.

Changes from Tera-WURFL 2.1.3

  • Switched to better XML Parser (old parser, SimpleXML will still be used if XMLReader is missing)
  • Improved matching for Apple, Android, Nokia, BlackBerry / RIM, DoCoMo, KDDI, LG and OperaMini
  • Allowed preloading a custom TeraWurflConfig so the default is ignored
  • Updated source documentation
  • Updated instructions
  • Greatly improved cmd_line_admin.php
  • Converted line endings to \n
  • Various bugfixes
  • Improved Remote Client
  • Added builtin regression testing (thanks digitalronin!). Tip: from the shell, go to the test/ dir and type “rake sanity” to make sure your patches are compatible with the loaded WURFL
  • Moved default config to “TeraWurflConfig.php.example” so your config isn’t overwritten
  • Fixed MSSQL DB Connector bug

If you rely on Android or BlackBerry detection, you will really want to get version 2.1.4 – it will significantly improve detection of these devices.

Even though RAM is cheap these days, there are some conditions in which your Linux server could run out of it completely. Just the other day, I noticed my main hosting server went down – I could still ping it, but DNS, SSH and Apache2 were not responding, so I had to call the datacenter to have them reboot my system. After analyzing the system, I realized that some unknown process ate up all the memory and all the swap space! I used Cacti to monitor my server’s performance, and so I could see that it took a nose-dive after getting hit with a few million requests in a couple days (these were raw mobile device detection requests).  After the system ran out of memory, it started swapping.  This lasted for about 2 weeks before depleting the swap space, at which point it struggled on for another 18 hours.  At this point, it was critically starved of memory and oom_killer (Out of Memory Killer) was invoked to start killing processes in a vain attempt to free up memory.  The oom_killer seems to have very little intelligence as to which processes to kill first, as sshd and named were early victims.  After this episode, I decided to create a script that adjusts the order in which key processes were killed, to make sure I have access to the server in the event of a memory leak or OOM condition.

oom_adjust.sh to the rescue!

I’ve created oom_adjust.sh to adjust (and periodically readjust via cron) the order in which the processes may be killed by oom_killer.  The script uses a config file called oom_adjust.conf (by default it looks for it in /etc) in which you can list processes and the oom_adj value that you want to give them.  The possible values are from -17 (never kill) to 15 (kill first).


# Adjust process oom_adj values so they are more or less likely to be killed in an oom event
# procname oom_adj

# Keep sshd ALIVE
sshd -17

# DNS is very important to me too
named -8

# I'd prefer that MySQL stays alive, but it's not required
mysqld -1

# Apache2 is a memory hog, but I'll give it a fighting chance
# I'm giving it 0 since the workers will respawn at 0 anyway
apache2 0

# Sphinx search is cool, but I can live without it if an oom occurs
searchd 3

# Memcache is in the same boat as Sphinx search
memcached 3

# I only use mongodb for testing on this server
mongod 5

# It would be nice if smtpd stayed up, so I still get alerts
smtpd 5

# These services can be killed first
pure-ftpd 10
pure-ftpd-mysql 10
snmpd 10
fail2ban-server 10
ntpd 10
authdaemond 10
saslauthd 10
qmgr 10
pickup 10

Here is the main script, which I’ve symlink’d into /usr/sbin for convinience.


# oom_adjust.sh Out of Memory Killer (oom_killer) Priority Adjustment Script
# by Steve Kamerman <stevekamerman@gmail.com>, Jan 2011
# http://www.stevekamerman.com


if [ ! -f $OOM_ADJ_FILE ]; then
        echo "oom_adjust.sh: config file $OOM_ADJ_FILE was not found" >&2
        exit 1

echo "oom_adjust.sh is setting oom_killer priorities"
for LINE in `cat $OOM_ADJ_FILE | sed -e '/^[# \t].*/d' | sed -e '/^$/d' | sed -e 's/ /:/'`; do
        NAME=`echo $LINE | cut -d":" -f1`
        ADJ=`echo $LINE | cut -d":" -f2`
        echo "  Setting $NAME to $ADJ"
        for PID in `pidof $NAME`; do
                echo $ADJ > /proc/$PID/oom_adj
exit 0

If your distro uses /etc/rc.local, you can put call this script there to apply the adjustments on startup.  I also call it on my servers via crontab every night to keep the processes in check, in case they have respawned/restarted with a different PID.

Arduino Duemilanove

Arduino Duemilanove

I recently bought  a 2010 Honda Accord Crosstour and was shocked to see how large it really is in my garage!  When it’s parked correctly, I have about 18 inches behind it and 26 in front of it, when it’s not, I can’t shut the garage door, or I can’t walk in front of it.  As a fun project, I decided to make a device that would let me know when I’m exactly in position!

Enter the Arduino.  The Arduino is an open-source microcontroller board that can programmed with a C-like language called Processing via USB.  It’s a great way to leverage your programming skills to interact with the world around you.  In order to help me park my car, I used an ultrasonic range finder from Radio Shack called the PING))) Parallax.  This device has a transmitter, receiver and a few processing chips on it.  It sends out a signal and listens for the echo, then returns the total time duration in microseconds.  Since sound travels one inch in about 74 microseconds, you can take this time and divide it by 74 to get the total distance that the sound traveled, then divide it by 2 since you only care about the distance too the object, not its return trip.  My plan was to put the range finder in the back of garage and use three LEDs to show me if my car was too far, too close or parked just the right distance away.  I wrote the code below to get a distance reading every 1.5 seconds, and if the distance changed since the last time it was polled, my car (or something in the garage) must have moved.  In this case, the Arduino starts polling for changes every 100ms and lights up an orange LED if it is over 26 inches away, a green LED if it is between 26 and 20 inches and a red LED if it is under 20 inches away.  I mounted the LEDs on a piece of aluminum and attached it to the wall so I could see it from my car while I’m parking.  After I’m finished parking and the Arduino sees that nothing has moved in over 10 seconds, it turns all the LEDs off and starts polling at 1.5 seconds again.  Simple and effective!

// PING))) Sensor
const int pingPin = 7;

// Indicator LEDs
const int tooFarPin = 13;     // Orange LED
const int justRightPin = 12;  // Green LED
const int tooClosePin = 11;   // Red LED

// Distance Thresholds
const int outOfRange = 108;
const int tooFar = 26;
const int tooClose = 20;

// Sensor Polling Delay (ms)
const int idleSenseDelay = 1500;
// High-Speed Polling Timeout
const int idleTimeout = 10000;
int senseDelay = 1500;

/*  States
0 = Out of range (no car present)
1 = Too Far
2 = Just Right
3 = Too Close

int currentState = 0;
int lastState = 0;
unsigned long lastDistanceChange;
long distance;

void setup() {
    lastDistanceChange = millis();
    pinMode(tooFarPin, OUTPUT);
    digitalWrite(tooFarPin, LOW);
    pinMode(justRightPin, OUTPUT);
    digitalWrite(justRightPin, LOW);
    pinMode(tooClosePin, OUTPUT);
    digitalWrite(tooClosePin, LOW);

void loop() {
    long duration, newDistance;

    // The PING))) is triggered by a HIGH pulse of 2 or more microseconds.
    // Give a short LOW pulse beforehand to ensure a clean HIGH pulse:
    pinMode(pingPin, OUTPUT);
    digitalWrite(pingPin, LOW);
    digitalWrite(pingPin, HIGH);
    digitalWrite(pingPin, LOW);

    // The same pin is used to read the signal from the PING))): a HIGH
    // pulse whose duration is the time (in microseconds) from the sending
    // of the ping to the reception of its echo off of an object.
    pinMode(pingPin, INPUT);
    duration = pulseIn(pingPin, HIGH);
    newDistance = duration / 74 / 2;
    if (distance != newDistance) {
        // Something is moving, increase polling rate
        lastDistanceChange = millis();
        senseDelay = 100;
    distance = newDistance;
    currentState = getState(distance);
    if (currentState != lastState) {
        // State changed
    } else {
        if (senseDelay != idleSenseDelay && (millis() - lastDistanceChange > idleTimeout)) {
            // Idle Timeout has Passed, turn off LEDs
            digitalWrite(tooFarPin, LOW);
            digitalWrite(justRightPin, LOW);
            digitalWrite(tooClosePin, LOW);
            senseDelay = idleSenseDelay;
    lastState = currentState;

int getState(long distance) {
    if (distance >= outOfRange) { return 0; }
    if (distance >= tooFar) { return 1; }
    if (distance <= tooClose) { return 3; }
    return 2;

void indicateState() {
    // Turn off the LEDs that are on
    switch (lastState) {
        case 1:
            digitalWrite(tooFarPin, LOW);
        case 2:
            digitalWrite(justRightPin, LOW);
        case 3:
            digitalWrite(tooClosePin, LOW);
        case 0:
        case 1:
            digitalWrite(tooFarPin, HIGH);
        case 2:
            digitalWrite(justRightPin, HIGH);
        case 3:
            digitalWrite(tooClosePin, HIGH);

And here it is, the finished product!

Fritzing Sketch

PING))) Parallax

Arduino ParkAssist

LED Indicator

I’ve been working hard on Tera-WURFL 2.1.3 lately.  I’ve been meaning to release it for over a month now, the problem is that I just keep finding improvements to make.  Since Tera-WURFL’s development is now hosted on GitHub, you can always see what I’m doing and grab a copy for yourself.  New in version 2.1.3 is a MongoDB Database Connector.  I keep getting pounded by people about NoSQL and I haven’t given it a fair chance until now.  Simon Harris from www.pointbeing.net was nice enough to contribute the initial MongoDB connector, and I’ve spent a few days working on it and testing it against MySQL5.  At first the performance seems outstanding – almost double that of MySQL5 – but I noticed some inconsistencies with my tests, so I built a benchmarking script and an comparison tool that automatically tests one installation against another and checks for consistency.  I’ve tested three different database connectors against each other and my memcached-based Tera-WURFL Enterprise.

Testing Setup

In order to test the Database Connectors, I brought up a virtual machine on an under-utilized ESXi server (dual 6-core Opterons and 32GB FB-DDR3).

Virtual Machine Specs:

  • 4x Opteron 2.2GHz cores
  • 12GB FB-DDR3 RAM
  • 20GB of storage on a SAS array over MPIO iSCSI
  • Ubuntu Server 10.04.1
  • MySQL 5.1
  • MongoDB 1.7.0

This machine doesn’t have extremely fast CPUs, just a lot of them.  At any rate, the specs were consistent so I was able to get a reasonable test result.  In order to test the performance of the DBs, I ran about 66,000 unique user agents (these are UAs that are not easily matched and create a lot of DB load) through the connectors and measured the rate at which they detected them.

The results

The following results are the average speed at which the different DB Connectors detected the test user agents.  The two graphs show the difference between uncached and cached detections.  For each database, I ran the test with the all the WURFL capabilities loaded (blue) and with only [‘product_info’][‘is_wireless_device’] loaded (red) to get an idea of the performance benefit of using Tera-WURFL’s CAPABILITY_FILTER option.

Uncached Device Detection Performance in Tera-WURFL 2.1.3

* note: memcached is not capable of serving uncached detections, so it is left out of this test.

Cached Device Detection Performance in Tera-WURFL 2.1.3


I was very surprised by these results so I tested them repeatedly just to be sure.  I was so excited that MongoDB would be the clear winner here, but it just isn’t the case for uncached detections.  Mongo did fare well when serving the cached full capabilities – 63% higher than MySQL5.  MS SQL Server 2005 was particularly painful to test.  By comparison, MongoDB took about 8 minutes to churn though all 66,000 user agents, but my initial test with MS SQL Server was 3 hours and 8 minutes.  Since I tested all the other connectors on Ubuntu Server, and the MS SQL Server connector requires Microsoft’s PHP SQL Server Driver, I had to run the tests on a different VM.  On this VM I was getting between 1-3 uncached detections per second, but I moved the testing to my development laptop (2.8GHz Core 2 Duo, 4GB RAM and a Samsung 250GB SSD) and was able to achieve better results.  MySQL fared very well overall and remains at the top of the list.  I have not mentioned the MySQL Nested Set connector before, but it’s been included in Tera-WURFL for a couple versions now.  It just extends the MySQL5 connector and adds right and left values to all of the devices in the WURFL so it can return the complete fallback tree for a given WURFL ID in a single indexed query.  Unfortunately, this seems to provide a negligible improvement in speed.  I would still recommend it for environments where the webserver is on a different host than the database.

Look for Tera-WURFL 2.1.3 to be released within a couple weeks.  I’m trying to get some unit-testing code integrated before I release it.

*** UPDATE ***

I’ve retested MongoDB and MySQL5 again on my production server against a typical distribution of 44887 user agents.  The performance was much better, but the MySQL <-> MongoDB performance was proportional to the first results:

Uncached Cached
All Capabilities Min Capabilities All Capabilities Min Capabilities
MySQL 303 510 2735 7740
MongoDB 195 255 3182 6154

A couple months ago I switched my version control system for PHP development from CVS to Git and moved the Tera-WURFL development tree to the public repository Github. Since then I had been using an early version of the Eclipse EGit plugin (I think it was 0.8.1, but I can’t be sure). The plugin was only partially implemented at the time and I used it for committing and pushing only. Other tasks like tagging, branching and merging we difficult or impossible. Today I updated it to 0.8.4 (still in it’s incubation) and I am pleased to see that it is much improved! Most of the functionality of Git is available right from the Eclipse IDE. The icons and windows have been reworked and are much more intuitive now. Also, many of the settings are stored in the standard .git respository, so if you are using the standard Git command-line interface, both systems are 100% compatible! Good work EGit team – thanks for making my life much easier!

Last night I finished a great book called Physics of the Impossible by Michio Kaku, a leading theoretical physicist.  As I was trying to get to sleep I was thinking about the relationship of space, time and gravity and came to some interesting conclusions.  This is my very infant theory of gravity’s relationship to space-time.

Picture a swimming pool full of water.  If you blew a bubble in the water you displace a very small amount of water with air, giving the water more potential energy.  On earth, since we have gravity, it is easy for the water to get back to its lower state of energy by pushing the bubble up to the surface since the air is less dense than the water, and the water gets more dense as you get closer to the bottom.  In space, if the water was constrained so it could not expand, the water would still want to push the bubble out, but there is no obvious direction for it to go since the density of water is fairly consistent.  Now, if space-time is like this water in space, its constraints are the edges of space and time, beyond which there is nothing (even if there was something we could not detect it because it would not follow our laws of physics).  Each subatomic particle with mass would be surrounded by a bubble in this water.  The bubble is the sphere of influence that it has on space-time, so matter itself is displacing space-time.  As a result of this displacement, space time “pushes back” at the particle resulting in gravity.  If you push matter together until it is very dense it will eventually fuse, creating other element(s).  If you keep forcing it with more and more energy, eventually, the subatomic particle’s spheres of influence (the bubbles) would touch and combine into one.  On a very small scale, this would result in a microscopic black hole, but since space-time is not pushing back very hard on this structure, it is unstable and it disintegrates back into its constituent particles.  On a large scale, there would be trillions and trillions of these particles, and they would create a very large sphere of influence.  In fact, we already have a name for this – it’s called the event horizon of a black hole.

If you assume this conceptual view of gravity is correct, then you can draw some pretty interesting conclusions.

  1. Gravity is a measurement of the potential energy of space-time that has been displaced by a mass.
  2. We cannot see inside a subatomic particle since space and time don’t exist or are so distorted that they would result in chaotic observations.
  3. When matter is in motion, it leaves a “wake” of space-time behind it as it is restored to an equilibrium.  This wake is visible in a particle accelerator – when particles feel their wake of their own gravity and are pulled into a tight circle.
  4. Perhaps the displacement of space-time itself is what is pushing the universe out in all directions (but it appears to be accelerating which would not  coincide).

I will continue to think about this concept and figure out what is logical and what is illogical.  Ideally someone in this world can integrate this “symptom of gravity” into the existing theories and put this whole gravity mystery to bed!

If you have an opinion about this concept, please comment on it!  I am not a theoretically physicist so I may just be way off in left field!