Latest Entries »

There’s been a lot of talk about WURFL lately and with good reason. On June 6, 2011 I launched ScientiaMobile along with my business partners, Luca Passani and Krishna Guda.  If you’re reading my blog you probably know that I am the author of the popular mobile device detection library Tera-WURFL.  I wrote Tera-WURFL back in 2006, when the world of mobile devices was completely scattered in every direction.  At the time, the vast majority of mobile devices did not support a common set of capabilities, like MP3, MMS or HTML.  Instead, your best bet was to write a terrible-looking, but fairly well behaved XHTML-MP site using very conservative tags.  WURFL solved this problem nicely by accurately matching a visiting user to a device profile that allowed you to serve a WAP page to a Nokia 3360, an XHTML-MP page to a Motorola RAZR and an HTML page to a 1st-gen iPhone.

Since 2006 there have been a lot of changes, particularly in the capabilities of mobile browsers.  Thanks to Apple Webkit, even lower-end phone are able to browse sites without catastrophic rendering issues (still a terrible UX, just not a catastrophe).  With all these changes, however, one thing has remained: mobile device fragmentation.  Device fragmentation means that the capabilities and methods of interacting with mobile devices are constantly spreading in different directions, none compatible with the other.  Technically this problem is driven by consumers, since we always want the next cool feature.  What happens is that manufacturers battle it out in an attempt to be the first to release a device that has an edge over the competition, like Near Field Communications and glasses-free 3D Displays.  The problem is that manufacturers release devices with these features before the industry has settled on a standard for interacting with them.

So what does this have to do with WURFL?  Luca Passani and I have seen the writing on the wall and we realize that this problem of device fragmentation has had some temporary relief by the massive market share by Apple and Android devices, but things are starting to get worse fast.  Every day we are seeing new devices and incomplete software features that make mobile web programming a nightmare!  For example, everyone is very excited about HTML5, but how do you know if the device you’re targeting supports the HTML5 features that you need?  One option is to use a high-level JavaScript abstraction framework like jQuery Mobile.  This is very slick on my super fast Snapdragon-powered HTC Inspire 4G, but how will it work on a typical off the shelf phone?  What about BlackBerry OS 6/7, MeeGo, Symbian^3, WebOS and WP7?  Good luck.

What Luca and I decided to do is to join forces and refocus our full attention on the problem.  In June we publicly announced the merger of the WURFL project and the Tera-WURFL project into a single company called ScientiaMobile (original WMLProgramming message).  Now that we’re on the same team, Luca and I are able to work more closely on the problem of device fragmentation and detection.  In addition to better APIs and more consistent data, we’re bringing some awesome new products to market that will make it easier for organizations of all sizes to get started with mobile device detection.  Right now, we are polishing up a product that has always been near the top of my feature requests:  WURFL Cloud.  WURFL Cloud is something that we’ve been secretly working on for some time and it is now in it’s Beta-testing phase.  We are spending a lot of time to make sure this service is very fast and accurate from day 1 (by the way, you can drop us a line at wurfl-cloud-beta@scientiamobile.com to get your Beta license).  Unlike the standard WURFL APIs, WURFL Cloud is a hosted device detection service that is always updated and maintained by the WURFL Team.

Throughout this process, we have chosen to follow our roots.  WURFL has always been a Free and Open Source project and it still is.  We value our community and are excited to bring WURFL to the next level while keeping it in the public domain.  At ScientiaMobile, in addition to the FOSS APIs, we are now selling commercial licenses and support for our APIs as well.  Under this new dual-licensing scheme we are able to help organizations that are not comfortable with the restrictions of the open source license.

As we say at ScientiaMobile: WURFL has Graduated!

If you’re not familiar with HAProxy, you’re missing out! HAProxy is a very intelligent high-availability reverse proxy that operates all the way up to Layer 7. Unlike Nginx (another good choice), HAProxy is not a webserver, it is designed with only high availability and load balancing in mind. Another great feature is that it’s a lot cheaper than your typical Layer 7 Load Balancer, free :).

There are plenty of good articles out there on HAProxy, but I couldn’t find any Debian/Ubuntu packages for the latest version (1.4.15). The Ubuntu 11.04 repo’s newest version is 1.4.8, but if you want some of the cool new features of 1.4.9-1.4.15, you have to build it from scratch. This process isn’t too painful, but I’ve got a lot of servers to put HAProxy on and I don’t want dpkg / apt-get to freak out and install the old HAProxy in order to satisfy a dependency.

For this reason, I’ve created an Ubuntu 11.04 x64 HAProxy package with TCP Splicing, Full Transparent Proxies and PCRE enabled:

HA-Proxy version 1.4.15 2011/04/08
Copyright 2000-2010 Willy Tarreau <w@1wt.eu>

Build options :
  TARGET  = linux26
  CPU     = native
  CC      = gcc
  CFLAGS  = -O2 -march=native -g -fno-strict-aliasing
  OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Available polling systems :
     sepoll : pref=400,  test result OK
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.

Feel free to grab haproxy_1.4.15_amd64.deb (502k SHA1: 8b2ecf05544e0f6531e50bf40261e0b112db61e9)

You can also grab the sources from here: http://www.stevekamerman.com/haproxy/

Recently the battery in my aging laptop (Sony Vaio VGN-NR498E) took a nose dive.  This is to be expected after 4 years or so, so I was not too upset about it.  What I am upset about, is the fact that Sony doesn’t seem to sell a genuine replacement for the battery.  The original is a VGP-BPS9/B and the only “Sony” batteries I found online were from unreputable sources and cost around $150.  I decided to take the battery apart and see if I could replace the cells directly, but in the process I had to break most of the battery connections and it did not seem trivial to replace them.  After whining about it for a week I opted to buy a third party battery off Buy.com for $59.

Battery Shipment Contents

I received the shipment within 5 days and inside I found the battery, a couple pages of instructions and a mini CD.  There was also a written warning in poorly translated English stating that I needed to update the BIOS in order to use the battery.  It was so poorly translated that I couldn’t even get a loose interpretation of some of the text.  For example, “You’re system may isn’t suitable for this software or has already not renewed at first the bios file of factory”  (side note: is it really that hard to get a fluent English-speaker to proof read your instructions?).  Anyway, the instructions suggested that I may damage the battery and/or system if I didn’t update my BIOS before plugging in the battery, so I popped in the CD which presumably contained the update.  The CD was blank.  I ended up finding the necessary software on a site mentioned in their instructions, but the software gave me a vague warning that my BIOS was not supported and even after I forced the update the software crashed.  At this point I gave up on bricking my laptop via a failed BIOS update and opted to plug it in and hope for the best.  The battery showed about a 40% charge but the laptop said it was not charging it (I don’t haveSony’s anti third-party battery software installed).  After much frustration and trying different combinations of removing and replacing the battery at different times, I decided to take the Steve approach and hack it :).

Original Sony battery (top) and replacement (bottom)

I had already taken my original battery apart and thrown the dead cells away, so I don’t have any pictures of it, but it looked almost exactly the same as the replacement: 6x 3.6V battery cells and a controller card.  Batteries are nothing but a means of storing power, so the controller must be responsible for communicating things like battery capacity and charging status.  I decided to transplant the new battery cells onto the old Sony controller and give it a shot; in short, it worked!  Here’s the process in pictures.

Step 1: Take batteries apart – remove 4 screws and pry top cover off, then gently pry out the batteries and controller card.

Replacement Battery

Step 2: Remove the controller card from the batteries by desoldering the four metal tabs that connect to the different battery junctions.  I used desoldering wick, flux and a hot soldering iron.

A closeup of the new controller (attached), and the original Sony controller next to it.

Step 3: Solder the battery leads to the new controller.

Solder Sony Controller

Finished Controller

Step 4: Reassemble the battery and secure it with some hot glue.

Glue the Batteries and Controller

Finished Battery

Step 5: Once the top cover is replaced, put the battery back in the computer – you’re done!

For best results you will also need to recalibrate the battery by letting it charge completely, then unplugging the power cable and let the battery discharge until it’s completely dead.  I let my new battery charge overnight, then discharged it in the morning.  Incredibly, the new battery kept the laptop alive for over 3 hours – a significant improvement over the 5-10 minutes I was getting before!

If you are going to attempt this, you should know that there is a very real possibility that things could go horribly wrong if, for example, you wired the battery pack in backwards.  You could destroy the charging circuit in your laptop or even melt it down into a heap of plastic and potentially burn down your house in the process.  That having been said, if you know what you’re doing, this is a great way to recover an investment in a low-grade battery!

After a lot of development and tons of regression testing, Tera-WURFL 2.1.4 is finally ready for release. There has finally been a break in commits on the GitHub repo for more that a week, and no bug reports on the development branch.

Changes from Tera-WURFL 2.1.3

  • Switched to better XML Parser (old parser, SimpleXML will still be used if XMLReader is missing)
  • Improved matching for Apple, Android, Nokia, BlackBerry / RIM, DoCoMo, KDDI, LG and OperaMini
  • Allowed preloading a custom TeraWurflConfig so the default is ignored
  • Updated source documentation
  • Updated instructions
  • Greatly improved cmd_line_admin.php
  • Converted line endings to \n
  • Various bugfixes
  • Improved Remote Client
  • Added builtin regression testing (thanks digitalronin!). Tip: from the shell, go to the test/ dir and type “rake sanity” to make sure your patches are compatible with the loaded WURFL
  • Moved default config to “TeraWurflConfig.php.example” so your config isn’t overwritten
  • Fixed MSSQL DB Connector bug

If you rely on Android or BlackBerry detection, you will really want to get version 2.1.4 – it will significantly improve detection of these devices.

Even though RAM is cheap these days, there are some conditions in which your Linux server could run out of it completely. Just the other day, I noticed my main hosting server went down – I could still ping it, but DNS, SSH and Apache2 were not responding, so I had to call the datacenter to have them reboot my system. After analyzing the system, I realized that some unknown process ate up all the memory and all the swap space! I used Cacti to monitor my server’s performance, and so I could see that it took a nose-dive after getting hit with a few million requests in a couple days (these were raw mobile device detection requests).  After the system ran out of memory, it started swapping.  This lasted for about 2 weeks before depleting the swap space, at which point it struggled on for another 18 hours.  At this point, it was critically starved of memory and oom_killer (Out of Memory Killer) was invoked to start killing processes in a vain attempt to free up memory.  The oom_killer seems to have very little intelligence as to which processes to kill first, as sshd and named were early victims.  After this episode, I decided to create a script that adjusts the order in which key processes were killed, to make sure I have access to the server in the event of a memory leak or OOM condition.

oom_adjust.sh to the rescue!

I’ve created oom_adjust.sh to adjust (and periodically readjust via cron) the order in which the processes may be killed by oom_killer.  The script uses a config file called oom_adjust.conf (by default it looks for it in /etc) in which you can list processes and the oom_adj value that you want to give them.  The possible values are from -17 (never kill) to 15 (kill first).

oom_adjust.conf

# Adjust process oom_adj values so they are more or less likely to be killed in an oom event
# procname oom_adj

# Keep sshd ALIVE
sshd -17

# DNS is very important to me too
named -8

# I'd prefer that MySQL stays alive, but it's not required
mysqld -1

# Apache2 is a memory hog, but I'll give it a fighting chance
# I'm giving it 0 since the workers will respawn at 0 anyway
apache2 0

# Sphinx search is cool, but I can live without it if an oom occurs
searchd 3

# Memcache is in the same boat as Sphinx search
memcached 3

# I only use mongodb for testing on this server
mongod 5

# It would be nice if smtpd stayed up, so I still get alerts
smtpd 5

# These services can be killed first
pure-ftpd 10
pure-ftpd-mysql 10
snmpd 10
fail2ban-server 10
ntpd 10
authdaemond 10
saslauthd 10
qmgr 10
pickup 10

Here is the main script, which I’ve symlink’d into /usr/sbin for convinience.
oom_adjust.sh

#!/bin/sh

# oom_adjust.sh Out of Memory Killer (oom_killer) Priority Adjustment Script
# by Steve Kamerman <stevekamerman@gmail.com>, Jan 2011
# http://www.stevekamerman.com

OOM_ADJ_FILE=/etc/oom_adjust.conf

if [ ! -f $OOM_ADJ_FILE ]; then
        echo "oom_adjust.sh: config file $OOM_ADJ_FILE was not found" >&2
        exit 1
fi

echo "oom_adjust.sh is setting oom_killer priorities"
for LINE in `cat $OOM_ADJ_FILE | sed -e '/^[# \t].*/d' | sed -e '/^$/d' | sed -e 's/ /:/'`; do
        NAME=`echo $LINE | cut -d":" -f1`
        ADJ=`echo $LINE | cut -d":" -f2`
        echo "  Setting $NAME to $ADJ"
        for PID in `pidof $NAME`; do
                echo $ADJ > /proc/$PID/oom_adj
        done
done
exit 0

If your distro uses /etc/rc.local, you can put call this script there to apply the adjustments on startup.  I also call it on my servers via crontab every night to keep the processes in check, in case they have respawned/restarted with a different PID.

Having been rather addicted to understanding electronics lately, I’m always looking for something to play with. Last week I was about to throw out a broken wireless Microsoft mouse, but I caught myself and ended up taking it apart.

Rotary Encoder from a Mouse

I dug down to the scroll wheel and found out it was a nice little rotary encoder – there’s an LED pointed at two phototransistors (light detectors) in it, as well as a circle of tiny black posts that pass between the LED and the detectors, as you can see to the left.  When the posts pass in front of the detectors, the Arduino reads them and is able to determine which direction they are going and increments or decrements a variable that I can access.

It took me about 4 hours to figure out how to hook this thing up, so I figured I would share the info in case someone else wants to use a rotary encoder from a Microsoft mouse in their project.


If you are holding the encoder so that the rotating half is facing you and the wires are facing down, the pinout is as follows:

Pin 1: Photo Sensor 1
Pin 2: Vcc
Pin 3: Photo Sensor 2
Pin 4: IR LED +
Pin 5: IR LED -

Unlike most of the examples I found for using a rotary encoder, this one requires +Vcc on Pin 2 – power will not flow the other way.  The LED power is very picky too;  I am powering it from the 5v regulated supply on my Arduino through a 1k resistor to keep the right amount of light in the encoder.  Too much light and both sensors will be high for too long and too little, neither will go high.  The hardest part to figure out was the voltage divider circuit for the sensors.  I could get them working ok on an analog input – swinging between about 80-600, but it wasn’t clean enough to input as digital until I found the right resistor: 10k.  This nicely divides the input between ground and +5v so the encoder can be read.

Here is the Arduino sketch for the project as shown in the video above:

// Arduino Rotary Encoder from Mouse Scroll Wheel
// and SparkFun 4 Digit Serial 7-Segment Display
// by Steve Kamerman 12/18/2010
// http://www.stevekamerman.com/2010/12/understanding-a-mouse-scroll-wheel/

// These MUST be on interrupt pins!
#define encoderPinA 2
#define encoderPinB 3

#define displayTXPin 10
#define displayRXPin 11

// used only for sprintf()
#include <stdio.h>

// If you don't have NewSoftSerial, grab it from:
// http://arduiniana.org/libraries/NewSoftSerial/
#include <NewSoftSerial.h>
NewSoftSerial displaySerial(displayRXPin, displayTXPin);

// the value - this must be volatile because it is modified
// during an interrupt function and used in a normal function
volatile unsigned int encoderPos = 0;

void setup() {
  // setup the encoder
  pinMode(encoderPinA, INPUT);
  pinMode(encoderPinB, INPUT);
  attachInterrupt(0, readEncoderA, CHANGE);
  attachInterrupt(1, readEncoderB, CHANGE);  
 
  // setup the display
  pinMode(displayTXPin, OUTPUT);
  displaySerial.begin(9600);
  // reset display
  displaySerial.print("v");
  // Set display brightness
  displaySerial.print(0x7A, BYTE);
  displaySerial.print(0x05, BYTE);
  // set the display to "   0"
  displaySerial.print("xxx0");
}
void loop(){
  displayNumber(encoderPos);
}

// prints the given number, right justified padded with spaces
// sprtinf() is not a great idea if you want to save space
// but I'm really not too worried about it :)
void displayNumber(int num){
  char buf[4];
  sprintf(buf, "%4d", num);
  displaySerial.print(buf);
}

void readEncoderA(){
  if (digitalRead(encoderPinA) == HIGH){
    encoderPos += (digitalRead(encoderPinB) == LOW)? 1: -1;
  }else{
    encoderPos += (digitalRead(encoderPinB) == HIGH)? 1: -1;
  }
}
void readEncoderB(){
  if (digitalRead(encoderPinB) == HIGH){
    encoderPos += (digitalRead(encoderPinA) == HIGH)? 1: -1;
  }else{
    encoderPos += (digitalRead(encoderPinA) == LOW)? 1: -1;
  }
}

Yes, I’m still on the electronics kick. This is something I’ve been wanting to do as part of a complete home automation project down the road: monitor the temperature of the different rooms / zones in my house. Now that I’ve got a few Arduinos laying around and some thermal probes (thermistors) from some broken Radio Shack temperature displays, I’ve got the hardware necessary to make it happen! I first installed the thermal probes in the ceilings of the master bedroom, my daughter’s bedroom and our living room in locations that I thought were least affected by vents, fans, lights and other sources of heat. Here’s a picture of the probe in my living room:

Thermal Probe in my Living Room

Next, I connected the three probes to my Arduino’s Analog Inputs using the thermistors as voltage dividers between +5VDC and Ground.  In order to detect the temperatures and make them accessible via the Arduino’s serial interface, I used a few examples on the Arduino site as the basis for this program, which waits for a character to be sent to the serial port, then outputs the temperatures:

#include <math.h>
//Schematic:
// [Ground] ---- [10k-Resister] -------|------- [Thermistor] ---- [+5v]
//                                     |
//                                Analog Pin 0

double Thermistor(int RawADC) {
 // Inputs ADC Value from Thermistor and outputs Temperature in Celsius
 //  requires: include <math.h>
 // Utilizes the Steinhart-Hart Thermistor Equation:
 //    Temperature in Kelvin = 1 / {A + B[ln(R)] + C[ln(R)]^3}
 //    where A = 0.001129148, B = 0.000234125 and C = 8.76741E-08
 long Resistance;  double Temp;
 Resistance=((10240000/RawADC) - 10000);
 Temp = log(Resistance);
 Temp = 1 / (0.001129148 + (0.000234125 * Temp) + (0.0000000876741 * Temp * Temp * Temp));
 // Convert Kelvin to Celsius
 Temp = Temp - 273.15;
 // Convert to Fahrenheit
 Temp = (Temp * 9.0)/ 5.0 + 32.0;
 return Temp;
}

void printDouble(double val, byte precision) {
  Serial.print (int(val));  //prints the int part
  if( precision > 0) {
    Serial.print("."); // print the decimal point
    unsigned long frac, mult = 1;
    byte padding = precision -1;
    while(precision--) mult *=10;
    if(val >= 0) frac = (val - int(val)) * mult; else frac = (int(val) - val) * mult;
    unsigned long frac1 = frac;
    while(frac1 /= 10) padding--;
    while(padding--) Serial.print("0");
    Serial.print(frac,DEC) ;
  }
}

void setup() {
  Serial.begin(115200);
}

#define ZoneOne 1   // Analog Pin 0
#define ZoneTwo 2   // Analog Pin 1
#define ZoneThree 3   // Analog Pin 2

double temp;
int inByte = 0;
int sensorDelay = 10;

void loop() {
  if (Serial.available() > 0) {
    inByte = Serial.read();
    Serial.print("Zone1:");
    printDouble(Thermistor(analogRead(ZoneOne)),3);
    Serial.print(" ");
    delay(sensorDelay);
    Serial.print("Zone2:");
    printDouble(Thermistor(analogRead(ZoneTwo)),3);
    Serial.print(" ");
    delay(sensorDelay);
    Serial.print("Zone3:");
    printDouble(Thermistor(analogRead(ZoneThree)),3);
    Serial.print(" ");

    Serial.println("");
  }
}

I had an old laptop with a dead battery that I wasn’t using, so I loaded it with Ubuntu 10.10 x64, Apache, MySQL, PHP and Cacti and put it up in the attic, connected to the Arduino via USB.  I went through the painfully dry Cacti Documentation and wrote a custom Perl script that Cacti uses to poll the Arduino for temperatures:

#!/usr/bin/perl

use Device::SerialPort;
use Time::HiRes qw( usleep );

# Serial Settings
$PORT = "/dev/ttyUSB0";
$ob = Device::SerialPort->new ($PORT) || die "Can't Open $PORT: $!";
$ob->baudrate(115200)   || die "failed setting baudrate";
$ob->parity("none")    || die "failed setting parity";
$ob->databits(8)       || die "failed setting databits";
$ob->handshake("none") || die "failed setting handshake";
$ob->write_settings    || die "no settings";
$ob->read_const_time(50);

while(1){
        $ob->write("x");
        if($_ = $ob->read(255)){
                chomp;
                print "$_\n";
                undef $ob;
                exit;
        }
        usleep(1000 * 10);
}

undef $ob;

Cacti Graph of my Home's Temperature

Lastly, I put a nice graph together that shows a comparison of the three temperatures over time, with a composite average of all three.  Now I can see that my daughter’s room spikes to nearly 90 deg when the heat comes on – ouch!  Using this data I am better able to adjust the vents in each room to keep the temperature consistent throughout the house.  Next I want to take one of those cheap LCD picture displays and tie it to the Cacti graph and put it near the thermostat so I can think twice about turning on the heat or AC!


Arduino Duemilanove

Arduino Duemilanove

I recently bought  a 2010 Honda Accord Crosstour and was shocked to see how large it really is in my garage!  When it’s parked correctly, I have about 18 inches behind it and 26 in front of it, when it’s not, I can’t shut the garage door, or I can’t walk in front of it.  As a fun project, I decided to make a device that would let me know when I’m exactly in position!

Enter the Arduino.  The Arduino is an open-source microcontroller board that can programmed with a C-like language called Processing via USB.  It’s a great way to leverage your programming skills to interact with the world around you.  In order to help me park my car, I used an ultrasonic range finder from Radio Shack called the PING))) Parallax.  This device has a transmitter, receiver and a few processing chips on it.  It sends out a signal and listens for the echo, then returns the total time duration in microseconds.  Since sound travels one inch in about 74 microseconds, you can take this time and divide it by 74 to get the total distance that the sound traveled, then divide it by 2 since you only care about the distance too the object, not its return trip.  My plan was to put the range finder in the back of garage and use three LEDs to show me if my car was too far, too close or parked just the right distance away.  I wrote the code below to get a distance reading every 1.5 seconds, and if the distance changed since the last time it was polled, my car (or something in the garage) must have moved.  In this case, the Arduino starts polling for changes every 100ms and lights up an orange LED if it is over 26 inches away, a green LED if it is between 26 and 20 inches and a red LED if it is under 20 inches away.  I mounted the LEDs on a piece of aluminum and attached it to the wall so I could see it from my car while I’m parking.  After I’m finished parking and the Arduino sees that nothing has moved in over 10 seconds, it turns all the LEDs off and starts polling at 1.5 seconds again.  Simple and effective!

// PING))) Sensor
const int pingPin = 7;

// Indicator LEDs
const int tooFarPin = 13;     // Orange LED
const int justRightPin = 12;  // Green LED
const int tooClosePin = 11;   // Red LED

// Distance Thresholds
const int outOfRange = 108;
const int tooFar = 26;
const int tooClose = 20;

// Sensor Polling Delay (ms)
const int idleSenseDelay = 1500;
// High-Speed Polling Timeout
const int idleTimeout = 10000;
int senseDelay = 1500;

/*  States
0 = Out of range (no car present)
1 = Too Far
2 = Just Right
3 = Too Close
*/

int currentState = 0;
int lastState = 0;
unsigned long lastDistanceChange;
long distance;

void setup() {
    lastDistanceChange = millis();
    pinMode(tooFarPin, OUTPUT);
    digitalWrite(tooFarPin, LOW);
    pinMode(justRightPin, OUTPUT);
    digitalWrite(justRightPin, LOW);
    pinMode(tooClosePin, OUTPUT);
    digitalWrite(tooClosePin, LOW);
}

void loop() {
    long duration, newDistance;

    // The PING))) is triggered by a HIGH pulse of 2 or more microseconds.
    // Give a short LOW pulse beforehand to ensure a clean HIGH pulse:
    pinMode(pingPin, OUTPUT);
    digitalWrite(pingPin, LOW);
    delayMicroseconds(2);
    digitalWrite(pingPin, HIGH);
    delayMicroseconds(5);
    digitalWrite(pingPin, LOW);

    // The same pin is used to read the signal from the PING))): a HIGH
    // pulse whose duration is the time (in microseconds) from the sending
    // of the ping to the reception of its echo off of an object.
    pinMode(pingPin, INPUT);
    duration = pulseIn(pingPin, HIGH);
    newDistance = duration / 74 / 2;
    if (distance != newDistance) {
        // Something is moving, increase polling rate
        lastDistanceChange = millis();
        senseDelay = 100;
    }
    distance = newDistance;
    currentState = getState(distance);
    if (currentState != lastState) {
        // State changed
        indicateState();
    } else {
        if (senseDelay != idleSenseDelay && (millis() - lastDistanceChange > idleTimeout)) {
            // Idle Timeout has Passed, turn off LEDs
            digitalWrite(tooFarPin, LOW);
            digitalWrite(justRightPin, LOW);
            digitalWrite(tooClosePin, LOW);
            senseDelay = idleSenseDelay;
        }
    }
    lastState = currentState;
    delay(senseDelay);
}

int getState(long distance) {
    if (distance >= outOfRange) { return 0; }
    if (distance >= tooFar) { return 1; }
    if (distance <= tooClose) { return 3; }
    return 2;
}

void indicateState() {
    // Turn off the LEDs that are on
    switch (lastState) {
        case 1:
            digitalWrite(tooFarPin, LOW);
            break;
        case 2:
            digitalWrite(justRightPin, LOW);
            break;
        case 3:
            digitalWrite(tooClosePin, LOW);
            break;
    }
    switch(currentState){
        case 0:
            break;
        case 1:
            digitalWrite(tooFarPin, HIGH);
            break;
        case 2:
            digitalWrite(justRightPin, HIGH);
            break;
        case 3:
            digitalWrite(tooClosePin, HIGH);
            break;
    }
}

And here it is, the finished product!

Fritzing Sketch

PING))) Parallax

Arduino ParkAssist

LED Indicator

I’ve been delaying the Tera-WURFL 2.1.3 release since mid-July because of the large number of feature requests and improvements coming in, but the time has come to release it. I’m still waiting for the unit test code to stabilize a bit before I put it into the tree, but if you are interested in it, follow me on GitHub and you should see it soon.

Here’s a short list of features and improvements in Tera-WURFL 2.1.3:

  • Native support for MongoDB
  • Better support and performance for Microsoft SQL Server
  • Command line administration utility
  • Better webservice performance using JSON
  • Better Python client with JSON support
  • Improved performance in MySQL5
  • Device Images are available via the PHP webservice client
  • Loaded WURFL version is now tracked in the DB
  • MySQL connector allows for alternate port
  • Better overall detection

If you’re a command line junky like me, I think you’ll really like the CLI administration tool.  To use it, just go to your Tera-WURFL/admin/ directory and type php cmd_line_admin.php

# php cmd_line_admin.php

Tera-WURFL Stable 2.1.3
The command line WURFL updater for Tera-WURFL
Loaded WURFL: www.wurflpro.com - 2010-09-09 04:08:06
Last Updated: Wed, 15 Sep 2010 20:38:59 -0500
---------------------------------------
Usage: php cmd_line_admin.php [OPTIONS]

Option                     Meaning
--help                    Show this message
--update=   The source of the WURFL file:
Update from your local wurfl.xml file:
--update=local
Update from wurfl.sourceforge.net:
--update=remote
--clearCache              Clear the device cache
--rebuildCache            Rebuild the device cache by redetecting all
cached devices using the current WURFL
--stats                   Show statistics about the Tera-WURFL Database

You can update the WURFL by passing the —-update=local or –update=remote parameter (crontab will like this):

# php cmd_line_admin.php --update=remote
Downloading WURFL from http://downloads.sourceforge.net/project/wurfl/WURFL/latest/wurfl-latest.zip ...

done (/testtw/2.1.3/mongo/data/wurfl.xml: 14.80 MB [897.30 KB compressed])
Downloaded in 4.6609511375427 sec @ 1.58 Mbps

Database Update OK
Total Time: 22.146492004395
Parse Time: 3.8051941394806 (TeraWurflXMLParser_SimpleXML)
Validate Time: 0.024373054504395
Sort Time: 1.2780990600586
Patch Time: 0.50953578948975
Database Time: 8.9265999794006
Cache Rebuild Time: 7.6026899814606
Number of Queries: 2456
PHP Memory Usage: 54.49 MB
--------------------------------
WURFL Version: www.wurflpro.com - 2010-09-17 15:50:54 (Fri Sep 17 15:56:32 -0500 2010)
WURFL Devices: 13687
PATCH New Devices: 39
PATCH Merged Devices: 1

You can also see stats on the Tera-WURFL database:

# php cmd_line_admin.php --stats
Tera-WURFL Stable 2.1.3
Database Type: MongoDB (ver MongoDB 1.6.2)
Loaded WURFL: www.wurflpro.com - 2010-09-17 15:50:54
Last Updated: Sat, 18 Sep 2010 15:28:05 -0500
Config File: /testtw/2.1.3/mongo/TeraWurflConfig.php
---------- Table Stats -----------

> MERGE
Rows:    13726
Devices: 6229
Size:    30.00 MB

> CACHE
Rows:    522
Size:    1.23 MB

Head over to Tera-WURFL.com and give version 2.1.3 a try!

I’ve been working hard on Tera-WURFL 2.1.3 lately.  I’ve been meaning to release it for over a month now, the problem is that I just keep finding improvements to make.  Since Tera-WURFL’s development is now hosted on GitHub, you can always see what I’m doing and grab a copy for yourself.  New in version 2.1.3 is a MongoDB Database Connector.  I keep getting pounded by people about NoSQL and I haven’t given it a fair chance until now.  Simon Harris from www.pointbeing.net was nice enough to contribute the initial MongoDB connector, and I’ve spent a few days working on it and testing it against MySQL5.  At first the performance seems outstanding – almost double that of MySQL5 – but I noticed some inconsistencies with my tests, so I built a benchmarking script and an comparison tool that automatically tests one installation against another and checks for consistency.  I’ve tested three different database connectors against each other and my memcached-based Tera-WURFL Enterprise.

Testing Setup

In order to test the Database Connectors, I brought up a virtual machine on an under-utilized ESXi server (dual 6-core Opterons and 32GB FB-DDR3).

Virtual Machine Specs:

  • 4x Opteron 2.2GHz cores
  • 12GB FB-DDR3 RAM
  • 20GB of storage on a SAS array over MPIO iSCSI
  • Ubuntu Server 10.04.1
  • MySQL 5.1
  • MongoDB 1.7.0

This machine doesn’t have extremely fast CPUs, just a lot of them.  At any rate, the specs were consistent so I was able to get a reasonable test result.  In order to test the performance of the DBs, I ran about 66,000 unique user agents (these are UAs that are not easily matched and create a lot of DB load) through the connectors and measured the rate at which they detected them.

The results

The following results are the average speed at which the different DB Connectors detected the test user agents.  The two graphs show the difference between uncached and cached detections.  For each database, I ran the test with the all the WURFL capabilities loaded (blue) and with only ['product_info']['is_wireless_device'] loaded (red) to get an idea of the performance benefit of using Tera-WURFL’s CAPABILITY_FILTER option.

Uncached Device Detection Performance in Tera-WURFL 2.1.3

* note: memcached is not capable of serving uncached detections, so it is left out of this test.

Cached Device Detection Performance in Tera-WURFL 2.1.3

Conclusion

I was very surprised by these results so I tested them repeatedly just to be sure.  I was so excited that MongoDB would be the clear winner here, but it just isn’t the case for uncached detections.  Mongo did fare well when serving the cached full capabilities – 63% higher than MySQL5.  MS SQL Server 2005 was particularly painful to test.  By comparison, MongoDB took about 8 minutes to churn though all 66,000 user agents, but my initial test with MS SQL Server was 3 hours and 8 minutes.  Since I tested all the other connectors on Ubuntu Server, and the MS SQL Server connector requires Microsoft’s PHP SQL Server Driver, I had to run the tests on a different VM.  On this VM I was getting between 1-3 uncached detections per second, but I moved the testing to my development laptop (2.8GHz Core 2 Duo, 4GB RAM and a Samsung 250GB SSD) and was able to achieve better results.  MySQL fared very well overall and remains at the top of the list.  I have not mentioned the MySQL Nested Set connector before, but it’s been included in Tera-WURFL for a couple versions now.  It just extends the MySQL5 connector and adds right and left values to all of the devices in the WURFL so it can return the complete fallback tree for a given WURFL ID in a single indexed query.  Unfortunately, this seems to provide a negligible improvement in speed.  I would still recommend it for environments where the webserver is on a different host than the database.

Look for Tera-WURFL 2.1.3 to be released within a couple weeks.  I’m trying to get some unit-testing code integrated before I release it.

*** UPDATE ***

I’ve retested MongoDB and MySQL5 again on my production server against a typical distribution of 44887 user agents.  The performance was much better, but the MySQL <-> MongoDB performance was proportional to the first results:

Uncached Cached
All Capabilities Min Capabilities All Capabilities Min Capabilities
MySQL 303 510 2735 7740
MongoDB 195 255 3182 6154