Latest Entries »

A couple months ago I switched my version control system for PHP development from CVS to Git and moved the Tera-WURFL development tree to the public repository Github. Since then I had been using an early version of the Eclipse EGit plugin (I think it was 0.8.1, but I can’t be sure). The plugin was only partially implemented at the time and I used it for committing and pushing only. Other tasks like tagging, branching and merging we difficult or impossible. Today I updated it to 0.8.4 (still in it’s incubation) and I am pleased to see that it is much improved! Most of the functionality of Git is available right from the Eclipse IDE. The icons and windows have been reworked and are much more intuitive now. Also, many of the settings are stored in the standard .git respository, so if you are using the standard Git command-line interface, both systems are 100% compatible! Good work EGit team – thanks for making my life much easier!

The new Barnes & Noble Nook is actually much cooler than I thought! After reading a few books on it, I’ve written an in-depth review on my findings.

Read the full review over at GadgetMe!

Last night I finished a great book called Physics of the Impossible by Michio Kaku, a leading theoretical physicist.  As I was trying to get to sleep I was thinking about the relationship of space, time and gravity and came to some interesting conclusions.  This is my very infant theory of gravity’s relationship to space-time.

Picture a swimming pool full of water.  If you blew a bubble in the water you displace a very small amount of water with air, giving the water more potential energy.  On earth, since we have gravity, it is easy for the water to get back to its lower state of energy by pushing the bubble up to the surface since the air is less dense than the water, and the water gets more dense as you get closer to the bottom.  In space, if the water was constrained so it could not expand, the water would still want to push the bubble out, but there is no obvious direction for it to go since the density of water is fairly consistent.  Now, if space-time is like this water in space, its constraints are the edges of space and time, beyond which there is nothing (even if there was something we could not detect it because it would not follow our laws of physics).  Each subatomic particle with mass would be surrounded by a bubble in this water.  The bubble is the sphere of influence that it has on space-time, so matter itself is displacing space-time.  As a result of this displacement, space time “pushes back” at the particle resulting in gravity.  If you push matter together until it is very dense it will eventually fuse, creating other element(s).  If you keep forcing it with more and more energy, eventually, the subatomic particle’s spheres of influence (the bubbles) would touch and combine into one.  On a very small scale, this would result in a microscopic black hole, but since space-time is not pushing back very hard on this structure, it is unstable and it disintegrates back into its constituent particles.  On a large scale, there would be trillions and trillions of these particles, and they would create a very large sphere of influence.  In fact, we already have a name for this – it’s called the event horizon of a black hole.

If you assume this conceptual view of gravity is correct, then you can draw some pretty interesting conclusions.

  1. Gravity is a measurement of the potential energy of space-time that has been displaced by a mass.
  2. We cannot see inside a subatomic particle since space and time don’t exist or are so distorted that they would result in chaotic observations.
  3. When matter is in motion, it leaves a “wake” of space-time behind it as it is restored to an equilibrium.  This wake is visible in a particle accelerator – when particles feel their wake of their own gravity and are pulled into a tight circle.
  4. Perhaps the displacement of space-time itself is what is pushing the universe out in all directions (but it appears to be accelerating which would not  coincide).

I will continue to think about this concept and figure out what is logical and what is illogical.  Ideally someone in this world can integrate this “symptom of gravity” into the existing theories and put this whole gravity mystery to bed!

If you have an opinion about this concept, please comment on it!  I am not a theoretically physicist so I may just be way off in left field!

I have often wondered why it’s not possible to go faster than the speed of light. My reasoning has been “well, according to Einstein’s theory of relativity (E=MC^2), you would require an infinite amount of energy to increase your speed by an infinitely small amount.” Until lately I have been been satisfied by this response, but I’ve always been stumped by this apparent paradox: Imagine you are in the back of a train going 1 MPH less than the speed of light. Now you get up and run to the front of the train at 8 MPH, would you not be going 7 MPH faster than the speed of light?. The answer is no, and here’s why. When you start moving forward you are perceiving yourself as moving normally, but in fact this is far from the truth. The closer you get to the speed of light, the slower time is.  What this means to you on the train is that you feel like you are going the 8 MPH because you are also perceiving time more slowly than normally.  If you were timing yourself with a watch, you would indeed get to the front of the train in the 10 seconds or so that you thought it took; however, if I were stopped on the outside of the train looking in, it would appear to me that you took many hours — maybe even days to run across the train.  What this means is that it actually took you a very long time to travel the distance of the train, let’s say 50 feet in 24 hours.  This speed is added to the speed of the train, but it’s far less than the 8 MPH you thought you were travelling at.  What if you shoot a bullet towards the front of the train?  It doesn’t matter what you do, the faster you go, the slower time goes.  If you managed to get to the speed of light, time would stop completely, so you would not actually be moving.

I recently created Tera-WURFL Explorer to allow people to browse through the WURFL, search for devices and upload images to the WURFL images collection. I originally used MySQL’s FULLTEXT index to let people search for devices, but quickly realized that it did not suit my needs. The main problem was that it does not index words smaller than what is specified in my.cnf (ft_min_word_len), and if you want to change it, you need to change it server-wide. This was not a good option for a large virtual host setup since it would affect all the FULLTEXT indices on the server; also, if you do change it, you need to reindex every FULLTEXT column in every database to prevent data corruption.

I did some research on search engines and eventually settled on Sphinx – mainly because it has a cool name, but also because there are some big-name success stories from companies like Craigslist who switched to it and never looked back.

Here’s how I installed it on Ubuntu 9.10:

First, you need to install the dependencies and download sphinx, then extract the archive and make it:

apt-get install g++ libmysql++-dev
cd /tmp
wget http://www.sphinxsearch.com/downloads/sphinx-0.9.9.tar.gz
tar -zxvf sphinx-0.9.9.tar.gz
cd sphinx-0.9.9
./configure --prefix=/usr/local/sphinx
make
make install

Now all the sphinx-related files are in /usr/local/sphinx.
Next, I created a system user and group called “sphinx”:

adduser --system --group sphinx

Note: on RedHat-like systems, you can use “adduser -r -n sphinx”

Now, I created an init script for it. I would recommend downloading my init.d script.

wget http://www.tera-wurfl.com/blog_assets/searchd
mv searchd /etc/init.d/
chown root:root /etc/init.d/searchd
chmod 755 /etc/init.d/searchd

This script adds the following functionality:

# Start the Sphinx service
service searchd start
# Stop Sphinx
service searchd stop
# Check if Sphinx is running
service searchd status
# Reindex every Sphinx index (works while started or stopped)
service searchd reindex

Now we’ll add sphinx to the startup and use the config option to setup sphinx to run as the sphinx user:

update-rc.d searchd defaults
service searchd config

Note: on RedHat-like systems you can use “chkconfig –add searchd”

Lastly, you need to configure sphinx. I would copy the default config file and edit that one:

cp /usr/local/sphinx/sphinx.conf.dist /usr/local/sphinx/sphinx.conf

You can follow along with the comments in the file, or jump on the documentation site and figure out what all the settings do.

Now everything is setup and should work properly!
If you followed my directions and put the tarball in /tmp, the sphinx PHP and Python APIs and some examples are in /tmp/sphinx-0.9.9/api/. You should put a copy of the PHP or Python API somewhere else on the system so you can use it from your applications.

To see my use of the Sphinx search engine, take a look at this site:
http://www.tera-wurfl.com/explore/browse/

I’m just about to release Tera-WURFL 2.1.0 (Stable) and I thought I’d give you an idea of what to expect. I was originally positioning this as a revisional update (i.e. 2.0.1), but decided to make it a minor version upgrade because it adds a lot of functionality and includes a few more settings than version 2.0.0.

Here’s what’s new:

1. Experimental support for Microsoft SQL Server 2005/2008: with the MSSQL2005 Database Connector, you can now use MS SQL Server as a Tera-WURFL backend! The support is experimental for now, although I’ve put it through my barrage of over 45,000 user agents and it works properly. The Reduction in String stored procedure still needs to be optimized since I just ported it from MySQL 5 to T-SQL. The MSSQL backend is considerably slower than MySQL, and I beleive it’s that procedure slowing it down.

2. SimpleDesktop Matching Engine: Tera-WURFL wasn’t originally designed to differnetiate between desktop and mobile browsers (either was WURFL for that matter), but with this release I’ve introduced the SimpleDesktop Matching Engine which, when enabled, uses keywords and regular expressions to detect 90% of desktop browsers without having to resort to searching the database for a matching WURFL entry. In my tests performance increased by 176% for detection of 45,000 actual unique user agents (both mobile and non-mobile). This feature also dramatically decreases the number of items in your cache by using a single cache item for all desktop browsers.

3. Capabilities Filter: I’ve been meaning to implement the Capability Filter for a long time, but two very high traffic clients of mine convinced me to sit down and get it finished. This adds a new setting in TeraWurflConfig.php called CAPABILITY_FILTER. If you set it to false it will be disabled and all the capabilities in the WURFL will be stored in the database and available to your scripts (this is the pre-2.1.0 behaviour). Here’s where the magic starts, you can set it to an array of the capabilities and groups of capabilities that you want to store and use, for example, if you just want the know what kind of device is visiting your site and whether or not it’s wireless, you can use this filter:

public static $CAPABILITY_FILTER = array(
"brand_name",
"model_name",
"is_wireless_device"
);

This will shrink your device database by more than a factor of 10. I tested the filter with 20 capabilities against 45,000 unique user agents and it reduced the size of the cache from 645MB (without filtering) to 92MB (with filtering), then down to 24MB with both filtering and SimpleDesktop.

You can look forward to the Tera-WURFL 2.1.0 release around February 10, 2010.

Tera-WURFL is available at it’s usual location, http://www.Tera-WURFL.com/

I happen to have an iPhone and an iPod Touch sync’d to my iTunes back in the US, but now I’m in Iraq and don’t have access to my computer with the iTunes library on it. I do have the music, but I don’t want to sync my iPod to my iTunes because all the play counts and ratings will be gone. I had a little free time last night so I wrote a script to update these attributes for me! It is written in JScript/JavaScript and uses the iTunes COM Interface to communicate with your device and the iTunes Library. It is not optimized very well so it will iterate over the entire iTunes Library for each track it encounters on your device, looking for a matching Artist, Album and Track Name to update. It also maintains a log file that lists every track that was updated including it’s old and new values, as well as any tracks on the iPod/iPhone that it could not find a match for in the iTunes Library. I created this script for my own purposes and decided to share it, so I don’t intend to provide much support for it. The zip file contains two files: UpdatePlayCount.bat and UpdatePlayCount.js. Extract the files somewhere, plug in your iPod/iPhone, then open iTunes and click on your device, then check “Manually manage music and movies” and restart iTunes, then double click on UpdatePlayCount.bat to update your library. You can see that progress is being made in iTunes by clicking on your Library’s Music folder then sorting the music by Play Count or Rating. This list is updated in real time.

Download UpdatePlayCount.zip (2KB)

Source Code (JScript/JavaScript)

/**
 * UpdatePlayCount.js
 *
 * Description
 * ---------------------------------------------------------------------------------------------------------
 * This script will update your iTunes library with the play counts and ratings from any iPod or iPhone.
 * It uses the iTunes COM Interface to communicate with your device.  In order to determine which tracks
 * match, the script will search through the iTunes Library looking for a matching Artist, Album and Title;
 * as a result, this process can take a long time.  Updates all the music and movies in the "Music" and
 * "Movies" playlists on your device.  If you need to update more folders you can modify the code.
 *
 * Usage
 * ---------------------------------------------------------------------------------------------------------
 * Double click on UpdatePlayCount.bat
 *
 *
 * @package UpdatePlayCount
 * @author Steve Kamerman, stevekamerman AT gmail.com
 * @version Alpha 1.0 $Date: 2009/12/22 13:19
 * @license http://www.mozilla.org/MPL/ MPL Vesion 1.1
 * @language Microsoft JScript / JavaScript
 *
 */


var logObject, logFile;
var logfileName = "UpdateItunes.log";
logObject = new ActiveXObject("Scripting.FileSystemObject");
logFile = logObject.CreateTextFile(logfileName, true);

var ITTrackKindFile = 1;
var iTunesApp = WScript.CreateObject("iTunes.Application");
var sources = iTunesApp.Sources;
var i;
var updateTracksCount = 0;
var missingTracksCount = 0;
var ipod;
var itunes;
var playlists;
var itunesPlaylists;
var iTunesXML;
iTunesXML = "";

for(i=1;i<sources.Count;i++){
    if(sources.Item(i).Kind == 2){
        ipod = sources.Item(i);
        playlists = ipod.Playlists;
    }
    if(sources.Item(i).Kind == 1){
        itunes = sources.Item(i);
        itunesPlaylists = itunes.Playlists;
    }
}

if(ipod == undefined){
    WScript.Echo("Error: No iPod or iPhone found.  Please make sure your device is listed in iTunes, then click on it's name and check "Manually manage music and videos".  Restart iTunes and rerun this script.");
    WScript.Quit(0);
}
WScript.Echo("iPod found, press OK to update iTunes with the play counts and ratings from your iPod.");
WScript.Echo("This process can take a long time to complete.  To monitor the progress,\nopen iTunes and click on your iTunes Music folder\nthen sort by play count or rating and watch them change.\nPress OK to continue");
var currentPlaylist;
var currentTracks;
var currentTrack;
for(i=1;i<playlists.Count;i++){
    currentPlaylist = playlists.Item(i);
    if(currentPlaylist.Name != "Music" && currentPlaylist.Name != "Movies")continue;
    currentTracks = currentPlaylist.Tracks;
    for(a=1;a<currentTracks.Count;a++){
        currentTrack = currentTracks.Item(a);
        if(currentTrack.PlayedCount > 0){
            updateITunesEntry(currentPlaylist.Name,currentTrack.Artist,currentTrack.Album,currentTrack.Name,currentTrack.PlayedCount,currentTrack.Rating);
        }
    }
}

WScript.Echo("Finished processing "+(updateTracksCount+missingTracksCount)+" tracks.\nUpdated Tracks: "+updateTracksCount+"\nMissing Tracks: "+missingTracksCount+"\nSee the logfile ("+logfileName+") for more details.");

function updateITunesEntry(playlist, artist, album, song, playCount, rating){
    var itunesPlaylist;
    var itunesTracks;
    var itunesTrack;
    var i;
    var a;
    for(i=1;i<itunesPlaylists.Count;i++){
        itunesPlaylist = itunesPlaylists.Item(i);
        if(itunesPlaylist.Name != playlist)continue;
        itunesTracks = itunesPlaylist.Tracks;
        for(a=1;a<itunesTracks.Count;a++){
            itunesTrack = itunesTracks.Item(a);
            if(itunesTrack.Artist == artist && itunesTrack.Album == album && itunesTrack.Name == song){
                logFile.WriteLine(itunesTrack.Name+": count: "+itunesTrack.PlayedCount+"->"+playCount+", rating: "+itunesTrack.Rating+"->"+rating);
                itunesTrack.PlayedCount = playCount;
                itunesTrack.Rating = rating;
                updateTracksCount++;
                return(1);
            }
        }
    }
    missingTracksCount++;
    logFile.WriteLine("WARNING: Track not found in iTunes Library: "+artist+" - "+song);
}

Just wanted to let you know that I'm here in Iraq and beginning my mission. I may provide more details about where I am and what I'm doing here in the future. It looks like I won't have as much time as I thought, but I still plan to get some work done on Tera-WURFL while I'm here!

I ran a database of 23,902 unique user agents through both Tera-WURFL 2.0.0 RC4 and the new WURFL PHP API (1.0.1-rc2). Here are the results:

Tera-WURFL
Total Time: 12.081017971039
Devices Processed: 23902
Total Queries: 23912

WURFL PHP API
Total Time: 217.57795381546
Devices Processed: 23902
Total Queries: N/A

Both results were obtained after repeated testing to verify that the user agents were cached.

Tera-WURFL started in late 2006 as a personal project to make the WURFL PHP Tools faster by storing the device capabilities in a MySQL Database instead of flat files. It’s now late 2009 and I’ve finally got Tera-WURFL 2.0 up and running. As of now I’ve released RC4 and I’m working on RC5. Hopefully I can get version 2.0.0 Stable out before I leave for Iraq. Here’s a breakdown of the features of Tera-WURFL 2:

Rewrote some of the UserAgentMatchers and deleted others to bring Tera-WURFL on par with the Java WURFL API. With the introduction of desktop browser UserAgentMatchers, we no longer need to use the large web patch; instead, you can just use the 8KB one from wurfl.sourceforge.net (included). Also, I fixed some typos and bugs here and there. NOTE: if you are upgrading from version 2.0.0 RC1-RC3 you should delete all your database tables before you update. You can leave the terawurflcache table if you want to retain your cache. Although it will still technically work even if you don’t delete the tables, you will be orphaning some unecessary tables in your database.

Complete code-rewrite from the ground up. The 2.x version of Tera-WURFL is loosely based on a pre-release of the Java WURFL Evolution Library, but the API is taken from Tera-WURFL 1.5.2. The following is a list of features found in Tera-WURFL 2.0:

  • User Agent Matchers have been created for each of the major manufacturers. These allow for specific matching methods to be applied to the user agent like string searching, RIS (Reduction in String) and LD (Levenshtein Distance).
  • Multiple patch files are now supported. Tera-WURFL ships with the current wurfl.xml, web_browsers_patch.xml and custom_web_patch.xml. Patch files can be added to TeraWurflConfig.php by separating them with semicolons in the TeraWurflConfig::PATCH_FILE directive. Patch files are loaded in order from left to right on top of the WURFL file, so if you want to override every other patch file, specify it last.
  • The custom_web_patch.xml file can be edited from the Web Administration page, and allows you to easily add non-mobile user agents to the patch file. The devices with these user agents will be detected as generic_web_browser (non-mobile).
  • Persistent Caching means that your cached devices stay cached. When you update the WURFL file or your patches, your device cache is also updated via the new database.
  • Cache Browser allows you to see what devices are hitting your site and what their capabilities were detected as.
  • Installation Script is better than 1.5.2. Once you download Tera-WURFL and extract it, edit TeraWurflConfig.php then go to /admin/install.php and follow the directions to finish installation.
  • PHP short_open_tags are no longer required to run Tera-WURFL. PHP has this feature disabled by default now.
  • Conclusive vs. Inconclusive Matching. If a device is matched with the UserAgentMatcher’s primary matching method it is considered a conclusive match, if it is detected via a recovery matcher or by the CatchAllMatcher it is an inconclusive match. This information is available via the tera_wurfl capability group.
  • tera_wurfl Capability Group. The TeraWurfl->Capabilities array now contains a group called “tera_wurfl”. This group contains the following Tera-WURFL related information:
    • num_queries – the number of database queries required to lookup the device.
    • actual_root_device – the WURFL ID of the actual device (not subrevision or generic), this can be null.
    • match_type – either conclusive or inconclusive.
    • matcher – the name of the UserAgentMatcher that detected the device.
    • match – whether or not there was an actual match. If there was no match, Tera-WURFL guessed which generic device is most similar to the device.
    • lookup_time – the time in seconds that it took to detect the device.
    • fall_back_tree – the complete fallback tree that built the capabilities of the device. This is a list of all the WURFL IDs from the detected device down to the base generic device.

Example Script

// Include the Tera-WURFL file
require_once("TeraWurfl.php");
// Instantiate the Tera-WURFL object
$wurflObj = new TeraWurfl();
// Get the capabilities from the object
$matched = $wurflObj->GetDeviceCapabilitiesFromAgent();
// Show whether there was a conclusive match
if($matched){echo "Match found";}else{echo "Match NOT found";}
// Print the capabilities array
echo "<pre>".htmlspecialchars(var_export($wurflObj->capabilities,true))."</pre>";