What's New?

  • Engineering Builds a Twitter Bot

    April 9, 2009 By Daniel 0 Comments

    Twitter is the new hotness, it seems every site you visit has some kind of "follow us on twitter" link or built in twitter functionality. People are hooking up cat doors and office chairs to the twitter API so it's only natural that Sparkart Engineering wants to play with this "exploding" new technology.

    The idea was simple, as anyone in support will tell you our fans have lots of followers (stalkers), but with twitter our clients can now stalk their fans right back! Twitter has built in search from their acquisition of Summize, that allows any tom, dick, or harry bot to search tweets for specific terms. If you haven't you should check out the advanced search features which allow you to search for terms with positive or negative attitudes, near places, between dates or that are asking a question. Arbitrarily we built our bot to search for tweets that have the terms sparkart or UFC in them, and in a couple of hours our first bot was born.

    Engineering Builds a Twitter Bot

    Since new technology deserves more new technology our twitter bot can't be written in ruby, that would be too easy for team engineering. To make the project more interesting and fun we decided to write the bot in Erlang. Erlang according to wikipedia is:

    A general-purpose concurrent programming language and runtime
    system. The sequential subset of Erlang is a functional language, with
    strict evaluation, single assignment, and dynamic typing. For concurrency
    it follows the Actor model. It was designed by Ericsson to support
    distributed, fault-tolerant, soft-real-time, non-stop applications.

    Wow, that's a mouthful. Erlang is starting to become the new "hot" language among startups, especially where concurrency is a major requirement. Everyone from Amazon.com, facebook, github, heroku, mochiweb, and last.fm are using it internally. Lets dive into some code and see if we can take the title of "nerdiest post on the blog". Warning really, really technical content ahead...

    First some Erlangism's to make the code below easier to understand. Variables in erlang start with capital letters, all lowercase symbols are called atoms (think ruby symbols). Square brackets denote lists and curly brackets denote tuples (ordered sequences).

      -module(twitter_search).

  -export([run/0, search/1]).

  -include_lib("xmerl/include/xmerl.hrl").

  -define(SEARCH_URL,
    "http://search.twitter.com/search.atom?rpp=50&q=").

   -define(SEARCH_STORAGE, "results-file").

  run() ->
    inets:start(),
    open_table(?SEARCH_STORAGE),
    SearchTerms = ["sparkart", "ufc"],
    [ store_result( Term, search_for( Term ) ) | Term <- SearchTerms ],
    timer:sleep(2000),
    close_table().

  search_for(Query) ->
    Xml = search(Query),
    retrieve_names_from(Xml).

  search(Query) ->
   URL = search_url_for(Query),
   { ok, {_Status, _Headers, Body }} = http:request(URL),
   Body.

  search_url_for(Query) ->
    ?SEARCH_URL ++ Query.

  retrieve_names_from(Xml) ->
    { Body, _Rest } = xmerl_scan:string(Xml),
    Names = xmerl_xpath:string("//name/text()", Body),
    UserNames = [ strip_out_full_name(Author) | {_xmlText,[{name, _},{author, _},{entry, _},{feed, _}],
                                             _,[],Author,text} <- Names],
    lists:usort(UserNames).

  strip_out_full_name(Author) ->
    [Name | _Rest ] = string:tokens(Author, " "),
    Name.

  store_result(Key, Results) -> dets:insert(?MODULE, {Key, Results}).

  open_table(File) ->
    io:format("dets opened:  ~p~n", [File]),
    io:format("dets name:  ~p~n", [?MODULE]),
    case dets:open_file(?MODULE, [{file, File},{type, bag}]) of
      {ok, ?MODULE} ->
        true;
      {error, _Reason} ->
        io:format("cannot open dets table~n"),
        exit(eDetsOpen)
    end.

  close_table() -> dets:close(?MODULE).

    Lets start breaking this bad boy down so it makes sense to more people than the person who wrote it.

    -module(twitter_search). 

-export([run/0, search/1]). 

-include_lib("xmerl/include/xmerl.hrl"). 

-define(SEARCH_URL, 
  "http://search.twitter.com/search.atom?rpp=50&q="). 

    -define(SEARCH_STORAGE, "results-file").

    The first line defines the name space for the the functions in our bot, the module name needs to match the file name, just like model files and classes in Rails. All functions in Erlang are name spaced, so run becomes twitter_search:run(), preventing any similarly named functions from colliding with each other. The export directive tells the run time what functions and their arity are available outside of the module. The arity of a function is the number of parameters it takes. So the twitter_search module exports the run and search functions. The include_lib line is like a ruby require, we need xmerl to do an Xpath search later. The two define lines create constants, for the twitter search API URL and for the path to a file for persistent storage of the results respectively.

    run() ->
  inets:start(),
  open_table(?SEARCH_STORAGE),
  SearchTerms = ["sparkart", "ufc"],
  [ store_result( Term, search_for( Term ) ) | Term <- SearchTerms ],
  timer:sleep(2000),
  close_table().
    The run function is the entry point into our system. It starts the inets process so we can make HTTP requests, it opens our persistent backing store, performs the search and finally closes the backing store. The line [ store_result( Term, search_for( Term ) )Term <- SearchTerms ] is an Erlang list comprehension. In English, for every item in the search terms list it maps that element to the variable Term and then runs the search_for and store_result functions. 
    search(Query) ->
 URL = search_url_for(Query),
 { ok, {_Status, _Headers, Body }} = http:request(URL),
 Body.

    Search builds the URL for our search including the query we are looking for. Lets explore the line { ok, {Status, Headers, Body }} = http:request(URL), since it makes use of pattern matching which is a fundamental aspect of Erlang programming. First throw out the notion you may have that http:request(URL) gets assigned to that mess on the left hand side of the equals sign. What's really happening here is http:request(URL) is called and the result is checked to see if it matches the tuple on the left hand side. The tuple has an atom (think ruby symbol) as the first element and then has another tuple, with the HTTP status, HTTP headers and body of the response. The underscore before the variable names tells the runtime that we don't really about that value. IF the result matches the tuple, then the variable Body will be bound to the HTTP response body.

    retrieve_names_from(Xml) ->
  { Body, _Rest } = xmerl_scan:string(Xml),
  Names = xmerl_xpath:string("//name/text()", Body),
  UserNames = [ strip_out_full_name(Author) | {_xmlText,[{name, _},{author, _},{entry, _},{feed, _}],
                                       _,[],Author,text} <- Names],
  lists:usort(UserNames).

    The retrieve_names_from function parses the XMl returned from our search, and uses an XPath expression to find all of the name nodes. We use another list comprehension to match the inner text of the name nodes and create a list of twitter usernames. Finally the list of user names is sorted and duplicates are removed. This list comprehension is a little more advanced than the one in run(). Instead of executing the strip out full name function for every element in the Names list, we use Erlang pattern matching to only match on the inner text of the author node. The format of the tuple is based on the internal parser representation of the node defined in xmerl header file.

    Simple right! One bright and shiny new bot in 56 lines of code. Hopefully this served as a good introduction to our first bot and Erlang the language it was written in. Feel free to drop questions, comments, kudos, and criticisms in the comments.

    Note: the single pipe in the list comprehensions should be a double pipe to be syntactically correct, but double pipes are reserved in markdown so a single pipe has been used.

  • Be Uniq

    January 26, 2009 By JamesL 0 Comments

    The Ruby programming language's built-in uniq function for Arrays offers a quick and easy way to remove duplicate values.

      a = [1, 1, 2, 2, 2, 3, 4, 4, 4, 4]
  a.uniq   #=> [1, 2, 3, 4]

    But, what happens if the elements in our array are more complex than integers, such as Rails' ActiveRecord objects? To implement an Address Book feature, we needed to fetch users' previously entered billing and shipping addresses, and then remove any duplicates. An interesting problem arose, since a person's billing and shipping addresses might be exactly the same, but since they are stored separately, they correspond to separate ActiveRecord objects.

    Say Address.find(1) is a user's billing address and Address.find(2) is their shipping address:

    >> address1 = Address.find(1)
  {"id"=>"1",
   "street_address"=>"5655 College Avenue",
   "city"=>"Oakland",
   "state"=>"CA"
   "postal_code"=>"94618",
   "address_type_id"=>"1"}
>> address2 = Address.find(2)
  {"id"=>"2",
   "street_address"=>"5655 College Avenue",
   "city"=>"Oakland",
   "state"=>"CA"
   "postal_code"=>"94618",
   "address_type_id"=>"2"}

    Calling

    >> [address1, address2].uniq.size
2

    returns both addresses because Ruby treats them as separate and unique objects (they have different id's and address_type_id's, after all). According to our rules, however, we want to treat the two addresses as identical if they have the same street_address, city, state, country, and postal_code, regardless of whether they are separate objects or have different id's. In order to make uniq use our rules, we need to implement Address#eql? and Address#hash:

    class Address
  def eql?(other)
    [street_address, city, state, country, postal_code].eql?([other.street_address, other.city, other.state, other.country, other.postal_code])
  end

  def hash
    [street_address, city, state, country, postal_code].hash
  end
end

    eql? and hash are the two underlying functions that uniq uses to determine uniqueness. With these two definitions in place, calling

    >> [address1, address2].uniq.size
1

    returns only one of the addresses, instead of both. Allowing uniq, eql?, and hash to do the work for us is certainly easier and more elegant than the alternative of manually looping through the addresses and comparing them all by hand.

    References

  • Air Linux

    January 22, 2009 By James 0 Comments

    Spotted this yesterday on a Delta Air Lines flight from JFK to San Francisco. Delta uses some flavor of Linux (maybe Red Hat) to run its inflight entertainment system. Here's a photo of it rebooting after landing:

    Air Linux

    The system includes music selections, games, a flight map, and about 20 channels of DISH network at each seat.

  • Top Gear's Top Car of 2008

    January 22, 2009 By Renate 0 Comments

    I don't know about you, but I'm obsessed with BBC America's show "Top Gear", the most hilarious and ever intriguing British car show you will ever see (currently you can view a few past episodes on On Demand, courtesy of the money-hungry, monopoly cable provider, also known as Comcast). TopGear.com announced the Car of Year 2008 is the VW Scirocco, a car my Dad had...before he had kids.

    "Top Gear" Top Car of 2008

    She's a cute little thing with a wide bottom.

    "Top Gear" Top Car of 2008

    This could be my next car, just not in green...once my lil old lady poops out on me. I might opt for a sexy lil Audi S5 though...or A5 seeing that most of my money is invested in shoes...and SF rent.

  • Snakeart Special

    December 18, 2008 By Don 0 Comments

    The snakes featured on this site presented many significant challenges, and presented a great opportunity to explore the real computing ceiling of Flash Player. By far the biggest task was in creating a robust inverse kinematics engine that would play well on networks as well as with thousands(!) of rotating sprites on the stage at once. In some cases this meant that network bandwidth and latency could be reduced by sending only the most crucial bits, and having the receiving clients interpolate the rest. For instance, when I begin to drag my snake around, instead of sending the other clients the coordinates and rotation for every single segment in the IK chain, only the coordinates of the head are being sent, and if everything is flowing correctly then the other clients should have no problem positioning all the other segments of the snake based on just the head's position.

    The project also had to be architected in a way to work with flash players' thorny garbage collector. With thousands of sprites being constantly fed in and out of memory, even a small memory leak can quickly spell the end of a snake session for the user, so this necessitated the creation of a pseudo design pattern that I'm calling the Sparkart Special, or the Modified Anonymous Factory to the layman. While it's common to store an array of objects in a pool and call on those when needed, this approach fails in situations when:

    1. The array is very large
    2. The array, or specific members of the array need to be accessed constantly
    3. The array is large and it needs to be quickly sliced, diced, chopped and blended

    A solution is to store an array of just the class definitions as a static constant in a class that's accessible to every other class in the project. That's maybe the nerdiest thing that will ever be written on these pages, so to to break it out of geek-speak:

    package com.sparkart.blog 
{ 
    import com.sparkart.blog.heads.*; 
    import com.sparkart.blog.manes.flames.*; 
    import com.sparkart.blog.scales.*; 

    public class Globals 
    { 

        public static const SNAKE_HEADS:Array = [ Head1 , Head2, Head3, Head4, Head5, Head6, Head8, Head7, SuperHead1, SuperHead2, SuperHead3 ]; 

        public static const SNAKE_SCALES:Array = [ Scale1, Scale2, Scale3, Scale4, Scale5 ]; 

        public static const MANE_FLAMES:Array = [ Flame1, Flame1, Flame1, Flame1, Flame1, Flame1, Flame1, Flame1, 
                                        Flame2, Flame2, Flame2, Flame2, Flame2, Flame2, 
                                        Flame3, Flame3, Flame3, Flame3, 
                                        Flame4, 
                                        Flame5, 

                                        //balls 
                                        Ball1, Ball1, Ball1, 
                                        Ball2, Ball2, Ball2, Ball2 ];   //<--- Array is weighted so that Flame1 comes up much more often than Flame5 
    } 

}

    Now any other class can simply import the Globals class and if it wants to create a random flame for the snake, it only needs to check against the MANE_FLAMES array. Whatever class is doing the constructing doesn't even need the exact definition for the flame it's creating; just the definition for it's superclass, Flame.

    public function makeFlames( num:int ) : void 
{ 
    for ( var i:uint = 0; i<num; i++ ) { 

        var pick:uint = uint( Math.random() * Globals.MANE_FLAMES.length ) 
        var f:Flame = new Globals.MANE_FLAMES[ pick ](); 
        snake.addChild( f ); 



    } 
}

    The superclass will automatically "self-destruct" after it excecutes whatever it needs to do, so at the end of this cycle there are zero references to the flame, it gets picked up by the garbage collector and everything keeps snaking along nicely.