{rel, "appname", "1",
[
  application1,
  application2,
  ....
  ....
  {tv, load}
  {appmon, load}
]}

Build your release and appmon and tv should be available :)

My first impression of Erlang wasn’t all that good. Looking at its syntax made me cringe a bit. Coming from a background in Ruby I was use to reading code as if it was written in plain old english. This wasn’t the case with Erlang. Staring at my first bit of Erlang code I had no clue what the heck was going on. However, I pushed on. About a week in I started familiarizing myself with the syntax. And you know what? Despite what everyone says I think the syntax is just fine. Try spending a bit of time learning Erlang and you’ll realize it isn’t half-bad.

That being said, the syntax in Erlang wasn’t the thing bothering me. It was because working in a functional language meant I would have to change the way I think.

To understand recursion, you must understand recursion

I remember a conversation I had with one of my Computer Science professors about recursion. She told me that although recursion is totally awesome its usually frowned upon when you try and sneak it into production code. She told me about her experience at IBM during her internship. In short, her co-workers weren’t too happy about her creative solutions.

Working in a functional language means you can’t avoid recursion. Its everywhere. Functional languages impose a share-nothing architecture. This means everything is immutable. In order to modify something you would have to create a new data structure. To prevent your code from looking like crap you would have to use recusive functions to build lists and other data structures.

Erlang vs Ruby

The great thing about Ruby is it allows you to use Metaprogramming. The horrible thing about Ruby is it allows you to use Metaprogramming. What I noticed about Erlang is if I wanted to know how something worked I could open up the source code of some library and take a peek. Within a few minutes I could grasp what the heck was going on. In Ruby, not quite. You would have to pray the library was documented well enough to tell you what the heck was going on. There are times when Metaprogramming helps readability however I often see it being misused by library authors. Ruby looks innocent on the outside but its a complex beast once you start getting into more advanced topics.

Working with Ruby Again

I started working on hobby projects again in Ruby sometime around April. Coming back after almost a year of working with a functional programming language was a bit weird.

What I love about Erlang was pattern matching. Writing Ruby code I would always have a sad face when I ran into cases where pattern matching would DRY up code. You’ve probably run into situations where you have to bust out the old case statement to switch on arguments passed into a function. It looks ugly and leads to your eyes having to jump around different logical paths.

I also noticed my way of thinking has changed. The first thing I think about is how to write something recursively instead of the other way aroud. Would it be fewer lines of code and more readable if I wrote this recursively? Most of the time the answer was no. When I did find oppurtunities to use recusion it simplified hundreds of lines of code :) My X11 library for Ruby constructs packets which contain sub-packets which was a good chance to use some recursion. Gotta love those moments when you can use recursion to simplify things.

Anyway, I’ve ranted enough. In retrospect becoming proficient in a functional programming language has improved my imperative programming style immensly.

Σ’ing things up

Suppose we want to compute a sum across every node in our cluster that gives us the total number of requests/second our distributed system is handling. Now we could just send messages periodically from each node in the cluster to some central point. However, our aim should always be for a fullly decentralized system where nodes don’t have to rely on a single machine to calcuate the global estimate.

That being said, were going to use a Gossip-based algorithm called the Symmetric Push-Sum Protocol or SPSP for short. Its a simple algorithm for computing aggregates across a cluster.

Symmetric Push-Sum Protocol (SPSP)

What does SPSP do for us? Well its a 2-in-1 algorithm since we can compute either a sum or an average with it. Its also damn simple which is why its a good place to start when talking about Gossiping algorithms.

SPSP works by keeping track of a weight and a value at each node. In our case, the value would be our request/second service rate at a node. And the weight acts like a percentage telling us how much of the overall sum we have. Different weight values are defined in the SPSP whitepaper. For example, if you wanted to calculate a sum across a cluster you would initialize the weight to be 1 at one node and 0 at the rest. I’ll explain why this is in a moment.

In the example below will pretend that nodes A and B are seeing 5 and 10 requests/second. Once our Gossip-based algorithm is finished we should be able to query either A or B to figure out that the system is seeing in total 15 requests/second.

Once the system has initialzied nodes can start gossiping.

Above I’ve drawn a little sequence diagram showing SPSP in action:

1. Node A will push half its weight and value to Node B
2. Node B receives a push message from Node A. Node B will take half its value and weight and send a symmetric push back to the caller Node A
3. Once Node A receives a symmetric push from Node B it simply adds the received value and weight to its own
4. Finally Node B will add Node A’s original push message to its own values

Note: this is all completely asynchronous. Theres no need to block between a push and a symmetric push

When is it finished?!

You know when the algorithm is finished when the weight values at each node converge to 1 / N where N is the number of nodes. Once you reach convergence you’re ready to calculate your sum. This part is easy! Take your value and divide by the weight. In the example above it would be 7.5 / 0.5 = 15

Next Page - Erlang Implementation

Reading his post made me remember my own evolution as a software developer. Some of the traps he describes I still suffer from. Often I run into situations where I find myself hacking away at something I haven’t thought through. What happens? I end up deleting the code and it never sees production. I’ve wasted so much energy in the past falling into this trap I still suffer from it today but slowly learning to avoid it.

I know for sure my boss is a very experienced software developer. He spends an enormous amount of time on research. This seems weird to me because I actually never see him programming, he’s constantly reading or whiteboarding things. When I do see him program or work on something he’s able to crank out elegant solutions to problems because of the amount of research and thinking he’s done beforehand.

I need to remember that code is a by-product. Thinking things out before hacking away seems obvious but I still find myself slipping from time to time. I’ll try keeping this in mind and remember that my real job is to think and write less code.

Where do we start?

Lets start with explaining Gossip communication and what it means in distributed systems. For example, perhaps a co-worker spreads a nasty rumour about you. Apparently you haven’t been testing your code (shame on you). Your colleagues A and B chit chat over lunch and A mentions your bad habit. Once A and B interact with more people the rumour spreads exponentially faster. Pretty soon the whole office is glaring at you and you become the scape goat for every single problem in the office. Eventually this rumour mutates and your boss hears your purposely unfactoring your code and using pig latin for your method definitions. You end up being fired :(

Similarily, in a distributed system its easy to tell one node a message and have that message quickly propogate out to a few servers, maybe thousands with amazing speed. Wikipedia provides an excellent example of a data center that houses 25,000 servers. A search term is passed to a central point where it is propogated out. Since one of the machines has the item being searched for its able to receive and send a result back within 3 seconds in the worst case scenario.

Next Page - The Push-Sum Protocol

Problem #1

Teensy USB device wasn’t being recognized. You can see the list of registred devices on Ubuntu using lsusb

Solution

Loaded program code may interfere with the USB registration process. Use the RESET button then check the output of lsusb again make sure the device is being recgonized. NOTE: you may have to hold it down and plug the USB device in then release it.

Problem #2

If that wasn’t enough my USB device was being registered under /dev/hidraw0 which isn’t recognized by the Teensy Loader Application. It only looks at /dev/ttyACM0

Solution

When the kernel registers the USB device it doesn’t give regular users write/read access to the device. You will need to setup appropriate udev rules to do this. udev rules are available from the Teensy website http://www.pjrc.com/teensy/49-teensy.rules you will need to download this file and place it in /etc/udev/rules.d/

Once you have ReviewBoard installed you can generate a website using rb-site install. I setup my website under /var/www/review.mysite.com but you can set it up anywhere if you prefer another location. The next step is to launch the FastCGI daemon script bundled with ReviewBoard

1

rb-site manage /var/www/review runfcgi method=threaded port=3033 host=127.0.0.1 protocol=fcgi

Once your FastCGI instance is up and running you just need to simply point Nginx at the FastCGI process

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

server {
    listen 80;    
    server_name review.mysite.com;
    root /var/www/review.mysite.com/htdocs/;    
    
    location /media  {     
      root /htdocs/media/;    
    }
   
    location /errordoc {      
      root /htdocs/errordocs/;    
    }

    location / {      
      # host and port to fastcgi server      
      fastcgi_pass 127.0.0.1:3033;      
      fastcgi_param PATH_INFO $fastcgi_script_name;      
      fastcgi_param REQUEST_METHOD $request_method;      
      fastcgi_param QUERY_STRING $query_string;      
      fastcgi_param CONTENT_TYPE $content_type;      
      fastcgi_param CONTENT_LENGTH $content_length;      
      fastcgi_pass_header Authorization;      
      fastcgi_intercept_errors off;    
    }  
}

Then restart your nginx server and reviewboard should be up and running. Did I miss anything?

For the project I was working on I couldn’t depend on Node.js for dependency management since I was creating a front-end browser game. At first I decided to implement my own hacked up solution in Ruby to substitute and concatenate coffee script source files… it wasn’t pretty :( After moving to sprockets I was able to refactor a few hundred lines into three :) For example, suppose you have a file that contains sprockets require syntax called Main.coffee

1
2
3
4

#= require "src/one"
#= require "src/two"
#= require "src/three"
MyApp ?= {}

Then all you need to do is create a Sprockets::Environment instance and add the source path

1
2
3
4

require 'sprockets'
env = Sprockets::Environment.new
env.paths << "src/coffee"
open("example.js", "w").write( env["Main.coffee"] )

Tada! It just works :)

For those of you who don’t know what a semaphore is it’s simply a variable that acts as a counter ( with two operations up/down ) which puts a process to sleep if it tries to down/decrement a 0 valued semaphore. The process will receive a wakeup signal once the 0 valued semaphore is up/incremented by another process.

In the producer-consumer example below we have a critical region (some shared buffer) that needs to be controlled using semaphores to prevent race conditions. We create two semaphores one called EMPTY which represents the number of empty slots available for the producer and another semaphore called FULL to represent the number of slots occupied with items.

If there are no items in our buffer a consumer will goto sleep since it will try to down the FULL counter which will be 0 because there are no items occupying any slots for it to consume. Once the producer creates an item it will down EMPTY and up FULL; vice-versa. In the case of the consumer when it consumes items it will up EMPTY and down FULL. This might seem a little confusing at first so I created a diagram to illustrate the process (note: producer/consumer are NOT running in parallel in this diagram).

It is important to note that EMPTY and FULL alone do not prevent race conditions.
This doesn’t cover the case when the buffer is neither full nor empty. This means we need a third semaphore to lock the shared buffer to prevent consumers and producers from accessing the shared buffer at the same time.

We call a semaphore with a single counter (initialized to 1) a binary semaphore or more commonly referred to as a mutex or lock. Using a mutex initialized to 1 when a consumer wishes to consume an item it simply needs to down the mutex before consuming the item which will prevent other producers/consumers from manipulating the shared buffer. There are two main semaphore implementations in UNIX-based systems: System V (available in older-systems) and POSIX. For my example program I used System V implementation.

Producer.c

Consumer.c

Shared.h

Shared.c