Chat so easy a t-rex could do it with his itty-bitty t-rex arms!

Over the years I’ve accumulated thousands of hours writing C#, Python, Java, Android, C, and a few others. So it might come as a shock that I wrote, my first real independent software venture, in Erlang, a language I had zero experience with when I started.

Having built a scalable, distributed content distribution system in C# for Cheezburger, you might think I would just do that again for, and initially I had thought I might do just that. My plan was to start with JabbR, an open source chat service written in C#, and customize it to my needs. After about an hour of poking through the JabbR codebase, I decided that JabbR wasn’t the best jumping-off point for There were two problems with it that I was facing. The first is just a problem with JabbR itself… it is kind of a mess. I didn’t want to start off with a codebase like JabbR.

The second problem was that as great a platform as ASP.NET MVC is to use, it does have its failings. From building scalable web apps in C# in the past, I knew the easiest thing to screw up was shared state. Shared state is just too easy in .NET. Whether it is using session state (something easy to track down after the fact) or static fields (something that can be quite hard to disentangle), ASP.NET’s philosophy on shared state is counter to what is best when building scalable web applications.

While I was lamenting the sad state of things (aka whining) to my longtime friend and colleague, Jacob Krall, he asked why even start with C# at all. This got me thinking about alternatives that might make it easier to just get it right from the get go.  So I started looking around, and at some point I found myself reading The more I read, the more excited I was, because Erlang was really starting to jive with what I had learned building stuff throughout my career.

Before I dig into why Erlang is so awesome, it is worth spending a little time on some related topics.

The CAP theorum states that in any distributed system, you can only guarantee that you have two of the following three things: Consistency, Availability, and Partition tolerance.  That is to say that when you are designing your system, you get to pick two but have to live without the third. Consistency means that every part of the system have a consistent view of the underlying data. Availability means that system is up for everyone. Partition tolerance means when parts of the system are either down or unable to communicate with each other, things continue to function properly. It should be pretty clear that a non-distributed system has only consistency, since there is only one partition; if it goes down the entire system is down, and so the system is neither available nor tolerant of partition failure. This is why we create distributed systems. We can make the system available by adding partitions, but then we have to choose between consistency and partition tolerance. While we can’t have both, there are many techniques that can ensure that we are eventually consistent and tolerant of partition failures.

One technique for providing eventual consistency is called Command Query Responsibility Segregation (CQRS). The basic idea is that each partition has a read-only database that it uses to query data, while commands are performed against the write-only database, and eventually data is propagated from the write-only database to the read-only databases. This approach ensures that if a partition fails, it still has a copy of the data to use, even if the partition failing is the partition with the write-only database.

In order to make CQRS work (or even most other approaches to eventual consistency), you need a message bus. A message bus is a system that can be used to send messages between the different components of your system. A CQRS system might have a series of components that each fire off a message to the next component in order to perform an command. For example, to perform an action, a message might trigger a change to the write-only database, which then sends a message that triggers all the read-only databases to be re-synced. There is plenty more to read and learn about the CAP theorem and CQRS on the internet. If this is your first time reading about either, then I recommend doing some more digging.

Now back to Erlang. The single biggest feature of Erlang that made me consider it for is the Erlang process model. Erlang provides cheap, lightweight processes — so cheap in fact that you can create hundreds of thousands in a matter of seconds. In addition, each process has its own message queue and powerful message routing abilities. Cheap processes and built in message queues are pretty cool, but alone they don’t tell the whole story. Messages are Erlang’s only mechanism for interprocess communication and shared state. So you are forced to write more flexible code. To top it off, Erlang can almost transparently send and receive messages with Erlang processes running on different nodes in a cluster. Given the processes and message passing awesomeness going on, Erlang could be thought of as the message bus language. (In fact, the very popular message bus RabbitMQ is written in Erlang!)

Another thing that makes Erlang great is its pattern matching mechanism. Almost all of Erlang’s flow control is built around pattern matching, most notably when calling functions and receiving messages, and this allows for clear, concise code and data filtering. Erlang isn’t unique in this regard. A number of other languages have similar structural pattern matching mechanisms, and it is pretty darn awesome wherever it is found. Once pattern matching and receiving messages are combined, as they are in Erlang, it is a recipe for amazing productivity (and talkative dinosaurs).

The last thing on my list is probably the most controversial and probably the least important, but I think it is particularly useful. In Erlang most everything is immutable, and so “variables” are write once, meaning once you set the value, it has the value forever inside the current scope. Some people don’t like this, and it is the thing people seem to complain about the most often after the way Erlang uses commas, periods, and semi-colons. As I’ve grown as a developer, I find myself defaulting more and more to making fields readonly in C# and making things final in Java. In fact, I mark literally every field, variable, and argument as final when I’m writing Java. I only remove the final modifier if I absolutely must. The benefit of this approach is that your code becomes cleaner. It does require more thought to avoid excess use of silly variable names (ie Client, Client2, Client3, Client4) as you mutate a values, but that also results in better code. Again your mileage may vary, but I find this aspect of the language benefits me.

Though it is still early, I am super happy with Erlang. I find it a joy to code in and am very glad I chose it for

A really great question I sometimes get is “Why build/use when you can just use X?” Where X is usually either HipChat, Campfire, or occasionally Skype. The simple answer is that these tools just aren’t good enough or innovative any more.

In 2008, I left Fog Creek and joined a small team at Cheezburger. The team was completely distributed and we needed a way to communicate. When we were three people, we could use just about anything and successfully shoot words at each other, but it didn’t take too long before we needed something more substantial.

Our first iteration was a combination of Skype and Campfire. Skype for voice and Campfire for group chat. Campfire lacks a lot of functionality, but it works great as a single chat room where people can post text and images and everyone else can see them. Campfire’s API is decent so we were able to write notification scripts, but bots had trouble staying connected. Campfire started to fail us when we had enough developers that we needed to break the dev team up into smaller teams working in certain knowledge domains. This meant that people started using Campfire for fun group watercooler stuff and Skype group chats for talking about work stuff. This opened a whole can of worms. Skype’s group chats are invite-only, so if someone was accidentally left out, they would be left out of the conversation. Additionally, we wanted these conversations to be available to the entire dev team, so other devs could use the discussion as a way to learn about our various knowledge domains. The end result was that we lost a lot of productivity due to repeated or missing communication.

Next we started looking for an alternative to Campfire that would solve our multi-room chat problem and possibly solve some of our other issues around scripts and bots. We looked into HipChat and a few others, but those didn’t quite fit our requirements. We eventually found had a lot of features we really liked and since it was built on top of IRC, it solved both our multi-room issue and our automation issues. The only problem with it was that it crashed every 5 minutes. This made it rather unusable, but we at least knew that it was possible to solve our problems.

Next we tried running ircd. This worked fantastically for most of the devs who were happy to fiddle with IRC clients, but for everyone else, it was just kind of a pain, because they were spending time making sense of IRC instead of getting work done. After IRC, we hosted JabbR for a while. JabbR solved the multi-room issue but had its own set of bugs, and it was eventually abandoned.

At this point, I feel like I have used most of the chat room tools companies have available, and I can say with confidence that they just aren’t that great, and could be a lot better.

So how is different?

First off, is built off of my experience using a bunch of chat services and from my years at Cheezburger. We didn’t just goof around and have fun in chat rooms; we built really cool powerful stuff. I led the team to build a giant distributed content network, spanning four continents, comprised of over a hundred servers, and fully capable of handling entire data center outages without affecting users. is written in Erlang, and built to scale from the ground up. We’ve incorporated numerous great features from and the rest to build a really solid chat experience. Additionally, we plan to include features like push-to-talk voice chat from within the web app and other clients. We are a fully distributed startup and we build as much for ourselves as for our customers. Want to give it a 30-day trial? Get started today at

In the past when I’ve used other chat services, the biggest hurdle was getting everyone logged in so we could give it a proper evaluation. Most of the time you create an account then invite your coworkers one at a time until everyone is signed in. Inevitably, a couple people will lose their invite email and you’ll forget to add others. This makes for a very frustrating evaluation.

Last week we released Google sign-in for We didn’t just make it so users could log in via google, we created the feature specifically to streamline the initial user experience. We have two settings related to Google sign-in. The first is the ability to automatically create accounts for people who sign-in via Google, and the other setting is to limit auto account creation to users who belong to a certain email domain.

For example, using these two features, I could setup a organization that automatically created accounts for users with email addresses. This means I can then send people the chat link, they can just log-in. No need to invite people or anything. 

In the case where you do need to invite people you can always use our easy bulk inviter, and save time that way too.

To make Google sign-in that much more awesome, if you register your organization via Google Apps sign-in, we automatically set both of those settings to reasonable defaults, meaning you can be up and running in less that a minute. Check out our demo video.

Give it a try for free today. Use coupon code googleauth to get a 3 month free trial instead of the normal one month! 

Today, joined the exclusive club of site that support SPDY/2 and SPDY/3. SPDY is built on HTTPS and provides some additional features for modern web applications.

What does get from SPDY?

If you are familiar with SPDY, you might be wondering what we get from using it. The most important feature of SPDY that we want to take advantage of is SPDY’s ability to transfer multiple simultaneous HTTP requests over a single SSL connection. This means that more active connections can be running so messages will never be delayed because you are doing something else in the application. Additionally, it means that the cost of establishing a connection and handshaking with the server is only paid once, so that will never delay message delivery either.

SPDY for the masses

Additionally, we don’t just want to be awesome. We want the entire internet to be more awesome. To that end, we have open sourced the code that makes it possible for us to support SPDY. You can already grab spdy_proxy from github ( Spdy_proxy is a lightweight, scalable SPDY to HTTP proxy. Run it on each of your web servers and then use HAProxy to handle SSL and load balancing. This means that you can keep running your current HTTP stack and use HAProxy and spdy_proxy to add SSL and SPDY support. (It should also be pointed out that you should consider using HAProxy to load balance your HTTP too, since it is pretty darn awesome.)

You might be wondering if using a tool like spdy_proxy would negate the benefit of SPDY since everything is still just getting translated into HTTP. The answer is a resounding no. This is because the client still has a single connection to HAProxy and HAProxy has a single connection to your web server. The only place where multiple connections occur is from the web server to the web server. This means that if there is a delay introduced by using spdy_proxy, it is just a few microseconds, while the benefit to users is at the very least several milliseconds.

We are very excited about adding SPDY to and providing a tool to make it easier for everyone else to support SPDY too! has been humming along for a bit now, but I wanted to take a moment and talk about three really exciting features that recently went out.

User & Custom Emoji

A large set of standard emoji has been supported since almost day one, but everywhere I’ve ever chatted, people want to define their own emoji. When I was a Cheezburger, we had a whole host of Cheezburger specific emoji, and it was dabes! At, we want everyone to have as much fun with emoji as they possibly can.

User emoji is a set of emoji that is composed of all of the users’ gravatars. You just type :<username>: and you get a picture of that person’s gravatar. So :stefan: would show Stefan’s face.

Custom emoji is an even bigger deal. You can take any word and turn it into any image, then you can make emoji-only sentences… :chatty: :heart: :erlang:

As an added bonus, whether you are trying to use the standard, user or custom emoji, you automatically get tab completion. Type : and then you can tab through all the emoji, or if you can type the first part of the emoji and tab to complete it.


Url Maps

Url Maps are kind of an advanced feature, but they are crazy powerful. A url map is a regular expression and a url generator. If you start with the regular expression map:”([^”]+)” and use$1 for the url replacement, you get fancy Google Maps integration.



Ok, I admit, IRC isn’t new to, but I wanted to take a moment to remind you, supports IRC!