“Goliath is an open source version of the non-blocking (asynchronous) Ruby web server framework powering PostRank. It is a lightweight framework designed to meet the following goals: bare metal performance, Rack API and middleware support, simple configuration, fully asynchronous processing, and readable and maintainable code (read: no callbacks).

The framework is powered by an EventMachine reactor, a high-performance HTTP parser and Ruby 1.9 runtime. One major major advantage Goliath has over other asynchronous frameworks is the fact that by leveraging Ruby fibers introduced in Ruby 1.9+, it can untangle the complicated callback-based code into a format we are all familiar and comfortable with: linear execution, which leads to more maintainable and readable code.

Each Goliath request is executed in its own Ruby fiber and all asynchronous I/O operations can transparently suspend and later resume the processing without requiring the developer to write any additional code. Both request processing and response processing can be done in fully asynchronous fashion: streaming uploads, firehose API’s, request/response, and so on…”



Scale Something: How Draw Something rode its rocket ship of growth

“We’ve learned to keep things simple. The original backend for Draw Something was designed as a simple key/value store with versioning. The service was built into our existing ruby API (using the merb framework and thin web server). Our initial idea was why not use our existing API for all the stuff we’ve done before, like users, signup/login, virtual currency, inventory; and write some new key/value stuff for Draw Something? Since we design for scale, we initially chose Amazon S3 as our data store for all this key/value data. The idea behind this was why not sacrifice some latency but gain unlimited scalability and storage.
The rest of our stack is pretty standard. Anyone who wants to build scalable systems will attempt to make every layer of the system scale independently from the rest. As the web frontend we use NGINX web server, which points to HAProxy software load balancer, which then hits our ruby API running on a thin web server. The main datastore behind this is MySQL – sharded when absolutely necessary. We use memcached heavily and redis for our asynchronous queueing, using the awesome ruby library called resque…”


OpenResty – a powerful web app server by extending nginx

“OpenResty (aka. ngx_openresty) is a full-fledged web application server by bundling the standard Nginx core, lots of 3rd-party Nginx modules, as well as most of their external dependencies. By taking advantage of various well-designed Nginx modules, OpenResty effectively turns the nginx server into a powerful web app server, in which the web developers can use the Lua programming language to script various existing nginx C modules and Lua modules and construct extremely high-performance web applications that is capable to handle 10K+ connections. OpenResty aims to run your server-side web app completely in the Nginx server, leveraging Nginx’s event model to do non-blocking I/O not only with the HTTP clients, but also with remote backends like MySQL, PostgreSQL, Memcached, and Redis.
See Components for the complete list of software bundled in OpenResty…”


See GettingStarted on how to quickly setup an OpenResty server that can say hello world over HTTP. Or you can go to the Download section to grab OpenResty’s source code tarball directly…”

Just how big are porn sites?

“While it’s difficult domain to penetrate — hard numbers are few and far between — we know for a fact that porn sites are some of the most trafficked parts of the internet. According to Google’s DoubleClick Ad Planner, which tracks users across the web with a cookie, dozens of adult destinations populate the top 500 websites. Xvideos, the largest porn site on the web with 4.4 billion page views per month, is three times the size of CNN or ESPN, and twice the size of Reddit. LiveJasmin isn’t much smaller. YouPorn, Tube8, and Pornhub — they’re all vast, vast sites that dwarf almost everything except the Googles and Facebooks of the internet…”


Sharding Postgres with Instagram

“On Tuesday last week we had a terrific SFPUG meeting at which Mike Kreiger of Instagram explained how they grew and eventually sharded their 2TB of Postgres data to support 27 million users.  It’s a great presentation which explains the growth process of a successful web/mobile startup, as well as horizontally scaling PostgreSQL. Yes, you too can use PostgreSQL to make One Billion Dollars!
Video is on UStream.   Sorry you can’t see the slides on the video; we had technical issues with the camera.  Slides are here, you can click along with Mike talking on your own…”


OpenStack Swift eventual consistency analysis & bottlenecks

Swift is the software behind the OpenStack Object Storage service. This service provides a simple storage service for applications using RESTful interfaces, providing maximum data availability and storage capacity. I explain here how some parts of the storage and replication in Swift works, and show some of its current limitations. If you don’t know Swift and want to read a more “shallow” overview first, you can read John Dickinson’s Swift Tech Overview…”


Functional Programming in C++

“Probably everyone reading this has heard “functional programming” put forth as something that is supposed to bring benefits to software development, or even heard it touted as a silver bullet.  However, a trip to Wikipedia for some more information can be initially off-putting, with early references to lambda calculus and formal systems.  It isn’t immediately clear what that has to do with writing better software…”


My Year of Riak

“Startups often ask my opinion on databases for their new application. In the past year we’ve launched a big CouchDB-based application, and we’ve helped build stylesclub.com, a Riak-based facebook app (but not yet launched). We’ve also helped launch ming.ly, an Amazon SimpleDB based application. Each of these has been an opportunity to further develop my philosophy about what makes a good database for a website or mobile app, and when to choose each one. I have a separate series coming on database tradeoffs but since there isn’t that much information on Riak out in the wild yet, I will start by profiling my thoughts on the database. These are my opinions and thoughts after a bit more than a year of using it at Inaka…”