Comprehensive Overview of Storage Scalability in Docker

“First, a brief backstory on the storage situation for Docker since it was open-sourced in early 2013. At that time, Docker relied on a filesystem called AUFS (advanced multi layered unification filesystem). This Union filesystem provided the necessary features to support several of Docker’s main selling points:
– container creation speed
– copy-on-write image->container

Trees as Linked Lists in Common Lisp

“If you are starting to learn Common Lisp, and have already read the chapter about “Cons” from an introductory book (also called cons cells), then you could try the following exercise. How to represent a tree using nothing but conses; and abstract that implementation with functions to create and traverse the tree.

PLEASE NOTE, that if you need to represent trees in a production program you shouldn’t use lists as described here unless you have a good reason. This is only an exercise in understanding how cons cells work…”

Python 3: threat or menace?

“I wish I still had my copy of this: a Harvard Lampoon parody of Life magazine from the ’60s, displaying a picture of a flying saucer and the ominous headline: “UFOs: Threat or Menace?”.

I was reminded of this when reading some recent articles worrying about the slow transition from Python 2 to Python 3, such as Python 3 is Killing Python. The authors of such articles, and Python developers in general, really like Python, and for the most part like Python 3. Their main concern is that the protracted 2-3 straddle will hurt Python’s popularity…”

Billion Messages – Art of Architecting scalable ElastiCache Redis tier

“Whenever we are designing a highly scalable architectures on AWS running thousands of application servers and supporting millions of requests, usage of NoSQL solutions have become inevitable part. One such solution we often been using for years on AWS is Redis. We love Redis. AWS introduced ElastiCache Redis on 2013 and we started using the same since it eased the management and operational efforts. In this article i am going to share my experience on designing large scale Redis tiers supporting billions of messages per day on AWS, step by step guide on how to deploy the same, what are the Implications you face at scale ? Best Practices to be adopted while designing sharded+replicated Redis Tiers etc…”

How Twitter Uses Redis to Scale – 105TB RAM, 39MM QPS, 10,000+ Instances

Yao Yue has worked on Twitter’s Cache team since 2010. She recently gave a really great talk: Scaling Redis at Twitter. It’s about Redis of course, but it’s not just about Redis.

Yao has worked at Twitter for a few years. She’s seen some things. She’s watched the growth of the cache service at Twitter explode from it being used by just one project to nearly a hundred projects using it. That’s many thousands of machines, many clusters, and many terabytes of RAM.

It’s clear from her talk that’s she’s coming from a place of real personal experience and that shines through in the practical way she explores issues. It’s a talk well worth watching.

As you might expect, Twitter has a lot of cache…”

Introducing mcrouter: A memcached protocol router for scaling memcached deployments

“Since any client that wants to talk to memcached can already speak the standard ASCII memcached protocol, we use that as the common API and enter the picture silently. To a client, mcrouter looks like a memcached server. To a server, mcrouter looks like a normal memcached client. But mcrouter’s feature-rich configurability makes it more than a simple proxy…”

A POSIX Queue Implementation

“In Unix Network Programming Volume 2, W. Richard Stevens [Stevens 99, Ch. 5] gives an eloquent explanation of the POSIX message queue. He makes the abstraction very accessible, using diagrams and a tight implementation of the API in about five-hundred lines of C. His work inspired me to examine the POSIX Queue implementation of my favorite operating system kernel, Illumos (the continuation of OpenSolaris after Oracle closed it down). The following is what I found…”