Go: g0, Special Goroutine

All the goroutines created in Go are under the management of an internal scheduler. The Go scheduler tries to give running time to all goroutines and keep all CPU busy with running goroutines when the current ones are blocked or terminated. It actually runs as a special goroutine.

Scheduling goroutine

Go limits the number of OS thread running thanks to GOMAXPROCS variable simultaneously. That means Go has to schedule and manage goroutines on each of the running threads. This role is delegated to a special goroutine, called g0, that is the first goroutine created for each OS thread:

Then, it will schedule ready goroutines to run on the threads.


Building a BitTorrent client from the ground up in Go

BitTorrent is a protocol for downloading and distributing files across the Internet. In contrast with the traditional client/server relationship, in which downloaders connect to a central server (for example: watching a movie on Netflix, or loading the web page you’re reading now), participants in the BitTorrent network, called peers, download pieces of files from each other—this is what makes it a peer-to-peer protocol. We’ll investigate how this works, and build our own client that can find peers and exchange data between them.

diagram showing the difference between client/server (all clients connecting to one server) and peer-to-peer (peers connecting to each other) relationships

The protocol evolved organically over the past 20 years, and various people and organizations added extensions for features like encryption, private torrents, and new ways of finding peers. We’ll be implementing the original spec from 2001 to keep this a weekend-sized project.


HTTP APIs is a new flavor of API Gateway. Build low-latency & cost-effective REST APIs

Announcing the API Gateway HTTP API

We talk to customers every day that use API Gateway for critical production applications. This includes everything ranging from simple HTTP proxies to full-blown API management with request transformation, authentication, and validation. API Gateway supports REST APIs and WebSocket APIs, but customers have told us they want more features, lower latency, and lower cost.

Customers have explained their need for the core features of API Gateway at a lower price along with an easier developer experience. Based on this feedback, we are excited to announce the availability of HTTP APIs (Preview).

HTTP APIs is a new flavor of API Gateway. It focuses on delivering enhanced features, improved performance, and an easier developer experience for customers building with API Gateway. Today, we’re launching the first phase, and we will continue to enhance HTTP APIs over the next few months.

We are introducing a new pricing model for HTTP APIs that better represents customer usage patterns. Staying true to our serverless principles, you will only pay for the requests you receive.  With existing REST APIs, you pay $3.50/million requests plus data transferred out.

With HTTP APIs, we have reduced request pricing to $1.00/million requests for the first 300 million requests, and $0.90/million for all requests after that. Most customers will see an average cost saving up to 70%, when compared to API Gateway REST APIs. In addition, you will see significant performance improvements in the API Gateway service overhead.


Go: Concurrency & Scheduler Affinity

Switching a goroutine from an OS thread to another one has a cost and can slow down the application if it occurs too often. However, through time, the Go scheduler has addressed this issue. It now provides affinities between the goroutines and the thread when working concurrently. Let’s go back years ago to understand this improvement.


Over 150 of the Best Machine Learning, NLP, and Python Tutorials

While machine learning has a rich history dating back to 1959, the field is evolving at an unprecedented rate. In a recent article, I discussed why the broader artificial intelligence field is booming and likely will for some time to come. Those interested in learning ML may find it daunting to get started.


Fast Cloudfront log queries using AWS Athena and Serverless

Recently we’ve needed to run queries on our Cloudfront web logs, for example, to determine the number of Googlebot hits to certain page types. We configured Cloudfront to write logs to an S3 bucket, and we set up an AWS Athena table to query those logs.

(By the way, if you haven’t used Athena before, I think it’s nothing short of amazing — the ability to use SQL to query fields from a bunch of gzipped text files sitting in S3? That’s incredible.)

But over time our queries became really slow. For every query, Athena had to scan the entire log history, reading through all the log files in our S3 bucket. This churned through a lot of data (about 120 GB) and made the queries slow and pretty expensive — taking over 65 seconds to run and costing about $0.70 per query.

So we did some searching and came across this AWS blog post, which describes how to set up Athena partitioning to vastly speed up date-based queries. We decided to use AWS Lambda to partition our logs, along with the Serverless framework to easily deploy the Lambda code.