Corral is a MapReduce framework designed to be deployed to serverless platforms, like AWS Lambda. It presents a lightweight alternative to Hadoop MapReduce. Much of the design philosophy was inspired by Yelp’s mrjob — corral retains mrjob’s ease-of-use while gaining the type safety and speed of Go.
Corral’s runtime model consists of stateless, transient executors controlled by a central driver. Currently, the best environment for deployment is AWS Lambda, but corral is modular enough that support for other serverless platforms can be added as support for Go in cloud functions improves.
Corral is best suited for data-intensive but computationally inexpensive tasks, such as ETL jobs.
More details about corral’s internals can be found in this blog post.
Every concurrency API needs a way to run code concurrently. Here’s some examples of what that looks like using different APIs:
go myfunc(); // Golang pthread_create(&thread_id, NULL, &myfunc); /* C with POSIX threads */ spawn(modulename, myfuncname, ) % Erlang threading.Thread(target=myfunc).start() # Python with threads asyncio.create_task(myfunc()) # Python with asyncio
There are lots of variations in the notation and terminology, but the semantics are the same: these all arrange for myfunc to start running concurrently to the rest of the program, and then return immediately so that the parent can do other things.
Another option is to use callbacks:
Again, the notation varies, but these all accomplish the same thing: they arrange that from now on, if and when a certain event occurs, then myfunc will run. Then once they’ve set that up, they immediately return so the caller can do other things. (Sometimes callbacks get dressed up with fancy helpers like promise combinators, or Twisted-style protocols/transports, but the core idea is the same.)
And… that’s it. Take any real-world, general-purpose concurrency API, and you’ll probably find that it falls into one or the other of those buckets (or sometimes both, like asyncio).
This repository provides the
dqlite Go package, which can be used to replicate a SQLite database across a cluster, using the Raft algorithm.
- No external processes needed: dqlite is just a Go library, you link it it to your application exactly like you would with SQLite.
- Replication needs a SQLite patch which is not yet included upstream.
- The Go Raft package from Hashicorp is used internally for replicating the write-ahead log frames of SQLite across all nodes.
The main differences from rqlite are:
- Full support for transactions
- No need for statements to be deterministic (e.g. you can use
- Frame-based replication instead of statement-based replication, this means in dqlite there’s more data flowing between nodes, so expect lower performance. Should not really matter for most use cases.
Everyone, the day has come.
AWS Lambda is finally. Compatible. With Golang. 🖖
Here’s how you can start using Go with the Serverless Framework RIGHT NOW and deploy Lambdas to your heart’s content.
A little over a year ago, I joined Cloud Foundry to work on Loggregator, Cloud Foundry’s application logging component. Its core concern is best-effort log delivery without pushing back on upstream writers. Loggregator is written entirely in Go.
After spending more than a thousand hours working with Go in a non-trivial code base, I still admire the language and enjoy using it. Nonetheless, our team struggled with a number of problems, many of which seem unique to Go. What follows is a list of the most salient problems.
Summary: To learn Go, I ported the backend of a small site I run from Python to Go, and had a fun, pain-free experience doing so.
I’ve been wanting to learn Go for a while now: I like the philosophy of a language that’s small, has a gentle learning curve, and compiles very fast (for a statically-typed language). What pushed me over the line to actually go and do it was seeing more and more fast, robust tools that are written in Go – Docker and ngrok are two I’ve used recently.
The philosophy of Go is not to everyone’s taste (no exceptions, no user-defined generics, etc), but it fit my mental model well. Simple, speedy, does things the obvious way. During the port, I was especially impressed with how robust the standard library and tooling was.