“You probably have heard of LevelDB it’s a blazing fast key value store (as a library not a daemon), that uses Snappy compression.
There is plenty of usages for it, the API is very simple at least in Go (I will be using Goleveldb).
The key is a
byte the value is a
byte so you can “get”, “put” & “delete” that’s it.
I needed a low memory low cpu system that could collect millions of geo data and query over them, Geohash has an interesting property you can encode longitude and latitude into a string :
f2m616nn this hash represents the lat & long 46.770, -71.304 f2m616nn, if you shorten the string to
f2m61 it still refers to the same lat & long but with less precisionsf2m61.
A 4 digits hash leads to 19545 meters precision, to perfom a lookup around a position you simply query for the 8 adjacent blocks.A Geohash library for Go.
Here you would store all of your data points matching a geohash to the same set.
Problem there is no such thing as a set in LevelDB.
But there is a cursor so you can seek to a position then iterate over the next or previous one (byte ordered).
So your data could be stored that way: 4 digits geohash + a uniq id.
Then you can perform proximity lookup by searching for the 8 adjacents hashes from the position your are looking with a precision of 20km, good but not very flexible.
We can have a more generic solution, first we need a key a simple int64 uniq id…”