Distributed SQLite for Go applications

This repository provides the dqlite Go package, which can be used to replicate a SQLite database across a cluster, using the Raft algorithm.

Design higlights

  • No external processes needed: dqlite is just a Go library, you link it it to your application exactly like you would with SQLite.
  • Replication needs a SQLite patch which is not yet included upstream.
  • The Go Raft package from Hashicorp is used internally for replicating the write-ahead log frames of SQLite across all nodes.

How does it compare to rqlite?

The main differences from rqlite are:

  • Full support for transactions
  • No need for statements to be deterministic (e.g. you can use time())
  • Frame-based replication instead of statement-based replication, this means in dqlite there’s more data flowing between nodes, so expect lower performance. Should not really matter for most use cases.

https://github.com/CanonicalLtd/dqlite

Advertisements

Amazon DynamoDB Continuous Backups and Point-In-Time Recovery (PITR)

The Amazon DynamoDB team is back with another useful feature hot on the heels of encryption at rest. At AWS re:Invent 2017 we launched global tables and on-demand backup and restore of your DynamoDB tables and today we’re launching continuous backups with point-in-time recovery (PITR).

You can enable continuous backups with a single click in the AWS Management Console, a simple API call, or with the AWS Command Line Interface (CLI). DynamoDB can back up your data with per-second granularity and restore to any single second from the time PITR was enabled up to the prior 35 days. We built this feature to protect against accidental writes or deletes. If a developer runs a script against production instead of staging or if someone fat-fingers a DeleteItem call, PITR has you covered. We also built it for the scenarios you can’t normally predict. You can still keep your on-demand backups for as long as needed for archival purposes but PITR works as additional insurance against accidental loss of data. Let’s see how this works.

https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-continuous-backups-and-point-in-time-recovery-pitr/

A Decade of Dynamo: Powering the next wave of high-performance, internet-scale applications

Today marks the 10 year anniversary of Amazon’s Dynamo whitepaper, a milestone that made me reflect on how much innovation has occurred in the area of databases over the last decade and a good reminder on why taking a customer obsessed approach to solving hard problems can have lasting impact beyond your original expectations.

“Dynamo: Amazon’s Highly Available Key-value Store” received the ACM SIGOPS 2017 Hall of Fame Award

http://www.allthingsdistributed.com/2017/10/a-decade-of-dynamo.html

Introducing Cloud Firestore: Our New Document Database for Apps

Today we’re excited to launch Cloud Firestore, a fully-managed NoSQL document database for mobile and web app development. It’s designed to easily store and sync app data at global scale, and it’s now available in beta.

Key features of Cloud Firestore include:

  • Documents and collections with powerful querying
  • iOS, Android, and Web SDKs with offline data access
  • Real-time data synchronization
  • Automatic, multi-region data replication with strong consistency
  • Node, Python, Go, and Java server SDKs

And of course, we’ve aimed for the simplicity and ease-of-use that is always top priority for Firebase, while still making sure that Cloud Firestore can scale to power even the largest apps.

https://firebase.googleblog.com/2017/10/introducing-cloud-firestore.html

Pandas and Python Real World Project (GPS data)

“To celebrate my sub-30 minute finish on the manitou springs incline, I figured I’d drop some python and pandas knowledge while analyzing the data using beautifulsoup, pandas, python dateutil, python googlemaps, python geopy and the standard library.

I try to look at things in a non-trivial way and reflective of actual problems you encounter analyzing real world data. The data file is here https://gist.github.com/jrjames83/4de9d124e5f43a61be9cb2a… come code along!”

https://www.youtube.com/playlist?list=PLImyDqSBQbdmicMPRW0Yo5QHbeOIwC765

Use all the Databases

Ever wanted to use a few different databases to build your app? Different types of databases are meant for different purposes, so it often makes sense to combine them. You might be hesitant due to the complexity of maintenance and coding, but it can be easy if you combine Compose and GraphQL: instead of writing a number of complex REST endpoints, each querying multiple databases, you set up a single GraphQL endpoint that provides whatever data the client wants using your simple data fetching functions.

This tutorial is meant for anyone who provides or fetches data, whether it’s a backend dev writing an API (in any language) or a frontend web or mobile dev fetching data from the server. We’ll learn about the GraphQL specification, set up a GraphQL server, and fetch data from five different data sources. The code is in Javascript, but you’ll still get a good idea of GraphQL without knowing the language.

https://www.compose.com/articles/use-all-the-databases-part-1/