curl statistics made simple

httpstat is a single file Python script that has no dependency and is compatible with Python 3.

https://github.com/reorx/httpstat

 

ifconfig vs ip: What’s Difference and Comparing Network Configuration

Linux based distributions have featured set of commands which provide way to configure networking in easy and powerful way through command-line. These set of commands are available from net-tools package which has been there for a long time on almost all distributions, and includes commands like: ifconfig, route, nameif, iwconfig, iptunnel, netstat, arp.

These commands are just about sufficient in configuring the network in a way any novice or an expert Linux user would want, but due to advancement in Linux kernel over past years and unmaintainable of this packaged set of commands, they are getting deprecated and a more powerful alternative which has ability to replace all of these commands is emerging.

This alternative has also been there for quite some time now and is much more powerful than any of these commands. Rest of sections would highlight this alternative and compare it with one of the command from net-tools package i.e. ifconfig.

http://www.tecmint.com/ifconfig-vs-ip-command-comparing-network-configuration/

Monitoring Kafka performance metrics

This post is Part 1 of a 3-part series about monitoring Kafka. Part 2 is about collecting operational data from Kafka, and Part 3 details how to monitor Kafka with Datadog.

What is Kafka?

Kafka is a distributed, partitioned, replicated, log service developed by LinkedIn and open sourced in 2011. Basically it is a massively scalable pub/sub message queue architected as a distributed transaction log. It was created to provide “a unified platform for handling all the real-time data feeds a large company might have”.1

There are a few key differences between Kafka and other queueing systems like RabbitMQ, ActiveMQ, or Redis’s Pub/Sub:

  1. As mentioned above, it is fundamentally a replicated log service.
  2. It does not use AMQP or any other pre-existing protocol for communication. Instead, it uses a custom binary TCP-based protocol.
  3. It is very fast, even in a small cluster.
  4. It has strong ordering semantics and durability guarantees.

Despite being pre-1.0, (current version is 0.9.0.1), it is production-ready, and powers a large number of high-profile companies including LinkedIn, Yahoo, Netflix, and Datadog.

https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics

hitchhiker-tree: Functional, persistent, off-heap, high performance data structure

Hitchhiker trees are a newly invented (by @dgrnbrg) datastructure, synthesizing fractal trees and functional data structures, to create fast, snapshottable, massively scalable databases.

What’s in this Repository?

The hitchhiker namespaces contain a complete implementation of a persistent, serializable, lazily-loaded hitchhiker tree. This is a sorted key-value datastructure, like a scalable sorted-map. It can incrementally persist and automatically lazily load itself from any backing store which implements a simple protocol.

Outboard is a sample application for the hitchhiker tree. It includes an implementation of the IO subsystem backed by Redis, and it manages all of the incremental serialization and flushing.

The hitchhiker tree is designed very similarly to how Datomic’s backing trees must work–I would love to see integration with DataScript for a fully open source Datomic.

https://github.com/datacrypt-project/hitchhiker-tree

Five Months of Kubernetes

For the past year, Descomplica moved towards a more service-oriented architecture for its core components (auth, search, etc) and we’ve been using Elastic Beanstalk from the start to orchestrate the deployment of those services to AWS.

It was a good decision at the time. In general, Elastic Beanstalk works fine and has a very gentle learning curve; it didn’t take long for all teams to start using it for their projects.

Fast-forward a few months, everything was nice and good. Our old problems were solved, but – as you might have guessed – we had new ones to worry about.

http://danielmartins.ninja/posts/five-months-of-kubernetes.html

How To Create a High Availability Setup with Corosync, Pacemaker, and Floating IPs on Ubuntu 14.04

This tutorial will demonstrate how you can use Corosync and Pacemaker with a Floating IP to create a high availability (HA) server infrastructure on DigitalOcean.

Corosync is an open source program that provides cluster membership and messaging capabilities, often referred to as the messaging layer, to client servers. Pacemaker is an open source cluster resource manager (CRM), a system that coordinates resources and services that are managed and made highly available by a cluster. In essence, Corosync enables servers to communicate as a cluster, while Pacemaker provides the ability to control how the cluster behaves.

Goal

When completed, the HA setup will consist of two Ubuntu 14.04 servers in an active/passive configuration. This will be accomplished by pointing a Floating IP, which is how your users will access your web service, to point to the primary (active) server unless a failure is detected. In the event that Pacemaker detects that the primary server is unavailable, the secondary (passive) server will automatically run a script that will reassign the Floating IP to itself via the DigitalOcean API. Thus, subsequent network traffic to the Floating IP will be directed to your secondary server, which will act as the active server and process the incoming traffic.

This diagram demonstrates the concept of the described setup:

Active/passive Diagram

Note: This tutorial only covers setting up active/passive high availability at the gateway level. That is, it includes the Floating IP, and the load balancer servers—Primary and Secondary. Furthermore, for demonstration purposes, instead of configuring reverse-proxy load balancers on each server, we will simply configure them to respond with their respective hostname and public IP address.

To achieve this goal, we will follow these steps:

  • Create 2 Droplets that will receive traffic
  • Create Floating IP and assign it to one of the Droplets
  • Install and configure Corosync
  • Install and configure Pacemaker
  • Configure Floating IP Reassignment Cluster Resource
  • Test failover
  • Configure Nginx Cluster Resource

https://www.digitalocean.com/community/tutorials/how-to-create-a-high-availability-setup-with-corosync-pacemaker-and-floating-ips-on-ubuntu-14-04