In last couple of years, we have observed evolution of several message brokers and queuing services which are all fast, reliable and scalable. While the list is long, in this blog, I will limit the discussion to SQS, Kinesis and Kafka. Simple Queuing Service (SQS) is a fully managed and scalable queuing service on AWS. Kinesis is another service offered by AWS that makes it easy to load and analyze streaming data and also provides the ability to build custom streaming data applications for special requirements. Apache Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system which is often used in place of traditional message brokers like JMS and AMQP because of its characteristics like higher throughput, reliability and replication.
While making decisions about which messaging system is right for you, it is important to understand not only the technical differences but also the implications of operational costs both in terms of running them at scale as well as monitoring them. In this blog, I will touch upon our experiences and learning at OpsClarity, based on our evaluation of messaging systems and our migration from SQS to Kafka.