Anomaly detection on Amazon DynamoDB Streams using the Amazon SageMaker Random Cut Forest algorithm

Have you considered introducing anomaly detection technology to your business? Anomaly detection is a technique used to identify rare items, events, or observations which raise suspicion by differing significantly from the majority of the data you are analyzing.  The applications of anomaly detection are wide-ranging including the detection of abnormal purchases or cyber intrusions in banking, spotting a malignant tumor in an MRI scan, identifying fraudulent insurance claims, finding unusual machine behavior in manufacturing, and even detecting strange patterns in network traffic that could signal an intrusion.

There are many commercial products to do this, but you can easily implement an anomaly detection system by using Amazon SageMaker, AWS Glue, and AWS Lambda. Amazon SageMaker is a fully-managed platform to help you quickly build, train, and deploy machine learning models at any scale. AWS Glue is a fully-managed ETL service that makes it easy for you to prepare your data/model for analytics. AWS Lambda is a well-known a serverless real-time platform. Using these services, your model can be automatically updated with new data, and the new model can be used to alert for anomalies in real time with better accuracy.

In this blog post I’ll describe how you can use AWS Glue to prepare your data and train an anomaly detection model using Amazon SageMaker. For this exercise, I’ll store a sample of the NAB NYC Taxi data in Amazon DynamoDB to be streamed in real time using an AWS Lambda function.

The solution that I describe provides the following benefits:

  • You can make the best use of existing resources for anomaly detection. For example, if you have been using Amazon DynamoDB Streams for disaster recovery (DR) or other purposes, you can use the data in that stream for anomaly detection. In addition, stand-by storage usually has low utilization. The data in low awareness can be used for training data.
  • You can automatically retrain the model with new data on a regular basis with no user intervention.
  • You can make it easy to use the Random Cut Forest built-in Amazon SageMaker algorithm. Amazon SageMaker offers flexible distributed training options that adjust to your specific workflows in a secure and scalable environment.

https://aws.amazon.com/pt/blogs/machine-learning/anomaly-detection-on-amazon-dynamodb-streams-using-the-amazon-sagemaker-random-cut-forest-algorithm/

Advertisements

How to Minimize AWS Lambda Cold Starts

Serverless architecture is the new kid on the block, and according to a recent surveyby Serverless, Inc., a vast majority of developers will start using it by the end of the year. The serverless paradigm involves running code in the cloud without managing any servers, allowing you to build business logic and create value without ever thinking about the infrastructure or underlying software. Essentially, it lets you focus on your code.

Serverless does not only cover AWS Lambda and other FaaS providers, but basically everything you can use to run code, host files, and store images and data. This means that you, as an engineer, don’t need to manage, scale, or operate any servers whatsoever. And here’s the icing on the cake: you only pay for the time your code is running!

Although serverless offers many benefits, there are still some pitfalls, such as latency. In this article, we’ll discuss how to minimize latency in AWS Lambda. This dreaded phenomenon is caused by cold starts, which are, by definition, slower initial responses from your serverless APIs.

Before we begin, let’s dig deeper into what FaaS is and how it works.

https://epsagon.com/blog/how-to-minimize-aws-lambda-cold-starts/

Compute, Database, Messaging, Analytics, and Machine Learning Integration for AWS Step Functions

AWS Step Functions is a fully managed workflow service for application developers. You can think & work at a high level, connecting and coordinating activities in a reliable and repeatable way, while keeping your business logic separate from your workflow logic. After you design and test your workflows (which we call state machines), you can deploy them at scale, with tens or even hundreds of thousands running independently and concurrently. Step Functions tracks the status of each workflow, takes care of retrying activities on transient failures, and also simplifies monitoring and logging. To learn more, step through the Create a Serverless Workflow with AWS Step Functions and AWS Lambdatutorial.

Since our launch at AWS re:Invent 2016, our customers have made great use of Step Functions (my post, Things go Better with Step Functions describes a real-world use case). Our customers love the fact that they can easily call AWS Lambda functions to implement their business logic, and have asked us for even more options.

https://aws.amazon.com/pt/blogs/aws/new-compute-database-messaging-analytics-and-machine-learning-integration-for-aws-step-functions/

https://www.tbray.org/ongoing/When/201x/2018/11/30/States-Language-Connectors

The Top Three AWS re:Invent Serverless Announcements

Last week was AWS re:Invent which is the most busy time of the year for those of us a part of the AWS ecosystem and arguably the most important. Every year Amazon inundates us with a large number of announcements and it can be overwhelming to keep track of them all. This year amazon announced new EC2 instance types, a time series database, and a slew of machine learning offerings… They also announced a service to retrieve data from orbiting satellites, a rack you can install in your data center with AWS services, an R/C car, and a blockchain service.

It’s easy to miss things in all of that so we’re going to recap what we see as the biggest announcements. Plus we’ll also briefly cover the fun we had with our “appearance” at Stackery’sre:Invent booth.

https://www.serverlessops.io/blog/the-top-three-aws-reinvent-serverless-announcements

re:Capping re:Invent: AWS goes all-in on Serverless

Last week I spent six incredibly exhausting days in Las Vegas at the AWS re:Invent conference. More than 50,000 developers, partners, customers, and cloud enthusiasts came together to experience this annual event that continues to grow year after year. This was my first time attending, and while I wasn’t quite sure what to expect, I left with not just the feeling that I got my money’s worth, but that AWS is doing everything in their power to help customers like me succeed.

There have already been some really good wrap-up posts about the event. Take a look at James Beswick’s What I learned from AWS re:Invent 2018, Paul Swail’s What new use cases do the re:Invent 2018 serverless announcements open up?, and All the Serverless announcements at re:Invent 2018 from the Serverless, Inc. blog. There’s a lot of good analysis in these posts, so rather than simply rehash everything, I figured I touch on a few of the announcements that I think really matter. We’ll get to that in a minute, but first I want to point out a few things about Amazon Web Services that I learned this past week.

https://www.jeremydaly.com/recapping-reinvent-aws-goes-all-in-on-serverless/

All the Serverless announcements at re:Invent 2018

re:Invent 2018 announcements

Latest:

Most Exciting:

Lambda:

https://serverless.com/blog/reinvent-2018-serverless-announcements/