Announcing the API Gateway HTTP API
We talk to customers every day that use API Gateway for critical production applications. This includes everything ranging from simple HTTP proxies to full-blown API management with request transformation, authentication, and validation. API Gateway supports REST APIs and WebSocket APIs, but customers have told us they want more features, lower latency, and lower cost.
Customers have explained their need for the core features of API Gateway at a lower price along with an easier developer experience. Based on this feedback, we are excited to announce the availability of HTTP APIs (Preview).
HTTP APIs is a new flavor of API Gateway. It focuses on delivering enhanced features, improved performance, and an easier developer experience for customers building with API Gateway. Today, we’re launching the first phase, and we will continue to enhance HTTP APIs over the next few months.
We are introducing a new pricing model for HTTP APIs that better represents customer usage patterns. Staying true to our serverless principles, you will only pay for the requests you receive. With existing REST APIs, you pay $3.50/million requests plus data transferred out.
With HTTP APIs, we have reduced request pricing to $1.00/million requests for the first 300 million requests, and $0.90/million for all requests after that. Most customers will see an average cost saving up to 70%, when compared to API Gateway REST APIs. In addition, you will see significant performance improvements in the API Gateway service overhead.
Recently we’ve needed to run queries on our Cloudfront web logs, for example, to determine the number of Googlebot hits to certain page types. We configured Cloudfront to write logs to an S3 bucket, and we set up an AWS Athena table to query those logs.
(By the way, if you haven’t used Athena before, I think it’s nothing short of amazing — the ability to use SQL to query fields from a bunch of gzipped text files sitting in S3? That’s incredible.)
But over time our queries became really slow. For every query, Athena had to scan the entire log history, reading through all the log files in our S3 bucket. This churned through a lot of data (about 120 GB) and made the queries slow and pretty expensive — taking over 65 seconds to run and costing about $0.70 per query.
So we did some searching and came across this AWS blog post, which describes how to set up Athena partitioning to vastly speed up date-based queries. We decided to use AWS Lambda to partition our logs, along with the Serverless framework to easily deploy the Lambda code.
Today we’re announcing AWS Lambda Destinations for asynchronous invocations. This is a feature that provides visibility into Lambda function invocations and routes the execution results to AWS services, simplifying event-driven applications and reducing code complexity.
When a function is invoked asynchronously, Lambda sends the event to an internal queue. A separate process reads events from the queue and executes your Lambda function. When the event is added to the queue, Lambda previously only returned a 2xx status code to confirm that the queue has received this event. There was no additional information to confirm whether the event had been processed successfully.
A common event-driven microservices architectural pattern is to use a queue or message bus for communication. This helps with resilience and scalability. Lambda asynchronous invocations can put an event or message on Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), or Amazon EventBridge for further processing. Previously, you needed to write the SQS/SNS/EventBridge handling code within your Lambda function and manage retries and failures yourself.
With Destinations, you can route asynchronous function results as an execution record to a destination resource without writing additional code. An execution record contains details about the request and response in JSON format including version, timestamp, request context, request payload, response context, and response payload. For each execution status such as Success or Failure you can choose one of four destinations: another Lambda function, SNS, SQS, or EventBridge. Lambda can also be configured to route different execution results to different destinations.
Serverless (in some people’s minds) currently encompasses:
- Anything that looks like “Function as a Service” like AWS Lambda, Google Cloud Functions, and Azure Functions
- Anything that can run a Function as a Service system, like OpenFaaS and similar
- Ok… lots of people think it’s a synonym for Function as a Service (spoiler: it’s not)
- Any solution that runs “on demand compute” such as Google App Engine (spoiler: it’s not)
- Anything that runs a container on demand like Google Cloud Run or Fargate (note: I like Cloud Run)
- Basically “on demand compute” of some description, some of which “scales to zero”
View at Medium.com
Step Functions state machine generator for AWS Lambda Power Tuning.
The state machine is designed to be quick and language agnostic. You can provide any Lambda Function as input and the state machine will estimate the best power configuration to minimize cost. Your Lambda Function will be executed in your AWS account (i.e. real HTTP calls, SDK calls, cold starts, etc.) and you can enable parallel execution to generate results in just a few seconds.
As we approach the end of 2018, I’m incredibly excited to announce that we at Serverless have a small gift for you: You can work with Amazon API Gateway WebSockets in your Serverless Framework applications starting right now.
But before we dive into the how-to, there are some interesting caveats that I want you to be aware of.
First, this is not supported in AWS CloudFormation just yet, though AWS has publicly stated it will be early next year! As such, we decided to implement our initial support as a plugin and keep it out of core until the official AWS CloudFormation support is added.
Second, the configuration syntax should be pretty close, but we make no promises that anything implemented with this will carry forward after core support. And once core support is added with AWS CloudFormation, you will need to recreate your API Gateway resources managed by CloudFormation. This means that any clients using your WebSocket application would need to be repointed, or other DNS would have needed to be in place, to facilitate the cutover.
I recommend you check out my original post for a basic understanding of how WebSockets works at a technical level via connections and callbacks to the Amazon API Gateway connections management API.
With all that out of the way, play with our new presents!
AWS Step Functions is a fully managed workflow service for application developers. You can think & work at a high level, connecting and coordinating activities in a reliable and repeatable way, while keeping your business logic separate from your workflow logic. After you design and test your workflows (which we call state machines), you can deploy them at scale, with tens or even hundreds of thousands running independently and concurrently. Step Functions tracks the status of each workflow, takes care of retrying activities on transient failures, and also simplifies monitoring and logging. To learn more, step through the Create a Serverless Workflow with AWS Step Functions and AWS Lambdatutorial.
Since our launch at AWS re:Invent 2016, our customers have made great use of Step Functions (my post, Things go Better with Step Functions describes a real-world use case). Our customers love the fact that they can easily call AWS Lambda functions to implement their business logic, and have asked us for even more options.
Last week was AWS re:Invent which is the most busy time of the year for those of us a part of the AWS ecosystem and arguably the most important. Every year Amazon inundates us with a large number of announcements and it can be overwhelming to keep track of them all. This year amazon announced new EC2 instance types, a time series database, and a slew of machine learning offerings… They also announced a service to retrieve data from orbiting satellites, a rack you can install in your data center with AWS services, an R/C car, and a blockchain service.
It’s easy to miss things in all of that so we’re going to recap what we see as the biggest announcements. Plus we’ll also briefly cover the fun we had with our “appearance” at Stackery’sre:Invent booth.
Last week I spent six incredibly exhausting days in Las Vegas at the AWS re:Invent conference. More than 50,000 developers, partners, customers, and cloud enthusiasts came together to experience this annual event that continues to grow year after year. This was my first time attending, and while I wasn’t quite sure what to expect, I left with not just the feeling that I got my money’s worth, but that AWS is doing everything in their power to help customers like me succeed.
There have already been some really good wrap-up posts about the event. Take a look at James Beswick’s What I learned from AWS re:Invent 2018, Paul Swail’s What new use cases do the re:Invent 2018 serverless announcements open up?, and All the Serverless announcements at re:Invent 2018 from the Serverless, Inc. blog. There’s a lot of good analysis in these posts, so rather than simply rehash everything, I figured I touch on a few of the announcements that I think really matter. We’ll get to that in a minute, but first I want to point out a few things about Amazon Web Services that I learned this past week.
For the first time ever, Serverlessconf was held in San Francisco! Serverlessconf is a community led conference focused on sharing experiences building applications using serverless architectures. Serverless architectures enable developers to express their creativity and focus on user needs instead of spending time managing infrastructure and servers. Watch the first release of talks from the main stage at Serverlessconf San Francisco 2018! The first 24 videos are now live, with more to come!