Benchmarking AWS Lambda runtimes

in 2019 (Part II)

We continue benchmarking AWS Lambda…

Tai Nguyen Bui Follow

Jul 15 · 8 min read

In Part I of this blog we tested the performance of a Hello World example for 8 different
runtimes and got us some very interesting metrics. However, we didn’t stop there. We
wanted to simulate a use case common in many applications — CRUD (Create Read
Update Delete) operations in a NoSQL database like AWS DynamoDB.

As a recap of Part I, we found that:

Cold-start time and memory usage vary depending on the runtime

 AWS Lambda execution durations are stable

API Gateway and AWS Lambda are reliable, no errors

API Gateway latency accounts for more than just code execution in AWS Lambda

Package size isn’t always an indicator for faster cold-start, demonstrated by

Golang and Ruby

Python and Node.js have great performance

Benchmarking process
In this case, we reduced the number of runtimes we benchmarked down to 5, Python
3.6, Node.js 8.10, Java 8, C# (.NET 2.1), and Haskell.

For each of these runtimes, we built CRUD operations with products, containing SKU,
name and description. We also created a DynamoDB table with autoscaling enabled;
the stack was deployed using Serverless Framework.

For testing, we continued to use Serverless Artillery , but created a different script that

generated a random SKU , name , and description for each of the requests and then
assigned those values to the following sequential HTTP requests:

Create > List > Update > Get > Delete

With the above flow, we could test all the functions and empty the table when the test
was completed, win win 😃
Once again, we used AWS CloudWatch to extract all the metrics we were interested in.

You can visit our repository to see the project we created for benchmarking.

Lambda Pre-requisites
AWS Lambda functions interact with other AWS services or APIs through libraries. In
this case, we needed a library to serialize and deserialize JSON inputs and outputs, a
library for date/time, and another library to interact with AWS DynamoDB. Official
AWS SDKs are available for C#, Python, Node.js, and Java and non-official AWS SDKs
for Haskell such as Amazonka or AWS SDK.

CRUD — Benchmark Result

For this benchmark, similar to the procedure we used in Part I, we performed 20
requests per second (rps) for a period of 1200 seconds. Making a total of 120,000
requests per runtime.

Cold-start
We were expecting an increase in the cold-start due to the addition of dependencies.
However, we were surprised by the dramatic increase for Python and Node.js, by
around 30x and 35x respectively in the worst case scenario.

Search this file…

1 Runtime CREATE GET LIST DELETE UPDATE Hello World
2 Python 61.1 55.7 118 75.5 101 1.93
3 Node.js 117 201 154 180 118 3.44
4
Haskell 653 656 1300 666 660 64.5
5 C# 761 778 784 739 765 461.31
6
Java 2150 4050 2140 1980 2100 825.19

aws-lambda-benchmark-crud-cold-start.csv hosted with ❤ by GitHub view raw

As can be seen in the above table, Java continues to be the slowest to warm up,
followed by C# and Haskell.

Execution durations
Once we get over the cold-start period, we start seeing very stable execution times for
C#, Python, and Java. However, we got some spikes in Haskell and Node.js for all the
CRUD operations that were performed.

Create Product— We can see some changes in the benchmark when creating items in
DynamoDB tables in comparison to Part I. C# takes the lead with an average execution
duration that’s almost 1 ms faster than Python and Java, and 2x faster than Node.js.
Additionally, we can see a maximum duration of 6 seconds in Haskell, which is where
our timeout was set.

Search this file…

1 Runtime Average duration (ms) Minimum duration (ms) Maximum duration (ms)
2 C# 7.47 4.77 230
3 Python 8.36 5.8 246
4
Java 8.57 5.45 249
5 Node.js 15.7 6.62 2040
6 Haskell 207 145 6010

aws-lambda-benchmark-crud-create-duration.csv hosted with ❤ by GitHub view raw

Create product — maximum execution duration

List Products — The ranking of results doesn’t change for listing all items in a table; as in
Part I, C# is the best performer. The execution durations increased for all the runtimes
due to the scan on DynamoDB tables, which requires the entire table to be read before a
result is returned. Furthermore, scan operations are limited to a 1 MB payload, which
means that if there is more than 1 MB of data to be retrieved, it will be paginated
through a nextToken key.

Early in the graph below, you can see another timeout in Haskell, when executing for
more than 6 seconds.

Search this file…

1 Runtime Average duration (ms) Minimum duration (ms) Maximum duration (ms)
2 C# 9.52 5.6 259
3 Python 11 7.44 242
4 Java 11.6 7.2 274
5
Node.js 19 7.88 2030
6 Haskell 214 145 6010

aws-lambda-benchmark-crud-list-duration.csv hosted with ❤ by GitHub view raw

List product — maximum execution duration

Get Product — The trend continues for reading items from DynamoDB. C# is the fastest,
closely followed by Python and Java. Moreover, we observe a lower average execution
duration for all the runtimes, between 1 and 2 ms, in comparison to the write operation.
 Search this file…
1 Runtime Average duration (ms) Minimum duration (ms) Maximum duration (ms)
2 C# 5.77 3.71 221
3
Python 7.21 4.52 250
4
Java 7.65 4.98 233
5 Node.js 14.6 5.35 2030
6 Haskell 205 142 2240

aws-lambda-benchmark-crud-get-duration.csv hosted with ❤ by GitHub view raw

Get product — maximum execution duration

Update Product — Python has the fastest average execution duration when updating
items in DynamoDB, with C# just 0.48 ms behind. In this operation, we need to specify
the partition key and attributes to be updated.

Search this file…

1 Runtime Average duration (ms) Minimum duration (ms) Maximum duration (ms)
2 Python 7.1 4.88 247
3
C# 7.58 4.98 252
4 Java 8.52 5.45 313
5 Node.js 14.9 5.63 2030
6 Haskell 204 137 2250
 aws-lambda-benchmark-crud-update-duration.csv hosted with ❤ by GitHub view raw

Update product — maximum execution duration

Delete Product — The impressive execution durations of C# and Python are reflected
one more time in the delete item operation from DynamoDB. Spikes of around 2 seconds
in Node.js and Haskell continue, but no timeout errors were recorded.

Search this file…

1 Runtime Average duration (ms) Minimum duration (ms) Maximum duration (ms)
2 C# 7.23 3.89 244
3 Python 8.5 4.81 238
4 Java 12.6 4.14 245
5
Node.js 16.2 5.82 2030
6 Haskell 206 134 2220

aws-lambda-benchmark-crud-delete-duration.csv hosted with ❤ by GitHub view raw

Delete product — maximum execution duration

CRUD — Results overview

It is apparent for all the CRUD operations above that Node.js and Haskell are being
penalized in the average execution duration due to a really high delta between the
maximum and minimum values. This negative impact is more noticeable in Node.js,
where the minimum execution duration is usually no more than 2 ms away from the best
performers. Furthermore, according to the data displayed above, C#, Python, and Java
have similar durations if we don’t take into consideration cold-starts.
The single digit execution durations of C#, Python, Node.js, and Java are great,
bearing in mind that a NoSQL database is being accessed. On the other hand, Haskell, in
spite of being slower than other runtimes, has acceptable execution durations that we
are very confident will improve in the future.

It can also be seen that Create, Get, Update and Delete operations have similar
performance, while the List operation has a slightly slower performance, around a 20
percent difference, due to the scan on the whole table.

Memory usage
Memory usage increases when performing AWS DynamoDB reads or writes. However,
it is still below 128 MB for all the runtimes except Java, which demanded a maximum
of 147 MB in some of the requests.

Search this file…

1 Runtime Create Max (MB) Get Max (MB) List Max (MB) Update Max (MB) Delete Max (MB) Hello World Max (M
2 Haskell 81 73 73 74 73 49
3 Node.js 73 73 73 73 73 68
4 Python 68 68 68 68 68 50
5
C# 97 97 97 97 97 83
6 Java 147 145 146 144 147 109

aws-lambda-benchmark-crud-memory.csv hosted with ❤ by GitHub view raw

Lambda read/write to DynamoDB, maximum memory usage

Errors
We performed a total of 720,000 requests against 25 different functions, 5 functions
per runtime, and we only registered 4 errors in Haskell functions, which were caused by
timeouts.

Package vs max cold-start

Python 5.1 KB 118 ms

Node.js 6.0 KB 201 ms

C# 633.5 KB 784 ms
 Haskell 5.8 MB 1300 ms

Java 10.1 MB 4050 ms

For the runtimes analyzed in this benchmark, we can see that the ones with bigger
packages also took longer to initialize. In Part I, we saw that Ruby and Go were the
two exceptions to this norm.

API Gateway latency

We continue to be interested in the full duration of the request from the point in which
API Gateway receives the request until a response is returned to the client.

Including cold-start, Python is the only runtime with a maximum API Gateway
average latency below 1 second — to be more precise, 635 ms. If we subtract the worst
cold-start execution duration of 118 ms, we end up with around half a second in which
AWS is getting the VMs ready to execute the code.

Note: The data below has been measured without cold-starts

Search this file…

1 Runtime Lat avg (ms) Lat max (ms) cold-start lat max (ms) Integration lat avg (ms) Integration lat max (ms)
2 C# 23.7 892 2700 20.5 889
3 Python 24.7 475 635 21.5 472
4 Java 25.9 3350 4890 22.7 3350
5
Node.js 32.4 2120 2120 29.2 2110
6 Haskell 224 6020 6020 221 6020

aws-lambda-benchmark-crud-latency.csv hosted with ❤ by GitHub view raw

 CRUD — API Gateway latencies without cold-start

We see some interesting results from C#, Python, and Java: their average latency is
20 to 30 percent faster than Node.js and nearly 10x faster than Haskell.
Additionally, the minimum latency experienced is 14 ms, which hints at the best
performance that could be achieved.

As we mentioned before, we have to add the time that it takes to transfer data from and
to the AWS region, i.e. us-east-1, to the above latencies. This latency could be measured
here.

Conclusion
We’re happy with the results and we feel that there’s good potential for improving not
only AWS Lambda but also the synergy with AWS API Gateway, for example, when
provisioning VMs to execute the code. We will continue to monitor how the different
runtimes evolve in this rapidly changing environment.

We found Python’s performance impressive, and the results from C# after cold-start
even more impressive. We feel that Node.js and Haskell runtimes will need to continue
improving, firstly, by finding more stability during executions, to avoid the spikes we saw
in the graphs, and second, by reducing the execution time to be more aligned with the
top performers. Additionally, in our benchmark, Haskell is using Amazonka as the AWS
SDK. From the performance of the Haskell Hello World example, it seems that some
improvements can be made in the non-official AWS SDK in order to reduce the duration
when interacting with DynamoDB.

We also believe that there is some more work to be done to improve Java and C# cold-
starts, which will make them more appealing when latencies are highly important at any
given point in time.
AWS Lambda and AWS API Gateway continue to be highly reliable: only 4 timeout errors
were experienced in Haskell out of a total of 600,000 requests to the 5 different
runtimes.

To sum up, we are very happy with the results that we’ve obtained for the runtimes
analyzed and especially proud of the Haskell runtime, which we will continue working
on to achieve great performance. Moreover, we encourage the community to contribute
to it and look forward to your ideas.

. . .

Thanks to Carlos Domínguez Padrón for working on this with me, and Annie Anderson
and to the rest of The Agile Monkeys for proofreading and giving feedback prior to
publishing this post. Special thanks to Javier Toledo.

Feedback from everyone else interested in the topic is welcome, I am open to discussion
:D

. . .

If you have questions, or would like to work with us, please contact us
www.theagilemonkeys.com

Thanks to Javier Toledo.

AWS Lambda Serverless Cloud Computing AWS Benchmarking

About Help Legal