Performance

Performance is one of the tent poles of aws-lite. We take it seriously because we want your applications to be as fast possible.

As such, we regularly test and publish open, reproducible, real-world metrics for every key aspect of performance, comparing aws-lite to AWS’s own aws-sdk (v2) and @aws-sdk (v3). Learn more and view source at the aws-lite performance project on GitHub.

All metrics are published below (or skip straight to the wrap-up).

Methodology

In addition to publishing our source, raw data, and final results, we believe it’s important to share the details of our performance testing methodology, too. Learn more here.


Stats last updated: 2024-03-01T01:41:18.085Z

Coldstart latency

Coldstart latency measures the impact of each SDK on AWS Lambda coldstarts – the pre-initialization phase where your code payload is loaded into the Lambda micro VM.

In these stats we expect to see lower values for either very small code payloads (such as aws-lite), or scenarios where we are using the AWS SDK included in the Lambda image (e.g. @aws-sdk v3 raw in nodejs20.x). Coldstart latency increases as code payload sizes increase; this is most clearly observed with bundled SDKs.

Benchmark statistics - Coldstart latency
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 121.33 132.27 122.26 120.9 142.76 125.33 121.99
stddev 34.32 67.5 40.2 37.34 47.95 41.55 35.76
p50 115.5 117.46 114.36 115.72 134.81 123.03 115.36
p90 151.29 157.63 149.81 153.01 199.65 149.66 146.27
p95 175.15 262.42 178.82 180.89 230.18 192.88 172.04
p99 249.81 401.96 294.03 253.83 309.86 255.28 267.86

Initialization latency

Initialization latency measures the impact of each SDK on the initialization phase of the Lambda lifecycle, including static analysis and execution of any code outside the scope of the Lambda handler.

Here we expect to see relatively similar values, as the performance benchmark has almost no static code or init-time execution.

Benchmark statistics - Initialization latency
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 140.18 159.02 157.01 151.52 142.09 156.36 157.56
stddev 7.29 8.05 7.52 9.11 10.56 8.29 7.66
p50 139.16 158.43 156.7 149.2 140.44 155.22 156.17
p90 149.32 170.59 165.64 157.99 149.06 163.9 166.71
p95 152.35 172.54 171.39 165.93 153.25 167.21 171.34
p99 159.73 180.64 177.15 188.06 170.09 173.49 176.89

Import / require

Here we measure the impact of importing / requiring each SDK. Ideally, all import / require operations should be sub-100ms to ensure fast responses in customer hot-paths.

It is important to note that import / require times are tied to individual services. In this benchmark, only the DynamoDB service client is imported. In real world use your business logic may rely on multiple AWS services – each of which necessitating additional imports, thereby compounding overall response latency.

Benchmark statistics - Import / require
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 56.49 44.53 245.6 119.4 847.57 253.81
stddev 6.59 5.93 13.04 6.61 35.17 14.37
p50 55 47 243 119 850 253
p90 65 49 262 125 894 267
p95 66 52 266 126 899 280
p99 71 58 278 141 927 302

Instantiate a client

Here we measure the impact of instantiating a new SDK client – a necessary step before making any service API calls. Ideally all operations should be sub-50ms to ensure fast responses in customer hot-paths.

Benchmark statistics - Instantiate a client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 16.96 20.89 21.94 22.51 10.3 12.34
stddev 5.22 3.86 4.3 4.07 5.82 5.6
p50 20 22 21 23 7 8
p90 22 23 32 24 20 20
p95 23 23 33 33 20 21
p99 24 32 34 34 32 21

DynamoDB - read one 100KB row

Here we measure the latency associated with reading a single 100KB row from DynamoDB, and parsing and returning results. All reads are identical across SDKs.

Benchmark statistics - DynamoDB - read one 100KB row
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 125.75 124.8 108.57 161.53 184.85 146.59
stddev 8.42 7.38 10.41 15.34 9.41 7.92
p50 125 124 104 162 183 145
p90 132 133 121 180 196 156
p95 136 136 125 181 201 160
p99 160 143 128 213 211 163

DynamoDB - write one 100KB row

Here we measure the latency associated with writing a single 100KB row into DynamoDB. All writes are identical across SDKs.

Benchmark statistics - DynamoDB - write one 100KB row
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 21.92 27.87 56.29 57.16 28.27 26.84
stddev 11.47 14.51 14.72 14.33 15.46 12.59
p50 18 24 53 52 22 22
p90 30 35 66 65 62 34
p95 39 65 91 98 64 63
p99 64 84 113 107 82 68

Peak memory consumption over Lambda baseline

Peak memory consumption measures each SDK’s peak memory usage throughout the above four steps (import / require, instantiation, read, and write).

To make it easier to assess the memory impact of each SDK, the graph is presented as a value over (thus, not including) the Lambda Node.js baseline. Baseline memory consumption would be expected to include Node.js itself, Lambda bootstrap processes, etc. The memory baseline used always corresponds to the equivalent peak memory of the control test (e.g. aws-lite peak memory p95 - control peak memory p95 = peak memory over baseline p95).

Benchmark statistics - Peak memory consumption over Lambda baseline
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 68.53 82.11 82.22 79.14 85.9 103.37 94.95
stddev 1.39 1.71 1.51 1.52 1.38 1.59 2.1
p50 69 83 83 80 86 104 96
p90 70 83 84 80 87 105 96
p95 71 85 84 81 88 105 98
p99 71 85 85 82 89 106 98

Time to respond, not including coldstart

Time to respond measures the total execution time of each SDK, not including coldstart or initialization. In real-world usage, Lambda coldstarts are usually less common than warm invocations, so this metric illustrates the most common case for most applications. Ideally all times should be sub-1000ms to ensure fast responses in customer hot-paths.

Benchmark statistics - Time to respond, not including coldstart
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 0 227.45 224.84 444.08 376.58 1081.04 453.72
stddev 0 17.41 17.86 26 22.93 43.59 21.73
p50 0 224 220 439 371 1079 450
p90 0 248 238 475 400 1136 478
p95 0 260 268 493 422 1153 495
p99 0 296 289 520 454 1166 511

Total time to respond, including coldstart

Total time to respond measures the total execution time of each SDK, including coldstart or initialization. In real-world usage, this metric represents a normalized “worst case” response time. Ideally all times should be sub-1000ms to ensure fast responses in customer hot-paths.

Benchmark statistics - Total time to respond, including coldstart
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 261.51 518.74 504.11 716.5 661.43 1362.73 733.27
stddev 34.26 72.62 45.13 48.27 54.95 62.79 47.04
p50 254 501 495 711 649 1362 726
p90 294 560 545 779 739 1420 773
p95 320 653 581 817 765 1471 802
p99 387 772 673 865 830 1531 894

Latest data

If you’d like to dig deeper, here’s the data from the latest performance run:

aws-lite is an Apache 2.0-licensed open source project under the umbrella of OpenJS Foundation Architect. aws-lite is not in any way affiliated with Amazon Web Services, Inc. (AWS). All names and trademarks are the property of their respective owners.