Performance

Performance is one of the tent poles of aws-lite. We take it seriously because we want your applications to be as fast possible.

As such, we regularly test and publish open, reproducible, real-world metrics for every key aspect of performance, comparing aws-lite to AWS’s own aws-sdk (v2) and @aws-sdk (v3). Learn more and view source at the aws-lite performance project on GitHub.

We currently track individual and aggregated performance benchmarking the following AWS service clients: DynamoDB, S3, IAM, CloudFormation, Lambda, and STS.

All metrics are published below (or skip straight to the wrap-up).

Methodology

In addition to publishing our source, raw data, and final results, we believe it’s important to share the details of our performance testing methodology, too. Learn more here.


Stats last updated: 2024-06-19T01:51:03.403Z

Coldstart latency

Coldstart latency measures the impact of each SDK on AWS Lambda coldstarts – the pre-initialization phase where your code payload is loaded into the Lambda micro VM.

In these stats we expect to see lower values for either very small code payloads (such as aws-lite), or scenarios where we are using the AWS SDK included in the Lambda image (e.g. @aws-sdk v3 raw in nodejs20.x). Coldstart latency increases as code payload sizes increase; this is often observed with bundled SDKs.

Benchmark statistics - Coldstart latency
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 126.83 125.65 117.94 123.34 125.5 127.02 126.3
stddev 51.14 38.21 26.59 45.15 31.52 36.95 37.9
p50 118 120 113.31 115.77 120.17 120.57 119.61
p90 146.14 146.66 140.15 142.83 150.78 153.1 149.72
p95 177.55 165.14 150.26 256.53 175.23 186.26 179.58
p99 353.96 321.31 237.45 265.73 253.18 266.83 276.57

Initialization latency

Initialization latency measures the impact of each SDK on the initialization phase of the Lambda lifecycle, including static analysis and execution of any code outside the scope of the Lambda handler.

Here we expect to see relatively similar values, as the performance benchmark has very little static code or init-time execution.

Benchmark statistics - Initialization latency
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 134.16 155.54 156.61 154.23 138.5 154.81 155.36
stddev 5.63 8.66 9.52 8.54 7.74 8.22 8.85
p50 134 154.54 154.71 152.31 137.99 153.73 154.7
p90 139.11 164.56 162.66 163.29 144.75 162.76 164.82
p95 143.59 167.71 168.9 170.93 149.5 166.41 168
p99 149.77 189.51 192.16 184.98 168.67 182.43 187.39

Peak memory consumption over Lambda baseline

Peak memory consumption measures each SDK’s peak memory usage throughout import / require, instantiation, read, and write.

To make it easier to assess the memory impact of each SDK, the graph is presented as a value over (thus, not including) the Lambda Node.js baseline. Baseline memory consumption would be expected to include Node.js itself, Lambda bootstrap processes, etc. The memory baseline used always corresponds to the equivalent peak memory of the control test (e.g. aws-lite peak memory p95 - control peak memory p95 = peak memory over baseline p95).

Benchmark statistics - Peak memory consumption over Lambda baseline
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 69.06 93.76 94.01 92.56 116.29 115.75 135.73
stddev 1.15 1.18 1.37 1.2 1.71 1.44 1.17
p50 70 94 94 93 117 116 136
p90 70 95 96 94 118 117 137
p95 70 95 96 94 118 118 137
p99 72 95 97 94 119 119 138

Time to respond, not including coldstart

Time to respond measures the total execution time of each SDK, not including coldstart or initialization. In real-world usage, Lambda coldstarts are usually less common than warm invocations, so this metric illustrates the most common case for most applications.

Results below show aggregate data for sequentially executing all six tested clients, followed by individual client response times. (Detailed client statistics can be found here.)

Note: Ideally, response times should be sub-1000ms to ensure fast responses in customer hot-paths. However, the aggregate benchmark simulates importing, instantiating, reading, and writing from six different AWS services in a single execution. When authoring customer-facing business logic, one should ideally utilize fewer services and/or calls to maintain a high degree of customer performance.

Benchmark statistics - Time to respond, not including coldstart
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 0 1298.36 1277.45 2411.9 2921.81 2223.64 2695.03
stddev 0 137.28 83.18 2224.1 2664.56 1562.05 120.55
p50 0 1261 1268 2059 2433 2030 2670
p90 0 1487 1408 2291 2613 2213 2831
p95 0 1558 1431 2487 2747 2492 2891
p99 0 1814 1486 17923 18041 2793 3097

Individual client response times

Benchmark statistics - Time to respond, not including coldstart (DynamoDB) Benchmark statistics - Time to respond, not including coldstart (S3) Benchmark statistics - Time to respond, not including coldstart (IAM) Benchmark statistics - Time to respond, not including coldstart (CloudFormation) Benchmark statistics - Time to respond, not including coldstart (Lambda) Benchmark statistics - Time to respond, not including coldstart (STS)

Total time to respond, including coldstart

Total time to respond measures the total execution time of each SDK, including coldstart or initialization. In real-world usage, this metric represents a normalized “worst case” response time.

Note: Ideally, response times should be sub-1000ms to ensure fast responses in customer hot-paths. However, the aggregate benchmark simulates importing, instantiating, reading, and writing from six different AWS services in a single execution. When authoring customer-facing business logic, one should ideally utilize fewer services and/or calls to maintain a high degree of customer performance.

Benchmark statistics - Total time to respond, including coldstart
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 260.99 1579.55 1552 2689.47 3185.81 2505.47 2976.69
stddev 51.55 148.23 94.34 2222.71 2660.45 1557.5 124.53
p50 249 1540 1538 2334 2693 2316 2957
p90 286 1778 1690 2563 2867 2508 3115
p95 313 1886 1741 2843 3021 2778 3178
p99 492 2109 1781 18194 18292 3062 3388

Client metrics

Performance criteria

Each SDK client is measured on the following criteria:

  • Import / require - Measurement of the impact of importing / requiring each SDK client.
    • It is important to note that import / require times are tied to individual services. In real world use your business logic may rely on multiple AWS services – each of which necessitating additional imports, thereby compounding overall response latency.
    • Ideally, all import / require operations should be sub-100ms to ensure fast responses in customer hot-paths.
  • Instantiate - Measurement of the impact of instantiating a new SDK client – a necessary step before making any service API calls. Ideally all operations should be sub-50ms to ensure fast responses in customer hot-paths.
  • Read - Measurement of a simple read operation to a service API. All reads are identical across SDKs.
  • Write - Measurement of a simple write operation to a service API. All writes are identical across SDKs.
    • Note: some clients may be read-only, based on the service in question (example: STS).
  • Total - Measurement of the total latency associated with all the above operations, thereby demonstrating the overall impact of using a given SDK client.

CloudFormation

Import / require

Benchmark statistics - Import / require CloudFormation
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 6.56 3.84 1.23 56.36 75.19 251.79
stddev 0.93 0.42 0.44 6.03 6.57 24.08
p50 7 4 1 59 76 259
p90 8 4 2 62 82 272
p95 8 4 2 66 83 282
p99 8 5 2 71 90 291

Instantiate

Benchmark statistics - Instantiate a CloudFormation client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 0.71 0.67 6.14 18.21 1.76 12.46
stddev 0.45 0.47 1.94 5.68 1.13 8.86
p50 1 1 6 20 2 7
p90 1 1 7 22 2 23
p95 1 1 7 25 2 25
p99 1 1 14 32 2 42

Read

Here we measure the latency associated with listing a single CloudFormation stack’s resources (ListStackResources()), and parsing and returning results.

Benchmark statistics - CloudFormation - read one stack
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 59.05 56.69 62.87 86.3 68.67 105.79
stddev 8.61 7.01 11.34 9.51 12.42 17.47
p50 57 56 60 85 66 100
p90 69 65 75 98 85 128
p95 74 69 82 104 86 140
p99 83 77 113 109 121 158

Write

Here we measure the latency associated with updating a single CloudFormation stack’s configuration (UpdateTerminationProtection()). This method was selected specifically to help limit the impact of stack update latency, which can be highly variable, from benchmarking routines.

Benchmark statistics - CloudFormation - write one stack
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 46.82 46.81 66.58 99.73 47.72 52.86
stddev 17.34 15.15 20.4 36.88 9.85 16.37
p50 41 42 61 92 44 47
p90 57 60 82 148 60 69
p95 72 72 97 157 67 95
p99 144 110 162 186 77 106

Total

Benchmark statistics - CloudFormation - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 113.4 108.19 137.34 271.16 195.16 439.74
stddev 21 17.49 23.78 39.63 18.83 28.42
p50 107 103 131 264 190 436
p90 131 125 160 319 214 471
p95 150 145 189 330 225 484
p99 210 170 224 389 254 531

DynamoDB

Import / require

Benchmark statistics - Import / require DynamoDB
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 63.63 55.77 249.85 117.13 363.04 241.03
stddev 6.14 5.27 14.69 9.99 19.01 12.03
p50 64 55 248 116 361 240
p90 69 61 267 125 385 256
p95 73 65 277 126 392 259
p99 81 72 290 157 418 268

Instantiate

Benchmark statistics - Instantiate a DynamoDB client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 2.79 2.77 22.72 21.33 12.21 12.47
stddev 1.86 1.14 5.27 3.38 5.19 5.58
p50 2 3 21 22 8 8
p90 3 3 33 24 18 20
p95 3 3 34 24 18 20
p99 13 4 41 25 18 21

Read

Here we measure the latency associated with getting a single 100KB row from DynamoDB (GetItem()), and parsing and returning results.

Benchmark statistics - DynamoDB - read one 100KB row
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 134.7 132.58 111.37 180.41 144.75 154.94
stddev 18.82 8.28 22.1 15.38 9.18 9.27
p50 132 131 105 180 143 155
p90 142 141 122 198 154 165
p95 145 145 134 202 161 171
p99 208 154 143 238 176 177

Write

Here we measure the latency associated with writing a single 100KB row into DynamoDB (PutItem()).

Benchmark statistics - DynamoDB - write one 100KB row
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 22.98 22.91 54.5 49.48 23.5 34.19
stddev 16.03 12.57 15.35 12.27 13.36 12.33
p50 17 18 50 47 18 31
p90 59 31 64 59 32 46
p95 64 57 90 86 61 73
p99 72 69 111 93 73 78

Total

Benchmark statistics - DynamoDB - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 229.57 218.27 451.06 375.61 553.14 461.25
stddev 25.84 16.39 34.24 28.53 27.23 23.73
p50 222 217 445 369 551 460
p90 260 238 486 402 590 485
p95 267 250 503 425 602 499
p99 325 275 532 458 618 521

IAM

Import / require

Benchmark statistics - Import / require IAM
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 5.25 3.74 1.86 58.16 105.06 231.83
stddev 1.03 0.77 3.47 7.75 8.96 12.19
p50 5 4 2 61 105 231
p90 6 4 2 64 117 245
p95 6 4 2 65 119 247
p99 7 6 2 76 124 272

Instantiate

Benchmark statistics - Instantiate an IAM client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 0.81 0.69 26.95 26.99 1.88 10.06
stddev 0.39 0.46 19.5 8.67 1.85 5.25
p50 1 1 19 24 2 7
p90 1 1 59 41 2 19
p95 1 1 63 42 2 20
p99 1 1 66 43 12 20

Read

Here we measure the latency associated with getting a single role from IAM (GetRole()), and parsing and returning results.

Benchmark statistics - IAM - read one role
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 312.61 311.65 328.42 486.92 472.05 337.43
stddev 17.51 12.88 20.45 1559.28 1550.63 12.22
p50 310 311 326 328 317 336
p90 325 324 354 345 328 348
p95 329 329 361 352 332 353
p99 388 360 373 402 413 383

Write

Here we measure the latency associated with updating a single IAM role (UpdateRole()).

Benchmark statistics - IAM - write one role
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 99.59 102.41 617.06 615.14 101.84 105.91
stddev 10.24 12.28 2213.45 2183.43 8.79 8.96
p50 96 99 299 300 99 104
p90 113 116 321 319 116 119
p95 115 118 324 326 119 120
p99 119 118 16005 15741 127 127

Total

Benchmark statistics - IAM - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 419.56 418.94 977.21 1200.58 684.79 693.84
stddev 20.86 19.57 2210.2 2663.65 1552.3 20.69
p50 417 417 665 729 529 693
p90 439 441 710 761 548 711
p95 448 448 723 785 554 715
p99 489 483 16374 16390 620 764

Lambda

Import / require

Benchmark statistics - Import / require Lambda
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 7.47 7.82 1.88 54.18 46.31 158.48
stddev 1.66 0.74 0.72 8.1 8.27 5.53
p50 7 8 2 50 48 158
p90 8 9 3 61 57 163
p95 8 9 3 63 58 164
p99 16 9 4 75 59 176

Instantiate

Benchmark statistics - Instantiate a Lambda client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 1.23 1.47 7.89 13.7 1.88 9.21
stddev 0.44 0.5 3.95 7.97 1.79 5.33
p50 1 1 6 8 2 6
p90 2 2 15 23 2 17
p95 2 2 17 25 2 19
p99 2 2 19 40 12 20

Read

Here we measure the latency associated with getting a single Lambda’s configuration (GetFunctionConfiguration()), and parsing and returning results.

Benchmark statistics - Lambda - read one configuration
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 61.54 63.92 74.17 100.06 69.26 91.1
stddev 18.76 19.33 21.13 25.11 20.91 22.12
p50 55 57 71 101 60 84
p90 87 90 103 131 99 117
p95 89 93 109 135 105 120
p99 92 100 123 146 112 135

Write

Here we measure the latency associated with updating a single Lambda’s configuration (UpdateFunctionConfiguration()).

Benchmark statistics - Lambda - write one configuration
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 141.13 140.67 154.21 156.87 143.64 141.16
stddev 21.74 15.37 14.88 25.48 18.11 12.87
p50 138 142 151 152 142 140
p90 164 155 173 176 164 154
p95 171 159 178 197 171 167
p99 222 177 197 259 202 178

Total

Benchmark statistics - Lambda - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 211.53 214.13 239.83 338.75 266.16 408.4
stddev 29.74 23.93 28.29 39.79 29.14 26.79
p50 211 211 235 332 261 404
p90 242 242 276 382 302 447
p95 249 249 287 401 310 449
p99 324 267 307 463 353 484

S3

Import / require

Benchmark statistics - Import / require S3
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 26.05 26.44 6.44 61.17 173.43 242.87
stddev 7.8 9.08 0.82 5.67 8.26 11.26
p50 27 29 6 61 172 241
p90 33 35 7 68 185 255
p95 35 35 7 70 187 258
p99 53 45 9 77 190 272

Instantiate

Benchmark statistics - Instantiate an S3 client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 2.05 1.87 13.97 24.38 7.13 9.63
stddev 0.46 0.34 4.91 3.15 5.04 4.69
p50 2 2 13 24 5 7
p90 3 2 20 26 15 17
p95 3 2 20 28 15 18
p99 3 2 32 40 16 19

Read

Here we measure the latency associated with getting a single 1MB object from S3 (GetObject()), and parsing and returning results.

Benchmark statistics - S3 - read one 1MB object
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 73.45 72.91 87.22 100.12 116.02 94.4
stddev 15.97 22.53 23.64 15.7 12.99 15.89
p50 68 65 84 98 117 93
p90 98 105 98 121 132 110
p95 105 119 105 133 137 121
p99 130 128 122 155 143 141

Write

Here we measure the latency associated with writing a single 1MB object into S3 (PutObject()).

Benchmark statistics - S3 - write one 1MB object
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 188.8 174.85 203.54 170.39 162.55 145.21
stddev 129.76 72.52 124.37 84.87 155.3 95.37
p50 151 155 160 145 124 124
p90 387 283 392 262 198 182
p95 439 362 486 348 538 336
p99 702 406 639 452 781 556

Total

Benchmark statistics - S3 - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 300.24 294.98 327.74 375.03 464 503.63
stddev 132.83 75.32 127.04 86.88 155.79 96.28
p50 264 277 281 351 422 479
p90 505 391 513 467 522 552
p95 573 495 600 547 839 694
p99 799 522 779 704 1084 908

STS

Import / require

Benchmark statistics - Import / require STS
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 3.99 3.05 0.26 52.82 22.88 118.7
stddev 0.46 0.48 0.44 7.5 6.75 6.82
p50 4 3 0 50 22 116
p90 4 3 1 60 28 126
p95 5 4 1 61 34 130
p99 5 4 1 72 41 137

Instantiate

Benchmark statistics - Instantiate a STS client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 0.46 0.28 1.96 3.12 1.83 10.6
stddev 0.5 0.45 0.56 1.8 1.27 6.47
p50 0 0 2 3 2 7
p90 1 1 2 3 2 17
p95 1 1 3 3 2 19
p99 1 1 3 13 5 20

Read

Here we measure the latency associated with reading one identity via STS (GetCallerIdentity()), and parsing and returning results.

Benchmark statistics - STS - read one identity
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 18.85 18.64 275.66 294.09 32.09 51.32
stddev 1.98 2.22 6.88 10.34 21.51 6.28
p50 18 18 276 295 30 53
p90 21 21 284 306 35 59
p95 21 22 287 309 36 59
p99 27 27 290 318 96 63

Total

Benchmark statistics - STS - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 23.39 22.08 277.95 359.15 59 186.49
stddev 2.03 2.41 6.97 11.11 25.48 9.68
p50 23 22 279 358 56 186
p90 26 24 286 374 63 197
p95 26 25 290 378 71 199
p99 32 32 292 384 118 216

Latest data

If you’d like to dig deeper, here’s the data from the latest performance run:

aws-lite is an Apache 2.0-licensed open source project under the umbrella of OpenJS Foundation Architect. aws-lite is not in any way affiliated with Amazon Web Services, Inc. (AWS). All names and trademarks are the property of their respective owners.