aws-lite

Performance

Performance is one of the tent poles of aws-lite. We take it seriously because we want your applications to be as fast possible.

As such, we regularly test and publish open, reproducible, real-world metrics for every key aspect of performance, comparing aws-lite to AWS’s own aws-sdk (v2) and @aws-sdk (v3). Learn more and view source at the aws-lite performance project on GitHub.

We currently track individual and aggregated performance benchmarking the following AWS service clients: DynamoDB, S3, IAM, CloudFormation, Lambda, and STS.

All metrics are published below (or skip straight to the wrap-up).

Methodology

In addition to publishing our source, raw data, and final results, we believe it’s important to share the details of our performance testing methodology, too. Learn more here.


Stats last updated: 2024-09-16T02:09:13.281Z

Coldstart latency

Coldstart latency measures the impact of each SDK on AWS Lambda coldstarts – the pre-initialization phase where your code payload is loaded into the Lambda micro VM.

In these stats we expect to see lower values for either very small code payloads (such as aws-lite), or scenarios where we are using the AWS SDK included in the Lambda image (e.g. @aws-sdk v3 raw in nodejs20.x). Coldstart latency increases as code payload sizes increase; this is often observed with bundled SDKs.

Benchmark statistics - Coldstart latency
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 136.66 143.3 141 125.08 142.15 136.58 147.49
stddev 31.06 37.49 29.55 42.81 29.28 24.84 34.01
p50 132.87 138.94 136.72 115.62 139.17 134.21 142.24
p90 156.93 164.8 162.65 142.32 169.74 157.89 181.97
p95 168.98 187.42 182.36 229.81 188.44 173.76 206.38
p99 273 324.69 287.68 289.97 247.85 228.94 287.79

Initialization latency

Initialization latency measures the impact of each SDK on the initialization phase of the Lambda lifecycle, including static analysis and execution of any code outside the scope of the Lambda handler.

Here we expect to see relatively similar values, as the performance benchmark has very little static code or init-time execution.

Benchmark statistics - Initialization latency
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 143.56 168.44 167.95 155.04 148.36 167.72 169.2
stddev 6.5 8.16 9.65 7.8 12.68 7.17 10.49
p50 142.79 167.34 166.81 153.68 146.23 167.39 167.27
p90 151.17 178.07 175.86 163.47 154.06 177.03 180.76
p95 153.8 179.2 185.64 169.65 159.27 178.17 187.97
p99 161.02 197.44 199.84 181.04 218.14 192.4 208.52

Peak memory consumption over Lambda baseline

Peak memory consumption measures each SDK’s peak memory usage throughout import / require, instantiation, read, and write.

To make it easier to assess the memory impact of each SDK, the graph is presented as a value over (thus, not including) the Lambda Node.js baseline. Baseline memory consumption would be expected to include Node.js itself, Lambda bootstrap processes, etc. The memory baseline used always corresponds to the equivalent peak memory of the control test (e.g. aws-lite peak memory p95 - control peak memory p95 = peak memory over baseline p95).

Benchmark statistics - Peak memory consumption over Lambda baseline
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 63.96 90.09 89.37 92.85 111.87 111.17 128.43
stddev 0.28 0.57 1.37 1.08 1.08 0.96 1.12
p50 64 90 89 93 112 111 129
p90 64 90 91 94 113 112 129
p95 64 92 91 94 113 112 130
p99 64 92 92 95 115 113 131

Time to respond, not including coldstart

Time to respond measures the total execution time of each SDK, not including coldstart or initialization. In real-world usage, Lambda coldstarts are usually less common than warm invocations, so this metric illustrates the most common case for most applications.

Results below show aggregate data for sequentially executing all six tested clients, followed by individual client response times. (Detailed client statistics can be found here.)

Note: Ideally, response times should be sub-1000ms to ensure fast responses in customer hot-paths. However, the aggregate benchmark simulates importing, instantiating, reading, and writing from six different AWS services in a single execution. When authoring customer-facing business logic, one should ideally utilize fewer services and/or calls to maintain a high degree of customer performance.

Benchmark statistics - Time to respond, not including coldstart
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 0 1505.59 1360.01 2433.19 2669.2 2323.87 2781.82
stddev 0 1552.27 154.71 2200.65 1539.23 1598.5 132.63
p50 0 1316 1331 2087 2494 2114 2762
p90 0 1503 1473 2273 2705 2492 2857
p95 0 1530 1586 2494 2784 2628 3026
p99 0 1952 2136 17624 2884 2810 3311

Individual client response times

Benchmark statistics - Time to respond, not including coldstart (DynamoDB) Benchmark statistics - Time to respond, not including coldstart (S3) Benchmark statistics - Time to respond, not including coldstart (IAM) Benchmark statistics - Time to respond, not including coldstart (CloudFormation) Benchmark statistics - Time to respond, not including coldstart (Lambda) Benchmark statistics - Time to respond, not including coldstart (STS)

Total time to respond, including coldstart

Total time to respond measures the total execution time of each SDK, including coldstart or initialization. In real-world usage, this metric represents a normalized “worst case” response time.

Note: Ideally, response times should be sub-1000ms to ensure fast responses in customer hot-paths. However, the aggregate benchmark simulates importing, instantiating, reading, and writing from six different AWS services in a single execution. When authoring customer-facing business logic, one should ideally utilize fewer services and/or calls to maintain a high degree of customer performance.

Benchmark statistics - Total time to respond, including coldstart
control aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 280.22 1817.33 1668.96 2713.31 2959.71 2628.17 3098.51
stddev 32.28 1553.43 160.16 2197.69 1540.26 1599.87 145.65
p50 276 1634 1636 2366 2790 2412 3071
p90 303 1818 1800 2554 3013 2801 3231
p95 314 1873 1910 2775 3080 2915 3320
p99 417 2229 2419 17894 3227 3114 3657

Client metrics

Performance criteria

Each SDK client is measured on the following criteria:

  • Import / require - Measurement of the impact of importing / requiring each SDK client.
    • It is important to note that import / require times are tied to individual services. In real world use your business logic may rely on multiple AWS services – each of which necessitating additional imports, thereby compounding overall response latency.
    • Ideally, all import / require operations should be sub-100ms to ensure fast responses in customer hot-paths.
  • Instantiate - Measurement of the impact of instantiating a new SDK client – a necessary step before making any service API calls. Ideally all operations should be sub-50ms to ensure fast responses in customer hot-paths.
  • Read - Measurement of a simple read operation to a service API. All reads are identical across SDKs.
  • Write - Measurement of a simple write operation to a service API. All writes are identical across SDKs.
    • Note: some clients may be read-only, based on the service in question (example: STS).
  • Total - Measurement of the total latency associated with all the above operations, thereby demonstrating the overall impact of using a given SDK client.

CloudFormation

Import / require

Benchmark statistics - Import / require CloudFormation
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 5.36 4.09 1.26 57.87 75.84 199.96
stddev 2.1 0.74 0.44 10.12 5.87 18.13
p50 5 4 1 59 76 196
p90 6 4 2 63 82 225
p95 6 5 2 73 84 235
p99 9 8 2 100 90 257

Instantiate

Benchmark statistics - Instantiate a CloudFormation client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 0.74 0.76 6.05 19.2 1.74 9.27
stddev 0.44 0.45 1.18 5.19 0.44 4.81
p50 1 1 6 20 2 7
p90 1 1 7 22 2 17
p95 1 1 8 25 2 17
p99 1 1 8 29 2 19

Read

Here we measure the latency associated with listing a single CloudFormation stack’s resources (ListStackResources()), and parsing and returning results.

Benchmark statistics - CloudFormation - read one stack
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 55.63 60.66 63.25 85.59 65.63 100.94
stddev 8.63 11.32 15.47 13.26 8.41 23.5
p50 54 58 60 82 64 95
p90 64 76 75 101 77 130
p95 69 77 82 106 80 163
p99 87 95 90 133 88 171

Write

Here we measure the latency associated with updating a single CloudFormation stack’s configuration (UpdateTerminationProtection()). This method was selected specifically to help limit the impact of stack update latency, which can be highly variable, from benchmarking routines.

Benchmark statistics - CloudFormation - write one stack
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 43.9 43.39 59.08 102.44 46.71 48.29
stddev 7.84 7.17 6.19 32.77 9.36 8.58
p50 43 41 58 95 44 46
p90 50 52 68 144 54 56
p95 53 58 72 149 67 62
p99 66 69 77 160 91 86

Total

Benchmark statistics - CloudFormation - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 105.91 109.16 130.14 276.4 191.83 369.32
stddev 11.95 14.8 17.39 34.11 16.3 39.7
p50 105 106 128 275 189 357
p90 119 125 144 319 210 426
p95 132 142 150 324 229 456
p99 140 148 171 336 238 480

DynamoDB

Import / require

Benchmark statistics - Import / require DynamoDB
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 64.27 54.73 256.91 118.41 369.74 253.03
stddev 6.14 4.62 20.97 8.58 17.47 20.58
p50 64 54 253 117 368 250
p90 73 62 277 126 389 267
p95 74 64 291 128 398 272
p99 76 72 331 150 423 355

Instantiate

Benchmark statistics - Instantiate a DynamoDB client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 2.89 5.29 23.33 21.9 14.55 13.71
stddev 1.79 3.14 5.3 5.43 4.92 5.31
p50 3 4 21 22 18 12
p90 3 10 33 25 19 20
p95 3 14 34 32 19 21
p99 13 15 36 40 20 21

Read

Here we measure the latency associated with getting a single 100KB row from DynamoDB (GetItem()), and parsing and returning results.

Benchmark statistics - DynamoDB - read one 100KB row
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 132.59 128.38 109.44 182.8 153.26 159.4
stddev 6.73 8.54 11.17 14.88 11.72 12.75
p50 131 128 105 180 150 159
p90 140 138 123 198 164 169
p95 145 141 125 204 180 176
p99 151 149 143 238 196 221

Write

Here we measure the latency associated with writing a single 100KB row into DynamoDB (PutItem()).

Benchmark statistics - DynamoDB - write one 100KB row
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 24.55 24.45 54.12 50.8 29.94 29.81
stddev 17.98 13.62 16.91 13.12 13.84 14.69
p50 16 19 49 47 29 28
p90 58 36 86 66 36 34
p95 63 59 92 83 72 69
p99 74 74 109 98 78 79

Total

Benchmark statistics - DynamoDB - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 230.34 219.76 455.49 379.38 577.72 470.27
stddev 21.28 18.81 31.98 27.21 29.9 34.31
p50 223 217 449 373 573 462
p90 264 241 500 408 616 491
p95 272 265 521 434 631 523
p99 285 281 536 479 675 614

IAM

Import / require

Benchmark statistics - Import / require IAM
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 11.96 11.42 1.57 59.94 103.83 231.43
stddev 3.47 3.87 1.82 7.08 8.93 16.4
p50 11 11 1 62 102 229
p90 12 12 2 65 114 248
p95 20 13 2 70 116 265
p99 26 22 2 84 131 289

Instantiate

Benchmark statistics - Instantiate an IAM client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 2.47 3.65 29.19 28.65 2.05 8.42
stddev 1.28 4.8 21.24 9.4 1.76 5.03
p50 2 2 18 24 2 6
p90 3 10 62 41 2 20
p95 3 12 63 42 2 20
p99 9 21 76 45 12 21

Read

Here we measure the latency associated with getting a single role from IAM (GetRole()), and parsing and returning results.

Benchmark statistics - IAM - read one role
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 466.13 309.97 634.12 480.75 471.05 344.75
stddev 1551.6 11.41 2184.89 1549.84 1584.89 14.5
p50 308 309 320 324 311 343
p90 323 322 349 341 326 363
p95 327 331 364 347 332 371
p99 359 338 15853 362 388 391

Write

Here we measure the latency associated with updating a single IAM role (UpdateRole()).

Benchmark statistics - IAM - write one role
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 97.29 99.78 296.16 300.36 100.32 103.15
stddev 8.79 13.34 11.12 12.85 7.79 9.61
p50 96 97 296 298 99 100
p90 107 109 310 313 110 115
p95 109 112 314 320 113 120
p99 133 129 321 334 119 126

Total

Benchmark statistics - IAM - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 581.82 428.03 965.92 882.92 681.96 697.68
stddev 1551.39 21.68 2184.72 1548.89 1585.25 25.04
p50 424 426 656 725 520 692
p90 447 451 691 754 550 732
p95 465 462 703 771 555 749
p99 487 493 16220 797 585 772

Lambda

Import / require

Benchmark statistics - Import / require Lambda
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 5.93 6.48 1.79 54.43 48.53 176.28
stddev 2.96 0.66 0.68 6.76 8.9 9.79
p50 6 6 2 51 48 176
p90 6 7 3 61 59 185
p95 6 7 3 64 60 189
p99 20 9 3 71 62 204

Instantiate

Benchmark statistics - Instantiate a Lambda client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 1.23 2.2 7.4 12.4 2.43 7.82
stddev 0.42 3.55 4.2 6.22 1.98 4.01
p50 1 1 6 8 2 7
p90 2 2 10 22 2 9
p95 2 13 19 23 3 20
p99 2 16 22 26 12 22

Read

Here we measure the latency associated with getting a single Lambda’s configuration (GetFunctionConfiguration()), and parsing and returning results.

Benchmark statistics - Lambda - read one configuration
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 77.82 76.15 79.54 109.13 76.93 101.82
stddev 25.36 21.8 25.08 37.54 21.49 25.19
p50 64 70 70 98 68 93
p90 108 108 110 167 110 134
p95 112 112 121 193 115 140
p99 154 123 131 204 123 150

Write

Here we measure the latency associated with updating a single Lambda’s configuration (UpdateFunctionConfiguration()).

Benchmark statistics - Lambda - write one configuration
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 177.51 178.86 195.76 198.67 179.47 179.3
stddev 16.02 19.04 21.63 24.56 16.22 17.1
p50 176 177 193 195 177 177
p90 199 201 224 218 198 198
p95 204 209 230 226 205 208
p99 222 243 264 281 238 235

Total

Benchmark statistics - Lambda - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 262.75 263.9 286.64 387.35 312.07 477.13
stddev 30.13 30.05 31.87 47.77 28.57 32.22
p50 260 259 281 377 307 470
p90 303 304 327 450 349 523
p95 313 316 338 485 360 533
p99 337 339 383 513 388 558

S3

Import / require

Benchmark statistics - Import / require S3
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 32.18 29.15 6.58 65.36 182.27 267.79
stddev 4.33 7.36 1.14 9.4 10.02 21.78
p50 33 28 6 63 180 263
p90 35 37 7 79 194 291
p95 36 46 8 84 199 299
p99 42 47 11 93 213 355

Instantiate

Benchmark statistics - Instantiate an S3 client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 7.48 2.32 14.52 24.39 6.81 24.28
stddev 6.27 1.45 5.23 4.13 5.03 17.37
p50 4 2 13 24 3 19
p90 17 3 20 27 14 48
p95 21 3 22 28 14 61
p99 21 4 30 41 16 64

Read

Here we measure the latency associated with getting a single 1MB object from S3 (GetObject()), and parsing and returning results.

Benchmark statistics - S3 - read one 1MB object
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 77.05 77.29 91.76 100.38 123.43 131.05
stddev 26.63 17.9 17.4 15.26 19.95 30.51
p50 68 73 87 100 121 127
p90 107 102 111 120 144 156
p95 121 111 124 125 160 168
p99 202 130 159 141 184 227

Write

Here we measure the latency associated with writing a single 1MB object into S3 (PutObject()).

Benchmark statistics - S3 - write one 1MB object
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 174.39 187.34 192.17 177.5 180.64 126.98
stddev 94.78 144.51 126.93 92.96 168.41 81.93
p50 148 151 161 149 122 117
p90 314 288 318 295 455 158
p95 363 363 382 400 633 175
p99 472 976 772 488 748 541

Total

Benchmark statistics - S3 - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 298.45 314.23 320.86 386.05 499.1 560.95
stddev 101.23 146.46 129.19 95.31 170.08 88.73
p50 268 281 284 356 440 548
p90 425 434 452 478 790 621
p95 489 515 514 631 963 639
p99 618 1108 891 699 1059 982

STS

Import / require

Benchmark statistics - Import / require STS
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 3.75 2.73 0.28 52.83 23.88 127.03
stddev 1.44 0.49 0.45 6.58 4.94 8.93
p50 4 3 0 50 23 127
p90 4 3 1 59 29 139
p95 4 3 1 61 30 140
p99 5 4 1 65 41 143

Instantiate

Benchmark statistics - Instantiate a STS client
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 0.52 0.35 2.11 3.07 2.55 9.63
stddev 0.5 0.48 0.44 1.52 2.9 4.85
p50 1 0 2 3 2 7
p90 1 1 3 3 2 17
p95 1 1 3 4 12 17
p99 1 1 3 12 13 20

Read

Here we measure the latency associated with reading one identity via STS (GetCallerIdentity()), and parsing and returning results.

Benchmark statistics - STS - read one identity
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 21.22 20.7 271.05 289.28 31.67 61.67
stddev 2.84 2.92 7.28 10.29 6.47 9.16
p50 21 21 271 289 31 60
p90 24 24 279 300 37 75
p95 27 24 280 305 41 77
p99 28 25 285 318 55 79

Total

Benchmark statistics - STS - time to execute, not including coldstart
aws-lite aws-lite (bundled) aws-sdk (v2) aws-sdk (v2, bundled) @aws-sdk (v3) @aws-sdk (v3, bundled)
mean 25.58 23.93 273.52 355.81 60.15 205.38
stddev 3.33 2.89 7.22 10.31 8.02 11.79
p50 25 24 274 356 60 205
p90 29 27 282 368 67 221
p95 31 27 282 372 70 223
p99 40 28 287 380 92 235

Latest data

If you’d like to dig deeper, here’s the data from the latest performance run:

aws-lite is an Apache 2.0-licensed open source project under the umbrella of OpenJS Foundation Architect. aws-lite is not in any way affiliated with Amazon Web Services, Inc. (AWS). All names and trademarks are the property of their respective owners.