Skip to content

S3: add heuristic for S3 requests from IaC/SDK with no prefix #12678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

bentsku
Copy link
Contributor

@bentsku bentsku commented May 28, 2025

Motivation

This PR implements a small workaround for S3 Virtual-Hosted requests done without an s3. prefixed endpoint.

We sometimes have the case that we cannot update the S3-only endpoint URL for some tools interacting with LocalStack, like IaC (CDK, for ex.) or SDKs.

For context, we implemented a prefixed endpoint for S3 to make sure we can properly parse the request, and know what is the bucket name in the host (anything before the s3. prefix, if there is anything, otherwise, we fallback to the path style access).

This can lead to issues where a request done to http://test-bucket.localhost.localstack.cloud:4566/test-key would not be properly parsed, because S3 would not try to parse the bucket name out of the host, and see test-key as the bucket name.

This workaround allows the S3 parser to parse the request if and only if there is something before the LOCALSTACK_HOST environment variable. This can enable some use-cases, but not all. We're still not entirely sure this could not lead to more complications, like requests being interpreted as virtual-hosted when not (what about the location service, which could have an endpoint named http://location.localhost.localstack.cloud:4566, this request would be directed to the location service even though it could be a valid bucket name. Or deadline)

We can park this work for now and revisit if we have a strong need for it. It was validated that it did fix some simple use-cases, but it has some other limitations:

  • non-signed requests in Virtual-Hosted mode won't work as they won't be flagged as S3 requests
  • CORS behavior will not work for non-prefixed S3 endpoint in virtual-hosted mode

\cc @joe4dev

Changes

  • add a workaround in the S3 Parser to properly detect Virtual-Hosted requests if the Host contains the LOCALSTACK_HOST value and be able to parse the request

TODO

We still need to add some tests around the limitations to have them documented and tested

@bentsku bentsku added this to the Playground milestone May 28, 2025
@bentsku bentsku self-assigned this May 28, 2025
@bentsku bentsku added aws:s3 Amazon Simple Storage Service semver: patch Non-breaking changes which can be included in patch releases labels May 28, 2025
Copy link

S3 Image Test Results (AMD64 / ARM64)

  2 files    2 suites   8m 32s ⏱️
496 tests 446 ✅  50 💤 0 ❌
992 runs  892 ✅ 100 💤 0 ❌

Results for commit 78e5c8a.

Copy link

Test Results - Preflight, Unit

21 579 tests  ±0   19 927 ✅ ±0   6m 18s ⏱️ +12s
     1 suites ±0    1 652 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit 78e5c8a. ± Comparison against base commit b90f172.

Copy link

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 7s ⏱️ ±0s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit 78e5c8a. ± Comparison against base commit b90f172.

Copy link

Test Results - Alternative Providers

597 tests   420 ✅  14m 41s ⏱️
  4 suites  177 💤
  4 files      0 ❌

Results for commit 78e5c8a.

Copy link

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 41m 52s ⏱️ +25s
4 470 tests +1  4 082 ✅ +1  388 💤 ±0  0 ❌ ±0 
4 472 runs  +1  4 082 ✅ +1  390 💤 ±0  0 ❌ ±0 

Results for commit 78e5c8a. ± Comparison against base commit b90f172.

Copy link

Test Results (amd64) - Integration, Bootstrap

    5 files      5 suites   2h 24m 0s ⏱️
4 825 tests 4 284 ✅ 541 💤 0 ❌
4 831 runs  4 284 ✅ 547 💤 0 ❌

Results for commit 78e5c8a.

@bentsku bentsku changed the title S3: add heuristic for S3 requests from IaC with no prefix S3: add heuristic for S3 requests from IaC/SDK with no prefix Jun 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws:s3 Amazon Simple Storage Service semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant