Skip to content

Update to Python 3.11 #8087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Sep 26, 2023
Merged

Update to Python 3.11 #8087

merged 15 commits into from
Sep 26, 2023

Conversation

dfangl
Copy link
Member

@dfangl dfangl commented Apr 6, 2023 •
edited by alexrashed
Loading

Motivation

This PR contains all changes necessary to upgrade our Python version for the runtime from 3.10 to 3.11.

Changes

  • Update all CircleCI runners (which use Python 3.11 by default).
  • Update all GitHub actions to install Python 3.11 instead of Python 3.10.
  • Install libsnappy-dev instead of libsasl2-dev when running against Pro (see changes there for a detailed explanation).
  • Update the dockerfile base image to the latest sha digest of Python 3.11.5.
  • Update all references to virtual environments which contain the Python version (f.e. /usr/lib/localstack/python-packages/lib/python3.11/site-packages).
  • Remove the SSL proxy (it's not used in Community anymore and the used library - python-proxy - is not maintained and not compatible with 3.11).
  • Update dill to a Python 3.11 compatible version.
  • Update the Lambda Runtime version used for local execution (in the runtime of LocalStack itself) from 3.10 to 3.11.

Impact

This PR only updates the runtime version of LocalStack itself. The LocalStack CLI will still be compatible with Python 3.7+.
However, Python Initialization Hooks and LocalStack Extensions will then run in a Python 3.11 environment instead of Python 3.10.

TODO:

  • Analyze Podman tests (unrelated)
  • Ensure compatibility with LocalStack Pro before merging this PR.

@thrau thrau mentioned this pull request Apr 9, 2023
Closed
@alexrashed
Copy link
Member

Just for the record: It seems like this is currently only blocked by https://github.com/qwj/python-proxy not being compatible with Python 3.11. The library seems to be unmaintained, there is an open PR for quite a while: qwj/python-proxy#163
If this PR isn't merged / a new release is published at PyPi we need to find an alternative or create a fork (MIT licensed).

@thrau
Copy link
Member

thrau commented Jun 25, 2023

what is the library used for @alexrashed ?

@dfangl
Copy link
Member Author

dfangl commented Jun 26, 2023

It is the base for start_ssl_proxy, which only usage left is by our EKS service.
To get rid of it, we would need to adapt our new gateway based proxy implementation to allow the use of client certificates, I think this is the only functionality that is still missing.

@alexrashed alexrashed mentioned this pull request Jun 26, 2023
Closed
@dfangl dfangl added this to the 3.0 milestone Jul 11, 2023
@alexrashed
Copy link
Member

alexrashed commented Jul 13, 2023 •
edited
Loading

I just rebased this PR and cherry-picked f1da6ac from #8577.
I would like to get this PR green to have a clear idea on what's left to do to get this over the line.

In another commit I removed all installations of SASL lib OS packages. Packages are moving to thobbs/pure-sasl due to cloudera/python-sasl#30, and the package should not be necessary anymore.

@alexrashed alexrashed added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Jul 13, 2023
@alexrashed alexrashed force-pushed the update-python-311 branch 2 times, most recently from 796ed61 to dac4c8d Compare July 13, 2023 08:55
@github-actions
Copy link

github-actions bot commented Jul 13, 2023 •
edited
Loading

LocalStack Community integration with Pro

       2 files         2 suites   1h 10m 46s ⏱️
2 220 tests 1 723 ✔️ 497 💤 0 ❌
2 221 runs  1 723 ✔️ 498 💤 0 ❌

Results for commit 99c7686.

♻️ This comment has been updated with latest results.

@localstack localstack deleted a comment from github-actions bot Jul 13, 2023
@alexrashed
Copy link
Member

alexrashed commented Jul 13, 2023 •
edited
Loading

It seems like the Podman tests might be a bit flaky (only the first podman ps command failed).
I retriggered them and started a run on master to find out what the current state on master is: https://github.com/localstack/localstack/actions/runs/5542533626

Edit: Yes, the tests also fail on master currently, they are unrelated and should be fixed in another PR.

@dfangl dfangl force-pushed the update-python-311 branch 2 times, most recently from 6a7c7af to 41b04b0 Compare July 25, 2023 16:43
@alexrashed alexrashed self-assigned this Aug 1, 2023
@alexrashed alexrashed force-pushed the update-python-311 branch 3 times, most recently from b39c93d to 86aafe4 Compare September 14, 2023 13:02
@alexrashed alexrashed modified the milestones: 3.0, 2.3 Sep 14, 2023
@coveralls
Copy link

coveralls commented Sep 14, 2023 •
edited
Loading

Coverage Status

coverage: 83.07% (+0.007%) from 83.063% when pulling 99c7686 on update-python-311 into 6287bfe on master.

@alexrashed alexrashed marked this pull request as ready for review September 26, 2023 08:39
@alexrashed alexrashed requested a review from joe4dev as a code owner September 26, 2023 08:39
dfangl and others added 15 commits September 26, 2023 10:46
@dfangl @alexrashed
first stab at python3.11
1a83433
@dfangl @alexrashed
fix sns_api snapshot transform regex
5f0447f
@dfangl @alexrashed
update dill
0d3e3f6
@alexrashed
update python in pipelines
dcec852
@alexrashed
remove sasl OS package installations, add libsnappy-dev
90d47a2
@alexrashed
remove pproxy
6f345f1
@dfangl @alexrashed
first version of a different approach (tls proxy)
549e8aa
@dfangl @alexrashed
fix tls proxy due to select only affecting underlying socket
dd9da53
@alexrashed
update newly created references to Python 3.10 to Python 3.11
0af9ee4
@alexrashed
upgrade default lambda python runtime for host mode tests
69b8a42
@alexrashed
downgrade Java from 17 to 11 for locally executed tests
2713fa0
@dfangl @alexrashed
slightly refactor tls proxy
78f6e3a
@dfangl @alexrashed
Revert "slightly refactor tls proxy"
d6fd7cc 
This reverts commit 3d42961.
@dfangl @alexrashed
Revert "fix tls proxy due to select only affecting underlying socket"
b9a7824 
This reverts commit 4fa3436.
@dfangl @alexrashed
Revert "first version of a different approach (tls proxy)"
99c7686 
This reverts commit f325375.
alexrashed
alexrashed approved these changes Sep 26, 2023
View reviewed changes
Copy link
Member

@alexrashed alexrashed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! We're finally there! Let's get this over the line! 🚀 :shipit:

@alexrashed alexrashed changed the title Update to python 3.11 Update to Python 3.11 Sep 26, 2023
@alexrashed alexrashed merged commit 9ccb39c into master Sep 26, 2023
@alexrashed alexrashed deleted the update-python-311 branch September 26, 2023 09:35
This was referenced Sep 26, 2023
Merged
Merged
@alexrashed alexrashed mentioned this pull request Feb 13, 2024
Draft
3 tasks
@lizard-boy lizard-boy mentioned this pull request Sep 20, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexrashed alexrashed alexrashed approved these changes

@joe4dev joe4dev Awaiting requested review from joe4dev

@dominikschubert dominikschubert Awaiting requested review from dominikschubert

@thrau thrau Awaiting requested review from thrau

Assignees

@alexrashed alexrashed

Labels
semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases
Projects
None yet
Milestone
2.3
Development

Successfully merging this pull request may close these issues.
4 participants
@dfangl @alexrashed @thrau @coveralls