Skip to content

fix flaky transcribe tests #12262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Mar 6, 2025
Merged

fix flaky transcribe tests #12262

merged 3 commits into from
Mar 6, 2025

Conversation

sannya-singal
Copy link
Contributor

@sannya-singal sannya-singal commented Feb 13, 2025 •
edited
Loading

Motivation

Fix flaky pipeline runs:

by making the downloading of vosk models more robust.

Changes

This PR:

  • adds provision to download the models from the vosk server and if it fails, it fallbacks to download the models from the Hugging Face.
  • removes skipping the flaky tests: test_transcribe_happy_path, test_transcribe_supported_media_formats and test_transcribe_start_job.

Successful pipeline runs: https://app.circleci.com/pipelines/github/localstack/localstack?branch=test-transcribe

@sannya-singal sannya-singal marked this pull request as draft February 13, 2025 14:43
@sannya-singal sannya-singal added the semver: patch Non-breaking changes which can be included in patch releases label Feb 13, 2025
@sannya-singal sannya-singal self-assigned this Feb 13, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 13, 2025 •
edited
Loading

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 50m 37s ⏱️ -13s
4 111 tests +4  3 793 ✅ +19  318 💤  - 14  0 ❌  - 1 
4 113 runs  +4  3 793 ✅ +19  320 💤  - 14  0 ❌  - 1 

Results for commit 23b7094. ± Comparison against base commit dd0d96e.

This pull request skips 1 and un-skips 15 tests. 
tests.aws.test_integration ‑ test_kinesis_lambda_forward_chain

tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_happy_path
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[None-None]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-2-None]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-3-test-output]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-4-test-output.json]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-5-test-files/test-output.json]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_start_job[test-output-bucket-6-test-files/test-output]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.amr-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.flac-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.mp3-hello my name is]
…

♻️ This comment has been updated with latest results.

@sannya-singal sannya-singal added this to the 4.3 milestone Feb 21, 2025
@sannya-singal sannya-singal marked this pull request as ready for review March 4, 2025 06:13
@sannya-singal sannya-singal mentioned this pull request Mar 4, 2025
Closed
viren-nadkarni
viren-nadkarni requested changes Mar 4, 2025
View reviewed changes
viren-nadkarni
viren-nadkarni reviewed Mar 5, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 5, 2025

S3 Image Test Results (AMD64 / ARM64)

  2 files    2 suites   8m 5s ⏱️
478 tests 426 ✅  52 💤 0 ❌
956 runs  852 ✅ 104 💤 0 ❌

Results for commit 23b7094.

viren-nadkarni
viren-nadkarni approved these changes Mar 6, 2025
View reviewed changes
Copy link
Member

@viren-nadkarni viren-nadkarni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! The mirrored models should add resiliency to the Transcribe service startup and test executions.

Thanks for working on this 💯

@sannya-singal sannya-singal merged commit 98e2924 into master Mar 6, 2025
40 checks passed
@sannya-singal sannya-singal deleted the test-transcribe branch March 6, 2025 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@viren-nadkarni viren-nadkarni viren-nadkarni approved these changes

@ackdav ackdav Awaiting requested review from ackdav

Assignees

@sannya-singal sannya-singal

Labels
semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Milestone
4.3
Development

Successfully merging this pull request may close these issues.
2 participants
@sannya-singal @viren-nadkarni