Skip to content

Fix race condition in workspace build API endpoint #18948

@blink-so

Description

@blink-so

Problem

As identified in PR #18932, there's a race condition in the workspace build API endpoint that can cause flaky tests and potentially inconsistent API responses.

The issue occurs in the /api/v2/workspacesbuilds/{workspacebuild}/ endpoint where two separate database queries happen outside of a transaction:

  1. Query for database.WorkspaceBuild in the route handler
  2. Call to GetProvisionerJobsByIDsWithQueuePosition in workspaceBuildsData

Because these calls happen outside of a transaction, the state can change between them, resulting in an in-progress workspace build having a completed provisioner job attached to it.

Code References

  • Route handler:

    coder/coderd/coderd.go

    Lines 1409 to 1415 in a3f64f7

    r.Route("/workspacebuilds/{workspacebuild}", func(r chi.Router) {
    r.Use(
    apiKeyMiddleware,
    httpmw.ExtractWorkspaceBuildParam(options.Database),
    httpmw.ExtractWorkspaceParam(options.Database),
    )
    r.Get("/", api.workspaceBuild)
  • workspaceBuildsData call:
    data, err := api.workspaceBuildsData(ctx, []database.WorkspaceBuild{workspaceBuild})
  • GetProvisionerJobsByIDsWithQueuePosition call:
    jobs, err := api.Database.GetProvisionerJobsByIDsWithQueuePosition(ctx, database.GetProvisionerJobsByIDsWithQueuePositionParams{
    IDs: jobIDs,
    StaleIntervalMS: provisionerdserver.StaleInterval.Milliseconds(),
    })
    if err != nil && !errors.Is(err, sql.ErrNoRows) {

Solution

The database queries should be wrapped in a transaction to ensure consistency, or the logic should be restructured to avoid the race condition.

Impact

  • Fixes flaky test TestAPI/ModifyAutostopWithRunningWorkspace
  • May fix other similar flakes in the test suite
  • Improves API consistency and reliability

Follow-up work identified from PR #18932

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions