Jump to content

Help:Toolforge/Deploy your tool

From Wikitech

This service is in a beta status, expect some bugs, weird behaviors and changes in the usage flows (both cli and API).

Test it and give feedback, though at this time we don't recommend using it for production services.

Please report any bugs/features/feedback here https://w.wiki/EYoq

Toolforge tools can use the Toolforge Components Service to configure and deploy their jobs (web services coming soon).

Current supported and non-supported features

For the beta there's a limited set of features that we support right away:

Non-supported features (for now, please feel free to check if they are [already on the plan] or request them here <link to phabulous template>):

  • Webservice components
  • One-off components
  • Specifying the component deployment order
  • Rollback of a deployment
  • Specifying your tool configuration from a url (ex. from your git repository)
  • Cancelling a deployment
  • Queuing multiple deployments
  • Deploying specific components only (instead of all of them)
  • Using the latest buildpacks/buildservice versions (on the works)

We will be updating this list during the beta. You can also keep an eye on the Toolforge changelog or if you want to keep a closer look, you can check the Toolforge phabricator dashboard.

Creating your tool configuration

In order to be able to deploy your tool, toolforge needs to know what jobs and services your tool should have, how to build them and how to run them.

For that, you can create a Toolforge Configuration using: 

$ toolforge components config create path/to/my/toolforge.yaml

If you want to import some existing jobs, or you want to see an up-to-date example of tool configuration, you can do so by running: 

$ toolforge components config generate

This will generate an example configuration from your existing jobs if possible, or show you a fake example. You can re-use the output to create your initial config like: 

$ toolforge components config generate > toolforge.yaml

The contents of the toolforge.yaml file will be something like (currently only continuous jobs are supported): 

config_version: v1beta1
components:
  backend-service:
    component_type: continuous
    build:
      repository: https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend
      ref: main
    run:
      command: backend
      port: 8080
      health_check_http: /healthz

You can find an up-to-date list with all the options in the API docs.

If you want to delete your tool config, you can also do so (it will not delete any existing jobs or builds): 

$ toolforge components config delete

Creating a deployment

Once you have a configuration, you can create a deployment to apply it. To do so, you can run: 

$ toolforge components deployment create

You can see the list of deployments with: 

$ toolforge components deployment list

And show the latest one with: 

$ toolforge components deployment show

Or a specific one by passing the id: 

$ toolforge components deployment show <deployment-id>

Forcing rebuilds and reruns

By default the toolforge deploy system will try to reuse any previous build for your components if there were no changes to the configuration and the commit where the given repository and ref point to at the moment of creating the deployment did not change. If that happens, you'll see that your build will say skipped and will still have a build id. That is the build that was used instead of rebuilding.

If you want to force the build to happen again, you can pass --force-build on the cli, or add the parameter force-build=true to the deployment url for the deploy hook (see #Triggering_a_deployment_from_your_CI_runner below).

Something similar happens to the runs, if there were no configuration changes, and the build was skipped, it will not recreate the job for that component.

To force it, you can pass --force-run on the cli, or add the parameter force-run=true to the deployment url.

CLI Example: 

$ toolforge components deployment create --force-build --force-run

And a deployment URL example: 

$ curl --fail-with-body -X POST "https://api.svc.toolforge.org/components/v1/tool/<your-tool-here>/deployment?token=$DEPLOY_TOKEN&force-run=true&force-build=true"

If the deployment fails

In case of failure, there's a chance that the system might be left inconsistent (ex. One of the jobs might have been created, but not the other). It will depend on the moment that the deployment failed. If the failure is a temporary error (ex. timeout, etc.) the easiest fix would be to try deploying again. In any other case it will depend on the precise error (see the top of the page notice on how to open a bug).

Cancel an ongoing deployment

If you want to stop a deployment that's ongoing, you can run: 

$ toolforge components deployment cancel <deployment-id>

This will flag the deployment for cancelling, the deploy process will then cancel any running builds (if any) and skip any new runs.

It will not roll-back any deployed components (yet), so it might leave the tool in an inconsistent state, so you should review the current status or fix the configuration and do another deployment to make sure everything is running as expected.


Triggering a deployment from your CI runner

Create the deploy token

For this we have to first create a deploy token: 

$ toolforge components deploy-token create
Token: 383ed7e1-259b-4f4f-8ba6-d9f0cdd57f56
Created: 2025-06-23T17:00:12.686250Z

This token is secret, and you should not make it public or share it with anyone (it's like a password).

If you forgot which one it is, you can show the current token with: 

$ toolforge components deploy-token show
Warning: You are using a beta feature of Toolforge.
Token: 383ed7e1-259b-4f4f-8ba6-d9f0cdd57f56
Created: 2025-06-23T17:00:12.686250Z

If you think it might have gotten leaked, you can recreate it (and deprecate the old one): 

$ toolforge components deploy-token refresh
Are you sure you want to refresh the deployment token? This will invalidate the existing token. [y/N]: y
Deploy token for tf-test updated successfully.
Warning: You are using a beta feature of Toolforge.
New deployment token:
Token: 8953a1e4-4333-4c3f-9a1a-02c639f7292d
Created: 2025-06-23T16:59:13.558272Z

If you don't use it anymore, consider deleting it (you can always recreate it later when needed): 

$ toolforge components deploy-token delete
Are you sure you want to delete the deployment token? This cannot be undone. [y/N]: y
Deploy token for tf-test deleted successfully.
Warning: You are using a beta feature of Toolforge.
Deleted deployment token:
Token: 8953a1e4-4333-4c3f-9a1a-02c639f7292d
Created: 2025-06-23T16:59:13.558272Z

Create a new deployment using the deploy token

With this token, you can trigger a deployment from anywhere, by using the url: 

https://api.svc.toolforge.org/components/v1/tool/<my-tool>/deployment?token=$TOOLS_DEPLOY_TOKEN

Where $TOOLS_DEPLOY_TOKEN is a variable containing your deploy token (see below on how to set that variable in gitlab/github), and <my-tool> is your tool name.

Retrieve the deployment status using the deploy token

You will be able to check the status of the deployment you just triggered too, if you want to wait see if it fails/succeeds: 

https://api.svc.toolforge.org/components/v1/tool/<my-tool>/deployment/<DEPLOYMENT_ID>?token=$DEPLOY_TOKEN

Where the <DEPLOYMENT_ID> should be the id of the deployment you created (returned when you create the deployment).

Gitlab example

Note that the process would be very similar in any CI system, you just need to trigger the curl command when there's a commit pushed to the branch you care about. This is an example of using gitlab.wikimedia.org.

Create .gitlab-ci.yml pipeline configuration

For gitlab, you can see an example pipeline in the sample-complex-app-backend repo.

The simplest pipeline to deploy on every merge or push to the default branch, would be like: 

deploy:deploy_tools:
  stage: deploy
  rules:
    - if: $CI_COMMIT_TAG
      when: never # Do not run this job when a tag is created manually
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch

  script:
    - curl --fail-with-body -X POST "https://api.svc.toolforge.org/components/v1/tool/<your-tool-here>/deployment?token=$TOOLS_DEPLOY_TOKEN"


You can reuse the scripts we provide to do the deployment which will also wait for the deployment to finish: 

include:
  - project: "repos/cloud/cicd/gitlab-ci"
    file: "toolforge-cd/deploy-to-toolforge.yaml"

Note that it will check for the variable TOOLS_DEPLOY_TOKEN to see if it has to run the deployment, see below on how to set it.

Create secret variable

Then you have to create a new secret named DEPLOY_TOKEN in gitlab. To do so you can go to Settings -> CI/CD -> Variables and create a new one. We recommend making it hidden, protected and masked for security reasons (you can always refresh it if you need it). Like the screenshot:

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Deploy_your_tool&oldid=2323471"