Skip to content

Commit 032f706

Browse files
authored
Local dev infog & clarifications (windmill-labs#469)
1 parent 03ef60b commit 032f706

File tree

13 files changed

+91
-47
lines changed

13 files changed

+91
-47
lines changed

docs/advanced/11_git_sync/index.mdx

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ For all details on Version Control in Windmill, see [Version Control](../../adva
1010

1111
:::
1212

13-
The first 5 minutes of this video show how to set up a Git repository for a workspace, the rest of the video shows how to use it to deploy to a Prod workspace using GitHub Actions:
13+
The first 5 minutes of this video show how to set up a Git repository for a workspace (Git Sync - workspace mode), the rest of the video shows how to use it to deploy to a Prod workspace using GitHub Actions (Git Sync - item mode):
1414

1515
<iframe
1616
style={{ aspectRatio: '16/9' }}
@@ -24,15 +24,9 @@ The first 5 minutes of this video show how to set up a Git repository for a work
2424

2525
<br/>
2626

27-
:::info Private Folders
27+
## Git Sync - Workspace mode
2828

29-
All scripts, flows and apps located in the workspace will be pushed to the Git repository, except the ones that are saved in private user folders (i.e. where the path starts with `u/`, use those with `f/` instead).
30-
31-
<br/>
32-
33-
Filtering out certain sensitive folders from the sync will be available soon.
34-
35-
:::
29+
All scripts, flows and apps located in the workspace will be pushed to the Git repository, except the ones that are saved in private user folders (i.e. where the path starts with `u/`, use those with `f/` instead). Filtering out certain sensitive folders from the sync will be available soon.
3630

3731
On each [deployment](../../core_concepts/0_draft_and_deploy/index.mdx#deployed-version), only the updated script/flow/app will be pushed to the remote Git repository.
3832

@@ -50,7 +44,7 @@ Run the following commands from the git repo folder to push the initial workspac
5044

5145
This feature is [Cloud & Enterprise Self-Hosted](../../misc/7_plans_details/index.mdx) only.
5246

53-
## Deploy to Prod using a Git Workflow
47+
## Git Sync - Item mode: Deploy to Prod using a Git Workflow
5448

5549
This feature can be used alongside GiHub Actions to adopt a robust development process for your Windmill scripts, flows and apps,
5650
with for example a Staging Workspace making automatically PRs on a repo that pushes to a Prod workspace upon merge.

docs/advanced/12_deploy_to_prod/index.mdx

Lines changed: 39 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,45 @@ import DocCard from '@site/src/components/DocCard';
44

55
Deployments in Windmill are commonly done from the same workspace using the [Draft and Deploy](../../core_concepts/0_draft_and_deploy/index.mdx) buttons.
66

7-
For finer control, you might want to deploy to a prod workspace from a staging area. Here are the options.
7+
For finer control, you might want to deploy to a prod workspace from a staging area.
8+
9+
## Deploy to Prod using a Git Workflow
10+
11+
Windmill integration with Git repositories makes it possible to adopt a robust development process for your Windmill scripts, flows and apps.
12+
13+
The process is as follows:
14+
- Users iterate and make their changes in a "staging" Windmill workspace.
15+
- Every time a Windmill App, Flow or Script is deployed to that workspace (via Windmill's UI), Windmill automatically commits it to this repo and creates one branch per app/flow/script.
16+
- On every commit from Windmill, PRs are automatically created via a GitHub Action. Approved GitHub users can review and merge those PRs.
17+
- Every time a PR is merged, another GitHub Action automatically deploys the change to a "production" Windmill workspace.
18+
19+
This gives the flexibility to fully test new Windmill scripts, flows and apps, while having them version-controlled and deployed in an automated way to the production environment.
20+
21+
Deploy to Prod using a Git Workflow is [Cloud and Self-Hosted Enterprise Edition](/pricing) only.
22+
23+
More details at:
24+
25+
<div class="grid grid-cols-2 gap-6 mb-4">
26+
<DocCard
27+
title="Deploy to Prod using a Git Workflow"
28+
description="Windmill integration with Git repositories makes it possible to adopt a robust development process for your Windmill scripts, flows and apps."
29+
href="/docs/advanced/deploy_gh_gl"
30+
/>
31+
</div>
32+
33+
This process can be used in particular for local development with a solid setup:
34+
35+
![Local Development Setup](../4_local_development/local_development_v0.png "Local Development Setup")
36+
37+
More details at:
38+
39+
<div class="grid grid-cols-2 gap-6 mb-4">
40+
<DocCard
41+
title="Local Development"
42+
description="Develop from various environments such as your terminal, VS Code, and JetBrains IDEs."
43+
href="/docs/advanced/local_development"
44+
/>
45+
</div>
846

947
## Deploy to Staging/Prod Web UI
1048

@@ -31,30 +69,6 @@ More details at:
3169
/>
3270
</div>
3371

34-
## Deploy to Prod using a Git Workflow
35-
36-
Windmill integration with Git repositories makes it possible to adopt a robust development process for your Windmill scripts, flows and apps.
37-
38-
The process is as follows:
39-
- Users iterate and make their changes in a "staging" Windmill workspace.
40-
- Every time a Windmill App, Flow or Script is deployed to that workspace (via Windmill's UI), Windmill automatically commits it to this repo and creates one branch per app/flow/script.
41-
- On every commit from Windmill, PRs are automatically created via a GitHub Action. Approved GitHub users can review and merge those PRs.
42-
- Every time a PR is merged, another GitHub Action automatically deploys the change to a "production" Windmill workspace.
43-
44-
This gives the flexibility to fully test new Windmill scripts, flows and apps, while having them version-controlled and deployed in an automated way to the production environment.
45-
46-
Deploy to Prod using a Git Workflow is [Cloud and Self-Hosted Enterprise Edition](/pricing) only.
47-
48-
More details at:
49-
50-
<div class="grid grid-cols-2 gap-6 mb-4">
51-
<DocCard
52-
title="Deploy to Prod using a Git Workflow"
53-
description="Windmill integration with Git repositories makes it possible to adopt a robust development process for your Windmill scripts, flows and apps."
54-
href="/docs/advanced/deploy_gh_gl"
55-
/>
56-
</div>
57-
5872
## Using Windmill CLI in GitHub Actions
5973

6074
Finally, you can define your own GitHub Actions to pull Windmill workspace regularly from GitHub using [Windmill CLI](../3_cli/index.mdx). To automatically deploy a PR to the Windmill workspace, the [push-on-merge.yaml](https://github.com/windmill-labs/windmill-sync-example/blob/main/.github/workflows/push-on-merge.yaml) can be used.

docs/advanced/13_version_control/index.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Manage changes to scripts workflows, apps and resources using commits & push on
88

99
From the workspace settings, you can set a [git_repository](../../integrations/git_repository.mdx) resource on which the workspace will automatically commit and push scripts, flows and apps to the repository on each [deploy](../../core_concepts/0_draft_and_deploy/index.mdx).
1010

11-
The first 5 minutes of this video show how to set up a Git repository for a workspace, the rest of the video shows how to use it to deploy to a Prod workspace using GitHub Actions:
11+
The first 5 minutes of this video show how to set up a Git repository for a workspace (Git Sync - workspace mode), the rest of the video shows how to use it to deploy to a Prod workspace using GitHub Actions (Git Sync - item mode):
1212

1313
<iframe
1414
style={{ aspectRatio: '16/9' }}

docs/advanced/4_local_development/index.mdx

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,25 @@ import DocCard from '@site/src/components/DocCard';
66

77
# Local Development
88

9-
Windmill has [its own integrated development environment](../../code_editor/index.mdx), but you can also develop and run scripts locally.
9+
Windmill has [its own integrated development environment](../../code_editor/index.mdx), but you can also develop and run scripts locally, on both [self-hosted](../1_self_host/index.mdx) and cloud instances.
1010

11-
Local development in Windmill (whether [self-hosted](../1_self_host/index.mdx) or on the cloud) has two distinct aspects:
11+
## Local Development Recommended Setup
1212

13-
1. **Editing scripts and flows**: This is done in [VS Code](../../cli_local_dev/1_vscode-extension/index.mdx) where Windmill has its own [extension](https://marketplace.visualstudio.com/items?itemName=windmill-labs.windmill). The extension allows for [UI pre-visualization](../../core_concepts/23_instant_preview/index.mdx), fetching resources, variables, scripts & flows, and testing them. However, these runs and executions are technically carried out from your remote instance.
13+
For a more complex and robust local development process, we recommend the following setup with one git repository, a staging and a prod workspace:
14+
15+
![Local Development v0](./local_development_v0.png)
16+
17+
It works as follows:
18+
- A git repository with 2 branches: staging and main.
19+
- A staging workspace with [Git Sync](../11_git_sync/index.mdx) workspace mode (all items stored in [folders](../../core_concepts/8_groups_and_folders/index.mdx) are commited to the repo) to staging branch and [Git Sync](../11_git_sync/index.mdx) item mode (any change is committed to an item-specific branch targeting main, with an associated PR, this can be done with GitHub Action [Open PR on Commit](https://github.com/windmill-labs/windmill-sync-example/blob/main/.github/workflows/open-pr-on-commit.yaml)) to main branch.
20+
- A prod workspace with [Git Sync](../11_git_sync/index.mdx) workspace mode to main branch.
21+
- Continuous Integration + [Windmill CLI](../../advanced/3_cli/index.mdx) between Main (resp. Staging) branch and Prod (resp. Staging) workspace to automatically push repo branch changes to workspace. This can be done with GitHub Action [Push on Merge](https://github.com/windmill-labs/windmill-sync-example/blob/main/.github/workflows/push-on-merge.yaml).
22+
23+
With this set up, users can concurrently iterate on both Windmill IDE and code editors locally and commit changes to an item-specific branch targetting main.
24+
25+
## Editing scripts and flows
26+
27+
Scripts and flow can be editied from any code editor. For VS Code, Windmill has a dedicated [VS Code Extnsion](../../cli_local_dev/1_vscode-extension/index.mdx) that can be found on [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=windmill-labs.windmill). The extension allows for [UI pre-visualization](../../core_concepts/23_instant_preview/index.mdx), fetching resources, variables, scripts & flows, and testing them. However, these runs and executions are technically carried out from your remote instance.
1428

1529
![VS Code Extension](../../../blog/2023-11-20-vscode/vscode_extension.png.webp "Windmill VS Code extension")
1630

@@ -24,7 +38,9 @@ Local development in Windmill (whether [self-hosted](../1_self_host/index.mdx) o
2438

2539
<br/>
2640

27-
2. **Run locally**. For iteration, integration with CI/CD and testing purposes you may need to [run a script locally](./run_locally.mdx) that also interacts with Windmill (for example, to retrieve resources). It will allow to integrate Windmill with any testing framework. To do this, you will need to fill out the context variables that would otherwise be filled out by the Windmill runtime for you.
41+
## Run locally
42+
43+
For iteration, integration with CI/CD and testing purposes you may need to [run a script locally](./run_locally.mdx) that also interacts with Windmill (for example, to retrieve resources). It will allow to integrate Windmill with any testing framework. To do this, you will need to fill out the context variables that would otherwise be filled out by the Windmill runtime for you.
2844

2945
<iframe
3046
style={{ aspectRatio: '16/9' }}
Loading

docs/advanced/4_local_development/run_locally.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,7 @@ then simply run the relevant command for your language.
191191

192192
### Metadata file
193193

194-
Metadata files are used with the [CLI sync](../11_git_sync/cli_sync.mdx) to push to a windmill workspace with additional data such as
194+
Metadata files are used with the [CLI sync](../11_git_sync/cli_sync.mdx) to push to a Windmill workspace with additional data such as
195195
summary, description, additional UI configs and the full lockfile. The metadata file ias at <script_path without extension>.script.yaml.
196196

197197
```

docs/advanced/9_deploy_gh_gl/index.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
1+
import DocCard from '@site/src/components/DocCard';
2+
13
# Deploy to Prod using a Git Workflow
24

35
Windmill integration with Git repositories makes it possible to adopt a robust development process for your Windmill scripts, flows and apps.
46

7+
This feature can be referred as [Git Sync](../11_git_sync/index.mdx) - Item mode.
8+
59
:::tip Deploy to prod
610

711
For all details on Deployments to Prods, see [Deploy to Prod](../../advanced/12_deploy_to_prod/index.mdx).
@@ -23,10 +27,26 @@ Check out the [windmill-sync-example repository](https://github.com/windmill-lab
2327

2428
:::
2529

30+
This process can be used in particular for local development with a solid setup:
31+
32+
![Local Development Setup](../4_local_development/local_development_v0.png "Local Development Setup")
33+
34+
More details at:
35+
36+
<div class="grid grid-cols-2 gap-6 mb-4">
37+
<DocCard
38+
title="Local Development"
39+
description="Develop from various environments such as your terminal, VS Code, and JetBrains IDEs."
40+
href="/docs/advanced/local_development"
41+
/>
42+
</div>
43+
2644
## Git Sync
2745

2846
Deploying to a prod workspace using git requires the [Git Sync](../11_git_sync/index.mdx) feature, which is is a [Cloud and Self-Hosted Enterprise Edition](/pricing)-only feature.
2947

48+
From the workspace settings, you can set a [git_repository](../../integrations/git_repository.mdx) resource on which the workspace will automatically commit and push scripts, flows and apps to the repository on each [deploy](../../core_concepts/0_draft_and_deploy/index.mdx).
49+
3050
## Setup
3151

3252
Note: this is the detailed setup steps for a [GitHub](https://github.com/) repository. It will need to be adapted for [GitLab](https://about.gitlab.com/).

docs/compared_to/airplane.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ Building scripts in Airplane involves transitioning
129129
from the web UI to a separate code editor and eventually employing the terminal.
130130
:::
131131

132-
Windmill has a Script Editor with an integrated [Monaco Editor](../code_editor/index.mdx). Only if you prefer, you can also use our [CLI](../cli_local_dev/index.mdx) and [VS Code](../cli_local_dev/1_vscode-extension/index.mdx), and write your code [concurrently with your team](../core_concepts/7_multiplayer/index.md).
132+
Windmill has a Script Editor with an integrated [Monaco Editor](../code_editor/index.mdx). Only if you prefer, you can also use our [CLI](../advanced/3_cli/index.mdx) and [VS Code](../cli_local_dev/1_vscode-extension/index.mdx), and write your code [concurrently with your team](../core_concepts/7_multiplayer/index.md).
133133

134134
<div class="grid grid-cols-2 gap-6 mb-4">
135135
<DocCard

docs/compared_to/peers.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ By comparison, in Windmill one would just write the canonical python or typescri
5353

5454
<br />
5555

56-
> [1]: Windmill is not just a workflow engine, it is also a function as a service (FaaS) infrastructure where it can run arbitrary scripts in typescript/python/bash/go. Contrary to lambda or gcp cloud functions, we do not need the functions to be pre-packaged and deployed in advance AOT. For typescript, we rely on the deno runtime that leverage v8 isolates and the immutable caching capabilities of deno. For python, we have implemented our own dependency resolver that will override the python virtual path and create a unique virtual environment for that specific script that will respect the lockfile generated at time of saving the script/flow for reproducibility. Given that those are interpreted languages, we pay no performance penalty to interpret that code on demand. So the only limiting factor for task execution is that in the events that dependencies are not cached by the worker, they need to be installed at time of execution. With a limited number of workers, the likelihood of a cache miss is low as soon as one script/workflow is executed more than once. With a large fleet of workers, cache miss increase and hence we have implemented a global caching mechanism that relies on syncing the cache through s3. It is only available in our [enterprise edition](../misc/7_plans_details/index.mdx). With it in place, we run tasks and workflows with 0 overhead versus running the same scripts on bare-metal. You can even leverage hardware acceleration without any additional configuration.
56+
> [1]: Windmill is not just a workflow engine, it is also a function as a service (FaaS) infrastructure where it can run arbitrary scripts in typescript/python/bash/go. Contrary to lambda or gcp cloud functions, we do not need the functions to be pre-packaged and deployed in advance AOT. For TypeScript, we rely on the deno runtime that leverages v8 isolates and the immutable caching capabilities of deno. For Python, we have implemented our own dependency resolver that will override the python virtual path and create a unique virtual environment for that specific script that will respect the lockfile generated at time of saving the script/flow for reproducibility. Given that those are interpreted languages, we pay no performance penalty to interpret that code on demand. So the only limiting factor for task execution is that in the events that dependencies are not cached by the worker, they need to be installed at time of execution. With a limited number of workers, the likelihood of a cache miss is low as soon as one script/workflow is executed more than once. With a large fleet of workers, cache miss increase and hence we have implemented a global caching mechanism that relies on syncing the cache through s3. It is only available in our [enterprise edition](../misc/7_plans_details/index.mdx). With it in place, we run tasks and workflows with 0 overhead versus running the same scripts on bare-metal. You can even leverage hardware acceleration without any additional configuration.
5757

5858
</details>
5959

docs/integrations/git_repository.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ More:
1414
/>
1515
</div>
1616

17-
The first 5 minutes of this video show how to set up a Git repository for a workspace, the rest of the video shows how to use it to deploy to a Prod workspace using GitHub Actions:
17+
The first 5 minutes of this video show how to set up a Git repository for a workspace (Git Sync - workspace mode), the rest of the video shows how to use it to deploy to a Prod workspace using GitHub Actions (Git Sync - item mode):
1818

1919
<iframe
2020
style={{ aspectRatio: '16/9' }}

0 commit comments

Comments
 (0)