Skip to content

Commit be8561b

Browse files
hcourdenthcourdent
authored
Dependencies in TypeScript (windmill-labs#614)
* Dependencies in TypeScript * Clarification doc * Video and clean up * Clarification * RF comments & tree * Final? changes
1 parent 3ff8dde commit be8561b

File tree

20 files changed

+725
-416
lines changed

20 files changed

+725
-416
lines changed

‎docs/advanced/14_dependencies_in_typescript/index.mdx

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
import DocCard from '@site/src/components/DocCard';
2+
3+
# Dependencies in TypeScript
4+
5+
In Windmill [standard mode](#lockfile-per-script-inferred-from-imports-standard), dependencies in [TypeScript](../../getting_started/0_scripts_quickstart/1_typescript_quickstart/index.mdx) are handled directly within their scripts without the need to manage separate dependency files.
6+
For TypeScript, there are two runtime options available: [Bun](https://bun.sh/) and [Deno](https://deno.land/).
7+
Both of these runtimes allow you to include dependencies directly in the script, and Windmill automatically handles the resolution and caching of these dependencies to ensure fast and consistent execution.
8+
9+
There are however methods to have more control on your dependencies:
10+
- Leveraging [standard mode](#lockfile-per-script-inferred-from-imports-standard) on [web IDE](#web-ide) or [locally](#cli).
11+
- Overriding dependencies [providing a package.json](#lockfile-per-script-inferred-from-a-packagejson).
12+
- [Bundling](#bundle-per-script-built-by-cli) per script with CLI, more powerful and local only.
13+
14+
Moreover, there are two other tricks, compatible with the methodologies mentioned above:
15+
- [Sharing Common Logic with Relative Imports](#sharing-common-logic-with-relative-imports-when-not-using-bundles) when not using Bundles.
16+
- [Private npm Registry & Private npm Packages](#private-npm-registry--private-npm-packages).
17+
18+
## Lockfile per script inferred from imports (Standard)
19+
20+
In Windmill, you can run scripts without having to [manage a package.json](#lockfile-per-script-inferred-from-a-packagejson) directly. This is achieved by automatically parsing the [imports](../6_imports/index.mdx) and resolving the dependencies.
21+
22+
When using Bun as the runtime for TypeScript in Windmill, dependencies are resolved directly from the script imports and their imports when using [sharing common logic](../5_sharing_common_logic/index.md). The TypeScript runtime Bun ensures 100% compatibility with Node.js without requiring any code modifications.
23+
24+
```ts
25+
// unpinned import
26+
import { toWords } from 'number-to-words';
27+
28+
// versioned import
29+
import * as wmill from 'windmill-client@1.147.3';
30+
```
31+
32+
Similarly, for TypeScript scripts using Deno as the runtime, the dependencies and their versions are specified directly in the script, and the resolution is managed by Deno. This method allows for direct use of npm imports and Windmill client imports without requiring any additional configuration for dependency management.
33+
34+
```ts
35+
// unpinned import
36+
import { toWords } from 'npm:number-to-words';
37+
38+
// versioned import
39+
import * as wmill from "npm:windmill-client@1.335.0";
40+
```
41+
42+
<div className="grid grid-cols-2 gap-6 mb-4">
43+
<DocCard
44+
title="TypeScript Client"
45+
description="The TypeScript client for Windmill allows you to interact with the Windmill platform using TypeScript in Bun / Deno runtime."
46+
href="/docs/advanced/clients/ts_client"
47+
/>
48+
</div>
49+
50+
### Web IDE
51+
52+
When a script is deployed through the Web IDE, Windmill generates a lockfile to ensure that the same [version of a script](../../script_editor/versioning.mdx) is always executed with the same versions of its [dependencies](../6_imports/index.mdx). To generate a lockfile, it analyzes the imports, the imports can use a version pin (e.g. `windmill-client@1.147.3`) if no version is used, it uses the latest version. Windmill's [workers](../../core_concepts/9_worker_groups/index.mdx) cache dependencies to ensure fast performance without the need to pre-package dependencies - most jobs take under 100ms end-to-end.
53+
54+
At runtime, a deployed script always uses the same version of its dependencies.
55+
56+
At each [deployment](../../core_concepts/0_draft_and_deploy/index.mdx), the lockfile is automatically recomputed from the imports in the scripts and the imports used by the relative imports. The computation of that lockfile is done by a dependency jobs that you can find in the [Runs](../../core_concepts/5_monitor_past_and_future_runs/index.mdx) page.
57+
58+
### CLI
59+
60+
On [local development](../4_local_development/index.mdx), each script gets:
61+
- a content file (`script_path.py`, `script_path.ts`, `script_path.go`, etc.) that contains the code of the script,
62+
- a metadata file (`script_path.yaml`) that contains the metadata of the script,
63+
- a lockfile (`script_path.lock`) that contains the dependencies of the script.
64+
65+
You can get those 3 files for each script by pulling your workspace with command [`wmill sync pull`](../3_cli/sync.mdx).
66+
67+
Editing a script is as simple as editing its content. The code can be edited freely in your IDE, and there are possibilities to even run it locally if you have the correct development environment setup for the script language.
68+
69+
Using [wmill CLI](../3_cli/index.mdx) command [`wmill script generate-metadata`](../3_cli/script.md#re-generating-a-script-metadata-file), lockfiles can be generated as files. The CLI asks the Windmill servers to run dependency job, using either the [package.json (if present)](#lockfile-per-script-inferred-from-a-packagejson) or asking Windmill to automatically resolve it from the script's code as input, and from the output of those jobs, create the lockfiles.
70+
When a lockfile is present alongside a script at time of deployment by the CLI, no dependency job is run and the present lockfile is used instead.
71+
72+
<div className="grid grid-cols-2 gap-6 mb-4">
73+
<DocCard
74+
title="Command Line Interface (CLI)"
75+
description="The Windmill CLI, `wmill` allows you to interact with Windmill instances right from your terminal."
76+
href="/docs/advanced/cli"
77+
/>
78+
<DocCard
79+
title="Local Development"
80+
description="Develop locally, push to git and deploy automatically to Windmill."
81+
href="/docs/advanced/local_development"
82+
/>
83+
</div>
84+
85+
## Lockfile per script inferred from a package.json
86+
87+
Although Windmill can [automatically resolve imports](#lockfile-per-script-inferred-from-imports). It is possible to override the dependencies by providing a `package.json` file in the same directory as the script as you would do in a standard Node.js project, building and maintaining a package.json to declare dependencies.
88+
89+
<iframe
90+
style={{ aspectRatio: '16/9' }}
91+
src="https://www.youtube.com/embed/AycoBCQyjUU"
92+
title="Perpetual Scripts"
93+
frameBorder="0"
94+
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
95+
allowFullScreen
96+
className="border-2 rounded-xl object-cover w-full dark:border-gray-800"
97+
></iframe>
98+
99+
<br/>
100+
101+
When doing [`wmill script generate-metadata`](../3_cli/script.md#re-generating-a-script-metadata-file), if a package.json is discovered, the closest one will be used as source-of-truth instead of being discovered from the imports in the script directly to generate the lockfile from the server.
102+
103+
You can write those package.json manually or through a standar `npm install <X>`.
104+
105+
Several package.json files can therefore coexist, each having authority over the scripts closest to it:
106+
107+
```
108+
└── windmill_folder/
109+
├── package.json
110+
├── f/foo/
111+
│ ├── package.json
112+
│ ├── script1.ts
113+
│ ├── # script1.ts will use the dependencies from windmill_folder/f/foo/package.json
114+
│ └── /bar/
115+
│ ├── package.json
116+
│ ├── script2.ts
117+
│ └── # script2.ts will use the dependencies from windmill_folder/f/foo/bar/package.json
118+
└── f/baz/
119+
├── script3.ts
120+
└── # script3.ts will use the dependencies from windmill_folder/package.json
121+
```
122+
123+
The Windmill [VS Code Extension](../../cli_local_dev/1_vscode-extension/index.mdx) has a toggle "Infer lockfile" / "Use current lockfile".
124+
125+
With this toggle, you can choose to use the metadata lockfile (derived from package.json after `wmill script generate-metadata`) instead of inferring them directly from the script.
126+
127+
![Toggle Lockfile](../../cli_local_dev/1_vscode-extension/toggle_lockfile.png 'Toggle Lockfile')
128+
129+
<div className="grid grid-cols-2 gap-6 mb-4">
130+
<DocCard
131+
title="Local Development"
132+
description="Develop locally, push to git and deploy automatically to Windmill."
133+
href="/docs/advanced/local_development"
134+
/>
135+
<DocCard
136+
title="Command Line Interface (CLI)"
137+
description="The Windmill CLI, `wmill` allows you to interact with Windmill instances right from your terminal."
138+
href="/docs/advanced/cli"
139+
/>
140+
<DocCard
141+
title="VS Code Extension"
142+
description="Build scripts and flows in the comfort of your VS Code Editor, while leveraging Windmill UIs for test & flows edition."
143+
href="/docs/cli_local_dev/vscode-extension"
144+
/>
145+
</div>
146+
147+
## Bundle per script built by CLI
148+
149+
This method can only be deployed from the [CLI](../3_cli/index.mdx), on [local development](../4_local_development/index.mdx).
150+
151+
To work with large custom codebases, there is another mode of deployment that relies on the same mechanism as similar services like Lambda or cloud functions: a bundle is built locally by the CLI using [esbuild](https://esbuild.github.io/) and deployed to Windmill.
152+
153+
This bundle contains all the code and dependencies needed to run the script.
154+
155+
Windmill CLI, it is done automatically on `wmill sync push` for any script that falls in the patterns of includes and excludes as defined by the [wmill.yaml](../../core_concepts/33_codebases_and_bundles/index.mdx#wmillyaml) (in the codebase field).
156+
157+
<div className="grid grid-cols-2 gap-6 mb-4">
158+
<DocCard
159+
title="Codebases & Bundles"
160+
description="Deploy scripts with any local relative imports as bundles."
161+
href="/docs/core_concepts/codebases_and_bundles"
162+
/>
163+
</div>
164+
165+
## Other
166+
167+
Two tricks can be used: [Relative Imports](#relative-imports) and [Private npm Registry & Private npm Packages](#private-npm-registry--private-npm-packages). Both are compatible with the methods described above.
168+
169+
### Sharing Common Logic with Relative Imports when not using Bundles
170+
171+
If you want to share common logic with Relative Imports when not using [Bundles](#bundle-per-script-built-by-cli), this can be done easily using [relative imports](../5_sharing_common_logic/index.md) in both Bun and Deno.
172+
173+
This applies for all methods above, except absolute imports do not work for [codebases and bundles](#bundle-per-script-built-by-cli).
174+
175+
Note that in both the webeditor and with the CLI, your scripts do not necessarily need to have a main function. If they don't, they are assumed to be shared logic and not runnable scripts.
176+
177+
It works extremely well in combination with [Developing scripts locally](../4_local_development/index.mdx) and you can easily sync your scripts with the [CLI](../3_cli/index.mdx).
178+
179+
It is possible to import directly from other TypeScript scripts. One can simply follow the path layout. For instance,
180+
`import { foo } from "../script_name.ts"`. A more verbose example below:
181+
182+
```typescript
183+
import { main as foo, util } from '../my_script_path.ts';
184+
```
185+
186+
Relative imports syntax is much preferred as it will work on [local editors](../4_local_development/index.mdx).
187+
188+
You may also use absolute imports (but won't work on local editors):
189+
190+
```typescript
191+
import { main as foo, util } from '/f/<foldername>/script_name.ts';
192+
193+
export async function main() {
194+
await foo();
195+
util();
196+
}
197+
```
198+
199+
Note that path in Windmill can have as many depth as needed, so you can have paths like this `f/folder/subfolder/my_script_path.ts` and relative imports will work at any level. Hence, it will work exactly the same as on local.
200+
201+
<div className="grid grid-cols-2 gap-6 mb-4">
202+
<DocCard
203+
title="Sharing Common Logic"
204+
description="It is common to want to share common logic between your scripts. This can be done easily using relative imports in both Python and Deno."
205+
href="/docs/advanced/sharing_common_logic"
206+
/>
207+
</div>
208+
209+
### Private npm Registry & Private npm Packages
210+
211+
You can use private npm registries and private npm packages in your TypeScript scripts.
212+
213+
This applies to all methods above. Only, if using Codebases & Bundles locally, there is nothing to configure in Windmill, because the bundle is built locally using your locally-installed modules (which support traditional npm packages and private npm packages).
214+
215+
![Private NPM registry](../6_imports/private_registry.png "Private NPM registry")
216+
217+
On [Enterprise Edition](/pricing), go to `Instance settings -> Core -> NPM Config Registry`.
218+
219+
Set the registry URL: `https://npm.pkg.github.com/OWNER` (replace `OWNER` with your GitHub username or organization name).
220+
221+
Currently, Deno does not support private npm packages requiring tokens (but support private npm registries). Bun however does.
222+
223+
If a token is required, append `:_authToken=<your url>` to the URL.
224+
225+
Combining the two, you can import private packages from npm
226+
227+
```
228+
https://registry.npmjs.org/:_authToken=npm_bKZp1kOKzWsNPUvx2LpyUzIJqi2uaw23eqw
229+
```
230+
231+
If the private registry is exposing custom certificates,`DENO_CERT` and `DENO_TLS_CA_STORE` env variables can be used as well (see [Deno documentaion](https://docs.deno.com/runtime/manual/getting_started/setup_your_environment#environment-variables) for more info on those options).
232+
233+
```dockerfile
234+
windmill_worker:
235+
...
236+
environment:
237+
...
238+
- DENO_CERT=/custom-certs/root-ca.crt
239+
```
240+
241+
<div className="grid grid-cols-2 gap-6 mb-4">
242+
<DocCard
243+
title="Private npm Registry & Private npm Packages"
244+
description="Import private packages from npm in Windmill."
245+
href="/docs/advanced/imports#private-npm-registry--private-npm-packages"
246+
/>
247+
</div>

‎docs/advanced/2_clients/ts_client.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ import TabItem from '@theme/TabItem';
33

44
# TypeScript Client
55

6-
The TypeScript client for Windmill allows you to interact with the Windmill platform using TypeScript in [Deno](https://deno.land/) / [Bun](https://bun.sh/) runtime. This client provides a set of functions and utilities to access Windmill resources and perform various operations.
6+
The TypeScript client for Windmill allows you to interact with the Windmill platform using TypeScript in [Bun](https://bun.sh/) / [Deno](https://deno.land/) runtime. This client provides a set of functions and utilities to access Windmill resources and perform various operations.
77

88
## Installation
99

‎docs/advanced/3_cli/script.md

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,13 @@ wmill script bootstrap --summary 'Great script' --description 'This script does
6363

6464
## (Re-)Generating a script metadata file
6565

66-
The wmill script generate-metadata command is used to read all the files that have been edited and gracefully update the metadata file and lockfile accordingly, inferring the script schema from the script content and generating the locks. Only the schema and the locks part of the metadata file will be updated. If a change was made to other fields like description or summary, it will be kept.
66+
The `wmill script generate-metadata` command is used to read all the files that have been edited and gracefully update the metadata file and lockfile accordingly, inferring the script schema from the script content and generating the locks. Only the schema and the locks part of the metadata file will be updated. If a change was made to other fields like description or summary, it will be kept.
67+
68+
This command can only be run at the same level of the `wmill-lock.yaml` of your workspace, so by default at top level of the repository.
69+
70+
```bash
71+
wmill script generate-metadata
72+
```
6773

6874
You can also generate metadata for a single script with `wmill script generate-metadata <path>`:
6975

@@ -73,9 +79,21 @@ wmill script generate-metadata [--lock-only] [--schema-only] [<path>]
7379

7480
Note that you can explicitly exclude (or include) specific files or folders to be taken into account by this command, with a [`wmill.yaml` file](https://github.com/windmill-labs/windmill-sync-example/blob/main/wmill.yaml).
7581

76-
### Package.json & requirements.txt
82+
### package.json & requirements.txt
83+
84+
When doing `wmill script generate-metadata`, if a `package.json` or `requirements.txt` is discovered, the closest one will be used as source-of-truth instead of being discovered from the imports in the script directly to generate the lockfile from the server.
85+
86+
Below is a video on how to override Windmill inferred dependencies by [providing custom package.json](../14_dependencies_in_typescript/index.mdx#lockfile-per-script-inferred-from-a-packagejson) or requirements.txt.
7787

78-
If you have a `package.json` or `requirements.txt` file in the same directory as your script, the `wmill script generate-metadata` command will automatically include the dependencies in the metadata file.
88+
<iframe
89+
style={{ aspectRatio: '16/9' }}
90+
src="https://www.youtube.com/embed/AycoBCQyjUU"
91+
title="Perpetual Scripts"
92+
frameBorder="0"
93+
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
94+
allowFullScreen
95+
className="border-2 rounded-xl object-cover w-full dark:border-gray-800"
96+
></iframe>
7997
8098
### Arguments
8199

‎docs/advanced/4_local_development/index.mdx

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -142,10 +142,12 @@ wmill sync push
142142
Once the workspace folder is pulled, edits can be made using any IDE.
143143

144144
When created and edited through the UI (Windmill App), the lockfile is automatically generated. On local development, each script gets:
145-
- a content file (`script_path.py`, `script_path.ts`, `script_path.go`, etc.) that contains the code of the script
146-
- a metadata file (`.script_path.yaml`) that contains the metadata of the script
145+
- a content file (`script_path.py`, `script_path.ts`, `script_path.go`, etc.) that contains the code of the script,
146+
- a metadata file (`script_path.yaml`) that contains the metadata of the script,
147147
- a lockfile (`script_path.lock`) that contains the dependencies of the script.
148148

149+
You can get those 3 files for each script by pulling your workspace with command [`wmill sync pull`](../3_cli/sync.mdx).
150+
149151
Editing a script is as simple as editing its content. The code can be edited freely in your IDE, and there are possibilities to even run it locally if you have the correct development environment setup for the script language.
150152

151153
#### Metadata file and lockfile
@@ -178,6 +180,8 @@ Locally, flows and apps are stored in a folder ending with `.flow` and `.app` re
178180

179181
If you have a `package.json` or `requirements.txt` file in the same or parent directory as your script, the `wmill script generate-metadata` command will automatically include the dependencies in the metadata file.
180182

183+
The lockfile for a file generated with generated-metadata will be overriden by the closest package.json from a parent or current folder if there is one.
184+
181185
![Generate Metadata Path](./generate_metadata_path.png 'Generate Metadata Path')
182186

183187
But what if you want to create a new script from scratch? It's also easy, just create a file with the correct extension (or run `wmill script bootstrap <path> <language>`), and simply run `generate-metadata` command to generate the metadata file. The name of the file and its location in the folder will become the script path in Windmill.

0 commit comments

Comments
 (0)