Skip to content

Fix dead links to the tokenizer's source code #1250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 19, 2023
Merged

Fix dead links to the tokenizer's source code #1250

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
401d365
Fixed token.c hyperlink in compiler design
MahadMuhammad Dec 17, 2023
ae8532b
Wrap lines; small grammar fixes
AlexWaygood Dec 17, 2023
6eaed71
Point grammer.rst to pure lexer tokenization code
MahadMuhammad Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion developer-workflow/grammar.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Note: sometimes things mysteriously don't work. Before giving up, try ``make cl
Then run ``make regen-ast`` to regenerate
:cpy-file:`Include/internal/pycore_ast.h` and :cpy-file:`Python/Python-ast.c`.

* :cpy-file:`Parser/tokenizer.c` contains the tokenization code.
* :cpy-file:`Parser/lexer/` contains the tokenization code.
This is where you would add a new type of comment or string literal, for example.

* :cpy-file:`Python/ast.c` will need changes to validate AST objects
Expand Down
2 changes: 1 addition & 1 deletion internals/compiler.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Abstract

In CPython, the compilation from source code to bytecode involves several steps:

1. Tokenize the source code (:cpy-file:`Parser/tokenizer.c`).
1. Tokenize the source code (:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`).
2. Parse the stream of tokens into an Abstract Syntax Tree
(:cpy-file:`Parser/parser.c`).
3. Transform AST into an instruction sequence (:cpy-file:`Python/compile.c`).
Expand Down
5 changes: 3 additions & 2 deletions internals/parser.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -599,8 +599,9 @@ If you are on Windows you can use the Visual Studio project files to regenerate

./PCbuild/build.bat --regen

How tokens are generated and the rules governing this is completely up to the tokenizer (:cpy-file:`Parser/tokenizer.c`)
and the parser just receives tokens from it.
How tokens are generated and the rules governing this are completely up to the tokenizer
(:cpy-file:`Parser/lexer/` and :cpy-file:`Parser/tokenizer/`);
the parser just receives tokens from it.

Memoization
-----------
Expand Down