stuff

Author: juleswritescode

crates/pg_lexer/README.md

@@ -0,0 +1,8 @@
+# pg_lexer
+
+The `pg_lexer` crate exposes the `lex` method, which turns an SQL query text into a `Vec<Token>>`: the base for the `pg_parser` and most of pgtools's operations.
+
+A token is always of a certain `SyntaxKind` kind. That `SyntaxKind` enum is derived from `libpg_query`'s protobuf file.
+
+The SQL query text is mostly lexed using the `pg_query::scan` method (`pg_query` is just a Rust wrapper around `libpg_query`).
+However, that method does not parse required whitespace tokens, so the `lex` method takes care of parsing those and merging them into the result.

crates/pg_lexer_codegen/README.md

@@ -0,0 +1,7 @@
+# pg_lexer_codegen
+
+This crate is responsible for reading `libpg_query`'s protobuf file and turning it into the Rust enum `SyntaxKind`.
+
+It does so by reading the file from the installed git submodule, parsing it with a protobuf parser, and using a procedural macro to generate the enum.
+
+Rust requires procedural macros to be defined in a different crate than where they're used, hence this \_codegen crate.