Adds better support for Typst

[2ta00ha3]

This PR adds basic support for Typst. It's pretty basic for now and still has many areas to improve, since I'm still discovering LPeg, I'd love to hear any thoughts or suggestions you have!!!
Typst Reference

Preview from vis editor

[orbitalquark]

Thanks for starting on this! When you think it's ready, request a review and I'll have a look. Also, please give a link to Typst documentation.

[2ta00ha3]

Thanks, will surely do

i know this is out of context but i have a question about a rule i have:

local method = lex:tag("UNKNOWN", '#' * lexer.word * '.') * (lex:tag(lexer.FUNCTION_METHOD, lexer.word) * P('('))
lex:add_rule('func_method', method)

this is supposed to match the following expression:
#dict.len()
and it does, but I don't want to tag the #dict. as anything for now (not even UNKNOWN), but only use it to match the whole expr and only tag len, similar to non-input-consuming operators in lpeg, which when i try to use them the expression isn't matched anymore

[2ta00ha3]

This is what I achieved as of now

some things are still not working correctly as I don't know enough LPeg to get them to work (like embedded parsing: notice the if (on the right pane) in the #for block not being tagged as keyword)

Typst's syntax crate (includes lexer, parser...)

[orbitalquark]

Thanks, will surely do

i know this is out of context but i have a question about a rule i have:

local method = lex:tag("UNKNOWN", '#' * lexer.word * '.') * (lex:tag(lexer.FUNCTION_METHOD, lexer.word) * P('('))
lex:add_rule('func_method', method)

this is supposed to match the following expression: #dict.len() and it does, but I don't want to tag the #dict. as anything for now (not even UNKNOWN), but only use it to match the whole expr and only tag len, similar to non-input-consuming operators in lpeg, which when i try to use them the expression isn't matched anymore

You always have to tag matched text with something in order to move on and tag stuff that follows. You can use the lexer.DEFAULT tag for the stuff you don't care about.

[2ta00ha3]

@orbitalquark, I think the work is ready for a review. I added comments in the code about things I wasn't sure how to do, so I’d love your feedback on those and any other suggestions you have

[orbitalquark]

Thanks for your work on this! It looks really promising. As I've noted in a comment below, you've partially solved the "lexer embedded in itself" problem, and I am eager to iterate on it. In the meantime, I've left some comments and suggestions.

This is not a thorough review, so I may have more to say in a subsequent look-over, but it's a start. Thanks again!

EDIT: I realize I haven't addressed any of your inline code comments. Sorry about that. I will do so in a subsequent review. Hopefully there's enough for you to work on until then.

[orbitalquark]

Modern lexers use local lexer = lexer now.

[orbitalquark]

Modern lexers don't need this anymore.

[orbitalquark]

I would suggest using lexer.EMBEDDED tag names so they are styled correctly.

[orbitalquark]

I would use a more descriptive name like 'typst_scripting'.

[orbitalquark]

I feel like we can bring the contents of build_rules() into this function. When trying to read this function, I find myself constantly scrolling up to see how each pattern is defined. Making changes would be a bit difficult.

[orbitalquark]

Is there a reason to keep these out of build_rules()?

[2ta00ha3]

I've tried to keep build_rules generic, so it has the shared rules in both scripting/text mode, as for bold/italic, these must only be applied in text mode, as an example:

#let countwords(s) = {
  if s == "" {
    return 0 * 0 * 5
  }
}

In the above example if we kept the bold/italic in the build_rules, it would tag the 0 in 0 * 0 * 5 as bold, which is not correct, since in scripting mode, -plain- text must be in brackets, and in consequence the italic/bold must be only applied in that case, as an example:

#let somefn(word) = {
   let text = [This is a *plain* text];
}

[orbitalquark]

Okay, I'm completely blown away. You've partially solved the "language embedded in itself" problem. I would really like to use something like

local self = lexer.load('typst', 'typst_scripting')
lex:embed(self, start_rule, end_rule)

You cannot do this now because of an endless loop, but I would totally work on a fix to try and make this viable.

[orbitalquark]

It's a bit of a nitpick for now, but in my brief testing,

#while n < 10 {
  n = (n * 2) - 1
  (n,)
}

the #while was not highlighting as a keyword.

I just thought I'd make a note.

EDIT: I saw you already noted something similar in an earlier comment, so this is just another instance to report.

[orbitalquark]

Sorry to commit directly to your PR. I was trying to figure out how to show a proof-of-concept for how we can embed this lexer within itself: 1a3f6f9. I thought I'd be on my own branch or something, but I guess not. I'm not very well versed with how git and GitHub work together.

Sorry about all the formatting changes. I forgot to turn off my autoformatter before I started making changes. Hopefully you get the idea.

[2ta00ha3]

Thanks so much for the notes, ill try to apply the changes required, as for the embedding mode, i'll look more into it later, and try to find something to fix the unbalanced brackets/curly braces and the recursive tagging in embedding mode

P.S: the docs are very very helpful and straight forward, probably one of the best docs i've came across for a long time since the BSDs' manpages, usually i'd spam some LLM w/ the man page or the docs and ask him what i want (i'm looking at you GNU manpages), but the Scintillua API docs had exactly what i needed