feat: serialize AST node back to wikitext string #8258

linonetwo · 2024-06-12T17:28:01Z

Currently just a demo, I will gradually move code from https://github.com/tiddly-gittly/wikiast/tree/master/packages/wikiast-util-to-wikitext to here.

Try it with tiddler (Well, if official website have share plugin installed, I can put a link here):

AAA test

---

\```js
var match = reEnd.exec(this.parser.source)
\```

end

and run this in console:

$tw.utils.serializeParseTree($tw.wiki.parseTiddler('New Tiddler').tree)

vercel · 2024-06-12T17:28:06Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Jun 12, 2024 5:39pm

… text" This reverts commit 8a12498.

pmario · 2024-06-13T08:24:26Z

The problem here is, that the output of you test does not produce the same rendered HTML output as the original source. See the screenshot.

linonetwo · 2024-06-13T09:27:44Z

First line is wrapped by a p element, but I haven't find the rule that produce the p element. And there was no p html in the source, so I may need to add some attribute to ast to show this p element is "fake".

Anyway, this is just a demo for API, do you like the API name and their usage? The further task is just adding exports.serailize = xxx to each rule.

pmario · 2024-06-13T09:33:59Z

First line is wrapped by a p element, but I haven't find the rule that produce the p element. And there was no p html in the source, so I may need to add some attribute to ast to show this p element is "fake".

IMO the current parser has a "systemic" problem with redundant P-tags, that we probably may need to solve first.

Working around this problem with "fake" elements will just increase your code complexity, instead of solving the underlaying problem. It will also create an other dependency, that will make solving the problem, in the core, in backwards compatible way even harder as it is now.

pmario · 2024-06-13T09:40:11Z

core/modules/wiki.js

+};
+
+exports.getParser = function(type,options) {
+	options = $tw.utils.extend({},options);


IMO your code will throw an RSOD if options is undefined
So options = options || {} should be used instead. We use this pattern all over the places.

I copy $tw.utils.extend({},options) from somewhere nearby, I thought this will work.

linonetwo · 2024-06-13T14:04:01Z

redundant P-tags

Can't remove then, many of my plugin style sheet is already depending on this structure. And p > text for block is following HTML5 semantic.

Another option is handle it like text node, regard it as a special case.

pmario · 2024-06-13T14:05:53Z

And p > text for block is following HTML5 semantic.

That's true, but we do have a lot of p > div which should not happen at all.

linonetwo · 2024-07-29T16:47:05Z

Handle the block rule by adding a rule: 'parseBlock' to WikiParser.prototype.parseBlock.

I've added some test for it.

…n of it

linonetwo · 2024-12-10T08:54:46Z

@rougsig Use edit button with a "T" in the view toolbar to enter WYSIWYG mode.

@BurningTreeC Those field is added in other PRs, this PR only generate type for them. start end is already in the core.

linonetwo · 2025-02-11T11:42:36Z

how the conditional shortcut syntax is handled

That's why I need this in the core, instead of in a plugin.

saqimtiaz · 2025-02-11T12:01:33Z

It would also be worth running every wikitext template in the core as test data too

@linonetwo from my perspective this is the missing next step, even if only to understand what if any shortcomings or peculiarities there are in the round trip from wikitext to parse tree and back.

linonetwo · 2025-02-11T12:34:33Z

@saqimtiaz "every wikitext template in the core" contains many "WikiScript" that this API will never meet, but in this PR I only trying to support WikiText cases that WYSIWYG editor will use. Current test cases already cover most of them.

Well, covering all coner cases may enable some creazy use cases like a visual "WikiScript" editor, so I can add the test, but I'm afraid this kind of test will make test very slow.

saqimtiaz · 2025-02-11T13:26:44Z

@linonetwo the goal is to better understand what use cases are and are not supported. Running a test using $:/core/ui/PageTemplate would be very informative.

netlify · 2025-02-19T07:39:08Z

✅ Deploy Preview for tiddlywiki-previews ready!

Name	Link
🔨 Latest commit	`a97aede`
🔍 Latest deploy log	https://app.netlify.com/sites/tiddlywiki-previews/deploys/67cdb55e5cf46c00087dc71a
😎 Deploy Preview	https://deploy-preview-8258--tiddlywiki-previews.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

linonetwo · 2025-02-19T07:39:52Z

@saqimtiaz I render it in the view template below text, looks the same, is this what you mean? I will revert this commit before merge. Or you suggest add this test to CI?

linonetwo · 2025-02-19T11:35:46Z

It takes 5min to run, when limit to non-system tiddlers, on my i9 MacBookPro. But it will cause FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory on netlify deploy...

And if runs for both system tiddlers, it seems takes forever.

And it OOM on my computer. So I update it to only run it on browser. Take a look.

DEBUG: render result below body only on browser DEBUG: render result below body DEBUG: fix build DEBUG: show render result as ViewTemplate

saqimtiaz · 2025-02-20T08:55:43Z

@linonetwo I suggest we add a test along the lines of those in https://github.com/TiddlyWiki/TiddlyWiki5/blob/master/editions/test/tiddlers/tests/test-wikitext-parser.js

So for example:

render the page template
and then serialize the HTML and compare it to the original wikitext

linonetwo · 2025-02-20T12:52:43Z

@saqimtiaz This will result in many failed tests. In current preview webpage, you can find it will add or delete some space or \n. And it is very slow, if go through every tiddlers.

I suggest we add a test along the lines of those in editions/test/tiddlers/tests/test-wikitext-serialize.js, do you mean change the file name?

There is already tests in https://github.com/TiddlyWiki/TiddlyWiki5/pull/8258/files#diff-ac833214819c79e6c8b85db9a4bc7cac9e6fcae0bc361f0373f42b514da91e8f

render the page template

What do you mean? This feature only works on wikitext on text field, it doesn't care about rendering. It is ast to wikitext, not wikitext to HTML.

linonetwo · 2025-02-28T18:15:10Z

WYSIWYG editor has been waiting this for too long!

When you are ready, I can revert commit DEBUG: render result and diff below body only on browser. So it can be merged. Before that you can see debug information on preview site.

linonetwo · 2025-03-09T07:10:49Z

Could this be merged before next release? WYSIWYG haven't update for nearly 2 years. Now I revert the debug message, so it is ready to merge.

pmario · 2025-03-09T13:58:42Z

Why did you also remove the filter: core/modules/filters/serializeparsetree.js -- You should only remove the ViewTemplate. IMO the filter needs to stay. IMO It is needed for debugging.

linonetwo · 2025-03-09T14:27:56Z

@pmario I don't like seeing core filters inflate more on my hand. It is only for debugging, no use cases. And this API is mainly for WYSIWYG editor, no many people other than me will use it, and I only use JS api, I will not using filter on this case. Other editor developers will probably only use JS API.

Of course I can add it back if you want to use it on your plugin.

pmario · 2025-03-09T14:01:55Z

core/modules/parsers/wikiparser/rules/commentblock.js

+	// Return a node representing the comment that is not rendered
+	var commentStart = this.match.index;
+	var commentEnd = this.endMatch.index + this.endMatch[0].length;
+	var commentText = this.parser.source.slice(commentStart, commentEnd);


IMO those variables are not needed. The code can directly be written into the return object. It is possible to debug the return object.

It will be text: this.parser.source.slice(this.match.index, this.endMatch.index + this.endMatch[0].length), which is redundent and too long. It reuses commentStart

I remove the commentText

pmario · 2025-03-09T14:02:28Z

core/modules/parsers/wikiparser/rules/commentinline.js

+	// Return a node representing the inline comment
+	var commentStart = this.match.index;
+	var commentEnd = this.endMatch.index + this.endMatch[0].length;
+	var commentText = this.parser.source.slice(commentStart, commentEnd);


Same as above. Code should be within the return object. No need for extra variables

pmario · 2025-03-09T14:08:09Z

core/modules/parsers/wikiparser/rules/list.js

+					var currentMarker = findMarker(node.tag, child.tag);
+					// Handle class attributes
+					var classAttr = child.attributes && child.attributes.class ? "." + child.attributes.class.value : "";
+					// same level text nodes may be split into multiple children, and separated by deeper list sub-tree. We collect same level text nodes into this list, and concat then submit them before enter deeper list.


Such a long comment should be split into 2 or more comment lines for easier readability.

pmario · 2025-03-09T14:18:15Z

editions/test/tiddlers/tests/data/serialize/FunctionDefinition.tid

+title: Serialize/FunctionDefinition
+type: text/vnd.tiddlywiki
+
+\function name(param:defaultvalue,param2:defaultvalue)


For the core we do have double quotes for parameter values.
IMO it should look like this:

\function name(param:"defaultvalue", param2:"defaultvalue")

It is from https://github.com/linonetwo/TiddlyWiki5/blob/8ab12738cf446f169609fd1d6c23d50ec10c2310/core/modules/parsers/wikiparser/rules/fnprocdef.js#L8-L20

So I will change the comment there too.

pmario · 2025-03-09T15:05:37Z

Of course I can add it back if you want to use it on your plugin.

The main problem is, that we have no easy to use possibility to test the stuff, if the filter is gone.

I did see your tests. They only contain plain text tiddlers and the test is: editions/test/tiddlers/tests/test-wikitext-serialize.js

That's OK. But it only can be tested using JS API. What we need is a possibility to use the new testcase-widgets to design new tests and also debug the existing tests.

IMO to be able to use the testcase-widget, we do need the filter. So we can do tests like the one you removed.

IMO There should be at least 1 test, that uses core/ui/ViewTemplate/body/default.tid in a testcase test.
For the structure see: https://github.com/TiddlyWiki/TiddlyWiki5/tree/master/editions/test/tiddlers/tests/data/conditionals

linonetwo · 2025-03-09T16:13:27Z

@pmario I spend some time tried this, and this is not possible. <$text can't render wikitext as-is. And <<text>> will render the wikitext.

And don't know why but test will wrap the Output with <p> which will break the test.

uses core/ui/ViewTemplate/body/default.tid in a testcase test.

Don't know what you mean, this PR is about turn AST back to wikitext with syntax. Not rendering wikitext or AST into plain text or HTML. So don't need to involve ViewTemplate which is about rendering.

pmario · 2025-03-09T19:43:31Z

And don't know why but test will wrap the Output with
which will break the test.

That's a known problem with the current renderer, which needs to be fixed in the future. That's the reason, why all the other tests also have this P tag, in the Expected tiddler.

Don't know what you mean, this PR is about turn AST back to wikitext with syntax. Not rendering wikitext or AST into plain text or HTML. So don't need to involve ViewTemplate which is about rendering.

We do not necessarily use the ViewTemplate. I only want to have a possibility for easy testing without using JavaScript.

linonetwo added 3 commits June 13, 2024 01:23

refactor: extract a new $tw.wiki.getParser

37338b1

feat: allow $tw.utils.getParseTreeText to render other rules' text

8a12498

feat: two example getText handler

09c2e71

linonetwo added 2 commits June 13, 2024 01:29

Revert "feat: allow $tw.utils.getParseTreeText to render other rules'…

a0708a1

… text" This reverts commit 8a12498.

refactor: keep original getParseTreeText not touched

f882cbd

vercel bot deployed to Preview June 12, 2024 17:31 View deployment

refactor: use serialize in rules

be3f037

vercel bot deployed to Preview June 12, 2024 17:39 View deployment

linonetwo changed the title ~~Feat/to string~~ feat: serialize AST node back to wikitext string Jun 12, 2024

pmario reviewed Jun 13, 2024

View reviewed changes

linonetwo added 5 commits July 29, 2024 21:43

Merge remote-tracking branch 'upstream/master' into feat/to-string

4ddcbc6

refactor: $tw.utils.extend({},options) -> options || {}

4e896d1

Update codeinline.js

179651a

Create test-wikitext-serialize.js

8cbf855

DEBUG: only run my tests for development, remove before PR merge

a075303

linonetwo mentioned this pull request Jul 29, 2024

Add start and end properties to WikiText AST nodes for all elements. #7866

Closed

linonetwo added 3 commits July 30, 2024 00:43

lint: if

05dec72

feat: add rule: 'parseBlock' metadata

1a6a071

feat: handle tailing \n that may be missing

84e27c0

linonetwo added 4 commits July 30, 2024 00:56

feat: allow recursive

078967c

feat: generate more rule and tests

2aff9fe

feat: generate more rule and tests

bfd116a

fix: remove pragma:true, otherwise following text will become childre…

d067871

…n of it

linonetwo force-pushed the feat/to-string branch 2 times, most recently from 880c0a5 to dee3132 Compare February 19, 2025 10:05

fix: link may not have attribute and children

2765e99

linonetwo force-pushed the feat/to-string branch from 34e3e6b to 45d36b0 Compare February 19, 2025 11:55

DEBUG: render result and diff below body only on browser

069181d

DEBUG: render result below body only on browser DEBUG: render result below body DEBUG: fix build DEBUG: show render result as ViewTemplate

linonetwo force-pushed the feat/to-string branch from 45d36b0 to 069181d Compare February 20, 2025 08:25

linonetwo added 2 commits February 28, 2025 23:11

fix: remove pad space in />

86ec4ca

test: remove pad space in />

74c9841

Revert DEBUG: render result and diff below body only on browser

8ab1273

pmario reviewed Mar 9, 2025

View reviewed changes

linonetwo added 3 commits March 9, 2025 23:31

refactor: fold commentText variable

711bf01

refactor: fold long comment

5c0198b

fix: double quotes for parameter values

a97aede

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: serialize AST node back to wikitext string #8258

feat: serialize AST node back to wikitext string #8258

linonetwo commented Jun 12, 2024 •

edited

Loading

vercel bot commented Jun 12, 2024 •

edited

Loading

pmario commented Jun 13, 2024

linonetwo commented Jun 13, 2024 •

edited

Loading

pmario commented Jun 13, 2024 •

edited

Loading

pmario Jun 13, 2024

linonetwo Jun 13, 2024

linonetwo commented Jun 13, 2024

pmario commented Jun 13, 2024

linonetwo commented Jul 29, 2024 •

edited

Loading

linonetwo commented Dec 10, 2024

linonetwo commented Feb 11, 2025

saqimtiaz commented Feb 11, 2025

linonetwo commented Feb 11, 2025

saqimtiaz commented Feb 11, 2025

netlify bot commented Feb 19, 2025 •

edited

Loading

linonetwo commented Feb 19, 2025 •

edited

Loading

linonetwo commented Feb 19, 2025 •

edited

Loading

saqimtiaz commented Feb 20, 2025

linonetwo commented Feb 20, 2025 •

edited

Loading

linonetwo commented Feb 28, 2025

linonetwo commented Mar 9, 2025 •

edited

Loading

pmario commented Mar 9, 2025

linonetwo commented Mar 9, 2025 •

edited

Loading

pmario Mar 9, 2025

linonetwo Mar 9, 2025

linonetwo Mar 9, 2025

pmario Mar 9, 2025

pmario Mar 9, 2025

pmario Mar 9, 2025

linonetwo Mar 9, 2025

pmario commented Mar 9, 2025

linonetwo commented Mar 9, 2025 •

edited

Loading

pmario commented Mar 9, 2025 •

edited

Loading

feat: serialize AST node back to wikitext string #8258

Are you sure you want to change the base?

feat: serialize AST node back to wikitext string #8258

Conversation

linonetwo commented Jun 12, 2024 • edited Loading

vercel bot commented Jun 12, 2024 • edited Loading

pmario commented Jun 13, 2024

linonetwo commented Jun 13, 2024 • edited Loading

pmario commented Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linonetwo commented Jun 13, 2024

pmario commented Jun 13, 2024

linonetwo commented Jul 29, 2024 • edited Loading

linonetwo commented Dec 10, 2024

linonetwo commented Feb 11, 2025

saqimtiaz commented Feb 11, 2025

linonetwo commented Feb 11, 2025

saqimtiaz commented Feb 11, 2025

netlify bot commented Feb 19, 2025 • edited Loading

✅ Deploy Preview for tiddlywiki-previews ready!

linonetwo commented Feb 19, 2025 • edited Loading

linonetwo commented Feb 19, 2025 • edited Loading

saqimtiaz commented Feb 20, 2025

linonetwo commented Feb 20, 2025 • edited Loading

linonetwo commented Feb 28, 2025

linonetwo commented Mar 9, 2025 • edited Loading

pmario commented Mar 9, 2025

linonetwo commented Mar 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmario commented Mar 9, 2025

linonetwo commented Mar 9, 2025 • edited Loading

pmario commented Mar 9, 2025 • edited Loading

linonetwo commented Jun 12, 2024 •

edited

Loading

vercel bot commented Jun 12, 2024 •

edited

Loading

linonetwo commented Jun 13, 2024 •

edited

Loading

pmario commented Jun 13, 2024 •

edited

Loading

linonetwo commented Jul 29, 2024 •

edited

Loading

netlify bot commented Feb 19, 2025 •

edited

Loading

linonetwo commented Feb 19, 2025 •

edited

Loading

linonetwo commented Feb 19, 2025 •

edited

Loading

linonetwo commented Feb 20, 2025 •

edited

Loading

linonetwo commented Mar 9, 2025 •

edited

Loading

linonetwo commented Mar 9, 2025 •

edited

Loading

linonetwo commented Mar 9, 2025 •

edited

Loading

pmario commented Mar 9, 2025 •

edited

Loading