Lexicon type for markdown notes

[tesseract1776]

From this discussion

We should create a lexicon type that supports markdown style notes, which could be used for things such as syncing your (public) obsidian/logseq notes on your pds, just having a place where people can view your research notes, personal knowledge management, or community wiki's, which people can later build custom front ends for.

Ideally we'd come up with a flexible type that works across these use cases, so that for instance if someone makes a plugin to synchronize notes with a pds, that can ideally be used wether you are just updating your pkm or using it to edit a community wiki.

It could be something as simple as

{
  "$type": "community.lexicon.mdg",
  "text": "# Title 
                blah blah **markdown**",
  "createdAt": "2023-08-07T05:44:04.395087Z",
  "langs": [ "th", "en-US" ]
}

but i'd love for some input on what to add.

One thing suggested in the discussion was a way to store backlinks, which would work well in the case the user manages the pages, as when you link to another page you own, you can just edit that post in your own repo to add the current record to an extra field that stores back links.
This wouldn't be as easy for the community wiki type use case, as you wouldn't be able to edit someone else's records (and probably shoudln't anyways), but it could just be dealt with the exact same way "likes you receive" or "followers" are now, where you dont actually store those on your pds, but it compiles that for you.

I imagine it shouldn't be too complex as a lot of data is just stored in the YAML frontmatter, and it doesnt require "facets" or "features" like bluesky posts, because it is in markdown already, so the front end can just parse most of the required data from that.

Anyways, drop your suggestions, changes, thoughts, ideas.

[tesseract1776]

i put "mdg" for "mark down graph", but that lexicon name can be changed too, i just wanted to choose something ultra generic that captures both the markdown aspect and the wiki interconnected aspect, but feel free to suggest changes on that too. I was considering "notes" but perhaps that might be used for some sort of community notes thing like twitter has

[bmann]

Thanks for kicking this off!

Naming is hard!

I think graph is an important part of it — backlinks and forward links between notes, transclusion, etc

my ideal goal for this would be:

• I can publish my public notes
• I can use at:// URIs to reference other notes and other Lexicon content
• backlinks

I end up adding quite a bit of structured data to my notes with custom “front matter”

so a “link” type would be interesting….but that might just be a Frontpage item or Bookmark? Structured data should probably just be other Lexicons?

Or perhaps a single at://URI, and the “note body” is about the linked element.

But if a lot of people use links then it should go in the Lexicon for simplicity.

I’ll come back to this after some more thought.

[ebwinters]

Why limit it to markdown? You can add a language attribute that can be markdown, text, etc, because people take notes in different formats.

I think it would be useful for performance purposes to store the text in a separate collection and reference by ID from community.lexicon.mdg. Otherwise you're going to be querying big text every time you want to display metadata about markdown

I am also curious about backlinks. I don't quite understand how backlinks would work, and it seems like a fairly complicated issue to solve.

[essential-randomness]

One of the issues with markdown is that markdown is way less defined as a standard than one would think. There's a few articles out there about the different flavors of markdown (see this comparison table for some example), but generally it's not sufficient to say "markdown" to know what should be supported.

Also, fun fact I only found out recently: HTML is valid markdown in many applications, including GitHub markdown (example: picture element, but so are link and divs etc.). I actually wanted to open a top-level discussion over interchange format for text, which I think is worth having separate from this.

I found this RFC from the IETF about a text/markdown media type, but I don't have time to look deep into what it's trying to establish or what the story behind it is right this moment.

[tesseract1776]

One of the issues with markdown is that markdown is way less defined as a standard than one would think. There's a few articles out there about the different flavors of markdown (see this comparison table for some example), but generally it's not sufficient to say "markdown" to know what should be supported.

Also, fun fact I only found out recently: HTML is valid markdown in many applications, including GitHub markdown (example: picture element, but so are link and divs etc.). I actually wanted to open a top-level discussion over interchange format for text, which I think is worth having separate from this.

I found this RFC from the IETF about a text/markdown media type, but I don't have time to look deep into what it's trying to establish or what the story behind it is right this moment.

Yea i want to hear more input from the community on which flavour of markdown to choose, or preferably allow some sort of flexibility that doesnt cause fracturing. e.g. if a certain flavour of MD doesnt have tables, since the table syntax likely wouldnt be used for something else, i'd imagine we could still want it to support tables. Does anyone have any suggestions on which flavours should be supported, or potentially if there is some way to indicate that esp to make this compatible with tools like obsidian and such? Maybe even field that tells you which flavour, and additional features your doc expects to support, although that might make it too complicated

[aeshna-cyanea]

perhaps we could use traits https://github.com/orgs/lexicon-community/discussions/11 to specify the markdown flavor/supported syntax extensions?

[essential-randomness]

Yea i want to hear more input from the community on which flavour of markdown to choose, or preferably allow some sort of flexibility that doesnt cause fracturing. e.g. if a certain flavour of MD doesnt have tables, since the table syntax likely wouldnt be used for something else, i'd imagine we could still want it to support tables. Does anyone have any suggestions on which flavours should be supported, or potentially if there is some way to indicate that esp to make this compatible with tools like obsidian and such? Maybe even field that tells you which flavour, and additional features your doc expects to support, although that might make it too complicated

We cannot choose a single markdown flavor for a community lexicon on markdown notes unless we assume that everything that people publish here will be atproto-first. If we assume people will want to publish their notes made on an already-existing software to the ATmosphere, then there needs to be a way for the publishing tool to give hints to the downstream renderers about what the notes mean.

Example: I use ~~strikethrough~~ (strikethrough) a lot in my markdown, which is a non-standard feature. If I published my notes and there was no hint, a renderer in some other ATmosphere service might display it as ~~strikethrough~~, which completely changes what my notes convey (something I find unacceptable for a widespread community lexicon to do). Conversely, someone whose software does not support it and writes ~~non strikethrough~~ would see it crossed off, which is also unacceptable. Not offering strikethrough at all is also unacceptable, as it severely limits what's a not universal but still fairly common mean of expression.

To do a markdown note type that the community can really get behind we have a few options:

• Have a way to say "here's the flavor that this markdown is in and what you should be using for rendering"
• Alternatively, rather than a flavor of markdown, you could specify which features are supported like, for example, an array of non-base markdown features like "tables, strikethrough" etc.
• Support a set of limited markdown, and request that other formatting be done via HTML tags (e.g. <s>strikethrough</s>). This way whatever transforms what's created in the origin software to our lexicon can do the conversion without ambiguity, even if it means what gets published won't be the same original file (which can instead be saved as a blob). Note that you can still restrict the number of HTML tags accepted

[essential-randomness]

If we don't want to support everything immediately because this is a hard problem to tackle we should:

1. Still define exactly what markdown flavor we support, so consumers in both the "converting existing markdown to lexicon format" and "rendering the lexicon contents" directions can make concrete choices as to what to do
2. Get a versioning history straightened up, so renderers and uploaders can make different choices as the expressivity of our markdown expands
3. Not consider the discussion closed, and seriously think about what the lexicon.community data-interchange format should be for user-generated content

[ngerakines]

Disclaimer: I don't feel strongly about this.

When I see markdown or html in records, my immediate thought is "that belongs in a blob". If something would be the "front matter" of a markdown file, then it probably belongs in the record itself. Otherwise, I'd lean towards putting anything that would be the "content" of a markdown record into an attached blob.

[bmann]

@ngerakines but also what about blob editing?

I want versioned text for notes for sure. And taking advantage of search.

And feed parsing and all the other stuff that is built for text / faceted rich text.

Or do you store in blobs and derive a record format? I'm still thinking about it, seems surprising to me.

[ngerakines]

Yeah, those are great points. Much like with records, when you update a blob, you're replacing the whole thing.

[verdverm]

I'm not sure what the general approach is, but I would put my vote behind having a dedicated markdown lexicon so we can avoid all the extra complexity in trying to support multiple encodings. From other comments here, Markdown has an amount of complexity or edge cases on its own

An extra level of scope could be helpful for keeping them organized community.lexicon.longform.{text,markdown,richtext,...}

[tesseract1776]

@ngerakines but also what about blob editing?

I want versioned text for notes for sure. And taking advantage of search.

And feed parsing and all the other stuff that is built for text / faceted rich text.

Or do you store in blobs and derive a record format? I'm still thinking about it, seems surprising to me.

@bmann i opened up another discussion on git/diff/edit records, that could potentially be used for editing blobs as well, and that used in conjuction with other record types to have versioning, rather than designing versioning seperately into every record type. Even if there are unique aspects to versioning, i imagine we can make several versioning record lexicon types, and we can take advantage of the combinations

[essential-randomness]

I'm not sure what the general approach is, but I would put my vote behind having a dedicated markdown lexicon so we can avoid all the extra complexity in trying to support multiple encodings. From other comments here, Markdown has an amount of complexity or edge cases on its own

An extra level of scope could be helpful for keeping them organized community.lexicon.longform.{text,markdown,richtext,...}

Strongly agree with this

[essential-randomness]

An important question here would be how to handle the markdown frontmatter (sample guide). This should probably be separated from the text of the note, as it is metadata and you wouldn't want every consumer to extract it on its own.

We're aware of at least 3 types of supported definition languages for markdown frontmatter in popular markdown-based CMSes: yaml, json, and toml.

The big questions is:

1. Would the frontmatter be a simple text field with maybe a "language" to help parse correctly? OR
2. Would the frontmatter be a blob + language? OR
3. Would the frontmatter be parsed and translated in "key/value" pairs before upload, with maybe an attached blob in case the original is needed?

Generally, if we did 3 then we'd be able to look at the notes and figure out whether they match some traits. For example, we could use that to look up notes featuring "tags" or "title" or "publicationDate" and assume they might be blogposts. However, #15 discusses issues with translating arbitrary JSON to records. I couldn't figure out exactly what the issue was from the discussion, but this would need to be considered.

[verdverm]

The various supported frontmatter formats definitely adds a complication. I'm not sure which / how different renderers are resilient to multiple frontmatter formats

Another thing to consider here is that Markdown renders typically expect the frontmatter to be supplied as part of the entire document. Do we really want to separate it, only to have to reassemble it later? Perhaps we duplicate it as structured content in the record (more work on the createRecord side of things)

[essential-randomness]

The various supported frontmatter formats definitely adds a complication. I'm not sure which / how different renderers are resilient to multiple frontmatter formats

Astro has just added TOML support in v5.2, so we have both "a popular destination of markdown content didn't support the most niche frontmatter format until now" and "a popular source of markdown content now supports the most niche frontmatter format". That is to say, renderers aren't very resilient to this, and it's more important than ever that the handling of unusual formats be considered.

Another thing to consider here is that Markdown renders typically expect the frontmatter to be supplied as part of the entire document. Do we really want to separate it, only to have to reassemble it later? Perhaps we duplicate it as structured content in the record (more work on the createRecord side of things)

There's 2 aspects to this:

1. Without markdown fields as structured content it becomes a lot harder for firehose consumers to look inside it to make choices on what to handle. For example, if someone wanted to just handle markdown notes with a specific tag, you'd have to parse the whole frontmatter of every entry rather than check if there's a frontmatter.tags entry in the record and check for a tag. This IMO is a lot of computational power compared to the usefulness of being able to make quick choices based on frontmatter content.
2. Whatever we do, we should include the original content as is, most likely in a blob. IMO consumers should always be able to look at what the note looked like initially. This could be separate "original frontmatter" + "original body" entries (ideally with an hint to the frontmatter format), or just a single "original entry" field, since it's nevertheless easy to split the two.

[essential-randomness]

Also re: more work on the createRecord side of things, I generally think this is ok. People/developers will be using community lexicon defs to enable QOL features for the whole ecosystem, which in theory would benefit both them and end users. IMO this is well-worth a bit of extra effort on their side.

Most importantly, whatever the choice we make we really need to consider providing utilities that developers can use to correctly create these records, which would lower the amount of work they need to do, but also ensure that it's done correctly and (more importantly) consistently.