paired md:myst format also works as MyST Markdown

[itcarroll]

Provides #1314.

Modifies the src/jupytext/myst.nb conversion functions to combine (for notebook_to_myst) and separate (for myst_to_notebook) the notebook metadata and MyST frontmatter. Notebook metadata includes fields like jupytext and language_info, while MyST frontmatter has all these fields. Note that jupyter is a "project only" field for MyST, so there should be no namespace conflict.

This PR is draft:

• no attempt has yet been made to understand and resolve the test failures it causes
• no attempt has been made to add configuration, such as the option to pop kernelspec metadata out of the introduced jupyter namespace in the markdown YAML block

[github-actions]

Thank you for making this pull request.

Did you know? You can try it on Binder: or .

Also, the version of Jupytext developed in this PR can be installed with pip:

HATCH_BUILD_HOOKS_ENABLE=true pip install git+https://github.com/itcarroll/jupytext.git@matching-myst-markdown

(this requires nodejs, see more at Developing Jupytext)

[codecov]

Codecov Report

Attention: Patch coverage is 5.26316% with 18 lines in your changes missing coverage. Please review.

Project coverage is 91.31%. Comparing base (8409313) to head (c035f59).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
src/jupytext/myst.py	5.26%	18 Missing ⚠️

❌ Your patch check has failed because the patch coverage (5.26%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (91.15%) is below the target coverage (97.00%). You can increase the head coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (8409313) and HEAD (c035f59). Click for more details.

HEAD has 11 uploads less than BASE

Flag	BASE (8409313)	HEAD (c035f59)
	12	1

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1315      +/-   ##
==========================================
- Coverage   97.04%   91.31%   -5.73%     
==========================================
  Files          29       29              
  Lines        4499     4513      +14     
==========================================
- Hits         4366     4121     -245     
- Misses        133      392     +259     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chrisjsewell]

To note, this will break all existing myst formatted notebooks, if you want to do this then there should be a flag for specifying you want the curvenote implementation of myst, as opposed to the current sphinx one

[itcarroll]

Thanks for taking a look and responding, @chrisjsewell. I don't see it yet however, in case you are able to elaborate.

My understanding of "md:myst" output is that it's not meant for use directly with the myst cli. It's the paired ipynb format that myst can use. The two yaml blocks in the "md:myst" format is not standard. If I'm wrong here, I'd agree with abandoning this PR.

To note, this will break all existing myst formatted notebooks

Do you mean fundamentally, or as drafted? As drafted, I agree. There's no mechanism yet to recognize that a .md file with two yaml blocks needs to be updated to a single yaml block format. (Or by notebooks above do you mean the paired .ipynb file? That would not make sense to me, because those are not meant to be modified.)

should be a flag for specifying you want the curvenote implementation of myst, as opposed to the current sphinx one

If both implementations follow the markdown spec described at https://mystmd.org/, then I don't understand this need. What would be the difference between the "md:myst" and a "md:curvenote" formats written by jupytext?

[mwouts]

Hi, I've not add much time to look into this but I agree with @itcarroll that having two consecutive yaml headers in the same document is not great. I also agree with @chrisjsewell that we don't want to break existing notebooks nor cause massive diffs when users transition to a new version of Jupytext.

@itcarroll I assume that the "curvenote" format directly has the MyST fields in the yaml front header?

@chrisjsewell do you think we could simply concatenate the two yaml headers, and use the explicit list of MyST fields to map the MyST fields back to the first Markdown cell when converting back to a Jupyter notebook? From what I see the kernelspec field should have the same meaning either in the Jupyter notebook metadata (where I would keep it in Jupyter) and in the MyST document?