Drawer folding problem in Emacs 31.0.50 and/or Org-mode 9.7.12

[natrys]

Problem

The :REVIEW_DATA: drawer gets unfolded due to the way table is written to it. And the way it happens renders the (org-fc-hide-drawers) call made afterwards preparing the card for review useless.

To demonstrate, if we have two (sibling) cards and we reviewed the first, when we move on to the second then (part of) the unfolded :REVIEW_DATA: from first card bleeds into view, which shouldn't happen.

Diagnosis

As it doesn't happen in Emacs 29.4 (with org-mode 9.6.x), at first I thought it could be a newly introduced org-mode bug, so took this to the org-mode mailing list. However the explanation provided convinced me that new behaviour is actually the sensible thing to do for org-mode, at least the part where it decides to unfold the drawer:

https://lists.gnu.org/archive/html/emacs-orgmode/2024-10/msg00145.html

Basically the way we presently write to the table temporarily leaves the drawer structure broken. That's enough to trigger org-mode to forcibly unfold the drawer, to prevent the situation where user wouldn't be able to so in a broken drawer.

That alone wouldn't be a problem. But unfortunately due to implementation complexity, org-mode can't reliably do the unfolding in the after-change-functions. It has to wait for all of our executing command to first finish, and then do the unfolding in the post-command-hook. So our attempt to fold and hide all drawers gets overridden by org-mode's forced unfolding which happens last.

Solution

We can do two things:

Don't break the drawer structure at any point

Unfortunately (insert a b) isn't atomic. It writes a and then b and after-change-function is triggered twice. Which means:

(insert "| " ......)

Will cause org-mode to run and consider the drawer at a time when its structure is broken:

:REVIEW_DATA:
....
| :END:

It will therefore queue a rescuing unfold operation (except that will only run after we already fixed the structure and folded the drawer).

We can avoid breaking the drawer at any point altogether. For example by perhaps using (format) to prepare whole row line instead of using multiple arguments to (insert). Or we can always keep a newline at the drawer end, which will be cleaned up once we are done with writing rows.

By wrapping our drawer editing with org-fold-core-ignore-fragility-checks macro

It signals that we are doing something dangerous temporarily but we pledge to restore integrity at the end so org-mode shouldn't be alarmed and take corrective measures. While this seems to be part of public API, I am not sure how far back it exists (or will exist in future) tbh.

---

I haven't had the chance to see if the bug happens in Emacs 29 and Org 9.7, though I presume it would. Might be worthwhile to replicate though. If someone not me can confirm it, I could create a PR if that's more convenient for you (given your preference for a solution).

[l3kn]

Thanks for the detailed bug report and your proposed fixes!

I prefer the format / insert solution because it relies just on some common and stable functions of Emacs.
One downside is that I we might need to rewrite some of the code (e.g. the dolist for each row) to work nicely with format or a multi-argument insert.

A third alternative could be the following, which feels a bit hacky but requires just a small change to the existing code:

(insert (with-temp-buffer
  (insert "foo")
  (buffer-string)))

I'm currently running org version 9.6.8 so I'll try to update, replicate your issue and check if the temp-buffer solution would also work.