Re: Completely hide properties drawer in 9.6

[Sterling Hooten]

So I've got a solution somewhat working but I'd like some help simplifying it.

The good news is that it's significantly faster than the previous Org
implementation I was using (the one you wrote on stack exchange), and
can now handle folding a file with 30k lines in around .065s for
hiding and .005s for showing. It does seem to be dependent on garbage
collection settings though.

I'm just going to walk through my solution and then ask some specific
questions about how to fix it.

Most of the work is actually in dealing with invisible edits and then
handling =org-return= when called in a heading with a hidden property
drawer.

Starting with a basic minor mode:

#+begin_src emacs-lisp
  (define-minor-mode org-hide-properties-mode
    "Toggle `org-hide-properties-mode'. Hide all property drawers in buffer and 
any further created."
    :init-value nil
    :lighter (" HP")
    (if org-hide-properties-mode
        (progn 
          (org-fold-add-folding-spec 'property-drawer-hidden-edges
                                     '((:ellipsis . nil)
                                       (:isearch-open . t)
                                       (:front-sticky . nil)
                                       (:rear-sticky . nil))
                                     nil
                                     'append)
          (org-fold-add-folding-spec 'property-drawer-hidden-interior
                                     '((:ellipsis . nil)
                                       (:isearch-open . t)
                                       (:front-sticky . t)
                                       (:rear-sticky . t))
                                     nil
                                     'append)
          (advice-add 'org-insert-property-drawer :after 
#'org-fold-hide-property-drawer)
          (org-fold-hide-property-drawer-all))
      (org-fold-show-property-drawer-all)
      (advice-remove 'org-insert-property-drawer 
#'org-fold-hide-property-drawer)))
#+end_src

In order to both have newly added properties automatically adopt the
invisibility text-property the interior characters of the properties
box needs to be sticky. But this conflicted with being able to type
with the point before the hidden text. To satisfy both these
requirements I applied =property-drawer-hidden-interior= as a folding
spec which was sticky to all but the first two and last two characters
of the property drawer. Then I applied =property-drawer-hidden-edges=
to these two remaining chunks. In this way you can add in new
properties and they'll be invisible, but typing at the edges appears
visibly.

Is there an alternative way of doing this without two separate folding
specs? Maybe an additional key for ":interior-sticky" which is sticky
on all characters sharing the same spec, but not sticky at the front
and rear?

I couldn't quite figure out what the :fragile setting was.

#+begin_src emacs-lisp
(defun org-fold--hide-property-drawers (begin end)
        "Hide all property drawers betweeen BEGIN and END."
        (org-with-wide-buffer
          (goto-char begin)
          (while (and (< (point) end)
                      (re-search-forward org-property-drawer-re end t))
    (with-silent-modifications
             ;; Hide interior of drawer to be sticky so org-entry-put will 
respect invisibility
             (org-fold-region (+ (match-beginning 0) 1) (- (match-end 0) 2) t 
'property-drawer-hidden-interior)
             ;; Hide first two and last two characters without sticky
             (org-fold-region (- (match-beginning 0) 1) (+ (match-beginning 0) 
1) t 'property-drawer-hidden-edges)
             (org-fold-region (- (match-end 0) 2) (match-end 0) t 
'property-drawer-hidden-edges)))))

(defun org-fold--show-property-drawers (begin end)
        "Unhide all property drawers in buffer between BEG and END"
        (org-fold-region begin end nil
                         'property-drawer-hidden-edges)
        ;; SWH 2022-12-15 HACK because I'm using two diferent folding specs
        (org-fold-region begin end nil
                         'property-drawer-hidden-interior))

(defun org-fold-hide-property-drawer-all ()  
        "Hide all property drawers in visible part of current buffer."
        (org-fold--hide-property-drawers (point-min) (point-max)))
      
(defun org-fold-show-property-drawer-all ()
        "Unhide all property drawers visible part of current buffer"
        (org-fold--show-property-drawers (point-min) (point-max)))

(defun org-fold-toggle-property-drawer-all ()
        "Show or hide all property drawers in buffer."
        (interactive)
        (require 'org-macs)
        (if org-properties-hidden-p
            (progn 
              (setq org-properties-hidden-p nil)
              (org-fold-show-property-drawer-all))
          (setq org-properties-hidden-p t)
          (org-fold-hide-property-drawer-all)))
#+end_src

Folding headings or subtrees:

#+begin_src emacs-lisp
      (defun org-fold-show-property-drawer (&optional arg)
        "Unhide property drawer in heading at point.
    With non-nil ARG show the entire subtree."
        (org-with-wide-buffer 
         (org-back-to-heading)
         (let* ((beg (point))
                (end (if arg (org-element-property :end (org-element-at-point))
                       (org-next-visible-heading 1)
                       (1+ (point)))))
           (org-fold-region beg end nil
                            'property-drawer-hidden-edges)
           (org-fold-region beg end nil
                            'property-drawer-hidden-interior))))

      (defun org-fold-hide-property-drawer (&optional arg)
        "Completely hide the property drawer in heading at point.
  With non-nil ARG hide the entire subtree."
        (org-with-wide-buffer 
         (org-back-to-heading)
         (let* ((beg (point))
                (end (if arg (org-element-property :end (org-element-at-point))
                       (org-next-visible-heading 1)
                       (1+ (point)))))
           (org-fold--hide-property-drawers beg end))))
#+end_src

Is there a faster or better way of getting the boundary points for a
heading without the subheadings? I've wanted this feature in a few
other situations but don't know a clean way to fetch it. I suppose you
could also just have that the regex stop searching for the beginning
of the property box after 2 lines past the heading, since the
properties box needs to be in the first two lines.

These next sections are more opinionated in how to handle the edge
cases when dealing with hidden text that shouldn't be moved during
normal editing (e.g., if the property drawer has an empty line between
it and the headline, it isn't recognized as such).

My intention was to match default behavior as much as possible without
breaking the parsing of the property drawer or needing to reveal the
invisible text.

The default =org-end-of-meta-data= will go past any blank lines and
only stop at a non-empty line.  I would prefer for hitting <RETURN> on
a headline to act as if the planning line and properties box were
immutable, and just pass directly after them. For this I wrote a
version that will stop after the logbooks or property drawer.

Is there a better way to do this? I tried using the match-data but it
seemed unreliable (if there's nothing but blank lines after the meta
data it matches on the next heading).

#+begin_src emacs-lisp
  (defun swh-org-end-of-meta-data (&optional full)
    "Skip planning line and properties drawer in current entry.

  Differs from `org-end-of-meta-data' in that it stops after the first logbook.

  When optional argument FULL is t, also skip planning information,
  clocking lines and any kind of drawer.

  When FULL is non-nil but not t, skip planning information,
  properties, clocking lines and logbook drawers."
    (org-back-to-heading t)
    (forward-line)
    ;; Skip planning information.
    (when (looking-at-p org-planning-line-re) (forward-line))
    ;; Skip property drawer.
    (when (looking-at org-property-drawer-re)
      (goto-char (match-end 0))
      (forward-line))
    ;; When FULL is not nil, skip more.
    (when (and full (not (org-at-heading-p)))
      (catch 'exit
        (let ((end (save-excursion (outline-next-heading) (point)))
              (re (concat "[ \t]*$" "\\|" org-clock-line-re)))
          (while (not (eobp))
            (cond ;; Skip clock lines.
             ((looking-at-p re) (forward-line))
             ;; Skip logbook drawer.
             ((looking-at-p org-logbook-drawer-re)
              (if (re-search-forward "^[ \t]*:END:[ \t]*$" end t)
                  ;; SWH 2022-12-19 Exit after finding the /first/ logbook
                  (progn (forward-line) (throw 'exit t))
                (throw 'exit t)))
             ;; When FULL is t, skip regular drawer too.
             ((and (eq full t) (looking-at-p org-drawer-regexp))
              (if (re-search-forward "^[ \t]*:END:[ \t]*$" end t)
                  (forward-line)
                (throw 'exit t)))
             (t (throw 'exit t))))))))
#+end_src

This tries to deal with hitting <RETURN> on a headline in a natural way:

#+begin_src emacs-lisp
      (defun org-return-newline-at-heading-skip-drawers (fun &rest args)
        "If at heading do nothing, if on heading skip the planning line,
  property drawer, logbook, and insert a newline after the last
  drawer or right before the first non-empty line."
        ;; FIXME Should also handle the case when on planning line
        (if (not (org-at-heading-p))
            (apply fun args)
          ;; Store point so undo will put it back in heading
          (push (point) buffer-undo-list)
          ;; HACK to get entry to unfold when hidden from org-cycle
          (org-fold-show-entry 'hide-drawers)
          (swh-org-end-of-meta-data 'logbook)
          (if (not (org-at-heading-p))
              (if (eobp)
                  (org-return)
                (beginning-of-line)
                (newline)
                (forward-line -1))
            (beginning-of-line)
            (newline)
            (forward-line -1))))

    (advice-add 'org-return :around 
#'org-return-newline-at-heading-skip-drawers)
#+end_src

I wanted the editing to be natural and the hidden text unobtrusive. I
added a new =org-catch-invisible-edits= flag of ='swh= and overwrote
=org-fold-check-before-invisible-edit= to accomodate this. To allow
for insertion before and after the hidden text I increased the
lookback for invisible characters from one to two. This allows for
having a trailing newline folded after the :END: and thus you can type
without appending to the end of property drawer (and messing up the
syntax). Deletion is still not allowed after the hidden text.

This is ugly but it seems to be working at the moment.

#+begin_src emacs-lisp
(defun org-fold-check-before-invisible-edit (kind)
  "Check if editing KIND is dangerous with invisible text around.
The detailed reaction depends on the user option
`org-fold-catch-invisible-edits'."
  ;; First, try to get out of here as quickly as possible, to reduce overhead
  (when (and org-fold-catch-invisible-edits
             (or (not (boundp 'visible-mode)) (not visible-mode))
             (or (org-invisible-p)
                 (org-invisible-p (max (point-min) (1- (point))))
                 ;; SWH 2022-12-19 Searching 2 characters back to allow for 
newlines in hidden property drawers
                 (org-invisible-p (max (point-min) (- (point) 2)))))
    ;; OK, we need to take a closer look.  Only consider invisibility
    ;; caused by folding of headlines, drawers, and blocks.  Edits
    ;; inside links will be handled by font-lock.
    
    ;; SWH 2022-12-14 Add property-drawer-hidden-edges to allow for completely 
hiding properties
    (let* ((invisible-at-point (org-fold-folded-p (point) '(headline drawer 
block property-drawer-hidden-edges)))
           (invisible-before-point
            (and (not (bobp))
                 (org-fold-folded-p (1- (point)) '(headline drawer block 
property-drawer-hidden-edges))))
           (invisible-before-point-2
            (and (not (bobp))
                 ;; SWH 2022-12-19 counting invisble things 2 characters away 
to allow for newlines in hidden property drawers
                 (org-fold-folded-p (- (point) 2) '(headline drawer block 
property-drawer-hidden-edges))))
           (border-and-ok-direction
            (or
             ;; Check if we are acting predictably before invisible
             ;; text.
             (and invisible-at-point (not invisible-before-point)
                  (memq kind '(insert delete-backward)))
             ;; Check if we are acting predictably after invisible text
             ;; This works not well, and I have turned it off.  It seems
             ;; better to always show and stop after invisible text.
             ;; (and (not invisible-at-point) invisible-before-point
             ;;  (memq kind '(insert delete)))
             )))
      (when (or invisible-at-point invisible-before-point 
invisible-before-point-2)
        (when (and invisible-before-point-2 (memq kind '(delete-backward)))
          (user-error "Can't delete after invisible areas is prohibited, make 
them visible first"))
        ;; (when (and invisible-before-point-2 (not invisible-before-point) 
(memq kind '(insert)))
        ;;   (message "SWH allowing modifying two places after invisible point 
editing"))
        (when (and (eq org-fold-catch-invisible-edits 'swh)
                   invisible-at-point (memq kind '(delete)))
          (user-error "Show hidden properties before editing invisible"))
        (when (or (eq org-fold-catch-invisible-edits 'error)
                  (and invisible-before-point (eq 
org-fold-catch-invisible-edits 'swh)))
          (user-error "Editing in invisible areas is prohibited, make them 
visible first"))
        (if (and org-custom-properties-overlays
                 (y-or-n-p "Display invisible properties in this buffer? "))
            (org-toggle-custom-properties-visibility)
          ;; Make the area visible
          (if (and (eq org-fold-catch-invisible-edits 'swh)
                   border-and-ok-direction)
              (message "Allowing modifying before invisible point editing")
            (save-excursion
              (org-fold-show-set-visibility 'local))
            (when invisible-before-point
              (org-with-point-at (1- (point)) (org-fold-show-set-visibility 
'local)))
            (cond
             ((eq org-fold-catch-invisible-edits 'show)
              ;; That's it, we do the edit after showing
              (message
               "Unfolding invisible region around point before editing")
              (sit-for 1))
             ((and (eq org-fold-catch-invisible-edits 'smart)
                   border-and-ok-direction)
              (message "Unfolding invisible region around point before 
editing"))
             ((and invisible-before-point-2 (not invisible-before-point) (memq 
kind '(insert delete)))
              (message "Allowing modifying two places after invisible point 
editing"))
             (t
              ;; Don't do the edit, make the user repeat it in full visibility
              (user-error "Edit in invisible region aborted, repeat to confirm 
with text visible")))))))))
#+end_src

I am also experiencing a weird (overlay?) problem with i-search. After
implementing when I search for things it will make all of the windows
greyed out, and then only show the matching characters.  I can see the
text again if I =mark-whole-buffer=. But I don't know what's causing
this.

Thanks again for your help, this is trickier than I initially thought
but I think it's coming together.