[BUG] org-goto slows down org-set-property

[Maxim Nikulin]

It is rather annoying that C-c C-x p `org-set-property' may be quite 
slow if a file is large enough. (I do not consider it huge however.) 
There is noticeable delay before the prompt for property name appears in 
minibuffer. Accidentally I have noticed that sometimes it is fast.

Problem appears after first jump using C-u C-c C-j `org-goto'. Actually 
a jump is not required, it is enough to get jump (refile) targets.

I hope, the following numbers are convincing enough. Second attempt to 
get list of properties is much slower than the first one.

#+begin_src elisp
  (require 'org-refile)
  (let ((org-refile-targets '((nil :maxlevel . 6)))
        (org-refile-use-outline-path 'file)
        (org-refile-use-cache t)
        (iters 10))
    (cons
     (list (org-version))
     (mapcar (lambda (f)
               (append (list f) (benchmark-run iters (funcall f))))
             (list #'org-buffer-property-keys
                   #'org-refile-get-targets
                   #'org-buffer-property-keys))))
#+end_src

git master:

#+RESULTS:
| 9.4.6                    |          |    |                     |
| org-buffer-property-keys | 0.065695 |  0 |                 0.0 |
| org-refile-get-targets   | 1.836473 | 12 | 0.39498671199999985 |
| org-buffer-property-keys | 0.563227 |  0 |                 0.0 |

elpa-org system package for Ubuntu-20.04 is even more slower:

#+RESULTS:
| 9.3.1                    |             |    |             |
| org-buffer-property-keys | 0.242574778 |  0 |         0.0 |
| org-refile-get-targets   | 2.669112479 | 13 | 0.285175928 |
| org-buffer-property-keys | 1.231223427 |  0 |         0.0 |

The effect is noticeable for the doc/org-manual.org file from org 
sources though absolute numbers are well below delay that makes 
interaction incomfortable.

In the configuration above effect of `org-refile-use-cache' is crucial, 
contribution of `org-refile-use-outline-path' is noticeable but not so 
important.

My expectation is that neither `org-refile-use-cache' nor `org-goto' 
command should make adding property slower.