demonstrated that the remaining slowdown can be caused purely by the
length of the buffer and how long 'parse-partial-sexp' takes to parse
it.
No, you haven't demonstrated that.
Apply the patch for xdisp.c that I have sent previously (it will be at
the end of this email too) and recompile Emacs.
Now try two different scenarios. 1,2a and 1,2b.
1. Visit dictionary.json. It will ask you whether to open such big file
literally, but after you answer 'y', it will display the beginning of
the file quickly.
2a) Evaluate (benchmark 1 '(save-excursion (parse-partial-sexp 1
(point-max)))), note the reported delay.
Kill and re-visit the file.
1. (same as before)
2b) Press M->, note the delay you see.
The delays in scenarios 1,2a and 1,2b should be ~the same. They are so
in my testing.
Or try this scenario: 1,2a,2b. Step 2b should work instantly here.