Re: Some testing issues

[Stephen Berman]

On Sat, 8 Jul 2017 00:20:23 -0400 Noam Postavsky <address@hidden> wrote:

> On Fri, Jul 7, 2017 at 5:22 PM, Stephen Berman <address@hidden> wrote:
>> Several of the tests call the function todo-toggle-view-done-items,
>> which contains this code:
>>
>>           (if (not (pos-visible-in-window-p shown))
>>               (recenter)
>>
>> (`shown' stores the position of the first done item in the category, if
>> it is shown.)  When using todo-mode and also when stepping through the
>> function with Edebug, it works as expected.
>
> If I comment out the set-window-buffer call it fails for me in Edebug
> as well (when running the first time).

You're right, I must have misrembered or been thinking of a different
case; sorry.

> By the way, in order to run successfully I did also need the following
> patch, otherwise all the tests failed (I think it only happens for me
> because I'm running from an Emacs that's living under my home
> directory).
>
> modified   test/lisp/calendar/todo-mode-tests.el
> @@ -46,6 +46,7 @@ (defmacro with-todo-test (&rest body)
>    "Set up an isolated todo-mode test environment."
>    (declare (debug (body)))
>    `(let* ((todo-test-home (make-temp-file "todo-test-home-" t))
> +          (abbreviated-home-dir nil)
>            (process-environment (cons (format "HOME=%s" todo-test-home)
>                                       process-environment))
>            (todo-directory todo-test-data-dir)

The only test file that sets abbreviated-home-dir is package-test.el, in
the macro with-package-test, which was indeed the inspiration for
with-todo-test.  I assume this would only effect cases like yours, and
hence make the test environment more robust, or are possible problems
that setting it to nil could raise?

>>  But when running the tests,
>> sometimes -- but not always -- it fails with the error "`recenter'ing a
>> window that does not display current-buffer."  In those cases, the test
>> failure is prevented by evaluating the following sexp immediately before
>> calling todo-toggle-view-done-items:
>>
>>           (set-window-buffer nil (current-buffer))
>>
>> When using todo-mode this is not necessary, because the selected window
>> always contains the current buffer when the function is called, but
>> apparently when running the tests, this isn't always the case, although
>> according to Edebug it is, AFAICT.
>
> When I evaluate (selected-window) in edebug before the call, I see
> that the selected window is showing the *ert* buffer.

Yes, I should have checked again before posting that.

>>  And again, the failure only happens
>> in some cases, though in those, it is reliably reproducible.  In
>> addition, in at least one case, when I rerun the test immediately after
>> it fails, then it succeeds.
>
> I think it succeeds the second time because the *ert* buffer is in a
> different state.

What state is that?  In Edebug it appears to be the same as on the first
test run: current-buffer is todo-test-1.todo and selected-window is the
one showing the *ert* buffer, yet now (pos-visible-in-window-p shown) is
non-nil, while on the first run it is nil.  I don't see what makes the
difference -- certainly not the value of the variable `shown', which is
226 and that position in window displaying *ert* is visible in both runs.

>>  And lastly, in one case, the test succeeds
>> without the set-window-buffer call when run interactively, but fails
>> without it when run in batch mode from the shell.  Any idea what's going
>> on here?  (If anyone wants to take a closer look, comments in the test
>> file point out the problematic cases.)
>
> todo-test-toggle-item-header04? I added a `message' call, and it seems
> that in batch mode the selected window shows *scratch* whereas in
> interactive mode it shows *ert*. I would say the success in
> interactive mode is just a coincidence.

Well, it's a reliably reproducible coincidence, which seems like a
contradiction in terms.

>> A second problem concerns trying to reference a message in a test.  The
>> function todo-toggle-view-done-items outputs a message if the category
>> contains no done items, and the test captures this by calling
>> current-message immediately after todo-toggle-view-done-items.  This
>> works when running the test interactively, but in a batch run,
>> current-message is nil (though the message is output in the shell).
>> Why?
>
> I think it's simply that there is no echo-area or any concept of a
> "current" message when running in batch mode.

Yes, as Martin also pointed out; I didn't realize (or stop to think
about) that.

> lread-tests--last-message might be worth looking at:
>
> (defun lread-tests--last-message ()
>   (with-current-buffer "*Messages*"
>     (save-excursion
>       (goto-char (point-max))
>       (skip-chars-backward "\n")
>       (buffer-substring (line-beginning-position) (point)))))

Thanks for the pointer, and for the feedback.

Steve Berman