[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [O] [test] Mark tests with missing dependencies as "expected to fail
Re: [O] [test] Mark tests with missing dependencies as "expected to fail"
Tue, 18 Oct 2011 10:22:34 -0600
Gnus/5.13 (Gnus v5.13) Emacs/24.0.90 (gnu/linux)
David Maus <address@hidden> writes:
> Hi all,
> Currently tests with missing dependency are silently skipped -- it
> might be worth changing this behavior to not skip them, but mark them
> as expected to fail. You can do this in ERT by placing the
> keyword :expected-result followed by either :passed or :failed before
> the test's body.
> Benefit of this is that the tests w/ missing dependencies will show up
> in the ERT result page (with a small letter f) but (obviously) don't
> count as failures.
> The following macro is a first shot at a convenient way to define
> tests with dependencies:
> #+begin_src emacs-lisp
> (defmacro org-test-deftest-with-dependency (dependency &rest body)
> (let* ((docstring (if (stringp (third body)) (list (third body))))
> (deftest (nthcdr (if docstring 3 2) body))
> (dependency-p (eval dependency)))
> `(ert-deftest ,@(append (list (first body) (second body)) docstring)
> ,@(if dependency-p
> '(:expected-result :passed)
> '(:expected-result :failed (error "Missing dependency")))
> Here DEPENDENCY is a simple form that evaluates to non-nil if the
> dependency is met. If marking the tests this way is the agreed way to
> go we can extend the syntax of a dependency to an alist with a
> human-readable description of the dependency as key and a list of
> forms that all must eval to non-nil as value. E.g.
> #+begin_src emacs-lisp
> (defvar test-ob-R-dependencies
> '(("R executable" (org-test-for-executable "R"))
> ("Package ESS" (featurep 'ess))))
> And change the expander code to map through this alist and as soon one
> dependency evals to nil sets a variable `dependency-missing' to the
> respective dependency.
> Any comments on this?
I agree it would be preferable to note that not all tests are run when
dependencies are missing, although I don't think it is extremely
important. I think some version of the above would be worthwhile if it
could be done in a file-wide manner (as are the current dependency
checks) and wouldn't require duplicating the dependency check or
changing every test form individually. Perhaps a file-local-variable
could be used to expect failures for every form defined in the file?
Cheers -- Eric