[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH v4 1/2] pickles: Add new pickle `pktest.pk`
From: |
Mohammad-Reza Nabipoor |
Subject: |
[PATCH v4 1/2] pickles: Add new pickle `pktest.pk` |
Date: |
Thu, 17 Dec 2020 19:43:33 +0330 |
This commit adds `pktest.pk` that provides testing facilities. It
is a TAP (Test Anything Protocol) producer. That means tests written
using `pktest` will generate TAP output. The output can be used by
any TAP consumer (test harnesses that read TAP).
2020-12-17 Mohammad-Reza Nabipoor <m.nabipoor@yahoo.com>
* pickles/pktest.pk: New file.
* pickles/Makefile.am (dist_pickles_DATA): Add `pktest.pk`.
* testsuite/poke.pktest/pktest.exp: New file.
* testsuite/poke.pktest/pktest-1.pk: Likewise.
* testsuite/poke.pktest/pktest-2.pk: Likewise.
* testsuite/poke.pktest/pktest-3.pk: Likewise.
* testsuite/poke.pktest/pktest-4.pk: Likewise.
* testsuite/poke.pktest/pktest-5.pk: Likewise.
* testsuite/poke.pktest/pktest-6.pk: Likewise.
* testsuite/poke.pktest/pktest-7.pk: Likewise.
* testsuite/poke.pktest/pktest-8.pk: Likewise.
* testsuite/poke.pktest/pktest-9.pk: Likewise.
* testsuite/poke.pktest/pktest-10.pk: Likewise.
* testsuite/poke.pktest/pktest-11.pk: Likewise.
* testsuite/poke.pktest/pktest-12.pk: Likewise.
* testsuite/Makefile.am (EXTRA_DIST): Add new tests.
* etc/hacking.org (Testing Pickles): Update.
* HACKING: Regenerate.
---
Hi, Jose!
Now `pktest` has only one function and one struct.
Tests are now check the output of `pktest_run` in `poke.pktests`.
Now it's the responsibility of the user to call `pktest_run` only once
and also return the correct return code (according to the TAP spec).
Regards,
Mohammad-Reza
ChangeLog | 21 ++++++++
HACKING | 9 +++-
etc/hacking.org | 9 +++-
pickles/Makefile.am | 2 +-
pickles/pktest.pk | 81 ++++++++++++++++++++++++++++++
testsuite/Makefile.am | 13 +++++
testsuite/poke.pktest/pktest-1.pk | 8 +++
testsuite/poke.pktest/pktest-10.pk | 31 ++++++++++++
testsuite/poke.pktest/pktest-11.pk | 33 ++++++++++++
testsuite/poke.pktest/pktest-12.pk | 17 +++++++
testsuite/poke.pktest/pktest-2.pk | 8 +++
testsuite/poke.pktest/pktest-3.pk | 8 +++
testsuite/poke.pktest/pktest-4.pk | 19 +++++++
testsuite/poke.pktest/pktest-5.pk | 19 +++++++
testsuite/poke.pktest/pktest-6.pk | 28 +++++++++++
testsuite/poke.pktest/pktest-7.pk | 28 +++++++++++
testsuite/poke.pktest/pktest-8.pk | 30 +++++++++++
testsuite/poke.pktest/pktest-9.pk | 31 ++++++++++++
testsuite/poke.pktest/pktest.exp | 22 ++++++++
19 files changed, 412 insertions(+), 5 deletions(-)
create mode 100644 pickles/pktest.pk
create mode 100644 testsuite/poke.pktest/pktest-1.pk
create mode 100644 testsuite/poke.pktest/pktest-10.pk
create mode 100644 testsuite/poke.pktest/pktest-11.pk
create mode 100644 testsuite/poke.pktest/pktest-12.pk
create mode 100644 testsuite/poke.pktest/pktest-2.pk
create mode 100644 testsuite/poke.pktest/pktest-3.pk
create mode 100644 testsuite/poke.pktest/pktest-4.pk
create mode 100644 testsuite/poke.pktest/pktest-5.pk
create mode 100644 testsuite/poke.pktest/pktest-6.pk
create mode 100644 testsuite/poke.pktest/pktest-7.pk
create mode 100644 testsuite/poke.pktest/pktest-8.pk
create mode 100644 testsuite/poke.pktest/pktest-9.pk
create mode 100644 testsuite/poke.pktest/pktest.exp
diff --git a/ChangeLog b/ChangeLog
index 665480f5..f6f2d768 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,24 @@
+2020-12-17 Mohammad-Reza Nabipoor <m.nabipoor@yahoo.com>
+
+ * pickles/pktest.pk: New file.
+ * pickles/Makefile.am (dist_pickles_DATA): Add `pktest.pk`.
+ * testsuite/poke.pktest/pktest.exp: New file.
+ * testsuite/poke.pktest/pktest-1.pk: Likewise.
+ * testsuite/poke.pktest/pktest-2.pk: Likewise.
+ * testsuite/poke.pktest/pktest-3.pk: Likewise.
+ * testsuite/poke.pktest/pktest-4.pk: Likewise.
+ * testsuite/poke.pktest/pktest-5.pk: Likewise.
+ * testsuite/poke.pktest/pktest-6.pk: Likewise.
+ * testsuite/poke.pktest/pktest-7.pk: Likewise.
+ * testsuite/poke.pktest/pktest-8.pk: Likewise.
+ * testsuite/poke.pktest/pktest-9.pk: Likewise.
+ * testsuite/poke.pktest/pktest-10.pk: Likewise.
+ * testsuite/poke.pktest/pktest-11.pk: Likewise.
+ * testsuite/poke.pktest/pktest-12.pk: Likewise.
+ * testsuite/Makefile.am (EXTRA_DIST): Add new tests.
+ * etc/hacking.org (Testing Pickles): Update.
+ * HACKING: Regenerate.
+
2020-12-17 Jose E. Marchesi <jemarch@gnu.org>
* libpoke/libpoke.h (struct pk_term_if): Change the prototype of
diff --git a/HACKING b/HACKING
index e80c5525..4d53a3ca 100644
--- a/HACKING
+++ b/HACKING
@@ -916,8 +916,13 @@ with GNU poke. If not, see
<https://www.gnu.org/licenses/>.
5.12 Testing Pickles
~~~~~~~~~~~~~~~~~~~~
- Each pickle in `pickles/FOO.pk' shall have a testsuite in
- `testsuite/poke.FOO' with a driver `testsuite/poke.FOO/FOO.exp'.
+ Each pickle in `pickles/FOO.pk' shall have a test file
+ `testsuite/poke.pickles/FOO-test.pk' written using pickle `pktest'.
+
+ If there are some features in the pickle that cannot be tested using
+ `pktest' (e.g., pickle uses `print'), that pickle also shall have a
+ testsuite in `testsuite/poke.FOO' with a driver
+ `testsuite/poke.FOO/FOO.exp'.
5.12.1 Command REPL tests
diff --git a/etc/hacking.org b/etc/hacking.org
index 485d9c2e..e63c18ae 100644
--- a/etc/hacking.org
+++ b/etc/hacking.org
@@ -664,8 +664,13 @@ Mohammad-Reza Nabipoor <m.nabipoor@yahoo.com>
** Testing Pickles
-Each pickle in =pickles/FOO.pk= shall have a testsuite in
-=testsuite/poke.FOO= with a driver =testsuite/poke.FOO/FOO.exp=.
+Each pickle in =pickles/FOO.pk= shall have a test file
+=testsuite/poke.pickles/FOO-test.pk= written using pickle =pktest=.
+
+If there are some features in the pickle that cannot be tested
+using =pktest= (e.g., pickle uses =print=), that pickle also shall
+have a testsuite in =testsuite/poke.FOO= with a driver
+=testsuite/poke.FOO/FOO.exp=.
*** Command REPL tests
diff --git a/pickles/Makefile.am b/pickles/Makefile.am
index aece5326..d5452e52 100644
--- a/pickles/Makefile.am
+++ b/pickles/Makefile.am
@@ -2,4 +2,4 @@ picklesdir = $(pkgdatadir)/pickles
dist_pickles_DATA = elf.pk ctf.pk leb128.pk bpf.pk btf.pk bmp.pk \
color.pk rgb24.pk id3v1.pk \
dwarf.pk dwarf-common.pk dwarf-frame.pk dwarf-pubnames.pk \
- dwarf-types.pk time.pk argp.pk
+ dwarf-types.pk time.pk argp.pk pktest.pk
diff --git a/pickles/pktest.pk b/pickles/pktest.pk
new file mode 100644
index 00000000..af2a1788
--- /dev/null
+++ b/pickles/pktest.pk
@@ -0,0 +1,81 @@
+/* pktest.pk - Facilities to write tests for pickles. */
+
+/* Copyright (C) 2020 The poke authors */
+
+/* This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+/* This is a TAP (Test Anything Protocol) producer.
+ * TAP is a simple text-based interface between testing modules in a test
+ * harness.
+ *
+ * For more info, see https://testanything.org/.
+ */
+
+type PkTestFn = (string) void;
+type PkTest = struct
+ {
+ string name;
+ /* Skip reason. If non-empty, test will be skipped */
+ string skip;
+ /* TODO reason. If non-empty, test will be marked as TODO */
+ string todo : todo == "" || (todo != "" && skip == "");
+ PkTestFn func;
+ };
+
+fun pktest_run = (PkTest[] tests, string skip = "") int:
+ {
+ var ok = 1;
+ var i = 0UL;
+
+ if (skip != "")
+ {
+ printf "1..0 # Skipped: %s\n", skip;
+ return ok;
+ }
+ else
+ printf "1..%u64d\n", tests'length;
+
+ for (t in tests)
+ {
+ ++i;
+
+ if (t.skip != "")
+ {
+ printf "ok %u64d %s # SKIP %s\n", i, t.name, t.skip;
+ continue;
+ }
+
+ fun todo = void:
+ {
+ if (t.todo == "")
+ print "\n";
+ else
+ printf " # TODO %s\n", t.todo;
+ }
+
+ try t.func(t.name);
+ catch (Exception ex)
+ {
+ ok = 0;
+ printf "not ok %u64d %s: %s", i, t.name, ex.msg;
+ todo ();
+ continue;
+ }
+ printf "ok %u64d %s", i, t.name;
+ todo ();
+ }
+
+ return ok;
+ }
diff --git a/testsuite/Makefile.am b/testsuite/Makefile.am
index e79361c0..950e2ecf 100644
--- a/testsuite/Makefile.am
+++ b/testsuite/Makefile.am
@@ -1801,6 +1801,19 @@ EXTRA_DIST = \
poke.pkl/xor-offsets-2.pk \
poke.pkl/xora-int-1.pk \
poke.pkl/xora-offset-1.pk \
+ poke.pktest/pktest.exp \
+ poke.pktest/pktest-1.pk \
+ poke.pktest/pktest-2.pk \
+ poke.pktest/pktest-3.pk \
+ poke.pktest/pktest-4.pk \
+ poke.pktest/pktest-5.pk \
+ poke.pktest/pktest-6.pk \
+ poke.pktest/pktest-7.pk \
+ poke.pktest/pktest-8.pk \
+ poke.pktest/pktest-9.pk \
+ poke.pktest/pktest-10.pk \
+ poke.pktest/pktest-11.pk \
+ poke.pktest/pktest-12.pk \
poke.repl/repl.exp \
poke.rgb24/rgb24.exp \
poke.rgb24/rgb24-1.pk \
diff --git a/testsuite/poke.pktest/pktest-1.pk
b/testsuite/poke.pktest/pktest-1.pk
new file mode 100644
index 00000000..82a48fa5
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-1.pk
@@ -0,0 +1,8 @@
+/* { dg-do run } */
+
+load pktest;
+
+/* { dg-command {var ok = pktest_run (PkTest[] ())} } */
+/* { dg-output "1..0\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "1" } */
diff --git a/testsuite/poke.pktest/pktest-10.pk
b/testsuite/poke.pktest/pktest-10.pk
new file mode 100644
index 00000000..339fda52
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-10.pk
@@ -0,0 +1,31 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "a failing test",
+ skip = "reason to skip this test case",
+ func = lambda (string name) void:
+ {
+ /* Because of the SKIP, the result will be "ok" */
+ assert (1 != 1);
+ },
+ },
+ PkTest {
+ name = "another descriptive name or phrase",
+ todo = "WIP",
+ func = lambda (string name) void:
+ {
+ assert (255 == 0xff);
+ assert ([1,2] + [3,4] == [1,2,3,4]);
+ },
+ },
+];
+
+/* { dg-command {var ok = pktest_run (tests)} } */
+/* { dg-output "1..2\\nok 1 a failing test # SKIP" } */
+/* { dg-output " reason to skip this test case\\n" } */
+/* { dg-output "ok 2 another descriptive name or phrase # TODO WIP\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "1" } */
diff --git a/testsuite/poke.pktest/pktest-11.pk
b/testsuite/poke.pktest/pktest-11.pk
new file mode 100644
index 00000000..6744c5de
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-11.pk
@@ -0,0 +1,33 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "a failing test",
+ skip = "reason to skip this test case",
+ func = lambda (string name) void:
+ {
+ /* Because of the SKIP, the result will be "ok" */
+ assert (1 != 1);
+ },
+ },
+ PkTest {
+ name = "another failing test",
+ todo = "WIP",
+ func = lambda (string name) void:
+ {
+ /* This will report "not ok".
+ Test harness should this as expected failure. */
+ assert (255 != 0xff);
+ },
+ },
+];
+
+/* { dg-command {var ok = pktest_run (tests)} } */
+/* { dg-output "1..2\\nok 1 a failing test # SKIP" } */
+/* { dg-output " reason to skip this test case\\n" } */
+/* { dg-output "not ok 2 another failing test: assertion failed at" } */
+/* { dg-output ".*:22:9 # TODO WIP\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "0" } */
diff --git a/testsuite/poke.pktest/pktest-12.pk
b/testsuite/poke.pktest/pktest-12.pk
new file mode 100644
index 00000000..29f81b95
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-12.pk
@@ -0,0 +1,17 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "test",
+ skip = "skip reason",
+ todo = "todo reason", /* `skip` and `todo` are orthogonal */
+ func = lambda (string name) void:
+ {
+ assert (1 == 1);
+ },
+ },
+];
+
+/* { dg-output "unhandled constraint violation exception" } */
diff --git a/testsuite/poke.pktest/pktest-2.pk
b/testsuite/poke.pktest/pktest-2.pk
new file mode 100644
index 00000000..4a535bc0
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-2.pk
@@ -0,0 +1,8 @@
+/* { dg-do run } */
+
+load pktest;
+
+/* { dg-command {var ok = pktest_run (PkTest[2] ())} } */
+/* { dg-output "1..2\\nok 1 \\nok 2 \\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "1" } */
diff --git a/testsuite/poke.pktest/pktest-3.pk
b/testsuite/poke.pktest/pktest-3.pk
new file mode 100644
index 00000000..d7e595e3
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-3.pk
@@ -0,0 +1,8 @@
+/* { dg-do run } */
+
+load pktest;
+
+/* { dg-command {var ok = pktest_run (PkTest[2] (), "Reason to skip")} } */
+/* { dg-output "1..0 # Skipped: Reason to skip\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "1" } */
diff --git a/testsuite/poke.pktest/pktest-4.pk
b/testsuite/poke.pktest/pktest-4.pk
new file mode 100644
index 00000000..ba5fbbf8
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-4.pk
@@ -0,0 +1,19 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "a descriptive name or phrase",
+ func = lambda (string name) void:
+ {
+ assert (1 == 1);
+ assert ("elf" != "ELF");
+ },
+ },
+];
+
+/* { dg-command {var ok = pktest_run (tests)} } */
+/* { dg-output "1..1\\nok 1 a descriptive name or phrase\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "1" } */
diff --git a/testsuite/poke.pktest/pktest-5.pk
b/testsuite/poke.pktest/pktest-5.pk
new file mode 100644
index 00000000..3e9400db
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-5.pk
@@ -0,0 +1,19 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "a failing test",
+ func = lambda (string name) void:
+ {
+ assert (1 != 1); /* `assert` starts at line 10, column 9 */
+ },
+ },
+];
+
+/* { dg-command {var ok = pktest_run (tests)} } */
+/* { dg-output "1..1\\nnot ok 1 a failing test: assertion failed at" } */
+/* { dg-output " .*:10:9\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "0" } */
diff --git a/testsuite/poke.pktest/pktest-6.pk
b/testsuite/poke.pktest/pktest-6.pk
new file mode 100644
index 00000000..5d59787b
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-6.pk
@@ -0,0 +1,28 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "a descriptive name or phrase",
+ func = lambda (string name) void:
+ {
+ assert (1 == 1);
+ assert ("elf" != "ELF");
+ },
+ },
+ PkTest {
+ name = "another descriptive name or phrase",
+ func = lambda (string name) void:
+ {
+ assert (255 == 0xff);
+ assert ([1,2] + [3,4] == [1,2,3,4]);
+ },
+ },
+];
+
+/* { dg-command {var ok = pktest_run (tests)} } */
+/* { dg-output "1..2\\nok 1 a descriptive name or phrase\\n" } */
+/* { dg-output "ok 2 another descriptive name or phrase\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "1" } */
diff --git a/testsuite/poke.pktest/pktest-7.pk
b/testsuite/poke.pktest/pktest-7.pk
new file mode 100644
index 00000000..08ea0267
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-7.pk
@@ -0,0 +1,28 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "a failing test",
+ func = lambda (string name) void:
+ {
+ assert (1 != 1); /* `assert` starts at line 10, column 9 */
+ },
+ },
+ PkTest {
+ name = "another failing test",
+ func = lambda (string name) void:
+ {
+ assert (255 != 0xff); /* `assert` starts at line 17, column 9 */
+ },
+ },
+];
+
+/* { dg-command {var ok = pktest_run (tests)} } */
+/* { dg-output "1..2\\nnot ok 1 a failing test:" } */
+/* { dg-output " assertion failed at .*:10:9\\n" } */
+/* { dg-output "not ok 2 another failing test:" } */
+/* { dg-output " assertion failed at .*:17:9\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "0" } */
diff --git a/testsuite/poke.pktest/pktest-8.pk
b/testsuite/poke.pktest/pktest-8.pk
new file mode 100644
index 00000000..3f5ebe92
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-8.pk
@@ -0,0 +1,30 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "a descriptive name or phrase",
+ skip = "my reason to skip this test case",
+ func = lambda (string name) void:
+ {
+ assert (1 == 1);
+ assert ("elf" != "ELF");
+ },
+ },
+ PkTest {
+ name = "another descriptive name or phrase",
+ func = lambda (string name) void:
+ {
+ assert (255 == 0xff);
+ assert ([1,2] + [3,4] == [1,2,3,4]);
+ },
+ },
+];
+
+/* { dg-command {var ok = pktest_run (tests)} } */
+/* { dg-output "1..2\\nok 1 a descriptive name or phrase # SKIP" } */
+/* { dg-output " my reason to skip this test case\\n" } */
+/* { dg-output "ok 2 another descriptive name or phrase\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "1" } */
diff --git a/testsuite/poke.pktest/pktest-9.pk
b/testsuite/poke.pktest/pktest-9.pk
new file mode 100644
index 00000000..9942fbcb
--- /dev/null
+++ b/testsuite/poke.pktest/pktest-9.pk
@@ -0,0 +1,31 @@
+/* { dg-do run } */
+
+load pktest;
+
+var tests = [
+ PkTest {
+ name = "a descriptive name or phrase",
+ skip = "my reason to skip this test case",
+ func = lambda (string name) void:
+ {
+ assert (1 == 1);
+ assert ("elf" != "ELF");
+ },
+ },
+ PkTest {
+ name = "another descriptive name or phrase",
+ todo = "WIP",
+ func = lambda (string name) void:
+ {
+ assert (255 == 0xff);
+ assert ([1,2] + [3,4] == [1,2,3,4]);
+ },
+ },
+];
+
+/* { dg-command {var ok = pktest_run (tests)} } */
+/* { dg-output "1..2\\nok 1 a descriptive name or phrase # SKIP" } */
+/* { dg-output " my reason to skip this test case\\n" } */
+/* { dg-output "ok 2 another descriptive name or phrase # TODO WIP\\n" } */
+/* { dg-command {ok} } */
+/* { dg-output "1" } */
diff --git a/testsuite/poke.pktest/pktest.exp b/testsuite/poke.pktest/pktest.exp
new file mode 100644
index 00000000..f579b167
--- /dev/null
+++ b/testsuite/poke.pktest/pktest.exp
@@ -0,0 +1,22 @@
+# pktest.exp - Tests for the pktest pickle
+#
+# Copyright (C) 2020 The poke authors
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301,
USA.
+
+load_lib ${tool}-dg.exp
+dg-init
+dg-runtest [lsort [glob -nocomplain $srcdir/poke.pktest/*.pk]] {} {}
+dg-finish
--
2.29.2
- [PATCH v4 1/2] pickles: Add new pickle `pktest.pk`,
Mohammad-Reza Nabipoor <=