test: fix test.h hidden output #193

Merged
OragonEfreet merged 3 commits from chore/test-framework-improvements into main 2026-11-09 12:24:33 +01:00
Owner

The house test framework (test/test.h) now reports failures with the values that caused them.

  • Framework output goes through a retained stdout stream: [FAIL] lines and the closing report stay visible even when a test writes to stdout.
  • Reports are colored: [OK] in green, [FAIL] in red, the memory report in cyan.
  • A failing CHECK_EQ or REQUIRE_EQ prints the run-time values of both operands (got 2, expected 3) next to the expression text.
  • Printed values are exact for every operand type: 64-bit integers keep every bit, pointers print as addresses, reals print with full precision.
  • CHECK_MEMEQ and REQUIRE_MEMEQ compare byte buffers and report the first mismatching byte: its offset, and both byte values in hex and decimal.
  • CHECK_REAL_EQ and REQUIRE_REAL_EQ compare reals within a caller-chosen tolerance: the boundary is inclusive, and NaN, infinities and negative tolerances never pass.
  • Every test binary accepts --stop-at-error to halt the run at the first failing case.
  • The closing report counts passed and failed asserts separately.
  • Under a compiler without introspection extensions the value display falls back to the plain expression text, so the framework still builds as strict C99.
  • The stdout redirection around each test is guarded: a test that closes stdout or exhausts descriptors can no longer leave output swallowed for the rest of the run.
  • The framework self-tests (redirection failure modes, parametric macros, real-equality semantics) live in one binary, unit_test_framework.
  • The MinGW CI jobs are named "build only" until they run the test binaries.
The house test framework (test/test.h) now reports failures with the values that caused them. - Framework output goes through a retained stdout stream: [FAIL] lines and the closing report stay visible even when a test writes to stdout. - Reports are colored: [OK] in green, [FAIL] in red, the memory report in cyan. - A failing CHECK_EQ or REQUIRE_EQ prints the run-time values of both operands (got 2, expected 3) next to the expression text. - Printed values are exact for every operand type: 64-bit integers keep every bit, pointers print as addresses, reals print with full precision. - CHECK_MEMEQ and REQUIRE_MEMEQ compare byte buffers and report the first mismatching byte: its offset, and both byte values in hex and decimal. - CHECK_REAL_EQ and REQUIRE_REAL_EQ compare reals within a caller-chosen tolerance: the boundary is inclusive, and NaN, infinities and negative tolerances never pass. - Every test binary accepts --stop-at-error to halt the run at the first failing case. - The closing report counts passed and failed asserts separately. - Under a compiler without introspection extensions the value display falls back to the plain expression text, so the framework still builds as strict C99. - The stdout redirection around each test is guarded: a test that closes stdout or exhausts descriptors can no longer leave output swallowed for the rest of the run. - The framework self-tests (redirection failure modes, parametric macros, real-equality semantics) live in one binary, unit_test_framework. - The MinGW CI jobs are named "build only" until they run the test binaries.
test: fix test.h hidden output
All checks were successful
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 9s
Documentation / build-docs (pull_request) Successful in 13s
QA / cert (pull_request) Successful in 37s
Build and Test / linux-linux-gcc / shared (pull_request) Successful in 24s
Build and Test / linux-linux-gcc / static (pull_request) Successful in 24s
Build and Test / linux-windows-mingw / shared (pull_request) Successful in 30s
Build and Test / linux-windows-mingw / static (pull_request) Successful in 31s
07d7d67389
RUN_TEST, RUN_TEST_ARGS and RUN_TEST_PARAM did not check the return of
dup, open and dup2. A failed dup left the saved descriptor at -1, and
if the null-device open still succeeded, the restoration dup2 silently
failed: stdout stayed pointed at the null device for the rest of the
run while the framework kept printing through its retained stream, so
the suite looked healthy while swallowing every library warning.

The redirection is now best effort. The null-device open only happens
when a valid backup descriptor was obtained, the redirection and its
restoration only happen when the null device opened, the backup close
only happens when it exists, and the test always runs. On failure the
suite degrades to noisy-but-visible output, never to a lost stdout.

Verified by exhausting the descriptor table before a run: the test
executes, stdout survives the test, and the closing report still
prints.
ci(test): name the mingw jobs for what they do
All checks were successful
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 11s
QA / cert (pull_request) Successful in 31s
Build and Test / linux-linux-gcc / shared (pull_request) Successful in 22s
Build and Test / linux-linux-gcc / static (pull_request) Successful in 24s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 31s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 31s
8bc31c4572
The workflow is called Build and Test, so a job named simply
linux-windows-mingw reads as testing coverage. The mingw jobs stop at
the build step: no test binary is built or run, because the tests are
not yet executed through wine. Until they are, the job names say
build only, and the cross-platform claims of the workflow stay honest.
The previous guard made the redirection conditional but still closed
the backup descriptor without checking the restoration dup2. A test
that closed fd 1 on its way out would have left the restoration
silently failing while the suite kept looking healthy through its
separate stream: the exact failure mode the guard was meant to
prevent.

The initial redirection dup2 is now checked too: its success is what
decides whether a restoration is owed at all. A failed restoration
prints a diagnostic through the framework stream, fails the case, and
the run continues; the backup descriptor is closed either way, since
nothing downstream would consume a preserved copy of a dead stream.

The descriptor failures reachable from outside a test (dup, open,
both initial and restoring dup2 on a hostile fd 1) are covered by the
new unit_stdout_redirection binary; the restoration failure itself is
not triggerable from a test body, only by a signal landing inside
dup2.
test(framework): cover the descriptor failure modes of RUN_TEST
All checks were successful
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 11s
QA / cert (pull_request) Successful in 31s
Build and Test / linux-linux-gcc / shared (pull_request) Successful in 22s
Build and Test / linux-linux-gcc / static (pull_request) Successful in 24s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 30s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 30s
8234274bdc
Four regression cases for the stdout redirection, all forced from the
test binary itself: rlimit ceilings make the backup dup and the null-
device open fail in turn, a hostile case closes fd 1 to prove the
restoration puts it back, and a final case verifies fd 1 is still a
valid descriptor after every failure mode. POSIX only (rlimit); the
suite is not built on Windows yet.
e5591f2 tested TST_DUP2's return against 0: the Windows _dup2
convention. POSIX dup2 returns the new descriptor on success, 1 here,
so the redirection flag was never set on POSIX platforms: the
restoration never ran, the backup was closed anyway, and stdout stayed
swallowed by the null device for the rest of the process while the
framework kept printing through its separate stream. The exact
regression the guard commits exist to prevent, invisible to ctest
because only the exit code is checked.

The only success test common to both conventions is against -1:
not -1 for the initial redirection, -1 for the restoration failure.
Measured before and after with an instrumented binary printing
fcntl(1, F_GETFD) from inside the cases: dead (-1) before, healthy
(0) after.
test(framework): assert fd liveness, not its inverse
All checks were successful
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 11s
QA / cert (pull_request) Successful in 31s
Build and Test / linux-linux-gcc / shared (pull_request) Successful in 21s
Build and Test / linux-linux-gcc / static (pull_request) Successful in 23s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 31s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 31s
8ad38ecbf2
The closing case of unit_stdout_redirection required the value of
fcntl(1, F_GETFD), which fails on 0. F_GETFD returns flags, not a
boolean: a healthy descriptor with no flags answers 0, a dead one
answers -1. The case therefore certified the inverse of its intent,
and it is what let the suite stay green while the dup2 convention bug
left fd 1 dead: each defect masked the other.

Require the descriptor to differ from -1 instead.
test(framework): port the assertion layer from feat/us-txt-1
All checks were successful
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 11s
QA / cert (pull_request) Successful in 32s
Build and Test / linux-linux-gcc / shared (pull_request) Successful in 23s
Build and Test / linux-linux-gcc / static (pull_request) Successful in 25s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 30s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 32s
2477b8a32d
Bring over the test.h improvements developed on the text stories branch
so this PR carries the complete framework work: failing equality
assertions print the run-time values of both operands (dispatched
without casting the operand, through a compound literal of its decayed
type read back at exact width; strict-C99 compilers fall back to the
plain stringified expressions), CHECK_MEMEQ and CHECK_REAL_EQ with
their REQUIRE variants, the --stop-at-error flag wired through
BEGIN_TESTS, passed/failed assert counts in the closing report, the
printf format attribute on PRINT, static mock_memory helpers, and the
seventeen spec migrations to REQUIRE_MEMEQ (str, mem, audio, event,
stream, net, bitmap; the bj_memcmp sites testing bj_memcmp itself
stay).

The descriptor guard from this branch's review rounds is re-applied on
top of the ported macros, so the guarded redirection lifecycle (dup,
open, both dup2 checked against -1, loud restoration failure) survives
the merge of the two lineages.

Verified on the ported tree: build with zero warnings, 42/42 tests,
operand values and byte diagnostics demonstrated by an external probe,
--stop-at-error halting at the first failing case.
Restoration was conditional on tst_redirected_, conflating descriptor
ownership with redirection status. The combined scenario the review
traced was real: dup saves stdout, the null-device open fails, the
test closes fd 1, and cleanup skipped the restoration because nothing
was redirected — while holding the very backup that could have
repaired the damage. The saved descriptor is now restored whenever it
exists; with fd 1 untouched the restoring dup2 is a harmless no-op
(same open file description on both ends), and when a test killed fd 1
it is a resurrection. The initial redirection dup2 stays best effort
and unchecked: its failure means visible output, not lost output, and
the tst_redirected_ flag is gone with the only branch that read it.

Write failures of the framework stream (vfprintf, fflush) are
deliberately ignored and the decision is now documented at PRINT: a
broken console leaves no channel to report through, and failing cases
over it would paint the suite red for an environment problem.

The pointer formatting of the value display reads its copy through
memcpy instead of a typed dereference: strict aliasing never comes
into play and pointer representations wider than void* would travel
too.

Two regression cases cover the combined scenario: the null-device
open refused by an rlimit ceiling while the backup dup succeeded, the
test closing fd 1, and the liveness case that follows. Both fail
against the previous header and pass against this one, verified by
compiling the binary against each.
ci(test): exclude the POSIX-only unit tests from Windows builds
Some checks failed
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 11s
QA / cert (pull_request) Successful in 31s
Build and Test / linux-linux-gcc / shared (pull_request) Failing after 22s
Build and Test / linux-linux-gcc / static (pull_request) Failing after 24s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 30s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 33s
153f2f0bf2
gen_test globs every unit_*.c unconditionally, so the descriptor-
regression binary (rlimit, fcntl, unistd) would be compiled into
Windows test builds. The glob is now guarded by NOT WIN32 at the CMake
level: the enforcement is in the build system, not in a file comment.
MinGW CI stays build-only: this keeps the shared/static jobs green the
day they gain a test step, and running these cases on Windows is work
for the Wine session.
The NOT WIN32 guard disabled every obj_*.c and unit_*.c test on
Windows instead of the one binary that needs rlimit and fcntl. The
gen_test calls are back to unconditional; gen_test itself filters
unit_stdout_redirection.c out when the target is Windows.

POSIX keeps all 42 targets, a mingw-w64 configure generates the 41
others, and the three remaining unit binaries compile clean under
x86_64-w64-mingw32-gcc.
test(framework): merge the framework self-tests into one binary
Some checks failed
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 10s
QA / cert (pull_request) Successful in 31s
Build and Test / linux-linux-gcc / shared (pull_request) Failing after 22s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Has been cancelled
Build and Test / linux-windows-mingw (build only) / static (pull_request) Has been cancelled
Build and Test / linux-linux-gcc / static (pull_request) Has been cancelled
83bfe551ea
unit_parametric.c and unit_stdout_redirection.c both test the test
framework itself, one for the parametric macros the property specs
rely on, the other for the stdout redirection failure modes (dup and
open refused by a lowered rlimit ceiling, fd 1 closed by a test
body). They now live in a single unit_test_framework binary; the
Windows exclusion in gen_test follows the renamed file.
OragonEfreet force-pushed chore/test-framework-improvements from 83bfe551ea
Some checks failed
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 10s
QA / cert (pull_request) Successful in 31s
Build and Test / linux-linux-gcc / shared (pull_request) Failing after 22s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Has been cancelled
Build and Test / linux-windows-mingw (build only) / static (pull_request) Has been cancelled
Build and Test / linux-linux-gcc / static (pull_request) Has been cancelled
to 0b02917de0
Some checks failed
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 9s
Documentation / build-docs (pull_request) Successful in 13s
QA / cert (pull_request) Successful in 37s
Build and Test / linux-linux-gcc / shared (pull_request) Failing after 25s
Build and Test / linux-linux-gcc / static (pull_request) Failing after 25s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 31s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 31s
2026-11-09 11:57:02 +01:00
Compare
OragonEfreet force-pushed chore/test-framework-improvements from 0b02917de0
Some checks failed
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 9s
Documentation / build-docs (pull_request) Successful in 13s
QA / cert (pull_request) Successful in 37s
Build and Test / linux-linux-gcc / shared (pull_request) Failing after 25s
Build and Test / linux-linux-gcc / static (pull_request) Failing after 25s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 31s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 31s
to 9607e10354
All checks were successful
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 11s
QA / cert (pull_request) Successful in 31s
Build and Test / linux-linux-gcc / shared (pull_request) Successful in 23s
Build and Test / linux-linux-gcc / static (pull_request) Successful in 25s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 30s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 30s
2026-11-09 12:07:04 +01:00
Compare
OragonEfreet force-pushed chore/test-framework-improvements from 9607e10354
All checks were successful
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 11s
QA / cert (pull_request) Successful in 31s
Build and Test / linux-linux-gcc / shared (pull_request) Successful in 23s
Build and Test / linux-linux-gcc / static (pull_request) Successful in 25s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 30s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 30s
to c9add33139
All checks were successful
C++ Compatibility / C++ Compatibility Check (pull_request) Successful in 8s
Documentation / build-docs (pull_request) Successful in 11s
QA / cert (pull_request) Successful in 32s
Build and Test / linux-linux-gcc / shared (pull_request) Successful in 23s
Build and Test / linux-linux-gcc / static (pull_request) Successful in 25s
Build and Test / linux-windows-mingw (build only) / shared (pull_request) Successful in 30s
Build and Test / linux-windows-mingw (build only) / static (pull_request) Successful in 31s
2026-11-09 12:21:29 +01:00
Compare
OragonEfreet added this to the 1.1 milestone 2026-11-09 12:23:12 +01:00
OragonEfreet deleted branch chore/test-framework-improvements 2026-11-09 12:24:33 +01:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
OragonEfreet/banjo!193
No description provided.