DMN-TCK Audit Methodology

This document explains exactly what dmn-tck-audit does so an auditor can decide whether the report it produces is meaningful.

Subject under test

The subject under test is a deterministic Business Rule Engine. Inputs:

• A DMN model (XML, DMN 1.3 or newer).
• An input document (JSON object, one entry per <inputData>).
• A target decision name.

Output:

• A materialized JSON document containing the decision's value (and the values of any decisions in its dependency closure).

The internal representation the engine uses to evaluate rules — how DMN gets compiled, what the bytecode looks like, how the runtime is deployed — is intentionally outside the audit scope. The audit verifies behaviour against the public DMN-TCK kit and nothing else.

Test corpus

The bundle ships the official DMN Technology Compatibility Kit unmodified, copied from github.com/dmn-tck/tck. The exact upstream commit used is recorded in tck-corpus.upstream.txt. Per-file SHA-256 hashes are recorded in tck-corpus.SHA256 so the auditor can confirm no DMN file or test XML has been altered.

The kit declares three buckets:

dmn-tck-audit verify --scope compliant (the default) runs Level 2 and Level 3 only. Use --scope all to also exercise the non-compliant bucket; results from that bucket must be reported separately and never rolled into a Level 2 / Level 3 figure.

What a single test case does

For each <testCase> in the kit:

1. The CLI reads the DMN file referenced by <modelName> and the <inputNode> values declared in the test XML.
2. The CLI builds a JSON input document keyed by inputNode name.
3. The CLI POSTs { dmn_xml, input, decision_name } to POST /dmn-tck/v1/verify on the configured rule engine endpoint.
4. The endpoint returns a signed envelope:
   • envelope_b64 — base64 of the canonical JSON envelope.
   • signature_hex — Ed25519 signature over those exact bytes.
5. The CLI verifies the signature against the bundled audit-server.pubkey. A bad signature aborts the case and records fail_bad_signature.
6. The CLI parses the envelope (decision, outputs, server_case_id, timestamp).
7. The CLI compares outputs against the <expected> value from the test XML. Comparison is performed locally — the rule engine cannot fake a pass.

Comparison rules

• Decision name lookup. The expected name from the test XML is searched in the materialized output by exact match, then by underscored alias (Full Name → Full_Name), then by dotted suffix. If the materialized output is a single-key object whose key does not match any candidate, that single value is taken as the actual.
• Numeric tolerance. Integers and floats compare by numeric value with a tight tolerance to absorb decimal/float roundtrip noise. 5 and 5.0 compare equal.
• List ordering. DMN lists are ordered. Strict order match.
• Object equality. Same set of keys, recursive equality on values.
• Null. null compares equal only to null.

Any mismatch is recorded with the expected value, the actual value, and the decision name searched. The report is the source of truth for what failed and why.

Trust model

Three independent properties hold the audit together:

1. Corpus integrity. Per-file SHA-256 in tck-corpus.SHA256 and the upstream commit hash in tck-corpus.upstream.txt allow the auditor to confirm that the kit shipped in this bundle is the same kit anyone can fetch from github.com/dmn-tck/tck.
2. Response correctness. The endpoint returns materialized output, not a boolean. The CLI compares output against the expected value declared in the kit. A server cannot cause the CLI to print "pass" without producing the right value.
3. Response authenticity. Each response is wrapped in an Ed25519-signed envelope. The CLI refuses to accept any case whose signature does not verify against the bundled public key.

The signature does not prove the engine is correct — it proves the response came from the holder of the matching private key. Correctness still rests on the comparison in property (2).

What the report contains

dmn-tck-audit verify --report-out report.json writes a structured JSON document with:

• Run metadata: binary version, endpoint label, corpus root, generated-at timestamp.
• A summary: total attempted / passed / failed, plus per-bucket counts.
• One entry per (test case, decision) pair, including:
  • bucket
  • DMN model name
  • test case id
  • decision name
  • status: pass / fail_mismatch / fail_not_found / fail_transport / fail_rejected / fail_bad_signature / fail_malformed_response / skipped
  • expected value (from the kit)
  • actual value (from the rule engine)
  • server case id and signature hex (for traceability)

What the audit does not do

• It does not re-execute DMN evaluation locally. The audit is comparing rule-engine outputs to kit-declared expectations.
• It does not modify the kit or the rule engine. The bundle is read-only from the engine's perspective.
• It does not run authoring, packaging, or provisioning operations. The CLI exposes none of those verbs and the endpoint refuses any HTTP path beyond the audit surface.
• It does not assess performance. Latency and throughput claims (if any) are not part of this audit.

Reproducibility

A run is reproducible to the extent that:

• The corpus SHA-256 manifest matches the kit you ran against.
• The binary SHA-256 in SHA256SUMS matches the binary you ran.
• The endpoint returns the same materialized output for the same input. Hot-reloaded rule changes on the engine side will produce different outputs; auditors should run dmn-tck-audit clean before each audit-grade run to refresh runtime state.