DMN-TCK Audit — Auditor Walkthrough

This bundle lets you independently verify Vastar's claim of 100% conformance against the official DMN-TCK compliant corpus (Compliance Level 2 + Level 3, 3,495 / 3,495 test cases).

The full trust model — comparison rules, signature scheme, trust boundaries, what the audit does and does not assess — is in METHODOLOGY.md next to this file. Read that before running, so the report you produce is meaningful.

If anything below does not work as advertised: contact auditor-support@midsolution.id with the failing command, the output, and your auditor token id. We do not modify the corpus, the rule engine, or this CLI for individual auditors — that would defeat the point of an independent run.

Prerequisites

Linux x86_64 (this bundle), or build for your platform from source. The binary is statically linked against musl; no extra packages are required at runtime.
Outbound HTTPS (TCP 443) to audit.vastar.id from where you run the CLI.
An auditor token from Vastar (see "Get your auditor token" below).
About 15 minutes of wall-clock time for the full Compliance Level 2 + Level 3 run (≈3,500 cases at ~250 ms each over normal Internet RTT). Resumable mid-run if the network drops.

The CLI itself only needs tar, sha256sum, and a network. Optional nicety: jq for inspecting the JSON report.

What you have

dmn-tck-audit-<version>-<os>-<arch>/
├── dmn-tck-audit               # the audit CLI (this is what you run)
├── audit-server.pubkey         # Ed25519 public key for response verification
├── tck-corpus/                 # bundled DMN-TCK test corpus (Apache-2.0)
├── tck-corpus.SHA256           # per-file integrity manifest of the corpus
├── tck-corpus.upstream.txt     # upstream commit hash + URL
├── SHA256SUMS                  # binary + pubkey integrity
├── README-AUDITOR.md           # this file (walkthrough)
└── METHODOLOGY.md              # methodology + trust model

Step 1 — Verify bundle integrity

The bundle is mostly bytes you have to trust. Confirm the bytes weren't tampered with in transit:

cd dmn-tck-audit-<version>-<os>-<arch>/

# 1a. Binary + pubkey checksums.
sha256sum -c SHA256SUMS
# Expected:
#   dmn-tck-audit: OK
#   audit-server.pubkey: OK

# 1b. Per-file corpus checksums (this prints ~6,500 lines; the
#     important thing is the trailing line + exit code = 0).
sha256sum -c tck-corpus.SHA256 | tail -3

# 1c. Confirm the binary is not a misleading copy. The version line
#     should match the version part of this directory's name.
./dmn-tck-audit version

If any line says FAILED, stop. Re-download the bundle from the download portal; if it still fails, email auditor-support@midsolution.id.

Optional — diff the corpus against upstream

The corpus shipped here is the unmodified DMN-TCK kit. If you want to confirm independently:

cat tck-corpus.upstream.txt
# url:    https://github.com/dmn-tck/tck
# commit: <some sha>
# bundled_at: <some iso-8601 utc>

# In a separate directory:
git clone https://github.com/dmn-tck/tck upstream-tck
git -C upstream-tck checkout <commit-from-tck-corpus.upstream.txt>

# Compare. The only differences should be in directories not
# included in the bundle (eg LICENSE locations, top-level docs).
diff -r upstream-tck/TestCases tck-corpus/ | head

Step 2 — Get your auditor token

Tokens authenticate auditors against the rule-engine endpoint. Each token is bound to a specific auditor account and program; you cannot share or transplant one between auditors.

Open the audit portal: https://programs.midsolution.id
Sign up (if you do not have an account yet) or sign in.
New accounts go through Vastar operator approval — once approved, you receive an email; first login forces you to change the password and enrol a TOTP authenticator.
On the dashboard, find the program card vrule-dmn-tck and click Generate token — give it a label (eg audit-2026-Q2).
Copy the plaintext token — it is shown once. The portal stores only its hash; if you lose it, revoke and regenerate.

Then export it in the shell where you will run the CLI:

export DMN_TCK_TOKEN='dmn_tck_<64 hex characters>'

If you never receive the approval email or your token returns 401 when used: contact auditor-support@midsolution.id with your auditor account email and the token id (the first 8 hex characters after dmn_tck_).

Step 3 — (Optional) Smoke-test the CLI offline

Before consuming network round-trips, you can confirm the binary runs on your machine and the corpus parses cleanly:

./dmn-tck-audit verify --corpus tck-corpus --scope compliant \
                       --mock --quiet \
                       --report-out /tmp/mock-report.json

--mock swaps in an in-process stub verifier. The mock will report all cases as failing — that is expected and intentional. The point is to confirm the binary loads, the corpus is readable, and the report file is writable. A real, signed run is in step 4.

Step 4 — Refresh the rule engine state

Hot-reloaded rule changes between runs would produce different outputs and therefore break comparison. Always run clean first:

./dmn-tck-audit clean --url https://audit.vastar.id/vrule-dmn-tck
# Expected:
#   ok: audit server cache cleared

If you see HTTP 401: invalid bearer token, your DMN_TCK_TOKEN is wrong, expired, or revoked. Generate a new one from the portal.

If the request just hangs: confirm DNS for audit.vastar.id resolves and outbound TCP 443 is open on your network.

Step 5 — Run the full conformance verification

This is the audit-grade run. Default --scope compliant runs Compliance Level 2 + Level 3 (3,495 cases). Add --checkpoint so a mid-run network hiccup is recoverable.

./dmn-tck-audit verify \
    --corpus     tck-corpus \
    --scope      compliant \
    --url        https://audit.vastar.id/vrule-dmn-tck \
    --checkpoint /tmp/dmn-tck-checkpoint.jsonl \
    --report-out my-audit-report.json

You will see one line per case (use --quiet to suppress) followed by a summary:

summary: attempted 3495, passed 3495, failed 0
  compliance-level-2: 126 / 126 (0 failed)
  compliance-level-3: 3369 / 3369 (0 failed)
report: my-audit-report.json

Exit code is 0 on full pass and 1 on any failure.

If the run dies mid-way (laptop goes to sleep, VPN drops), resume:

./dmn-tck-audit verify \
    --corpus tck-corpus --scope compliant \
    --url https://audit.vastar.id/vrule-dmn-tck \
    --checkpoint /tmp/dmn-tck-checkpoint.jsonl --resume \
    --report-out my-audit-report.json

Other useful scopes

--scope l2          # Compliance Level 2 only
--scope l3          # Compliance Level 3 only
--scope compliant   # L2 + L3 (default; this is the conformance claim)
--scope all         # add the negative-path "non-compliant" bucket;
                    # report results separately, never roll up

Step 6 — Read the report

my-audit-report.json is canonical-ordered JSON. The shape is documented in METHODOLOGY.md; quick recipes for the common things you will want to see.

Render to PDF (offline, client-side)

The CLI ships an offline PDF renderer — no network calls — so an auditor can hand a board, regulator, or client a single printable file. Render runs entirely on your machine and adds zero load to the rule engine:

# Render an existing report.
./dmn-tck-audit render --in my-audit-report.json --out my-audit-report.pdf
# rendered: my-audit-report.pdf
# from:     my-audit-report.json
# sha:      <sha256 of the JSON, also embedded in the PDF footer>

# Optional auditor label (engagement, audit cycle, internal id):
./dmn-tck-audit render --in my-audit-report.json \
                       --out my-audit-report.pdf \
                       --label "Engagement 2026-Q2"

# Or one-shot — verify + render in a single command:
./dmn-tck-audit verify \
    --corpus tck-corpus --scope compliant \
    --url https://audit.vastar.id/vrule-dmn-tck \
    --report-out my-audit-report.json \
    --pdf-out    my-audit-report.pdf \
    --pdf-label  "Engagement 2026-Q2"

The PDF includes:

A cover page with PASS / FAIL banner, run metadata (timestamp, endpoint, binary version, corpus root + SHA-256), and the per-bucket summary table.
A trust + methodology one-pager so a non-technical reader has the audit's claim and what it does NOT cover on the same page as the result.
A failures section — one block per non-pass case, with bucket / model / case_id / decision name / status / signature prefix / expected / actual (truncated for layout). Only present when the run found failures; omitted on a clean pass.
A footer on every page showing the SHA-256 of the source JSON, so the printed PDF and the JSON it was rendered from can be paired forensically.

The JSON remains the canonical artefact; the PDF is for printing, archiving, and attaching to compliance packs.

Inspect the JSON directly (jq recipes)

# Top-line: did anything fail?
jq '.summary' my-audit-report.json
# {
#   "total_attempted": 3495,
#   "total_passed": 3495,
#   "total_failed": 0,
#   "by_bucket": {
#     "compliance-level-2": {"attempted": 126, "passed": 126, "failed": 0},
#     "compliance-level-3": {"attempted": 3369, "passed": 3369, "failed": 0}
#   }
# }

# All failures, with bucket / model / decision name / status.
jq '.cases[] | select(.status != "pass")
       | {bucket, model, case_id, decision_name, status, expected, actual}' \
   my-audit-report.json

# Failure count grouped by status code (helps spot the failure mode).
jq -r '.cases[] | .status' my-audit-report.json | sort | uniq -c | sort -rn

# Single-case deep-dive (replace with a real case_id from the report).
jq '.cases[] | select(.case_id == "0001-input-data-string-test-01")' \
   my-audit-report.json

The per-case status values you can encounter:

Status	Meaning
`pass`	Materialised output matches the expected value.
`fail_mismatch`	Output differed from the expected value.
`fail_not_found`	Decision name from the test XML had no match in the output object.
`fail_transport`	Network error talking to the rule engine.
`fail_rejected`	Engine refused the request (400 / 403 / 422 etc).
`fail_bad_signature`	Response did not verify against `audit-server.pubkey`. Treat as serious.
`fail_malformed_response`	Engine returned valid HTTP but a non-conformant body.
`skipped`	Case excluded by scope or unreadable test XML.

fail_bad_signature is the only status that suggests a trust boundary was crossed (response did not come from the holder of the matching private key). Treat it as a stop-the-world finding — collect the offending case envelope and email auditor-support@midsolution.id.

Step 7 — Cross-verify against Vastar's published report

Running verify once and reading the result is enough to certify your own audit conclusion. Vastar also publishes the report from its most recent internal run at:

https://programs.midsolution.id/programs/vrule-dmn-tck/evidence/report.json
https://programs.midsolution.id/programs/vrule-dmn-tck/evidence/report.pdf

Cross-comparing the two runs is the strongest test that the published claim is not just a number on a page. The verification chain has four independent layers; each one sharpens trust in a different direction.

Layer 1 — bytes integrity

Confirm the file you downloaded is the file Vastar published:

curl -fsS https://programs.midsolution.id/programs/vrule-dmn-tck/evidence/report.json \
     -o vastar-report.json
sha256sum vastar-report.json
# Compare with the JSON sha256 displayed under "Latest internal verification run"
# at https://programs.midsolution.id/programs/vrule-dmn-tck — they must match.

Layer 2 — `expected` values match TCK upstream

The bundled tck-corpus/ is supposed to be the unmodified DMN-TCK kit. Confirm:

# 2a. corpus shipped here matches the SHA-256 manifest
sha256sum -c tck-corpus.SHA256 | tail -1     # last line: 0 failed

# 2b. corpus matches public upstream at the recorded commit
cat tck-corpus.upstream.txt
git clone https://github.com/dmn-tck/tck /tmp/upstream-tck
git -C /tmp/upstream-tck checkout <commit-from-tck-corpus.upstream.txt>
diff -r /tmp/upstream-tck/TestCases/compliance-level-2 tck-corpus/compliance-level-2 \
    | head
diff -r /tmp/upstream-tck/TestCases/compliance-level-3 tck-corpus/compliance-level-3 \
    | head
# (no output = corpus matches upstream — `expected` values cannot be tampered)

Layer 3 — re-execute against the engine and diff with Vastar's run

This is the strongest layer. Run verify yourself with your own token, then compare both reports field-by-field with the bundled compare subcommand:

# Run yourself — produces my-own-report.json in CWD.
export DMN_TCK_TOKEN='dmn_tck_<your token from the portal dashboard>'
./dmn-tck-audit verify \
    --corpus tck-corpus --scope compliant \
    --url https://audit.vastar.id/vrule-dmn-tck \
    --report-out my-own-report.json

# Diff against Vastar's published JSON.
./dmn-tck-audit compare \
    --official  vastar-report.json \
    --mine      my-own-report.json \
    --report-out compare-diff.json

# Sample output on a clean match:
#   compare summary:
#     cases in official:  3495
#     cases in mine:      3495
#     matched:            3495
#     differs:            0
#       status differs:   0
#       actual differs:   0
#       expected differs: 0  (non-zero = corpus drift, investigate)
#     only in official:   0
#     only in mine:       0
#
#   ✓ no differences — the two runs agree on every comparable field.

What the four counts mean:

Counter	What it tells you
`expected differs`	The TCK-declared `<expected>` value differed between Vastar's run and yours. Non-zero is a stop-the-world finding — somebody altered the corpus on at least one side.
`actual differs`	The engine returned different bytes for the same input on the two runs. Often resolved by running `clean` before each run; if it persists, the engine is non-deterministic on that case.
`status differs`	One run said `pass`, the other said `fail_*`. Usually paired with `actual differs`.
`only in official` / `only in mine`	The other run's case set covered different ground. Should be zero when both runs use the same `--scope`.

Exit code is 0 on a clean match and 1 on any difference. The --report-out compare-diff.json flag writes the full structured diff (every disagreeing case, both sides' values) to a file you can share with Vastar to triage a finding.

Layer 4 — verify Vastar's well-known public key

Each case in the report carries a 128-hex-char signature_hex — Ed25519 signature over the case envelope. The CLI verifies these inline during verify (default --require-signature); a tampered response causes the case to record fail_bad_signature instead of pass. To pin the bundle's audit-server.pubkey against an independent source:

# Vastar publishes the verifying key at a well-known URL.
curl -fsS https://audit.vastar.id/.well-known/ed25519/vrule-dmn-tck.pub
# 9a42c6dc84b8a6930446aff28e2c742568614dc3e852742311fefd39943ff29c

# Diff against the bundle copy — they must be identical.
diff <(curl -fsS https://audit.vastar.id/.well-known/ed25519/vrule-dmn-tck.pub) \
     audit-server.pubkey
# (no output = the bundle's pubkey is the one the live endpoint signs with)

Manage stored remote reports (optional)

The rule engine retains a copy of each report for 30 days, indexed by an opaque report_id. Useful when an auditor and Vastar need to compare the report Vastar saw to the one the auditor saw.

./dmn-tck-audit report list \
    --url https://audit.vastar.id/vrule-dmn-tck

./dmn-tck-audit report download rpt_<id> -o report.json \
    --url https://audit.vastar.id/vrule-dmn-tck

./dmn-tck-audit report delete rpt_<id> \
    --url https://audit.vastar.id/vrule-dmn-tck

Server-side reports are auto-deleted after 30 days; auditors should keep their own local --report-out file as the source of truth.

What this CLI cannot do

The audit surface is intentionally narrow. The CLI cannot:

Upload custom rules, decisions, or any other artefact.
Configure or modify the rule engine.
Run anything other than the preinstalled DMN-TCK verification path.

The endpoint enforces the same restrictions independently; even a hand-crafted HTTP request will be refused. See METHODOLOGY.md for the full trust model.

Troubleshooting

Symptom	Likely cause + fix
`HTTP 401: invalid bearer token`	Token wrong / revoked / expired. Generate a fresh one in the portal.
`Resolving timed out` for `audit.vastar.id`	Local DNS resolver issue. Try `dig +short @1.1.1.1 audit.vastar.id` to confirm; add `38.47.184.59 audit.vastar.id` to `/etc/hosts` as a workaround if the resolver is flaky.
`SSL certificate problem: self-signed certificate`	Your machine's CA bundle is incomplete. Update `ca-certificates` (`apt-get install -y --reinstall ca-certificates`). The endpoint uses Let's Encrypt; the LE root is in any current bundle.
`Connection refused` / `timed out` on the endpoint	Outbound 443 blocked at your firewall, or the auditor LXC is down for maintenance. Email `auditor-support@midsolution.id`.
Run is slow	Latency between you and the endpoint dominates wall-clock time (each test case is one HTTP round-trip). 250 ms RTT × 3,495 cases ≈ 14 minutes is normal. Use `--checkpoint` so a mid-run drop is recoverable.
`fail_bad_signature` on any case	The response did not come from the holder of the private key matching `audit-server.pubkey`. Stop and report — do not roll the report up as "pass".

Need help

Email auditor-support@midsolution.id with:

Your auditor account email.
Your token id (first 8 hex characters after dmn_tck_).
The exact command you ran.
The full output (or the first 50 lines if it is very long).

Do not include the plaintext token in support emails — only the id.

Auditor walkthrough