io.github.benseverndev-oss/goldenmatch

What we checked

Six static checks, weighted by risk. Every result reflects only what could be observed in the published package and repository — never intent.

• Injection surface

  unscannable25% of grade

  Tool descriptions/manifest scanned for instruction-injection patterns (imperatives at the model, hidden text, 'ignore previous', data-exfil URLs).

  No tool descriptions, server instructions, prompts, or resources were fetched; nothing to scan for injection.

• Supply chain

  fail25% of grade

  Package provenance: namespace verification, repo linkage, maintainer count, account age, postinstall scripts, typosquat distance.

  Python build script executes at build time: network call in build script

• Credential hygiene

  pass15% of grade

  How the server takes secrets (env vs plaintext config vs hardcoded); secrets appearing in tool schemas.

  Reads secrets from the environment (e.g. dbt-goldensuite/spcs/server.py) — the recommended intake shape. PASS = no static red flag; static analysis cannot prove the code honors env-based secret handling at runtime.

• Permission scope

  warn15% of grade

  Declared tools vs. breadth (filesystem, network, exec); flags shell-exec and unbounded filesystem access.

  has both shell/exec and network egress — capable of download-and-run, though no such path was observed (dbt-goldensuite/scripts/run_goldencheck.py: subprocess.* call; goldenmatch/cli/serve_ui.py: outbound HTTP/socket call). has both shell/exec and destructive filesystem writes (dbt-goldensuite/scripts/run_goldencheck.py: subprocess.* call; goldenmatch/backends/score_duckdb.py: destructive filesystem call (shutil.rmtree/os.remove/etc.)). discloses shell/exec capability: 7 files (dbt-goldensuite/scripts/run_goldencheck.py, scripts/bench_data_gen_server.py, scripts/bench_pair_stream_columnar.py, +4 more) discloses dynamic code execution: goldenmatch/core/matchkey.py: uses eval()/exec() discloses broad/destructive filesystem access: 11 files (goldenmatch/backends/score_duckdb.py, goldenmatch/config/settings.py, goldenmatch/core/cloud_ingest.py, +8 more) Capability DISCLOSURE, not a verdict: static analysis sees the primitive is present and reachable, not whether its use is attacker-controlled.

• Version behavior

  fail10% of grade

  Diff of tool definitions between versions; new permissions or changed descriptions in a patch release (the postmark-mcp class).

  vs prior 3.5.0 (minor bump): new capability primitive(s) in source: child_process, dynamic-eval, fs-write, http-client, net, subprocess; new hardcoded outbound host/BCC destination(s): ...blob, 0.0.0.0, 127.0.0.1, api.anthropic.com, api.example.com, api.hubapi.com, api.openai.com, avatars.githubusercontent.com, cloud.google.com, docs.aws.amazon.com, example.com, fake, github.com, goldenmatch-bench-gen-production.up.railway.app, gpu, oauth2.googleapis.com, react.dev, www.googleapis.com, www.kaggle.com, www.w3.org, www2.census.gov, x.com, your-gpu-server, your-server.

• Transport config

  warn10% of grade

  Remote servers: TLS and auth mode (none/token/OAuth). Local servers: whether the manifest indicates it phones home.

  manifest/source references telemetry behavior — disclosure