io.github.giachi85it-ux/mcp-judge

What we checked

Six static checks, weighted by risk. Every result reflects only what could be observed in the published package and repository — never intent.

• Injection surface

  warn25% of grade

  Tool descriptions/manifest scanned for instruction-injection patterns (imperatives at the model, hidden text, 'ignore previous', data-exfil URLs).

  tool "quick_check" description: off-vendor URL in a description (host); capability text should not name where data goes. tool "quick_check" param "url" description: off-vendor URL in a description (motherwise-api.onrender.com); capability text should not name where data goes. tool "evaluate_mcp" description: off-vendor URL in a description (host); capability text should not name where data goes. tool "evaluate_mcp" param "url" description: off-vendor URL in a description (motherwise-api.onrender.com); capability text should not name where data goes. tool "compare_mcps" description: off-vendor URL in a description (a); capability text should not name where data goes. tool "compare_mcps" description: off-vendor URL in a description (b); capability text should not name where data goes. tool "compare_mcps" param "urls" description: off-vendor URL in a description (motherwise-api.onrender.com); capability text should not name where data goes. tool "compare_mcps" param "urls" description: off-vendor URL in a description (mcp-judge.onrender.com); capability text should not name where data goes.

• Supply chain

  warn25% of grade

  Package provenance: namespace verification, repo linkage, maintainer count, account age, postinstall scripts, typosquat distance.

  no resolvable repository linked to the package (provenance unverifiable)

• Credential hygiene

  pass15% of grade

  How the server takes secrets (env vs plaintext config vs hardcoded); secrets appearing in tool schemas.

  No hardcoded secrets, secret-shaped tool parameters, or leaked keys found. PASS = no static red flag; static analysis cannot prove the code honors env-based secret handling at runtime.

• Permission scope

  unscannable15% of grade

  Declared tools vs. breadth (filesystem, network, exec); flags shell-exec and unbounded filesystem access.

  No source files, package.json, or pyproject were fetched; cannot infer capability scope.

• Version behavior

  fail10% of grade

  Diff of tool definitions between versions; new permissions or changed descriptions in a patch release (the postmark-mcp class).

  vs prior 0.1.0 (minor bump): new tool "list_criteria" appeared; new tool "quick_check" whose description contains embedded outbound URL; new tool "evaluate_mcp" whose description contains embedded outbound URL; new tool "compare_mcps" whose description contains embedded outbound URL.

• Transport config

  fail10% of grade

  Remote servers: TLS and auth mode (none/token/OAuth). Local servers: whether the manifest indicates it phones home.

  remote MCP endpoint declares NO authentication (unauthenticated remote — nginx-ui/CVE-2026-33032 class)