The scoring rubric is a versioned, public contract. Every grade records the rubric_version it was computed under, so scores stay reproducible as the standard evolves. Newest revision first.
How the data is gathered. Static analysis of the published package and manifest, plus runtime probing where a server can be reached: remote endpoints are connected live, and local servers are run in an isolated sandbox to read their real tool list. Anything that can't be reached or booted stays insufficient — never a silent pass.
v1.1.0PlannedUpcoming
Weight calibration from real findings
The v1 weights are a documented starting point. As the corpus accumulates real findings, the next revision re-balances them against observed severity — and folds in the first dynamic-probe signals (live tool lists) as coverage rises.
Re-weight checks from measured finding distributions, not first principles.
Fold remote/local probe results into injection_surface and version_behavior.
No silent changes: ships as 1.1.0, and old scores keep their 1.0.0 stamp.
v1.0.0Current
Initial public rubric
The first published standard: six static checks, each weighted by how directly a failure enables a real attack, scored 0–100 and mapped to a letter grade. Analysis is static only — the scanner never executes server code — so the rubric is honest about what it can and cannot see.
Six checks spanning injection, supply chain, credentials, permissions, version drift, and transport.
Coverage-aware: unscannable is a distinct state, never a silent skip or a free pass.
Every score is recomputable from its findings — no black box.
Checks & weights
Check
Weight
What it evaluates
injection_surface
25%
Tool descriptions/manifest scanned for instruction-injection patterns (imperatives at the model, hidden text, 'ignore previous', data-exfil URLs).
How the server takes secrets (env vs plaintext config vs hardcoded); secrets appearing in tool schemas.
permission_scope
15%
Declared tools vs. breadth (filesystem, network, exec); flags shell-exec and unbounded filesystem access.
version_behavior
10%
Diff of tool definitions between versions; new permissions or changed descriptions in a patch release (the postmark-mcp class).
transport_config
10%
Remote servers: TLS and auth mode (none/token/OAuth). Local servers: whether the manifest indicates it phones home.
Grade bands
A≥ 90B≥ 75C≥ 60D≥ 40F< 40unscannabledistinct state
Scanner · what changed
Changelog
The rubric version identifies the standard. These are changes to how the scanner applies it — no check or weight changed, but grades moved, so they are published rather than shipped quietly.
permission_scope
Capability disclosure is no longer scored as a failure
Referencing a credential path (~/.ssh, .aws/credentials), or having both shell execution and network egress somewhere in a repository, was scored as a HIGH failure. Those are capabilities, not evidence: an SSH manager touching ~/.ssh is doing its job, and an exec in one file plus an HTTP call in another is not a download-and-run path. A failure now requires a shape that evidences itself — a download piped into a shell, or a network response passed into exec. Test and fixture files are no longer scanned at all, since they are not the shipped surface.
Impact. About a quarter of D and F grades cited one of these findings. Affected servers were re-scored.
transport_config
Servers using MCP OAuth were recorded as having no authentication
Native MCP OAuth allows an unauthenticated initialize handshake and enforces authorization at the tool-call layer. The probe read the successful handshake as 'no auth required' and failed the transport check. Authentication is now read from the RFC 9728 protected-resource metadata, which is the authoritative signal.
Impact. Every OAuth-protected remote server in the registry was affected, and all were re-scored. Reported by a maintainer.
Think a grade is wrong? Tell us and it gets looked at.
The check set and weights are a versioned, public contract. See the dataset export for the machine-readable data, and the register to browse graded servers.