9. Metrics

Organizations SHALL establish baselines by measuring for at least 3 Increments before setting targets. Targets are organization-specific, not universal.

9.1 Mandatory (SHALL)

A post-done defect is a defect task explicitly linked to a completed parent task that introduced the defect. The link SHALL be recorded in the task tracker.

9.2 Recommended (SHOULD; SHALL in Team+)

Manual Intervention Rate is computed from recorded Gate Bypasses, not from self-assessment. Section 8.6(j) makes direct modification of an artifact a Gate Bypass that SHALL be recorded, so the boundary of “manual intervention” is set normatively and is no longer left to each organization to define.

Relationship to Gate Bypass rate: 8.6(i) recommends rather than requires that figure, so an organization may hold this metric without it, and this Standard does not fix how it is computed. What the definitions give is a containment of events, not of rates: a modification recorded under 8.6(j) is a Gate Bypass under 8.6(h), so every such record belongs to the population Gate Bypass rate is drawn from, while the reverse does not hold — that population also contains admissions of any other gate’s declared effect. Whether the two rates stand in the same relation depends on the denominator the organization gives Gate Bypass rate, which 8.6(i) leaves to it: they are comparable where that denominator is Tasks, the unit this metric uses, and not otherwise. Figures on different denominators answer different questions and SHALL NOT be added together or presented as one nested inside the other.

NOTE: The record this metric counts is made by the party that acted. It is therefore a measure of what was recorded, and organizations SHALL NOT present it as a measure of what occurred — the general form of this is 9.4(b). Organizations that collect this metric SHALL set a target for it and document the basis (9.4(c)) — at Foundation, which 11.1 does not ask to report it, the duty arises only where the organization reports it; a figure that crosses the target is escalated under 9.4(d). This standard does not fix the number. No measurement exists that would justify one figure for every organization, and a ceiling stated without such a measurement would be an arbitrary number carrying the force of a requirement. What the standard does require is that the ceiling exist, be reasoned, and not be moved in silence — which is what a normalizing bypass rate looks like from the outside.

An unrecorded modification does not appear in it, and nothing in this standard detects one automatically. Where assurance about the figure itself is required, it comes from comparing the modifications recorded in version control against the gate’s admission records (8.6(g)): every modification within a Task’s scope should fall under an admission the gate recorded, and one that does not is either an admission that was not recorded or a bypass that was not declared. Authorship does not settle it — 8.6(j) makes the route the origin of the content, not the identity of the process that wrote the file, so a comparison of who committed what is blind to precisely the case the clause was written for. This is an audit the organization performs, not a property the metric has.

Adversarial Detection Rate (ADR) measures the density of CRITICAL-severity findings discovered by adversarial review (Section 10.15, L3) per task that underwent L3 review. Target: organizations SHOULD aim for ADR < 0.5 (fewer than one CRITICAL finding per two reviewed tasks). An ADR of 0 indicates either excellent AI output quality or insufficient review rigor — organizations SHOULD distinguish between the two.

For ADR computation, “L3_reviewed_tasks” means tasks that underwent L3 Adversarial Review. Tasks that did not receive L3 review (e.g., Low-risk tasks where L3 was skipped per Section 10.15) are excluded from the denominator. This ensures ADR reflects review effectiveness, not review coverage.

Knowledge Capture Rate target calibration: organizations with established knowledge bases SHOULD set KCR targets that reflect diminishing returns. A target of 1.0 (one entry per task) is appropriate for greenfield projects. For mature projects (>500 tasks), a target of 0.33 (one entry per three tasks) better reflects the natural rate of novel knowledge discovery. Organizations SHALL document their KCR target rationale.

9.3 Collection

Metrics 1–5, 8 and 9 SHALL be collected automatically from task tracking and session data. Metric 8 is derived from recorded Gate Bypasses under 8.6(j) and requires no separate reporting act. Metrics 6–7 require cost tracking integration. Cost Predictability requires Flow Manager assessment at Increment Retrospective.

Metric 10 (ADR) SHALL be collected from adversarial review findings recorded during L3 review (Section 10.15). Organizations SHALL maintain a record of review findings per task, classified by severity, to compute ADR.

Metric computation SHALL include all tasks without epoch-based filtering or exclusion of historical data. Organizations SHALL NOT exclude tasks from metric computation based on creation date, migration status, or tooling version. Rationale: epoch filters introduce complexity, create maintenance burden, and mask data quality issues; they also make a figure unauditable, because the population it was computed over stops being the population the tracker holds. A cumulative figure does not shed its early data on its own: those tasks keep the weight their share of the total gives them, and that share falls only as the total grows — not because the tooling changed. Where the change of tooling is exactly what makes early data unrepresentative, waiting does not fix the figure. That is a reason to state the limitation rather than to filter it away. If early data is known to be unreliable (e.g., pre-automation manual entries), this SHALL be documented as a known limitation, not filtered.

Cost Predictability (metric 6) requires organizations to record planned cost before implementation begins. In practice, planned cost estimation for AI-assisted tasks is unreliable — AI execution time is non-deterministic and model pricing varies. At Team configuration, Cost Predictability SHALL be tracked but baselines may be provisional during the first 3 Increments. At Enterprise configuration, Cost Predictability SHALL be tracked with established baselines. Cost per Task (metric 7) provides a more actionable proxy for cost management in early adoption.

Multi-agent and multi-session configurations: when multiple AI agents work in parallel (multiple concurrent sessions), per-session metrics (throughput, session duration, cost per session) reflect individual agent performance, not aggregate team output. Organizations using multi-agent configurations SHOULD additionally track aggregate metrics at the Increment level. Token and cost attribution in multi-agent scenarios SHOULD be recorded per-agent, with aggregate totals available for Increment-level reporting.

9.4 Reporting a figure

A metric is a claim about what happened. These requirements apply to every metric of 9.1 and 9.2, and to any further figure an organization reports as a SENAR metric.

a) The method is disclosed with the figure. Organizations SHALL state how a reported figure is computed: the population it is drawn from, the record type it is counted over, and the period it covers. A figure whose definition is not disclosed is an assertion rather than a measurement, and SHALL NOT be presented as a SENAR metric.

b) A figure computed from self-made records measures what was recorded. Where the records a metric is counted over are made by the party being measured, the figure measures what was recorded and not what occurred, and organizations SHALL NOT present it as a measure of what occurred. First-Pass Success Rate, Defect Escape Rate and Manual Intervention Rate share this property: each improves when a record is not made, and nothing in this standard detects an unmade record automatically.

c) A self-set target or threshold carries its basis. Where this standard leaves a target, threshold or baseline to the organization, the organization SHALL document the basis on which it was set and the data that basis rests on. A number set without a basis cannot be assessed, and an assessor acting under 13.7 is left with nothing to ask for.

d) A target that is crossed is escalated, not quietly moved. Where a reported figure crosses a target set under (c), the organization SHALL record the crossing and the action taken, or the decision to change the target together with the basis for the change. Revising a target to accommodate the figure, without stating that this is what happened, is a defect of the record and not a change of practice.

NOTE: (b) is a property of the class of records, not of a particular metric, which is why 9.2 states it where Manual Intervention Rate is defined and does not argue it again here. Assurance about such a figure comes from comparing the records against an independent trace — 9.2 names that comparison for Manual Intervention Rate — and not from adding further checks to the metric.