A bridge isn't competent because the blueprint is elegant, but because the bridge holds — and continues to hold when trucks cross, winds rise, and inspectors check the bolts.
Tronto insists that "assuming responsibility is not yet the same as doing the actual work of care." Competence is about execution: working code that does what it promised, audited, explainable, and safe-to-fail. And crucially — a point Tronto presses on technologists in particular — "to be competent to care is not simply a technical issue, but a moral one." A system that ships broken care with good intentions has failed morally, not just technically. The promise was made; the bridge did not hold.
The illustration's framing: We check the process — not "just trust us," but with transparency and fast operational feedback on how care is delivered. Trust, in a Civic AI context, is not a disposition you extend once to a vendor; it is something earned incrementally, through demonstrated, inspectable practice.
Definition
- Safety is a property of practice. Competence is demonstrated in operation, not assumed from design.
- Proof before promotion — the Apprentice Model. Features graduate like apprentices: shadow mode → canary → general with guardrails.
- Observability over opaqueness. A "show your work" approach with decision traces, source citations, uncertainty scores, and explainable summaries tied to decisions promotes observability. (Observability means the system's reasoning is inspectable, not that anyone sees individual private interactions; privacy and transparency are not in conflict.)
- Least power. The simplest mechanism is used to meet the need; complexity grows attack surface.
- Fail safely. When evidence is weak or a component drifts, the system narrows scope, hands off to a human, or pauses instead of extrapolating and bluffing.
Why it matters
Competence matters because public promises fail unless execution is visible, testable, and reversible. Pack 2 binds commitments; Pack 3 asks whether the system actually delivers on them. For Civic AI, safety is something people should be able to inspect in operation, not infer from vendor intent. A system that ships broken care with good intentions has failed morally, not just technically.
The Apprentice Model
The staged-competence pattern has a name and a lineage: the apprentice. A master machinist in Taichung knows from the sound of the spindle whether a tool is about to chatter; a ward nurse hears, at three in the morning, the difference between ordinary sleep-apnoea noise and something that warrants waking the doctor. This is tacit knowledge — Michael Polanyi's term for what we know but cannot fully tell. Civic AI enters such settings as a digital apprentice, not a replacement: it observes in shadow mode, records decision traces, and learns the tacit knowledge of this workshop, this ward, this community before it is trusted to suggest, let alone act. And the master keeps four things: the steering wheel (execution stays under human direction — a co-pilot that can override the operator has inverted the care relationship), the brake (one prominent control that stops the machine now, wired directly and tested regularly), the maintenance manual (traceability of which rules fired, which inputs guided the output, and who repairs the model when it fails), and the right to exit (portable interaction history, settings, and fine-tuned weights, so the care relationship is one you can leave).
What it looks like in practice
- Graduated release — the Apprentice Model. New policies run as apprentices: shadow mode first, then canary release to a stratified, representative slice (not a random one), then general rollout with rollback primed.
- Decision traces. Every denial, recommendation, or escalation has a trace: which rule, which sources, uncertainty score, and a receipt link.
- Guardrails as code. Rights and red lines expressed as machine-checkable rules (deny-by-default when ambiguous).
- Security as a care obligation. An agent with filesystem or network access runs in a strict sandbox with least-privilege permissions, validated inputs, and no implicit trust of upstream content. Least privilege is a care principle, not merely a security heuristic.
- Prompt injection, privilege escalation, and lateral movement are competence failures — moral responsibilities of those who build and deploy these systems, not mere technical oversights; they expose the people the system is meant to care for to harms its builders accepted when they chose not to prevent them.
- Working fallbacks. If confidence drops or a dependency fails, the system uses a reversible default, routes to a human, or pauses within the promised window.
- Data minimalism. Only what the remedy needs is collected; delete on handoff; consent honoured at every stage. Every byte of unnecessary retention is a promise of security the system has not made and may not be able to keep.
- Reproducible builds. Configs are versioned; one-click replays re-create results.
From ideas to practice
- Derive specs from contracts. Convert Pack 2 engagement contracts into acceptance tests.
- Instrument for observability. Emit decision traces with links to sources and receipts (from Pack 1).
- Run shadow mode. New policy sees inputs and proposes actions but doesn't act. Compare to human/previous system. Shadow mode is the apprentice watching over the master's shoulder.
- Canary safely. Release to a small, stratified, representative group with automatic rollback if drift exceeds bounds.
- Audit before general. Conduct independent audit of evals, logs, and guardrails; publish attested report.
- Generalise & monitor. Enable for all; watch drift monitors; keep pause wired.
- Post-incident learning. Maintain blameless reviews; fixes become tests.
Buildable tools
- Shadow/canary orchestrator with rollback switches.
- Decision trace schema. Inputs, rules fired, sources, uncertainties.
- Guardrail engine. Policy-as-code for rights/consents.
- Drift monitors. Data, performance, fairness.
- Eval registry. Versioned tests, provenance, and localised test suites.
- Replay tooling. One-click re-runs for audits, incidents, and appeals.
- Fallback router. Confidence thresholds that trigger human handoff or pause.
One case: the flood-bot
- Shadow → canary. A new "medical receipts waiver" runs in shadow for a week; then canaries to a stratified 10% slice of livelihood claims, with demographic and geographic balance enforced; rollback bound: appeals >15%.
- Observability. Every denial has a trace: which rule, which sources, uncertainty score, and a receipt link for the claimant.
- Safe fallback. When uploaded documents are unreadable or confidence drops, the bot uses a reversible default and routes the claim to a human caseworker rather than guessing.
What could go wrong
- Unsafe confidence. The system acts despite weak evidence. Fix: Confidence thresholds, fallback routing, and pause on ambiguity.
- Train/test leakage. Evals look good; reality fails. Fix: Hold-out datasets, randomised spot checks, live A/Bs with rollback.
- Opaque "black box." "Trust us" explanations. Fix: Traceable summaries + public examples; auditors can reconstruct decisions.
- Canary bias. Canary slice is unrepresentative. Fix: Stratify sampling; publish canary demographics. A canary that proves performance only on easy cases is not a canary; it is a ceremony.
- Security drift. Permissions expand quietly as the system is extended, turning a once-adequate least-privilege footprint into an attack surface. Fix: Regular permission audits; treat scope creep as a care failure, not an engineering convenience.
Interfaces
- From Responsibility (Pack 2): specs, SLAs, brakes.
- To Responsiveness (Pack 4): competence delivers; responsiveness checks whether it worked. Incident loops and eval results feed Pack 4. Competence without responsiveness is a system that checks itself; responsiveness without competence is a feedback process with nothing reliable to act on.
- To Solidarity (Pack 5): dependable, well-instrumented systems make cooperation and public audits credible.
- To Symbiosis (Pack 6): competence proves an agent is ready to stay local.
A closing image: the bridge with inspection tags
Imagine a well-kept bridge with inspection tags — date, load test, next check — visible to anyone crossing. Competence is not the absence of failure; it is the presence of proof that someone checked, and will check again. The deployment stages are the inspection schedule; the decision traces, the inspection records; the blameless post-mortems, the reports filed when something was found wrong.
