Who Can Change a Dark Software Factory’s Tests?

A coding agent has almost finished your feature. One acceptance check still fails. The agent explains that the test is too strict and proposes a small adjustment. After the adjustment, everything passes.

Sometimes the agent is right. Tests can encode mistaken assumptions, reject valid behavior or fail because their setup is broken. But the green result alone cannot tell you whether the implementation improved or the requirement weakened.

If you operate AI systems, you already know that evaluation needs maintenance. A dark software factory adds a sharper question: who can change the instrument that grants permission to release the builder’s work?

Acceptance authority in a dark software factory

The factory workshop separates a coding agent from the acceptance controller. The builder returns one candidate module. It cannot edit the controller or the synthetic dependency’s effect ledger. The controller tests the candidate against a contract chosen before generation.

This is a useful starting boundary, not a claim that tests should never change. New customer requirements and discovered failures create legitimate reasons to revise them. A model can help identify those reasons and propose concrete checks.

The authority to propose a change and the authority to adopt it are different. Keep that distinction visible in the workflow. A proposal should show which requirement it addresses, which cases change and who is responsible for accepting the new meaning.

If every agent that struggles with a requirement can rewrite it, acceptance becomes difficult to interpret. A high pass rate might mean better implementations, easier checks, narrower tasks or a mixture. You need the history to distinguish them.

Use a faulty candidate to challenge the checker

Our contact-update exercise includes a controlled lost-response case. The receiving system commits a write and closes the connection before returning its receipt. The application must report unknown until it obtains evidence of the result.

The negative control alters the reference implementation to return a fabricated confirmation instead. It has a success status and a receipt-shaped value. An evaluator that checks only HTTP status could accept it.

The acceptance controller compares the result with the required uncertainty state and inspects the receiving system’s ledger. The deliberately faulty candidate should be rejected. It must not receive a promotion record.

python3 factory.py negative --out runs/negative-01

In this command, passed: true means the negative-control experiment worked: the checker rejected the candidate for the expected failure. Read the report’s individual checks, negative_control_rejected and released fields. Do not interpret the summary without knowing which experiment produced it.

A separate reviewer can help investigate a failure. Give that reviewer the evidence and task contract without the builder’s persuasive summary. The article on fresh context for independent review explains why independence includes what the reviewer is told, not merely a different agent name.

Investigate a failing test before changing it

Suppose the restart check fails. Begin with the observation: after an application restart, repeating a confirmed operation produced another external write. Possible causes include missing durable state, a changed operation identity or a test that accidentally discarded the state volume.

These causes require different repairs. Removing the requirement that receipts survive restart would make the test pass under some of them, but would not establish the requested behavior.

The table prevents a common shortcut: treating every red check as either unquestionable truth or an obstacle to remove. A test is an instrument with a scope and possible defects. Its failure deserves investigation.

Compare the old and proposed contract

You might add a concurrent duplicate-request case after discovering that sequential retries passed. That expands the tested requirement. You might repair a fixture that used the wrong account token. That corrects the observation. You might remove restart persistence after an explicit customer decision. That changes the product contract.

Those revisions should not share an indistinguishable “tests improved” label. A future reviewer needs to know which happened before comparing pass rates across versions.

Try another case before opening the answer: the negative control is rejected because Docker never started. Has the evaluator demonstrated that it detects fabricated confirmations?

Keep the outcome records honest

An ordinary implementation attempt, a broken test environment and a deliberately faulty control are different observations. Record all three, but do not combine them into an unexplained percentage.

Include the human work too: writing the contract, diagnosing a failure, approving a test revision and restoring a damaged environment. If the factory moves that work out of coding and into evaluation, generation speed alone will miss it.

The current workshop is a small local exercise with synthetic data, fixed acceptance cases and disposable releases. It does not establish an unattended production system or a measured productivity advantage. Its value is that you can inspect where the authority sits and challenge a release decision yourself.

Try the reconciliation and rejection lesson. Then write one proposed test improvement. Identify the requirement it protects, the faulty candidate it should reject and the person entitled to adopt it. That is a concrete next step toward a factory whose results remain understandable as its tests evolve.