Testing Symphony on live work

The only interesting test for an agent coding tool is live work with real constraints.

Toy tasks are useful for learning the interface. They do not show whether the tool can survive a messy repo, narrow write scope, existing local changes, project-specific instructions, and a verification command that actually matters.

That is why I tested Symphony on live work instead of treating it as a demo surface. The question was not simply “can it write code?” Most modern coding agents can produce plausible code. The better question is whether the surrounding flow helps the operator scale work without losing control.

In my trial, the most useful observation was structural. Compared with a more manual GitHub Projects style flow, the work felt cleaner when the issue loop and execution path were more explicit. That does not mean a tool automatically solves orchestration. It means the evaluation should focus on the operating layer, not only the generation moment.

For agent coding tools, the hard problem is often operator scaling.

Can the system keep task structure clear? Can it route work without turning the workspace into noise? Can it preserve local changes it did not make? Can it show what passed, what failed, and what remains unknown? Can another operator inspect the result without reading an entire chat transcript?

Those questions are not exciting in a demo, but they decide whether a tool is useful in real delivery.

The minimum live-task evaluation should include a few constraints:

• a real repo with local rules;
• a scoped file list;
• existing work that must not be reverted;
• a source document or task record to follow;
• a verification command that can fail;
• a final report that separates evidence from confidence.

This kind of test exposes behavior that toy tasks hide. An agent may write good code but ignore repository instructions. It may pass a narrow check but leave unrelated files dirty. It may summarize a change convincingly without running the right command. It may over-edit because the task boundary was weak. It may continue patching after the evidence says the assumption is wrong.

Those are not minor workflow details. They are the difference between a useful delivery tool and a generator that creates review debt.

Useful tools help the operator see and steer the work. They make the current task explicit, attach execution to a source of truth, preserve nearby state, and leave behind a result that can be reviewed. They also make it easier to stop. If the agent is blocked, uncertain, or outside scope, the system should surface that instead of encouraging a longer and less inspectable run.

For me, Symphony is interesting as a test surface for this broader question: what does the coding tool do around the model?

The answer matters because the next level of AI-assisted software delivery is not only better code generation. It is better execution design:

• clearer task intake;
• stricter scope handling;
• stronger verification habits;
• cheaper handoff;
• more visible operator control.

A tool that wins on generation but leaves unresolved state still costs the team. A tool that leaves the operator with a clear task record, a bounded diff, and honest verification is much more valuable.

Live work is the test because live work has friction. That friction is not noise. It is the product requirement.

This is also why I do not treat tool comparisons as purely model comparisons. The model matters, but the surrounding workflow decides whether the operator can trust the result. A less dramatic generation moment can still be the stronger delivery experience if the system makes scope, checks, and handoff easier to manage.