Code Engineered

Claude Opus 5 vs Qwen3.8

I ran an experiment. Have Qwen3.8 and Claude Opus 5 try to optimize a codebase for performance. I wanted to see which was better and how far the gap was between them.

These models are very different from one another. Opus 5 is one of the top frontier models. Qwen3.8 is a model I could run locally, in my case on a Framework Desktop. For the harnesses, I used the claude CLI and opencode (for Qwen3.8).

Going in I knew Opus was the better model. This wasn’t about finding the winner. It was about seeing how far local models have come and what I could squeeze out of Qwen3.8.

The Experiments

I did numerous different runs that included:

  1. Ask Claude to “one shot” improve performance of the Go codebase.
  2. Ask Qwen3.8 with no skills to “one shot” improve performance.
  3. Ask Claude Opus to create a plan to improve performance. Then have both Opus and Qwen3.8 implement the plan.
  4. Ask Qwen3.8 (with and without Go skills) to create a plan to improve performance where both Open and Qwen3.8 would implement the plan.

These were tests on usefulness rather than trying to be scientific.

The Results

Before and after change benchmarks

Analyzing the results was both manual and using LLMs. Some things were straight forward as there were benchmarks that could be run before and after any changes. The allocations and time to run a function makes it easy to measure this kind of change.

The ordering from best to worst results:

  1. Claude Opus creating a plan and implementing the plan.
  2. Claude Opus creating a plan and opencode (including Go skills) with Qwen3.8 implementing the plan.
  3. Claude Opus “one shot” improving performance
  4. Everything where Qwen3.8 had to figure out how to improve performance.

The big surprise was the Qwen3.8 implementing a plan created by Opus was better than Opus “one shotting” (i.e. a single prompt to improve performance) a change. This was my position, the position of AIs I asked about it, and the position of Opus when it reviewed the various changes.

When Qwen3.8 was reviewing the codebase and planning changes it missed numerous obvious candidates for improvements. It just didn’t know what to look for.

Useful local AI keeps looking more promising. Not everything needs to be done by the most capable models.