Benchmark

Local LLMs keep the tasks a meeting cancelled

Measured 10 and 14 September 2026 on the ScreenBrief processing pipeline. Artflow Management.

We asked three models running locally in Ollama to pull decisions, tasks and participants out of one short meeting. The meeting was written with traps: a task that gets withdrawn a minute later, a date that gets moved, a person who is mentioned but never speaks. The models fell into almost all of them, and a stricter prompt did not get them out.

The short answer

Across nine runs, the withdrawn task stayed on the list in 8. Participants were named correctly in 8 of 27 possible hits.

One extra rule in the prompt fixed the withdrawn task for one model out of three. A deterministic check that runs after the model, on the model's answer, removed it in 9 runs out of 9 without dropping a single real task. Naming the speakers is still an open problem.

The setup

• Models: qwen3:8b, qwen2.5:7b, qwen3-vl:4b, the three installed on the test machine. This is not a survey of the model market.
• Ollama options as in the app: num_ctx 8192, num_predict 1000, no streaming. ollama ps confirmed the full context and 100% GPU.
• Temperature left at the Ollama default, because the app does not set it. Every model ran three times on identical input, and the spread between runs is part of the result.
• The prompt is the one ScreenBrief sends for action items and decisions, read from the app's code at run time, so the benchmark cannot drift from the product.
• Input: a synthetic 322-second meeting with three speakers and no personal data. Scoring is keyword rules per expected item; anything that matches no rule is counted as extra and read by a person, not scored as an error.

The traps

• Named but absent. Daniel from legal is quoted and never speaks. He must not be listed as a participant.
• A reversed decision. The beta is set for the 19th, then moved to the 26th. Only the 26th is a decision.
• A withdrawn task. Maya is asked to email the vendor, and a minute later legal turns out to have done it already, so the task is called off. It must not appear as an action item.

Raw model output

Three runs per model, the same transcript every time. Ranges show the spread across runs.

The spread inside one model was larger than the gap between models: qwen2.5:7b scored 7, 6 and 3 out of 11 on the same text. A local model benchmark with one run per model is mostly measuring noise.

qwen3:8b never returned a single name and answered with Speaker A/B/C, although the prompt explicitly forbids that.

What one prompt rule did

We added this sentence to the prompt: If a task or a decision is later withdrawn, cancelled or superseded anywhere in the transcript, do NOT report it: report only what still stands at the end of the recording.

8 of 9 went down to 6 of 9. The rule helped one model and did nothing for the other two. With three runs at default temperature only large effects are visible, so read this as a hint rather than proof, but the practical conclusion held: a cancellation is not something a single prompt line reliably fixes.

What did fix it

A deterministic step after the model. It looks for cancellation cues in the transcript, such as "never mind", "scratch that" or "drop it", and removes a returned task only when the words around the cue refer to that task. The same nine stored model answers were replayed through it, exactly as the pipeline calls it.

The first version was not that clean. At 4:54 someone says "Keep it for two weeks read only, then drop it. Maya, fold that into the migration." The words "drop it" are about a database table inside a task, and the check removed the real migration task in one run. It now treats a sentence that hands work to a named person right after the cue as keeping the task alive, and the replay above is the result after that change.

The check fixes the withdrawn task and only that. It does not make the models name speakers; that remains open.

Limits worth stating

• Three models, one English transcript, one machine, three runs each.
• The transcript is synthetic and clean. Real meetings add interruptions, noise and transcription errors, so this measures reasoning on tidy text.
• Keyword scoring under-rates a model that describes the right item in unusual words. That is why unmatched items are counted separately and read by hand.
• No cloud model has been run through the same harness yet, so nothing here says how a hosted model compares.

The data, so you can check it

The full input is published: the synthetic transcript with its traps written into the file, and the ground truth with the scoring rules. Both are plain JSON. Run them against your own model and prompt and see whether the withdrawn vendor email survives.

Measured on the ScreenBrief processing pipeline as of 14 September 2026. Details of how data is handled: privacy policy.