Cloud or Local for Voice AI? Here's Our Benchmark.

Earmark now runs on local models. The AI that turns your meeting into notes, action items, and follow-up documents can run entirely on your laptop. Audio goes in, notes come out, and nothing leaves the machine.

We've been asked for this for a while. Security teams at larger customers want it, people who work on anything under NDA want it, and some folks just don't like the idea of their meetings sitting on someone else's server. That's a reasonable thing to want.

But "local" has a reputation. The assumption is that it's the compromise option: fine for privacy, worse at everything else. We didn't want to ship it on vibes, so we ran the same meetings through local and cloud models and measured the difference. This post is the full version of what we found. The short version is that the reputation is about a year out of date.

Why we ran this

Meeting notes are only useful if you can act on them.

After a meeting, people use the notes to decide what happens next: who owns a task, which deadlines are firm, which plans are still proposals. A summary that gets one of those wrong doesn't just read badly, it sends work in the wrong direction. Someone builds the wrong thing, or nobody builds the right thing because the summary said it was still being discussed.

So we weren't interested in which model writes the prettiest paragraph. We asked four questions:

1. How good are the notes? Overall quality, scored out of 100.

2. How often are they wrong? Specifically, errors that could change a decision.

3. How long do they take? Median generation time.

4. Did it hear the right words? Transcription accuracy on real meeting audio.

How we tested

We took 30 real meeting transcripts and ran each one through four setups. Two cloud models: GPT 6.1 Sol, which is what Earmark uses in the cloud today, and GPT-4.1, a widely-used model from the previous generation. Two local models: Gemma 4 12B and Gemma 4 E4B, both running on-device with thinking enabled. For each model we used the best configuration we tested, so this is a comparison of each one at its best, not a tie-your-hands-behind-your-back test.

Every output was reviewed twice by an AI judge against a 100-point rubric: 40 points for factual quality, 30 for including what you actually asked for, 15 for organization, 15 for writing clarity. Where the two reviews disagreed on error counts, we report the range rather than picking the friendlier number.

One honest caveat up front: these are AI-judged scores, not human-verified percentages. They're provisional judgments. We think they're directionally right because we read a lot of the outputs ourselves, and the examples below are the ones that stuck with us.

1. How good are the notes?

GPT 6.1 Sol scored 98.2 out of 100. Nothing else was close, and that's the result we expected.

The result we didn't expect is the second line. Gemma 4 12B, running on a laptop, scored 84.1. GPT-4.1, running in OpenAI's cloud, scored 77.8. The smaller Gemma 4 E4B landed at 78.6, which is also above GPT-4.1.

Read that again, because it reframes the whole question. A local model you can run on a MacBook now produces better meeting notes than a cloud model that most of the industry was using as a default a year ago. Local isn't the budget tier anymore. It's roughly last year's cloud, running on your own hardware.

The breakdown shows where the gap actually lives. GPT 6.1 Sol got a perfect 40/40 on factual quality and perfect marks on organization and clarity. The local models aren't bad at organizing or writing. Their deficit is almost entirely in factual quality (34.5 and 32.9 vs 40.0) and in "includes what you asked for" (21.2 and 19.3 vs 28.2). In plain terms: the local notes are well-structured and readable, but they miss more of what was said and get more of it slightly wrong.

2. How often are they wrong?

This is the number I care most about. Not "did it score well," but "how many of these 30 summaries contained an error that could change a decision?"

GPT 6.1 Sol: zero out of 30. Across two independent reviews, not a single output had a material or critical error.

Gemma 4 12B: 9 to 11 out of 30. Gemma 4 E4B: 12 to 14. GPT-4.1: 14 to 16.

Two things to take from that. First, the gap between the best cloud model and everything else is bigger on errors than on score. A 98 vs 84 looks like a modest difference. Zero errors vs ten errors in thirty meetings is not modest. If you're using notes to make decisions, that's roughly one in three summaries you'd want to double-check against the recording.

Second, local again beats the older cloud model. Gemma 4 12B made fewer decision-changing errors than GPT-4.1.

None of the four produced what we classified as a critical error, the kind that would be actively dangerous to act on. The errors were material: wrong but plausible.

What these errors look like

Numbers are abstract, so here are two examples from the actual outputs.

GPT 6.1 Sol, keeping a proposal a proposal. In one meeting, the team had a firm pre-launch target for active shipping routes and a proposed two-week deadline for everything else. GPT 6.1 Sol preserved that distinction. The reader could see which plans still needed confirmation. That's exactly the kind of nuance that most summaries flatten, and flattening it is how a proposal becomes a commitment nobody agreed to.

Gemma 4 12B, giving the task to the wrong person. In another meeting, someone was assigned to own the daily update, with a named backup. One Gemma 4 12B summary put the wrong teammate in charge. Everything else in the summary was fine. But if you followed it, you'd send the task to the wrong person, and the right person would assume it was handled.

That second example is why we don't treat "84 out of 100" as "84% as good." The 16 points you lose aren't spread evenly. They show up as specific, confident, wrong facts.

3. How long do they take?

Here's where local pays for itself, in the other direction.

GPT-4.1 returned notes in a median of 7.7 seconds. GPT 6.1 Sol took 14.4. Gemma 4 E4B took 48.4 seconds. Gemma 4 12B, the local model with the best quality, took 107.3 seconds.

So the local model that competes on quality is about seven times slower than the cloud model it beats, and more than that compared to the cloud model it loses to. These timings exclude model loading, were measured in separate sessions, and depend on your hardware and network, so treat them as rough.

Whether that matters depends entirely on when you need the notes. If you want them before you've walked to your next meeting, a 107-second wait is a real cost. If a meeting is processed in the background while you're doing something else, or overnight, it isn't a cost at all.

4. Did it hear the right words?

Everything above started from a written transcript. In a real meeting, audio has to become text first, and if the transcription is wrong, no summarizer can fix it. So we tested transcription separately.

We measured word error rate on 1,579 reference words across five real meeting clips, about ten minutes of audio, with no dictionary boosting. We used an adjusted WER that ignores declared fillers, repetitions, and formatting, because "um" and "the the" don't change the meaning of a meeting. We kept conventional WER in our evidence archive too.

AssemblyAI 3.6 Pro, a cloud service, came in at 3.0%. Nemotron English, running locally, was 3.5%. Parakeet (local) was 4.0%, Nemotron 3.5 was 4.6%, and Moonshine v2 was 7.4%.

A half-point gap between the best cloud and best local transcription is close to nothing. For transcription, local is essentially at parity.

One detail worth knowing: technical terms still count. In one clip, someone said "FedRAMP," a security certification. AssemblyAI kept it as FedRAMP. Nemotron English wrote "feed ramp." That's a wrong term, so we counted it, even with our forgiving scoring. If your meetings are full of product names, acronyms, and customer names, that's the failure mode to watch for with local transcription, and it's where a custom dictionary earns its keep.

What this means for choosing

Here's how I'd summarize the tradeoff.

Cloud (GPT 6.1 Sol) gets you the best notes we've ever measured, with zero decision-changing errors in 30 meetings, in about 15 seconds. If your organization is comfortable with meeting audio and transcripts going to a cloud model, this is the better product and it isn't close.

Local (Gemma 4 12B) gets you notes that are better than a leading cloud model from a year ago, with about one in three summaries containing something you'd want to verify, in about two minutes. Transcription is nearly as good as cloud. Nothing leaves your device.

The honest framing isn't "cloud is better" or "local is good enough." It's that these are different products for different constraints. If audio can't leave the building, local is now a real option rather than a consolation prize. If it can, cloud is still meaningfully better, and the gap shows up exactly where it matters most: in the facts.

We'll keep running this benchmark as models change. The local side of this chart has moved a long way in twelve months, and we don't expect it to stop.

Methodology notes: 30 meeting outputs per model, each scored by two independent AI reviews on a 100-point rubric. Error counts are means across the two reviews, not verified error rates. Transcription WER is adjusted to exclude declared fillers, repeats, and formatting; reference transcripts were reviewed after exposure to model outputs. Generation times are medians, exclude model loading, and were measured in separate sessions.