A Scorecard for Measuring Work an AI Removes

TL;DR

• Measure a task people genuinely complete, end to end, rather than a feature that appears in a demo.
• Define the quality gate before looking at any output, or the bar quietly moves to match whatever you got.
• Give every product a competent configuration. Most comparisons measure setup effort and then report it as capability.
• Keep failures visible in the arithmetic. Averaging away the cases where a tool produced nothing usable is how weak tools win evaluations.

A note appears after a meeting. That tells you the application produced something. It tells you very little about whether the rep's work is finished.

The rep may still need to correct the owner, find the latest account context, rewrite the email and put the result in the right system. An evaluation that stops at the first generated summary misses those steps.

Measure the complete task instead. Define what an acceptable result looks like, give each candidate a fair setup and record the work required to get there. You can use the method below for Magic Screen, another assistant or your current process.

Pick a task people actually complete

Start with a specific output and recipient. "Take better notes" is too vague to evaluate. "Produce a reviewed follow-up email and update the account's current blockers" is much easier to judge.

Write down the expected content before running the test. For a follow-up, that might include the buyer's stated priority, confirmed next steps, owners and dates where agreed. It should exclude internal negotiation notes and invented commitments.

For account preparation, define which history is available and what the next call needs. For live assistance, define the approved knowledge and the type of buyer question the rep should handle.

Use the same requirement for the existing workflow and every candidate. Do not ask one product for a complete account update while accepting a one-paragraph recap from another.

Define the quality gate before looking at the result

A fluent output can contain an important error. Research on generated meeting summaries has examined omission, irrelevant content and other error categories; the COLING study cited here was not a ranking of commercial meeting assistants. 1

Create a simple ground-truth sheet from the test conversation. Record the facts that must be preserved, the details that remain unknown and the statements that should not become commitments.

An output passes only after the reviewer accepts it against that sheet. Record the changes needed to reach acceptance. If it never becomes acceptable within the test's limit, record a failure rather than quietly removing that attempt from the results.

Set critical failures in advance. Examples include sending internal material to a customer, assigning work to the wrong person or writing to the wrong account. Keep those failures visible even when they are later corrected.

Choose cases that expose the routine and the exceptions

Use fictional or permissioned conversations. Include enough variation to exercise the actual job: a straightforward call, an unresolved decision, a changed deadline and a conversation with ambiguous ownership.

For an account-history test, supply multiple meetings in order. Include a superseded statement and a missing piece of context. The evaluator should know which facts are current without expecting the tool to infer information it never received.

For live help, include a relevant trigger, an irrelevant mention and a question the approved knowledge does not answer. A system that always displays something may look active without being useful.

A few scenarios can reveal a workflow problem. They do not establish a market-wide reliability rate or a statistically robust performance advantage. Treat a small pilot as a decision aid for the tested conditions.

Give every product a competent configuration

Record the plan, app version, operating system, meeting platform, capture mode and enabled features. Load the same approved source material where each product supports it.

Use the feature intended for the task. For example, Fireflies distinguishes bot-free Live Assist from its bot-dependent automatic Sales Assist workflow. Fathom documents different capture choices in its newer experience. Comparing a disabled or irrelevant mode would answer the wrong question. 2 3

Magic Screen's setup includes desktop permissions, prepared knowledge and selecting or attaching the appropriate resources and templates. Count that work rather than presenting the first session as configuration-free. 4 5

Give evaluators basic training in each candidate. An expert in the incumbent tool and a first-time user of the alternative do not provide a clean comparison of routine use. Preserve setup observations, but distinguish learning from the steady-state task.

Record five kinds of effort

Use this blank scorecard for each task and product. It contains no observed results.

For each stage, log active human minutes and the actions performed. Record processing delays in a separate elapsed-time field.

A thirty-second wait is different from thirty seconds of active editing. It can still matter when someone needs an answer during a call, but the distinction should remain visible.

Also note recurring maintenance. A saved prompt may remove repeated instructions while requiring occasional updates. An account document may reduce reconstruction while requiring association checks. Assign that work to its actual owner.

Count attention without pretending every click is equivalent

A single action can interrupt a conversation more than several actions after the call. Keep an event log for live use: what the seller did, why they did it and whether it interrupted the exchange.

Ask the seller and observer to record useful or distracting moments using a consistent rubric. Treat those assessments as subjective observations, not objective elapsed-time measurements.

Do not convert every notification into an invented monetary cost. A helpful cue and an irrelevant alert have different consequences. The log should preserve enough context to distinguish them.

Where possible, counterbalance the order of tools across evaluators or use comparable fresh cases. Replaying the same script repeatedly can make later attempts easier because the participant already knows the answer.

Use calculations that keep failures visible

Calculate recurring effort only after deciding what belongs in the numerator and denominator.

Human effort per accepted output
= Total active human minutes for all attempts / Accepted outputs

Acceptance rate
= Accepted outputs / Attempted outputs

Recurring effort difference
= Baseline minutes per accepted output
 - Candidate minutes per accepted output

Include time spent on failed attempts, corrections and retries. If no output is accepted, the first measure is not calculable; report "No accepted output" rather than zero minutes.

Show the acceptance rate beside the effort figure. A product that completes only the easy cases may appear fast if failures disappear from the analysis.

For larger pilots, report the distribution as well as the average. Keep raw task-level observations so a reviewer can inspect outliers, different call types and changes after configuration.

A hypothetical break-even example

Assume three fictional tasks all pass the same quality check. The baseline takes 39 active minutes in total. The candidate takes 21 active minutes, including review and corrections.

The baseline average is 13 minutes per accepted task. The candidate average is 7. The illustrative difference is 6 minutes per accepted task.

Now assume the candidate requires 60 additional minutes of setup. Under those assumptions, ten further accepted tasks would offset the extra setup time:

60 additional setup minutes / 6 minutes saved per task = 10 tasks

These are invented numbers explaining the calculation. They are not results for Magic Screen or another product. The example excludes subscription cost and recurring maintenance, and assumes the task mix and effort difference remain stable.

A commercial evaluation should add those costs separately using the total-cost model. Released capacity also needs to be distinguished from a reduction in cash spending.

Apply the scorecard to a Magic Screen workflow

Start with a reusable note template. Record the setup, attach it to the intended meeting and review the resulting output. Count how much restructuring remains compared with the existing routine. 6 5

Then test account continuity separately. Create a Living Note for an account, associate the intended meetings and inspect the update after a changed decision. Count linking and correction work along with any reconstruction the persistent document removes. 7

For live resources, choose a specific approved response and test whether the rep can use it in context. Record resource preparation and the rep's actual interaction. Do not infer improved revenue from a successful prompt appearing on screen.

These tests answer different questions. Keep their results separate before deciding which matters most to the team.

Publish enough detail for someone else to disagree

A useful evaluation report includes its scenarios, requirements, source material, configurations, task order, quality rules and failed attempts. It explains what was observed and what remains untested.

Do not combine privacy approval, correctness, speed and subjective preference into one opaque score. Set mandatory requirements first. Then compare the effort and cost of the candidates that meet them.

Keep the scorecard after the purchase. Repeat a small version when the product, team process or required output changes. The purpose is to know whether the assistant continues removing work the team would otherwise have to do.

Frequently asked questions

How do I compare AI meeting assistants objectively?

Score a task people actually complete, end to end, against a quality gate defined before you look at any output.

What makes most tool comparisons unreliable?

Uneven configuration and a moving quality bar. Most comparisons measure setup effort and then report it as capability.

How should failures be counted?

Visibly. Averaging away the cases where a tool produced nothing usable is the single most common way a weak tool wins an evaluation.

Sources

1. Kirstein, Ruas and Gipp / COLING. What's Wrong? Refining Meeting Summaries with LLM Feedback. January 2025. Checked September 10, 2026.
2. Fireflies. Live Assist / Sales Assist desktop guide. Checked September 10, 2026.
3. Fathom. Fathom 3.0. Checked September 10, 2026.
4. Magic Screen. Installing the Desktop App. Checked September 10, 2026.
5. Magic Screen. Creating Note Templates. Checked September 10, 2026.
6. Magic Screen. Notes and Templates Overview. Checked September 10, 2026.
7. Magic Screen. Living Notes. Checked September 10, 2026.