Data and reporting
Measure the manual process first, or you will never know it worked
Most automation is built without a baseline. The work goes live, everyone agrees it feels faster, and nobody can say by how much — or whether the change did anything at all. This case has almost no tooling in it. It is a fixed sample, one row per run, and the same measurement taken again afterwards.
- Triggers
- Two
- Runs
- Two fixed windows
- Stack
- 2 tools
- Durable execution
- Not required
The flow
A loop, not an integration. Sample, record, compare, decide. The top track runs before anything is built; the bottom track runs after, against the same definition.
Someone proposes automating a task
Manual
Fix the sample in advance
Manual
One row per run
Sheets · Airtable
The change goes live
Manual
Measure again, the same way
Sheets · Airtable
Moved beyond ordinary variation?
Manual
Keep it, and record the number
Sheets · Airtable
Say so, and roll it back
Slack · WhatsApp
Step by step
- 01
Someone proposes automating a task
ManualWrite down one task and one number before anything else. "Handling supplier emails" is not a task; "minutes from email arriving to order entered" is a number.
- 02
Fix the sample in advance
ManualDecide the window before you start looking: the next thirty runs, or two full weeks. Deciding when to stop after you have seen the numbers is how a baseline stops being one.
- 03
One row per run
Sheets · AirtableStart time, end time, who did it, what the outcome was. Nothing clever. The value is that the rows exist and were written down as they happened, not recalled afterwards.
- 04
The change goes live
ManualThe automation, the new template, the extra person — whatever the change is. One change at a time, or the comparison cannot attribute anything.
- 05
Measure again, the same way
Sheets · AirtableSame fields, same window length, same definition of a run. Changing the definition between the two windows produces a difference that is entirely your own doing.
- 06
Moved beyond ordinary variation?
ManualCompare the two sets of rows against how much the number bounced around inside each window. A shift smaller than the week-to-week spread is not yet a result.
- 07
Keep it, and record the number
Sheets · AirtableThe recorded before-and-after becomes the baseline for the next change. This is the only step that compounds.
- 08
Say so, and roll it back
Slack · WhatsAppTell the people who use it that the change did not move the number, and remove it. Automation kept on faith is upkeep with no return.
What measurement will not tell you
A baseline is worth having and is still not proof. Two limits are worth stating plainly before anyone reads too much into the difference.
How many runs you need is not a fixed number: it depends on the spread of the process itself and on the margin of error you are willing to accept — the required sample size rises with the population standard deviation and falls as the acceptable margin widens. A noisy process needs more observations than a steady one before any difference means anything.
NIST/SEMATECH e-Handbook — Sample sizes requiredData taken over time "may have an internal structure (such as autocorrelation, trend or seasonal variation) that should be accounted for". A before-and-after comparison across two periods cannot on its own separate your change from the season the two periods fell in.
NIST/SEMATECH e-Handbook — Introduction to time series analysis
When this is not worth building
Three cases where the measuring costs more than the answer is worth.
The task runs a handful of times a month. You will not accumulate enough runs to tell a real change from an ordinary one before the decision has to be made anyway.
The decision is already made and funded. Measuring to justify a choice nobody will reverse produces a number written to agree with the plan, which is worse than no number.
Nobody can define what a run is. If two people would time the same piece of work differently, the two windows are measuring different things and the comparison is arithmetic on unlike quantities.