Testing & Experiments

SEO Tests

Metricstab's SEO Tests let you measure whether an on-page change actually moved the needle. You define a deployment date, a URL pattern, and two comparison windows (before and after). Metricstab pulls clicks, impressions, CTR, and average position from your existing Google Search Console data, runs statistical tests, and tells you whether the difference is real or noise. There are two test types: a time-based test compares the changed pages against their own past; a split test also holds out a control group of pages you leave untouched, so site-wide movement can be subtracted out.

How the windows work

Every test has three key dates:

  • Before window: the chosen number of days ending the day before your change went live.
  • Change date: the day your change went live; the after window starts here.
  • After window: the chosen number of days from the change date.

The default is 28 days on each side. Longer windows give more statistical power at the cost of waiting longer for results. The minimum recommended window is 14 days; fewer than 7 days rarely produces meaningful results.

Backdated tests. The deployment date can be any day in the past, so you can test a change you made months ago, as far back as the Search Console history Metricstab holds for the site. Pages are matched among those with data in the last 90 days, or since the start of the test's before-window if that is earlier — so a page that was later removed or redirected is still included.

Google Search Console data has a 2–3 day lag. If your deployment date is today, post-window metrics will not appear in Metricstab for 2–3 days.

Two test types

Time-based (before / after)

The changed pages are compared against their own pre-window. This is the simplest test and works with any URL set, but it has no control: seasonality, a core update, or a competitor's move over the same days is indistinguishable from your change. Treat a time-based result as "something moved after the change", not as proof of cause.

URL change / redirect

Moved a page to a new address and redirected the old one? Choose URL change / redirect and enter the old and the new URL. Metricstab treats the two as one page: each day's clicks and impressions are added together, so the old URL's history before the change is compared with the new URL's performance after it, and traffic that is still reported on the old URL during the move is not lost. It is a before/after test — the redirect moves the page, so there is no unchanged copy to use as a control.

Split test (vs control group)

You additionally scope a control pattern — a pool of comparable pages you will not touch for the duration of the test. Metricstab draws the control group from that pool automatically: candidates are stratified by pre-period traffic and by site section, then randomised within each stratum (seeded from the test id, so a recompute cannot quietly reshuffle the group). The result is a difference-in-differences estimate: (test post − test pre) − (control post − control pre). Whatever moved the whole site moved both groups, so it cancels out.

Before claiming significance, Metricstab checks parallel trends — that the two groups were tracking each other before the change. If they were already drifting apart, the groups weren't comparable and no verdict is given; the numbers are still shown, but no significance is claimed from them.

Reading a split test's results. Each metric card leads with the estimate against the controls — for example +9.4% vs control — because that is the number the significance check is about. Underneath, each group's own before/after change is shown (Test pages +12% · Control pages +2.6%). The chart draws both groups as a percentage of their own pre-change average, so a small test group and a large control group share one scale, and a second chart shows the daily gap between them with its pre-change average as a dashed line: the test asks whether that gap moved after the change. The LLM Sessions metric, when shown, is not compared with the control pages and is labelled as a plain before/after.

Before you start: "Ready to test?"

Preview URLs on the create form checks the pages you picked against the days before your change. It tells you roughly the smallest change in clicks your test could detect — for example ±14% over 28 days — and how many days you would need to detect a 10% change. Pages with few clicks or very uneven days need longer tests. For a split test it also shows an example control group, how closely it moved with your pages, and whether the two were already drifting apart. The real control group is drawn the same way when you save the test, so its pages will differ from the example. The estimate assumes each day is independent, which real traffic rarely is, so treat it as a rough guide rather than a guarantee.

A split test is a quasi-experiment, not a randomised A/B test in the web-analytics sense. Google does not let you show two versions of one URL to different searchers, so you choose which pages get the change; only the control side is randomised. It is a large step up from before/after — but if the pages you chose to change differ systematically from the pool, that difference is still in the result.

The statistical tests

For time-based tests Metricstab runs two tests on each metric and reports both p-values. For split tests, Welch's t-test is run on the daily gap between the test and control groups (pre-window vs post-window), which yields the difference-in-differences estimate and its p-value directly.

Welch's t-test

Compares the mean daily value in the pre-window against the mean in the post-window. Welch's version (unlike Student's t) does not assume equal variance — appropriate since SEO metrics often change seasonally or due to indexing spikes. The result tells you whether the average daily performance changed.

Mann-Whitney U test

A non-parametric alternative that compares the distributions of the two windows without assuming normality. It is not thrown off by one unusual day (e.g. a viral news spike). Metricstab uses a normal approximation for the U statistic, which is accurate when each window has at least 8 data points.

Interpreting p-values

  • p < 0.05 — result is statistically significant at 95% confidence. The observed difference is unlikely to be random noise.
  • p ≥ 0.05 — not significant yet. This could mean: the post-window is too short, the effect size is small, or there was genuinely no change.
  • p < 0.001 — very strong evidence of a real change.

The 95% range

Every result also shows a 95% range, for example +8.0% to +21.0%: the effect is likely somewhere in it. It comes from the same Welch test as the p-value, so the two always agree — a result is significant exactly when its whole range sits on one side of zero. A wide range that crosses zero means the test could not tell the change apart from noise, not that the change did nothing. For time-based tests the range is a percentage of the pre-change daily average; for split tests it is in the same units as the vs control estimate. Mann-Whitney U has no range; it is shown as a cross-check only.

Statistical significance does not mean business significance. A CTR change from 2.0% to 2.01% might be significant but is not worth acting on. Always evaluate the effect size (the Δ% column) alongside the p-value.

The four scenarios

Significant win

p < 0.05 and the metric moved in the desired direction (e.g. CTR increased, position improved). Your change worked. What to do: roll out to similar pages and document the win in the Site Changelog.

Significant loss

p < 0.05 and the metric moved in the wrong direction. Your change hurt performance. What to do: revert the change, check for contamination (did something else change?), and revisit the hypothesis.

Not yet significant

p ≥ 0.05. The post-window may not be complete, or the effect is small. What to do: wait for the post-window to finish. If still n.s. after a full window, the change likely had no measurable impact on GSC metrics. Consider testing a larger change.

Contaminated

The simple crawler detected that title, meta description, H1, or canonical changed on one or more URLs after the test started. What to do: treat results with caution — you cannot isolate which change caused the metric movement. Consider archiving the test, reverting to a known baseline, and re-running.

Page snapshots & contamination detection

Every night Metricstab crawls the pages of each running test and stores a page snapshot: title, meta description, first H1, canonical URL, and HTTP status code. A new test gets its first snapshot on the next nightly run.

Each snapshot records the page's title, meta description, main heading and canonical link. If any of them changes between crawls, the page is flagged as contaminated and a warning banner appears on the test detail page.

To keep the load on your site low, the crawl is limited: up to 200 test pages per test (always the same 200, so each page is compared with its own earlier snapshot), split-test control pages at most once a week, a page that sits in several tests fetched once, about one request per second, and robots.txt is obeyed. The test page tells you how many of its pages were checked; a page we did not fetch cannot show contamination. What our crawler fetches.

Has Google seen the change yet? Google keeps showing the old version of a page until it re-crawls it, so a test can look like "no change" simply because Google hasn't visited yet. Once a day Metricstab asks Search Console's URL Inspection tool when Google last crawled each test page (the first 50 test pages per test; control pages aren't needed), and the test page shows "Google has re-crawled N of M checked test pages since …". Google limits URL Inspection to about 2,000 checks per property per day; Metricstab uses at most 1,500 and stops checking a page once it has been re-crawled. It needs the Search Console connection you already gave Metricstab.

If no baseline snapshot exists (i.e. the crawler had not run before your deployment date), contamination detection is unavailable for that test. The results are still computed from GSC data, but you will not receive hash-change alerts.

Caveats & limitations

  • GSC lag: Data is typically available 2–3 days after the date. Results for the last 3 days of the post-window will appear with a delay.
  • Time-based tests have no control group: they compare the same URLs before and after, so seasonality, algorithm updates and competitor changes can be mistaken for your change. Use a split test where you can, and record other concurrent changes in the Site Changelog either way.
  • Split tests randomise the control side only: you choose which pages receive the change, so the design is a quasi-experiment rather than a randomised A/B test. The parallel-trends check catches groups that were never comparable, but it cannot catch every selection effect. A control page that changes mid-test contaminates the result and is flagged.
  • Query-level data not included: Tests compare page-level aggregates. Individual query movements are not broken out.
  • Maximum 500 URLs per test: URL sets larger than 500 are capped. For very large sites, use a narrower URL pattern or split into multiple tests.
  • Position is position-weighted average: Avg. position from GSC is weighted by impressions per query per day. It may differ from a simple median.

Test lifecycle & statuses

Every test moves through a simple lifecycle. You control transitions manually from the test detail page.

Status What it means Nightly automation Typical next step
Draft Test is saved but not yet active. No crawls or metric recomputes run. Excluded Click Activate when you're ready to start tracking.
Running Test is live. Pages are crawled every night and the results recomputed every evening. Included Wait for the post-window to elapse, then read results.
Concluded Automatically set by the system when the full post-window has elapsed. Results are final. Excluded Review findings; archive when done or reactivate to extend the window.
Archived Retired test. All historical results and URL snapshots are preserved for reference but the test is dormant. Excluded No action needed. Use Reactivate to resume tracking if required.

Draft vs Archived: Draft = not yet started (typically no results). Archived = finished or retired (results are preserved). Neither counts toward the "Running Tests" badge in the navigation.

Verdicts & tags

A concluded test gets one verdict, decided on clicks — the metric that matters most. The other metrics are shown as context but don't change the verdict, so a significant rise in impressions can't make a test a "winner" while clicks stayed flat. The same verdict appears on the test list, the test page and the share card.

  • Winner — clicks rose and the whole 95% range is above zero (for a split test: rose compared with the control pages).
  • Loser — clicks fell and the whole range is below zero.
  • No change — the range crosses zero: no detectable effect over this window. That is not proof the change did nothing.
  • No verdict — split test whose groups were already drifting apart before the change, so they couldn't be compared.
  • Not enough data — no clicks result could be computed yet.

Each test page also opens with a short In plain English summary: what was changed and on how many pages, the clicks verdict with its range, what impressions and position did, and any caveat that applies (still running, pages changed again mid-test). It is written by fixed rules from the numbers on the page, not by AI, and it never claims what caused a change: a time-based summary always reminds you that other things happening in the same weeks are mixed into the result.

Testing overview. The top of the test list sums up your testing: how many tests have finished, the win rate (winners divided by finished tests that reached a verdict), how many changes you tested each month, and a deliberately cautious "at least N extra clicks a day" from winning tests — for each winner, the low end of its 95% range times its pages' daily clicks before the change. Time-based winners are not adjusted for seasonality, so read that figure as a rough lower estimate, not a guarantee.

When a test finishes — the night its post-window has fully passed — the site owner and site members get one email and one notification in the bell menu with the verdict and the plain-English summary. Turn these off under Account → Notifications → Threshold & test alerts. No email is sent for a backdated test that was already over when you created it, or when you mark a test concluded yourself.

Tags are free-text labels (comma-separated) you can add when creating or editing a test — for example a client name or a quarter. Click a tag to filter the list.

Related features

  • Robots.txt Monitor — detect accidental crawl blocks that could invalidate your test.
  • Site Changelog — record other changes that happened during your test window to document confounders.
  • Winners & Losers — see which pages moved most in the same period to spot correlated changes.