Observability is a word from IT Land. We don’t usually hear about it much in the world of SEO and AI search, although we should and I think we’re going to hear about it a lot more.
As we’ve been discussing recently, the landscape of search data is changing in many ways, but one key change is that it is becoming an increasingly volatile data set to work with.
This volatility is, in our view, directly correlated with the value it provides AI and data teams. It’s a scarce resource, it’s hard to get at scale, and it provides a look at the forces that reflect, influence, and shape consumer behavior on a scale that cannot be found elsewhere.
It is also our data, like yours and mine. This goes back to our belief that if there is data about you or something that affects you, then it is your data. Same thing goes for your websites.
The key players who claim to “own” this data thrive on information asymmetry and, as a result, aren’t particularly inclined to make it easy to get.
Hence the volatility.
As this volatility has increased, especially over the last year, it has become increasingly important to build data about this data to answer questions like: what is the error rate, sampling rate, number of pages being fetched, and so forth, on a daily basis.
And the shape of these questions and answers differs depending on the engine in question. If we’re talking about Google SERPs, then it has one set of characteristics, and data from ChatGPT has another. And so on.
To date, as far as we are aware, there isn’t much out there from vendors such as DemandSphere (or anyone in our market) providing answers to these questions as a part of their offering.
We feel that it is time to change that and, as a result, we are announcing a new feature within our platform called Observability. We’ve been working on this for a few months. It’s not what we would consider “done” as it will be evolving every time there is a new big change and there is still a lot more that we want to do with it.
First, let’s establish a formal definition of what Observability is in the IT world.
What is Observability in IT and data?
The goal of Observability is to understand a system’s internal state and the key characteristics that shape how data within the system manifests itself. All data has a shape. Observability describes that shape and the forces that act upon it. Traditionally, there are a number of key aspects to Observability, including:
- Logs: the raw data, typically row-level records of individual events
- Metrics: aggregated measurements taken over time, such as counts, rates, and distributions
- Traces: the end-to-end record of a single request as it moves through the system, assembled from timed spans, so you can follow one operation across every stage it passed through
Traces are the one that needs translating for our world. In a web application, a trace follows a single request across services and shows you where the time went or where it broke. In search data collection, the equivalent is following a single fetch: which region and route it was requested from, when it ran, what came back, how it was parsed, which imputation steps touched it, and how all of that became the position number sitting on a chart.
That is the part that matters when a number looks wrong. Without it you are staring at an aggregate and guessing. With it you can take one suspicious data point, walk it back to the collection event that produced it, and find out whether the SERP moved or the pipeline did.
Observability vs. Monitoring
They may sound similar, but Observability and Monitoring are two different things. Traditionally Monitoring focuses on predefined alerts and looking for certain patterns. Observability is designed to provide more ad hoc levels of flexibility to analyze the data.
In the context of SEO and AI search and, specifically for our feature, we are adding an additional facet to this definition: first-order vs. second-order.
First-order is the data that everybody pays for: SERP data, LLM data, etc.
Second-order is the data about this data. Again, this will be data points like error rate, sampling rate, etc.
In a perfect world, there wouldn’t be a need for much second-order data about the first-order data but, as of October 2026, we’re in a new world.
We can confidently predict that we are entering a new era of volatility and that it will be key for customers to understand what is happening within their data systems on a regular basis.
This does add a large new burden on platform vendors that did not exist before and it is something that will take time to build up. But we’ve decided to do it and we’re announcing this new feature set today.
Observability in DemandSphere
There have been a number of volatile changes with data, particularly from Google, over the last year. We log the confirmed ones on the Google Algorithm and AI Search Update Tracker.
The newest view in Observability covers the problem we wrote about in how we detect poisoned SERPs: result pages that come back looking complete but with the usual results swapped out. This panel shows how many delivered ranks were read from a SERP the fetcher flagged as poisoned, after re-fetching.
The scale is deliberate. Plotted against the full 0 to 100 range, the poisoned share is small, and it should be. The point of measuring it is to know that, and to know the day it stops being true.
One that we haven’t had a chance to talk about too much publicly yet, and it hasn’t affected everybody, is what we call “drop-and-pops” (“DnPs” for short).
A drop-and-pop occurs when Google temporarily tanks a given domain’s rankings for a few days and then brings them back to where they were. This is something that has occurred, to varying degrees, for years. But clients started to notice an increase in these in May. We needed a way to show the extent, impact, and volatility levels of these events. We also needed to be able to show when some percentage of results happened to recover.
We set a 7-day window for this, on a rolling basis, and have developed our first version of Observability around this phenomenon.
Here are some views from REI (not yet a client but we’d love to talk to you), a domain we use a lot for demos.
End-state by tier, with the drop-and-pop episode counts alongside:
Two numbers above look like they disagree and do not. 64.8% of Top 1-3 keywords had at least one drop-and-pop across the period; on any given day only about 1.3% are sitting inside one. Episodes are short and spread out, so a tier can be almost entirely stable at any moment and still have most of its keywords affected at some point. The chart below is the daily view.
Trended view:
Volume-weighted trends:
Position trends by tier, which show whether damage is concentrated or spread evenly:
An additional view on confirmed lost keywords (DnP did not “pop” again within 7 days):
We have a similar one for recovered keywords.
We are, this week, adding some additional metrics, one of which we will discuss in our next blog post about Google serving spurious results.
These metrics will cover both the spurious results rate and also the overall sampling percentage, along with details around which keywords have sampled data on a day-to-day basis.
Conclusion
Given the world we’re heading into, we believe strongly that transparency and Observability are critical aspects of any data strategy and that is doubly true when it comes to the world of search. Let me know any questions or feedback!