At DemandSphere, we pride ourselves on responding quickly to changes in the SEO landscape and solving the challenges that our customers face in a holistic manner. 

We take a three-pronged approach to our product roadmap: 

• our vision / mission guidance
• customer requests and challenges
• changing market conditions

We focus heavily on enterprise SERP monitoring with a particular focus on the needs of ecommerce and programmatic SEO teams, both in-house and agency. 

This focus permits us to do what we do best and we don’t make the mistake of trying to boil the ocean. 

2023 was a year that left its mark on the SEO industry in a way that none of us will forget. 

We saw the juggernaut of AI consume much of the attention in our space. This is because the impact that it has already had on the internet as a whole is profound. This impact will only expand in coming years. We’ve covered this in many of our posts so I won’t go into too much detail here, with the exception of our new product innovations mentioned below. 

Another major factor in SEO was the sheer velocity and impact of Google’s algorithm changes, ranking system updates, new product releases (including SGE), and many changes to SERP layouts. 

The prevailing feeling among our customers and partners as we head into the New Year is that this rate of change will be the new constant. This is certainly reflected in our data which covers SERP layout changes and visual ranking volatility on a daily basis. 

In response to these changes and our roadmap guiding principles mentioned above, we are proud of the innovations that we have released in 2023.

Our team works harder than any team I have ever led and consistently punches far above our weight class. 

The following list covers the highlights of what we provided to our customers, although it is not comprehensive. 

Massive infrastructure upgrades

One of the biggest projects we had this year was to further upgrade our infrastructure.

This is one of the most important and yet least-visible aspects of our platform. But I mention it first because this is where our customers get a ton of value from our services. We have been able to achieve the most scalability and best cost / performance in the market, hands-down. Our customers are able to manage SERP volumes at price-points that are simply not achievable anywhere else, with an incredible degree of accuracy. 

A key means for us to achieve these results comes from our decision more than ten years ago to exit the cloud for our core infrastructure. Back then, we were on AWS and, as our data volumes grew, we quickly saw that their platform would eat into all of our profits. We moved to bare metal servers and have built our own private cloud and networks, relying heavily on Kubernetes and a number of other great tools for automation and orchestration.

We strongly believe that the trend of cloud exit / repatriation will continue and we frequently advise our customers on how to manage and scale their digital products with this trend in mind. We believe that the future, for many teams, will be Hybrid. 

Going forward, with the proliferation of AI and data products, the winners will be those teams and companies who can handle the most amount of data in the most innovative ways without breaking the bank on budget. 

Google Cloud and BigQuery upgrades

Speaking of the cloud, one cloud provider we really like is Google Cloud Platform. In particular, we see BigQuery as a total game-changer. 

We have based our SERP Intelligence data warehouse product on BigQuery and continue to innovate on our schema, automation, and extensions to this product and the platform just never lets us down. 

In 2023, we added a lot more SERP Intelligence customers and it is one of the most powerful sets of features that we have ever delivered. It has been a game-changer for us too because our Managed Services team can go from waiting on some new feature to be developed (which can take weeks) to being able to prototype and launch new insights for our customers in a matter of minutes or days. 

GA4 Upgrade

I have yet to talk to one person who actually likes GA4 but it was obviously necessary for us to handle it so we upgraded to the new GA4 API over the summer. 

We were able to achieve seamless integration with historical UA data, so customers didn’t need to do very much in order to reap the benefits of this upgrade. 

We have already had a comprehensive, 360-degree view of content performance in DemandMetrics for years, even before the GA4 upgrade. 

This feature enables customers to get a URL-centric view of their SERP visibility, combined with ranking data, GSC metrics, and data from Google Analytics, all in a single view. Additionally, our Content Groups feature allows customers to create site segments in real-time with easy-to-use matching rules, which can be applied / modified without any negative impact on the underlying data. 

For developers and the curious, Google’s API Query Explorer also supports GA4, so you can use this to quickly prototype views and confirm that data matches your reports.

Unified Visibility

In addition to the upgrade to Content Analytics mentioned above,  we released our Unified Visibility feature. 

Unified Visibility leverages our TrueKeywords feature, which imports ALL of the core data from Google Search Console, allowing customers to far exceed the limitations in the UI.

Google does offer bulk export capabilities for GSC to BigQuery but, unfortunately, they don’t support (and don’t plan to support) historical imports. 

When a customer connects GSC, DemandMetrics will automatically import all the historical data as well as all of the keywords and pages, properly filtered and broken out into dimensions for additional filtering. 

Unified Visibility takes all of this one step further. 

While TrueKeywords, as a feature, is valuable, it is still almost all post-click analysis.

What this means is that while GSC does provide a couple of pre-click metrics (position and impressions), it relies on Google’s view of these two metrics. Anyone who has been in SEO for any amount of time knows that you don’t want to base all of your decisions on the purported accuracy of Google’s data. 

This is the reason why companies continue to invest in rank tracking software. 

But traditional rank tracking doesn’t cut it anymore either because of the changing nature of the SERPs. 

If you don’t understand the impact that SERP Features have on your CTR in the form of Visual Ranking, pixel depth, and more, you’re flying blind. 

Unified Visibility, true to its name, unifies all of these core metrics into a single view. This was a long-awaited feature and we’re thrilled that we were able to release it in 2023. 

Updated Content Gap

We released our new Content Gap feature last year and have made some improvements on it this year. In particular, we have connected it to SERP Rewind so that it’s easy to go from our traffic modeling tools into Content Gap, into SERP Rewind, to see the exact impact of the SERP on this competitive set of metrics.

We will likely be renaming this feature to Visibility Gap in 2024, just as a heads up. 

SERP Rewind Updates

Speaking of SERP Rewind, as this is one of our most popular features, we’ve made some nice improvements. 

SERP Rewind, for those unfamiliar, maintains a copy of the fetched SERP for as long as you’re a customer, according to whatever retention policy you select. 

We maintain the original HTML, as well as the parsed JSON and also a very useful details view of the parsed SERP data.

Additionally, we now have an API that supports screenshots of the SERP with a visual overlay showing how Google has built the SERP with its various features, broken out into ads, organic search results, SERP Features, and more. 

SERP Rewind has a comparison feature that allows you to compare any two SERPs from your historical record, across dates, search engines, and devices. 

SERP data can be downloaded, exported, and is, of course, available in all of its detail in our SERP Intelligence data warehouse. 

Top 20+ Reporting

One of the things we’ve wanted to build into our platform for quite some time was a deeper view of the SERP in greater detail. This became a priority in 2023 as we had a number of customers migrate to our platform away from another well-known (and also quality) platform provider in the industry. One of the features needed to support this migration is commonly known as the Top 20 report. 

The Top 20 report is multi-faceted but, at its core, goes beyond the traditional top ranking reports you see with your site’s ranking vs. competitors. Instead it looks as the top 20 (as these are the most relevant) positions in the SERP, including SERP Features. This enables customers to get a very detailed view on what we are always preaching with understanding the full structure of the SERP. 

This is a nice feature to complement what we have in SERP Intelligence. SERP Intelligence does way deeper as it has every single element indexed but sometimes you just want a higher level report on what is impacting visual ranking, CTR, etc. 

This is what the Top 20 feature set does. 

Not satisfied with this, however, we are now working to expand this data to deeper levels (deeper than 20) with the SERP and also make this dataset easier to query and navigate in the UI. 

This new set of features is based on an entirely updated schema that will support not only this but some additional features that we believe will be game-changers in 2024. 

More to follow. 

Visual Share of Voice

In addition to the Top 20 report mentioned above, this new schema enabled us to build a Visual Share of Voice feature. 

Visual SoV goes beyond traditional Share of Voice metrics, which only focus on organic rankings. This builds a SoV model based on the visual impact that the user sees on the SERP, accounting for any and all SERP Features. 

As with many of the features we’ve released over the last two years, if you don’t have this view, you’re not getting the full picture.

SERP Feature Explorer

Another new feature enabled by our new schema improvements is SERP Feature Explorer.

We’ve had SERP Feature metrics in our platform for years, of course, but we wanted to go even deeper. 

SERP Feature Explorer (SFE for short) looks at the presence of customer and competitor URLs within every SERP Feature available. 

This all gets broken out into an incredible amount of detail and is available in our custom dashboards and reporting features, in addition to the main part of Keyword Insights in DemandMetrics. 

We’re also adding in live SERP rendering as part of our fetching process, so we will be including PAA answers (and more) for customers who need this in 2024. 

As this is a newer feature, we haven’t rolled it out yet to all of our customers while we gather feedback but it is available to any qualified customers that request it. 

SGE Data Collection

One of the most common questions we got in 2023 was “how do you handle SGE?” 

With good reason. Google SGE has the potential to have the largest ever impact on rankings and CTRs. 

As most in the industry are aware, SGE has not been released into General Availability. You have to enable access to it in Search Labs.

This means that it’s not yet easily available for data vendors such as ourselves to include it in daily reports. 

However, we have built an internal research scraper for this data so that our customers can request datasets based on SGE results. 

As mentioned above, with our live SERP rendering capabilities, we are prepared to roll out SGE support as soon as it becomes available so customers will be able to immediately measure the impact once it goes live. 

New Metrics Added to Data Explorer

Data Explorer is a popular widget in our reporting and dashboard suite as it enables customers to get customized views of most of our common metrics and easily make comparisons, slice and dice, and more. 

As part of our improvements this year, we’ve added the following metrics into data explorer:

• Visual rank
• Average pixel depth
• Average position (GSC)
• Impressions (GSC)
• Clicks (GSC)
• CTR (GSC)

We’re going to be doing a lot more with this feature in 2024. 

ProseVector

ProseVector is a brand new product that we released this year. We didn’t even know we were going to be building it a year ago. However, with the advent and release of the GPT LLMs and related APIs, we saw an opportunity to solve a longstanding problem that our customers were experiencing. 

Our programmatic and ecommerce customers have this issue where they need to create or optimize content at massive scale. We’re talking 10s of thousands to hundreds of thousands of URLs. Traditional content creation budgets do not support this very well. 

ProseVector enables us to create and / or optimize exactly this sort of content at very reasonable budgets for customers. We use a combination of custom-built knowledge graphs and our expertise in prompt engineering to create content that is impossible to detect as AI content. Humans reading it don’t experience any of the usual “uncanny valley” effects that one typically experiences when reading AI-generated content. 

v5 Updates

v5 is the next iteration of the DemandMetrics platform. We are making significant updates to the underlying architecture and modernizing the framework. Because it is such a large project, we took an iterative approach to building it, successfully leveraging our internal SSO solution to launch v5 features side by side with existing features, so customers can easily switch between v4 and v5 views. 

This year, we launched our new Launchpad, which makes it much easier to get right into your account and get the data you need. It supports large organization and account trees, with global hub rollups, which is a feature that was built with our agency and large conglomerate customers in mind. 

Our next features to live on the v5 architecture will be our freshly rebuilt keyword research and discovery features, combined with an updated SERP Rewind. Rest assured that we will be announcing this as soon as it goes live. It is already undergoing internal testing. 

v5 API

The v5 UI is, of course, supported by our new, v5 API. This updated API will also represent significant performance and developer usability capabilities. 

Customers who want to begin using the v5 API can get access to it today, they just need to ask their account manager. 

Python Data Apps

One of the features that I have wanted to support forever has finally come to fruition. Python is the de facto standard language in the data science and data engineering world. Naturally, the SEO world has taken notice and we are seeing massive adoption among technical and developer-focused SEO teams of Python and related technologies. 

Internally, our Managed Services team makes heavy use of Python for reports, data analytics, and custom features. We use libraries such as advertools, Dash, Streamlit, and a variety of deep learning libraries (see below) to solve customer problems. This is great but it was always annoying that this work wasn’t tightly integrated with the rest of our platform capabilities. 

In Q4, we finally had the opportunity to make progress on this. We built a new Python library called DemandSphere-web. This library is not publicly available and is used currently by our internal team but we do have plans to make it available to customers in the future. 

DemandSphere-web lets developers build custom Python apps in Flask or other web frameworks, authenticate with DemandMetrics for SSO, and we will even host these apps so that they are directly integrated with our customer’s normal UI experience. We have a whole range of new tools we plan to build with this toolkit. 

This has been one of our favorite examples of eating our own dogfood because we can now prototype and launch new platform features, iterate on them faster based on customer feedback, and then actually launch them without needing to disturb our critical path, core product roadmap. 

One of our core values at DemandSphere is speed and this feature directly addresses our desire to always accelerate the delivery of additional benefits to our customers. 

Clustering

An example of how our Managed Services team  has made use of these Python data app capabilities has been in the iteration of our keyword clustering capabilities. We’ve been through a few iterations of this process now and it has turned into quite an interesting project. 

There are various strategies when it comes to keyword clustering. The two most common are semantic (or topical), which look at the keyword meanings and group semantically, and URL-based clustering, which clusters keywords based on SERP analysis to find similar or overlapping keywords. 

Clustering is valuable for many reasons, ranging from planning to monitoring, optimization, and more. 

The challenge that we had with clustering is that the volume of keywords and URLs that our customers manage are so large that it quickly becomes a big data problem. 

We’ve optimized our processes on this to the point that we can cluster millions of rows in a matter of minutes. This greatly accelerates time to insight. It’s a feature that we use in our Managed Services practice now to deliver custom data feeds to customers but this will be getting bundled into our core platform in 2024. 

Conclusion 

As we head into thew New Year, we can promise that next year will bring even more changes in the industry and even more great innovations from our team. 

We thank everyone that has been part of our world this year and wish our readers a very happy and safe New Year. We wish you all the best for a successful 2024.