Prompt trackers: The most expensive way to learn nothing

Every few months, another hype cycle promises to change marketing forever.
This time, it’s Prompt Tracker – tools designed to show how often your brand appears in ChatGPT, Claude or Perplexity.

Sounds useful. Who wouldn’t want to know?

The problem is that the technology behind these tools is fundamentally flawed – at least if you intend to use the data to make serious decisions.

Why Prompt Trackers Don't Work

Let's start with the most obvious problem:
AI is non-deterministic.

Put simply: ask ChatGPT the same question twice and you will not necessarily get the same answer. Sometimes only the nuance changes; sometimes the focus is completely different.

For statisticians, this is a nightmare.
To draw even reasonably reliable conclusions, you would need to repeat every prompt many times – not once or five times, but dozens or hundreds of times, depending on the level of accuracy you need.

In practice, this means:
Even a few hundred runs per prompt can quickly cost a high two- or three-figure sum. Per prompt. Per model.

Track 30, 40 or 50 prompts across several models and monthly costs can quickly run into five figures – for data that remains highly variable.

The API is not ChatGPT

Most prompt trackers use APIs from OpenAI, Anthropic or other providers. That makes sense – any other approach would risk breaching their terms of use.

But there is a catch:
The API does not reproduce what users see on ChatGPT.com.

The web interface uses proprietary system prompts, internal tools, added context and orchestration that changes regularly. None of these layers is visible or reproducible from outside.

So what does that mean in practice?
A prompt tracker therefore measures responses from a technical setup that differs fundamentally from the actual user experience.

Or, in other words:
You analyse data from a parallel universe – and wonder why it doesn't match reality.

The query fan-out problem

Another point that is rarely discussed:

When users ask complex questions, an AI system does not necessarily process them as a single search query. It may split, rephrase and evaluate parts in parallel, depending on the model, prompt and context.

From
„What are the best wireless headphones under £200?“
may be expanded into several related searches, such as:

• „Wireless headphones under £200“
• “Bluetooth headphone reviews”
• “Headphone comparisons”

In other words, you are tracking a prompt that may never serve as the actual basis for the answer.
The AI follows its own process while your tracker measures something else.

What works instead

Rather than guessing how a model behaves, you can measure several concrete outcomes – directly and without interpretation.

• Where AI traffic actually comes from
• What these visitors do on the site
• Which content AI systems link to most often
• How AI traffic changes over time

The reassuring part:
Some data can be measured directly. No estimates. No interpretation.

AI referral traffic

When someone visits your website through a link from ChatGPT, Perplexity or another AI platform, that visit is a measurable fact.
No model, no simulation – a real visit.

And a closer look reveals even more.

Breakdown by AI source

AI platforms do not all work the same way:

• ChatGPT primarily uses Bing
• Claude uses Brave
• Perplexity operates its own search logic

Depending on where your content is visible, visitors can arrive from very different AI ecosystems.

Separate the sources properly and you can quickly answer:
Which AI platform actually sends relevant traffic – and which only creates noise?

Conversion tracking per AI source

Traffic is useful.
But the question that matters is: what happens next?

• Do these visitors buy?
• Do they sign up?
• Do they continue reading or leave immediately?

If visitors from one AI source convert significantly better than others, that is not a hunch. It is solid evidence of where investment may pay off.

Landing page analysis

Which content do AI systems actually link to?

If guides consistently attract AI referrals while product pages barely do, that is unlikely to be a coincidence.
It shows which types of content perform well in AI-driven search – and which do not.

AI share over time

A simple but often overlooked question:

How is your share of AI traffic changing from week to week and month to month?

Is it rising steadily?
Has it stalled?
Or has it suddenly dropped?

These shifts can be early warning signs.
If your share of AI traffic is shrinking while the overall market is growing, something is wrong.

AI share over time

A simple but often overlooked question:

How is your share of AI traffic changing from week to week and month to month?

Is it rising steadily?
Has it stalled?
Or has it suddenly dropped?

These shifts can be early warning signs.
If your share of AI traffic is shrinking while the overall market is growing, something is wrong.

Conclusion

Prompt trackers sell an illusion of control. Dashboards update, figures change and everything looks measurable. Yet beneath the surface, most of it is noise – expensive to collect and difficult to interpret.

These tools may be useful for experimentation. They are not a sound basis for business decisions.

What Trackboxx already measures

We focus on AI referral tracking. Less flashy, but grounded in real behaviour.

You can see which AI platform sends you traffic – ChatGPT, Claude, Perplexity or Gemini – which pages it links to and which visits convert.

Real visitors. Real data. No estimates.

It is all cookieless and GDPR-compliant, without costing thousands of pounds a month.