Competitor Website Monitoring: Better ways to do it

For the past 4 years, I've been working on marketing teams in niche markets*. And at each stop I've worked at, our sales and marketing teams followed this pattern:

1. Someone on sales is told by a prospect about how this competitor has x
2. Sales person messages the company Slack
3. Marketing and product processes how this affects positioning

Wouldn't it be nice if this process wasn't so reactive? With Claude, LLMs, and new AI tools, monitoring competitor websites should be something that's fairly automatable. And as I went down this rabbit-hole of creating a robust - but NOT overengineered - AI-driven, here are the stages of competitor website monitoring.

*SaaS integration platforms and embedded hardware SDKs in case you were interested.

Stage 1: Ask Claude every month

In the past, LLMs and harnesses were not good enough to just "ask chat" to analyze your competitors. But now, ask any half-decent AI product like Claude Code or Codex, and you can see that it can use a flurry of web searches and fetches to scan your competitors' website and analyze them against your own.

This works pretty well, but it is a surface-level scan. After all, a surface-level prompt will only get a surface-level result. This moves us to the next stage: Prescriptive prompting.

Stage 2: Prescriptive instructions for an agent workflow

When writing a prescriptive prompt, it's almost like writing code. You're describing loops, ifs, and writes. I started with something like:

1. Go through every competitor page, including docs, blog, and landing pages (for loop)
2. Evaluate each competitor on the following axes: feature parity, pricing, content quality, SEO reach, integrations
3. If page is relevant, write each link with a summary in "competitor-x-09-23-26.md" (if statement)
4. Compile a matrix of the results in "competitor-monitoring-09-23-26.md" (write out)

This is really good. It's still a bit reactive. You have to remember to run this once every few weeks or every month, but it's a lot more automated and you can specify what you care about.

At this point, you can stop here, but stage 3 is sort of the perfectionist's solution to this problem.

Stage 3: An Agent Product for Competitor Monitoring

The biggest differentiator between an Agent Product vs Agent Harness, is that a product leverages harnesses with UI views and a user experience built for a specific use case. For competitor website monitoring, this means:

1. An agent harness equipped with not just web search and fetch, but also web crawling. This is the most comprehensive way to monitor every page.
2. A database to track historical changes. Every scan is a snapshot, so we can track our competitors over time.
3. Efficient indexing and retrieval. Reading every web page when a competitor has 500+ pages is a lot of context to hold. We need vector search and graphs to make connections between website pages over time.
4. Charts and configurable prompt templates: Some of us are visual learners, sometimes people process information better with graphics and not a block of text.
5. Cron-based scans and email pushes: I don't want to have to run the same Claude prompt every month. Do it for me.

Is this an over-engineered solution for monitoring competitor websites? Maybe... But hey, I built it so you wouldn't have to.

You can take my over-engineered solution for a spin with DiffSignals.