If you run a site that relies on organic search for leads or sales, crawl budget can feel like a niche technical detail right up until it starts costing you money. Totally Digital London helps clients fix this exact issue: you can have genuinely strong pages that should be performing, but Googlebot ends up spending its time crawling the wrong URLs instead.
Because when Googlebot ignores the pages you care about, it is rarely random. It is usually your site quietly telling Google to crawl the wrong things.
What crawl budget actually means
Google defines crawl budget as the set of URLs that Google can and wants to crawl, driven by 2 forces: crawl capacity and crawl demand.
- Crawl capacity is how much your site can handle without slowing down, timing out, or throwing errors.
- Crawl demand is how much Google wants to crawl, based on how important, unique, and useful your URLs look.
If either side is weak, Googlebot backs off or gets distracted. The end result looks the same: slower discovery, slower re-crawling, and important pages sitting there like they are invisible.
Why this matters commercially in the UK
This is not just enterprise SEO theory. In the UK, timing is everything, because online buying behaviour is massive and often seasonal.
The Office for National Statistics has reported that the proportion of retail sales made online was 28.6% in November 2025. And Reuters reported UK online shoppers spent £3.8bn over the 4-day Black Friday to Cyber Monday period in 2025.
So if Google is slow to crawl your:
- new category pages
- updated product availability
- refreshed service landing pages
- seasonal campaign pages
…you can miss demand windows that are worth serious money. A crawl budget problem quickly becomes a revenue problem.
The signs you have a crawl budget issue
You will usually see a few of these together:
- New pages take ages to get indexed
- Updated pages do not refresh in search results for weeks
- Important pages drop in and out of the index
- Search Console shows lots of “discovered, currently not indexed” or “crawled, currently not indexed”
- Google is crawling parameter URLs, filters, tags, or internal search pages more than your core pages
- Your server logs show Googlebot spending a big chunk of visits on low-value URLs
If you are only checking rankings, you can miss this entirely. Crawl budget issues often sit underneath performance dips that look like “Google updates” or “content isn’t working”.
The 2 levers you control: capacity and demand
Crawl capacity: stop making it hard for Google to crawl you
If your site is slow, unstable, or error-prone, Googlebot will reduce crawl rate. Capacity problems commonly come from:
- Spikes of 5xx errors (hosting, application crashes, overloaded servers)
- Slow time to first byte on key templates
- Heavy pages that take ages to render
- Redirect chains that force Google to waste requests
- Poor caching and inconsistent performance
Your goal is not perfection. Your goal is reliability. You want Googlebot to hit pages and consistently get fast 200 responses, without timeouts or drama.
Crawl demand: stop making your site look like it has endless low-value URLs
Demand drops when Google thinks a lot of your site is duplicated, thin, or not worth revisiting. Common demand killers include:
- Infinite filter combinations (faceted navigation gone wild)
- Tracking parameters creating duplicates
- Multiple versions of the same page (HTTP vs HTTPS, www vs non-www, trailing slashes, case sensitivity)
- Tag archives producing near-identical lists
- Internal search results that generate thousands of low-quality pages
- Pagination and sorting URLs that create variations with no real unique value
If Google sees that 70% of what it crawls is repetitive, it learns to care less.
Why Googlebot crawls the wrong pages
This is the bit people underestimate: Googlebot is not reading your strategy doc. It is following signals from your website.
1. Your internal linking is steering Google to rubbish
Googlebot follows links. If your navigation and template links point heavily at:
- tag pages
- filter URLs
- internal search URLs
- endless calendar archives
- “related posts” widgets that multiply thin pages
…Googlebot will crawl those. Especially if those URLs are closer to the homepage than the pages you actually want to rank.
A quick test: can you reach your most valuable pages within 2 to 3 clicks from the homepage without using search? If not, you have a crawl priority problem.
2. You have index bloat (too many pages competing for attention)
Index bloat is when your site generates a large number of URLs, but only a fraction deserve to be indexed.
Typical causes:
- Ecommerce filters (size, colour, brand, price, availability, sorting)
- Multiple URL paths to the same content
- Blog tag pages, author pages, and date archives
- Duplicate location pages that differ by 1 sentence
- “Near-identical” service pages across multiple towns
This wastes crawl budget and weakens your overall SEO signals. Google spends time crawling pages that do not help you, while also becoming less confident about what your site is really about.
3. Your canonicals are inconsistent or incorrect
Canonicals are meant to tell Google which version of a page is the main version. But in the real world, canonicals often cause chaos:
- Canonical points to a URL that redirects
- Canonical points to a noindexed page
- Canonicals vary across duplicated pages (so Google sees conflict)
- You internally link to non-canonical versions (mixed signals again)
When canonicals are messy, Google spends extra time crawling to figure out what you meant. It also reduces demand because your site looks indecisive.
4. Your sitemaps are clean, but not trusted
A sitemap is a hint, not a command. If you include lots of URLs that are:
- redirected
- blocked in robots.txt
- returning non-200 codes
- thin or duplicated
…Google learns the sitemap is not a great signal. Your important URLs then do not get the priority boost you expect.
5. Redirect chains and broken links are silently draining crawl budget
Every redirect is a small tax. A chain is multiple taxes. If Googlebot hits:
- old URLs that redirect to newer URLs that redirect again
- internal links pointing to redirected URLs
- large volumes of 404s and soft 404s
…it wastes crawl requests that could have gone towards your important pages.
How you diagnose this properly
Step 1: Start with Search Console patterns
You are looking for trends, not one-off issues:
- Crawl stats: are crawl requests stable, falling, or spiking?
- Response codes: are there lots of 3xx, 4xx, 5xx?
- “Discovered, currently not indexed”: are your important pages sitting there?
- URL inspection on key pages: when were they last crawled?
If your top service pages have not been crawled in ages, that is a signal.
Step 2: Use server logs to see what Googlebot is actually doing
Logs are the truth.
When you analyse logs, you can answer:
- What percentage of Googlebot crawls go to parameter URLs, filters, tags, or internal search
- Which templates are being crawled most (product pages, categories, blog, assets, search)
- Whether Googlebot is stuck in redirect chains
- Whether Googlebot is crawling low-value pages more than key pages
- How crawl behaviour changes after site updates
If you do not have log access, it is still worth asking your dev team. This is one of the highest value datasets you can use for technical SEO.
Step 3: Crawl your site like a bot to map the problem
A proper crawl (using a crawler tool) helps you find:
- orphan pages (important pages with no internal links)
- depth issues (money pages buried too deep)
- duplicate clusters and near-duplicate pages
- canonical and indexation conflicts
- parameter patterns creating infinite URLs
- internal links that point to redirects
Think of it as your “how your site looks to a machine” view.
The fixes that usually make the biggest difference
Fix 1: Reduce URL spam at the source
If your CMS or platform generates infinite URL variations, you need to get control of it.
Common actions:
- Decide which filter combinations should be indexable (usually only a small subset)
- Use canonicalisation on duplicate variations back to the core category page
- Add noindex to pages users need, but search does not (for example internal search results)
- Limit internal linking to non-indexable filter pages so Googlebot is not tempted
Be careful with blanket robots.txt blocking. Blocking can stop crawling, but it does not always solve indexing issues cleanly if links still exist and signals conflict. Aim for consistency.
Fix 2: Make your priority pages unmissable
If you want Google to crawl something, it needs to look important:
- Link to key pages from navigation or prominent hub pages
- Use contextual internal links from related pages
- Reduce sitewide links to low-value pages (tags, archives, thin lists)
- Fix orphan pages immediately
Internal linking is one of the fastest ways to increase crawl demand for the right URLs, because you are changing the path Googlebot follows.
Fix 3: Clean up redirects properly
You want:
- one-step redirects wherever possible
- no internal links pointing to redirected URLs
- old redirect rules removed when they are no longer needed
- consistent canonical URLs across the site
Redirect tidying is boring, but it is one of the quickest crawl budget wins you can get.
Fix 4: Improve performance where it matters most
You do not need to chase vanity scores. You need stability and speed on the pages Googlebot is trying to crawl:
- reduce server response time on key templates
- make sure caching is working consistently
- investigate spikes in 5xx errors
- avoid heavy pages that time out under bot load
A site that handles crawling smoothly can earn more crawl capacity over time.
Fix 5: Stop publishing pages that should not exist
This is the awkward conversation, but it is often the real answer.
If your site is full of:
- duplicate service pages
- weak location pages
- thin blog posts
- tag archives that add no unique value
…Googlebot will spend time on them, and your important pages will suffer.
Sometimes the best technical fix is content pruning plus better information architecture. Fewer, stronger pages often win.
A simple crawl budget action plan you can run this month
Week 1: Identify waste
- Pull a list of the most crawled URLs (logs are best)
- Group them by pattern: filters, parameters, tags, search, duplicates
- Compare that to your priority pages list
- Quantify the gap: what share of crawling is wasted?
Week 2: Fix the biggest crawl traps
- Tackle parameter and filter explosions first
- Remove internal links into crawl traps
- Fix redirect chains
- Ensure sitemaps only contain clean, indexable 200 URLs
Week 3: Increase crawl demand for the pages you care about
- Build or improve hub pages that link to priority pages
- Add internal links from high-authority pages to key pages
- Update key pages with meaningful changes (not just swapping a sentence)
Week 4: Measure change
- Recheck Search Console crawl stats
- Check whether key pages are being crawled more frequently
- Review logs again and confirm Googlebot behaviour has shifted
This is how you make crawl budget improvements measurable, rather than just hoping.
The outcome you are aiming for
You are not trying to make Google crawl every URL. You are trying to make Google crawl the right URLs more often.
That usually means:
- fewer low-value URLs
- clearer internal linking paths to priority pages
- consistent indexation signals
- fast, stable responses for key templates
- clean sitemaps and fewer redirect distractions
When those pieces come together, you will often see faster indexing, more reliable rankings, and stronger SEO performance.
Next steps
If you suspect Googlebot is spending its time on pages that do not matter while your highest-value pages get ignored, you do not need more guesswork. You need a crawl budget plan grounded in real crawl data, a clean indexation strategy, and fixes you can prioritise by impact.
If you want help diagnosing where crawl budget is being wasted and turning that into a practical fix list, speak to Totally Digital.