TheScrumMaster.co.uk is a busy WordPress website supporting a global training business. It combines WooCommerce, LearnDash, multilingual content and a substantial catalogue of courses, products and learning resources.
The site had become increasingly unstable. At unpredictable times it would slow down or become unavailable altogether, producing 502, 503 and 504 errors. Some outages lasted several minutes and others considerably longer. The hosting platform reported high resource usage and exhausted PHP workers, but that only explained what was happening — not why it was happening.
Simply increasing server resources might have hidden the problem temporarily. We wanted to understand the real cause and stop it from happening.
Finding the Real Cause
We began by analysing server logs, Cloudflare traffic, cache behaviour, WordPress requests and the traffic reaching the origin during periods of instability. What we found was not one problem, but several interacting problems.
A significant amount of traffic was bypassing cache and reaching WordPress directly. This mattered because a cached page is inexpensive to serve, while an uncached request may require PHP, WordPress, the database and multiple plugins to execute.
We found automated traffic generating large numbers of URL variations through query strings. These variations fragmented the cache and caused repeated cache MISS requests to reach the server.
The multilingual structure of the website amplified the problem further. Crawlers could discover translated versions of thousands of pages, multiplying the number of URLs available to crawl and creating sudden bursts of activity across translated sections of the site.
We also identified unusually high traffic to WordPress's admin-ajax.php endpoint. This endpoint was legitimately required by LearnDash, WooCommerce and other site functionality, but every request could consume server resources. During periods of heavy automated traffic, that contributed to PHP worker exhaustion.
WooCommerce added additional dynamic requests around baskets, checkout and AJAX activity. LearnDash also needed to continue processing course activity reliably.
The challenge was therefore much more complicated than simply “blocking bots”.
We needed to reduce unnecessary traffic while making sure customers could still purchase courses, learners could complete lessons, translated content remained available and search engines could continue indexing the site.
Building Protection Around the Website
We created a layered protection strategy rather than relying on a single firewall rule or performance plugin.
At Cloudflare, we built and refined Web Application Firewall rules targeting the traffic patterns we had identified. These included protection for suspicious query strings, translated URL abuse, malformed requests, selected WooCommerce traffic and excessive activity against admin-ajax.php.
Where appropriate, suspicious visitors were challenged rather than automatically blocked. This allowed us to reduce automated traffic while minimising the risk of affecting legitimate visitors.
We also introduced rate limiting around admin-ajax.php. This was carefully tuned because an aggressive limit could easily break legitimate LearnDash or WooCommerce activity.
Some of the work had to go deeper than Cloudflare. Custom LearnDash functionality was changed so that legitimate AJAX requests could operate correctly alongside the new security rules. We improved compatibility with translated course pages and protected lesson-completion functionality while maintaining stronger edge protection.
We reviewed the site's robots.txt configuration and discouraged crawlers from wasting resources on low-value areas such as translated search pages, baskets, checkout pages, account pages and other dynamic URLs.
We also reduced unnecessary translated sitemap exposure in Google Search Console, lowering the crawl surface without removing important multilingual content.
Fixing the Caching Layer
Caching turned out to be one of the most important parts of the investigation.
We analysed which requests were returning cache HITs and which were reaching the origin as MISSes. We investigated query strings, cookies, browser cache headers and optimisation plugins to understand where cache effectiveness was being lost.
This work included extensive testing of WP Rocket and SiteGround caching, as well as trials with alternative optimisation approaches.
The objective was not to chase a slightly higher performance score. The objective was to make the site's caching behaviour predictable and ensure that as much legitimate public traffic as possible could be served without repeatedly executing WordPress.
We also diagnosed individual problems that initially appeared to be server issues but were actually occurring elsewhere in the caching chain. In one case, a stale homepage affecting a particular browser and device was traced to local browser caching rather than the WordPress server.
That distinction matters. Changing the wrong layer can make a website less stable rather than more stable.
Improving Our Ability to Diagnose Problems
A major part of the project was developing better visibility into what was happening when an incident occurred.
We created custom tools and analysis processes to examine 50x errors, bot traffic, cache behaviour, translated requests, top IP addresses, suspicious traffic patterns and cache MISS storms.
Instead of seeing an outage and guessing at the cause, we could determine which requests were hitting the site, whether they were cached, which endpoints were being targeted and what traffic patterns immediately preceded the outage. This allowed us to move from broad defensive measures to increasingly precise protection.
During severe incidents, Cloudflare's Under Attack Mode could be used temporarily to stabilise the origin. Once the immediate pressure had been removed, we could analyse the traffic and replace the broad protection with more targeted rules.
We also improved the way uptime was monitored so that monitoring itself did not repeatedly exercise expensive dynamic WordPress endpoints.
The Result
TheScrumMaster.co.uk moved from repeated periods of instability and prolonged 502, 503 and 504 errors to a much healthier operating state.
The work resulted in:
- dramatically fewer outages
- substantially shorter incidents when traffic spikes did occur
- reduced bot and crawler pressure
- better protection of PHP workers
- improved cache effectiveness
- reduced multilingual crawl amplification
- stronger protection around WordPress AJAX requests
- continued operation of WooCommerce and LearnDash functionality
- much faster diagnosis when unusual traffic appeared
Most importantly, the website was no longer dependent on simply throwing more server resources at unexplained traffic spikes. We understood what was causing the load and had protection in place to deal with it.
What This Project Demonstrates
WordPress downtime is rarely solved well by looking at just one part of the stack.
A hosting company may see exhausted PHP workers. A WordPress developer may suspect a plugin. A security provider may see bot traffic. A caching plugin may report poor cache performance. The real problem can involve all of them.
Resolving the issues on TheScrumMaster.co.uk required us to work across Cloudflare, WordPress, server logs, caching, WooCommerce, LearnDash, multilingual content, crawler behaviour, AJAX requests and application code.
That ability to investigate the whole request path — from the visitor at the edge through to WordPress and PHP at the origin — is what ultimately made the difference.
Is Your WordPress Site Experiencing Similar Problems?
If your WordPress site suffers from recurring 502, 503 or 504 errors, unexplained CPU or PHP worker spikes, bot traffic, poor cache performance or intermittent downtime, simply upgrading your hosting may not solve the underlying problem.
WebsiteFort investigates what is actually reaching your WordPress origin, identifies the requests responsible for the load and puts targeted protection in place without unnecessarily blocking genuine customers or search engines.
If your host keeps telling you that your site has “high traffic” but nobody can explain exactly why it keeps going down, talk to us.