Recently we had an overworked machine with around 60% of the traffic coming from search engine indexing bots. We needed to stabilise the load but didn’t want to sacrifice all indexing and thus our SEO entirely.

This is the solution we came up with.
Read more