Bots on the internet

Did you get that annoying box to click you are not a robot? It's obvious that you are not and you just wanted to visit a website real quick.

Next part of Belfast DevOps meetup was held by Eve Brennan, she talked an incident caused by non-human traffic on internet. In fact in report by Imperva they published a number of 51% of network traffic is generated by bots. Non-organic. The bad news is 37% are bad bots, crawlers, scrapers, scalpers, DDoS attackers...

January 2025, a website suddenly down. Traffic spikes coming in all of a sudden and HPA couldn't keep up. Business Insider website got attacked by bots. It's traffic overloaded node.js, couldn't scale that fast and it crashed due to OOM. This not even the ad companies like as it can make false advertisement.

Temporarily the cluster was defended by manual restrictions over the incoming IPs. This is where WAF comes in place. Web application firewall. Cloud team at business insider was using Fastly's CDN, to prevent extra hops and layers on the network, that's why they went with Fastly's WAF offer. Their WAF approach is signal based and tags traffic behaviour in real-time speed.

To prevent false positives during the rollout they had to differ between good bots and the bad ones. Rollout strategy went incrementally month by month. Now there were no spikes over monitored past 30days.

The message of this story is to scale real traffic from people that has impact and not for bots.