There is nothing like sitting back at Xmas with a Yuletide Log4j on the fire and an episode of the latest DDoS (Distributed denial of service) thriller to help you unwind.
Running up to the end of year holidays is always stressful for those of us that have those jobs that “keep the lights on”. First you are looking for a well-earned rest like everyone else. Next, you know that there will be lots of last-minute requests and you work hard to keep these managed and realistic so that you can finish and relax without worrying about running past the “change freeze” or having a mountain of work to come back to. Last but by no means least, you have that instinctive fear that the minute you switch off, the alarms will go off and call you back on duty.
So, this Xmas was no different to previous years, but we did manage to go into the holidays without too much to worry about. It nearly went all Pete Tong and Sod’s Law was trying to impose itself for a moment there. A large two-headed monster rose from the ocean and threatened to spoil our holiday down time. Fortunately, the monster did not show its two heads at the same time and we were able to deal with them separately, one after the other. Two at once could have created a high-pressure situation and we were lucky there. Dealing with them consecutively for the most part allowed us enough time to avoid making decisions under duress and keep our balance. Still, you must expect the worst and be prepared and there was a moment (more like a week) where we were looking over our shoulder to see if there was a third head or if the monster had a sting in its tail!
Having dealt with the issues, we would like to share the stories and the lessons learnt.
The Threat
The first head on the monster came in the form of a threat of DDoS (Distributed denial of service) against the platform. We saw an unusual spike in traffic on our edge one evening around the last week of November. There was nothing particular to note other than an unusual traffic pattern. It was not persistent or long lived. Our load balancers kicked in and managed the traffic.
When we analyzed the event, however, we noticed that there was more to this than an unannounced customer launch or sudden press release about new products. Inside the headers of the requests, we saw messages which warned that the website in question was the target for a DDoS attack. It included other instructions related to payments which may head off any further attack. Our customer had received the same message through other channels and was already alert as we notified them. We quickly set up an Incident Management room and a joint team that co-ordinated our response.
We are glad to report that the story ended well. Our combined efforts and our technical controls ensured that the customer’s web site was not disrupted, and the subsequent efforts of the attackers were thwarted. There was never a sense that the attack would result in a security breach and the Threat risk was quite low. Although this was not the most severe DDoS attack that the world has ever seen, you cannot tell how big the attack will be and how long these things will take to resolve at the outset. In fact, you do not really know when it will start for real or when it will end once it starts. What we do know in hindsight is that we have tested our plans and management processes to a successful conclusion.
Collaboration is key
We also know how our customers and our vendors will respond and work with us to address these types of Threat. Our customer was both supportive and collaborative, which gave us an edge in terms of planning and executing many actions during this period. Cloudflare play a significant role in the protection of our platform against DDoS attacks and their support was second to none in terms of both Threat Intelligence and mitigation through their network. The job was done, and we allowed ourselves an acknowledgement of the fact that we won this battle, but we were not about to become complacent.
It was just not the time for complacency because right on the heels of this DDoS attack came the announcement that has become known as the “biggest cybersecurity threat in history” … Log4j or Log4Shell as it was dubbed. We turned around and faced down the second head of the monster.
Day Zero and Log4j
For those that are unaware of why this was so significant, there are a couple of easy-to-read indicators. This was a zero-day vulnerability which means there was no patch at the time it was announced. The exploits gave hackers widespread access to underlying systems. Log4j is an embedded software library that is widely used. It underpins a considerable proportion of legacy systems which in turn support many modern-day platforms. It is harder to find companies that do not have some implementation using Log4j than those that do. It has been reported that 50% of all companies have been subject to breach attempts and millions will have been breached. We had at least 12 specific customer requests regarding this within the first week of the announcement even though we sent a customer notification out within 2 days explaining our position and the Threat risk to our platform. I have not seen that level of engagement before.
Image source: ifunny.co
Vulnerability Scans
We have several tools at our disposal to support us in these situations. As with the DDoS attack, vendor support played a key role and made our job much easier. Immediately, we looked at our vulnerability scanning service from AppCheck. Before we even turned to their support team, they confirmed that the service had been updated to detect and report vulnerability. Scans identified that we were not exposed, and this provided some initial reassurance although we knew from our inventory that Log4j libraries were included in some internal applications. We did have one customer report that an application we hosted for them was vulnerable based on a scan they performed themselves. This was a false positive and it shows the level of anxiety that can be generated in these situations. In situations like this, you always need to look twice at the data you receive. You can be fooled into thinking you are safe because you have scanned and found nothing, or you can be convinced you are vulnerable when you pick up these anomalous results.
Despite the scan results, we were not about to sit back and assume everything was all right. We knew that we had Log4j components, and we needed to patch regardless. Patching was not going to be straight forward either. We were running into the “change freeze” window and that usually means late changes start queuing up which we might disrupt. Regardless of this kind of pressure, we were about to find out that the first patch that was issued had yet another vulnerability. It was clear that rushing into emergency changes would not be the smart option for our circumstances even though this could be an option on another day. Instead, we turned first to our Cloudflare WAF and made sure that we had rules in place to mitigate any attempted exploits and monitored this closely. As with AppCheck, Cloudflare was on top of this and gave us the protection we needed. However, this is never enough, and we turned to our newly operational SIEM to extend the monitoring for internal events and to identify any Log4j libraries, their version state and potential risk status. Critical to this aspect of the SIEM monitoring was data provided by the AWS GuardDuty service. This would confirm where and how we patched our systems. Working with this data and vendors who provided those internal systems was straightforward from here on.
Image source: ifunny.co
Keeping the lights on with SIEM
Our SIEM played a big part in both events and we set it up to monitor and alert all the way through the holidays with automated messaging and integration using our Ops Genie on call management and escalation process. Whilst none of us like that call in the middle of the night, “keeping the lights on” means 24/7 coverage to protect and operate our platform. The temporary process, specific to the Log4j monitoring alerts has been wound back now but demonstrated another test of our response strategies but the on-call function continues to ensure we are ready to manage other operational issues if required.
For further information on our security protocols and features, please reach out by leaving your details and a descriptive comment from our Information Security team will get back to you. Alternatively, you can request a demo of how our tools and services can add value to your digital transformation.
