My Experience With Log4j
Ok — let’s talk about the shit storm that was and is Log4j and what, as an attacker — I saw and learned about security over the past couple months.
We’re all used to getting alarm bells and fighting fires, but I don’t tend to jump to arms quite as quickly as most security — because the reality is that the majority of these alarms are more smoke than fire. What I mean is that most bugs aren’t actually exploitable. But when the alarm bells started ringing around Log4j — I jumped.
When I sat down to dinner with my founding team the night of December 8th, 2021, very few things could have kept me from the margarita and taco salad I had ordered. But Log4j was one of those things.
Because the nature of our research work at Randori is to identify applications we feel can be exploited, if not now, then in the near future — we had somewhat of a headstart when Log4j broke as we’d already been looking into systemic risks in java applications. Our Principal Scientist, Aaron Portnoy, was tracking some chatter kicked around on Chinese twitter, and the research team was busy looking for new ways to attack VMware targets at a few key accounts.
When I sat down for dinner, nothing suggested I’d soon be pulling an all nighter. Halfway through dinner, our VP of Hacker Operations, Eric McIntyre got a text from Portnoy. It was happening. We called VMware immediately.
How We Responded to Log4j
After literally sprinting back to our office from dinner, the Randori team broke up and began a three-pronged approach.
- Understand our own internal exposure.
- Ensure our customers knew their exposure ASAP.
- See if we could prove exploitability and test our customers’ defenses.
I ran a triage of the security incident or potential security incidents with our engineering and security teams. Meanwhile, Eric assembled our hacker operations crew and went into attack mode while our targeting team began notifying customers and figuring out what software was truly vulnerable. We quickly discovered — it was a lot.
Our attack team had a Log4j exploit within the hour, were able to confirm VMWare and Jamf were exploitable within the next hour and establish footholds in several customer environments that same night. Once confirmed, I immediately got on the phone and began working our network to escalate with affected partners and vendors.
At the same time, our marketing, comms and customer teams were engaged in notifying and working with customers to understand their exposure. We had to alert our customers because we saw log4j instances all over the exposed internet and we wanted to immediately secure our customers and take them off the chessboard for attackers.
Over the next 48 hours, our team identified thousands of vulnerable targets, worked with our customers to get those removed before widespread exploitation began later that week, and partnered with the team at Greynoise to help keep the broader community informed on the latest updates and recommendations. Within 7 days, we’d published our team’s internal list of vulnerable applications online at www.canilog4j.com to help folks trying to understand what applications are vulnerable and we’re offering free scans to anyone looking to understand their exposure. We continue to update that inventory as new information arises and there for organizations looking to understand their exposure.
We ultimately determined that our own systems as well as most of our customers were relatively safe. Even though they had log4j on their networks, they were blocking outbound traffic. This means even if an attacker were to gain access to their valuable systems, they still wouldn’t be able to immediately exfiltrate any data.
What Can We Learn?
Just over a month out, I’ve found myself starting to reflect. The most poignant words I have heard about Log4j 2 so far are, “It’s a lot easier to fix a bug than it is to remove a feature.” That is essentially what Log4j 2 is: a feature. It is a (sometimes vital) logging tool. The big question facing the cybersecurity community in recent months has been, “What happens when systems are exposed and we can’t patch our way out of trouble?”
Log4j was one of the worst vulnerabilities I’ve seen in my lifetime, and we may still be months away from adequate remedies for many of those affected. While a good deal of folks can shut off their affected systems, many can’t take this simple step because they don’t know if their application is affected. Or even worse, owners of appliances might not even know if they are affected.
When these roadblocks inevitably occur, they cannot be the downfall of the entire system. We must build resilient systems that can continue to function in the modern threat landscape, in which occasional compromises are a given. With stopgaps in place like blocking or monitoring outbound traffic, implementing default-deny and taking affected systems offline, we hope the next generation of security programs will be able to absorb exposures the way that ours and our customers’ did, even with something as ubiquitous and easy to exploit as Log4j.
