Black October Debrief

Brian Moody: You know, so much happened in October that Cybersecurity Month got kind of nicknamed Black October. Why? You know, and we look at PHP incident, right? You had ConnectWise incident. We got a Beyond Trust incident, right?

Shahin Pirooz: LiteLLM.

Brian Moody: I mean, it was just one after another and it seemed like the headlines each week just continued to be kind of another security issue, another cybersecurity issue. So the industry called October now Black October.

So that's the topic today is really with respect to this is the breach is walking through the front door. And I think you guys have heard both Shahin and I say on multiple occasions, hackers are no longer breaking into your network. They're logging in.

Shahin Pirooz: Yep.

Brian Moody: And, you know, if you look at the capability with respect to each one of, in a sense, the security incidences that we talked about, they weren't break-ins. They were using credentials. They were using application credentials. They were using the capabilities of flaws that existed in code to get in and get access to the environment, without security tools really recognizing what's going on.

Shahin Pirooz: 'Cause it looks like normal behavior.

Brian Moody: It looks like normal behavior.

Shahin Pirooz: And we're not monitoring application keys the same way we monitor user behavior.

Brian Moody: Well, you know, we talk about MFA and how MFA helps us, right? So we're gonna dive into this a little bit. You know, AI, you know, AI is so dramatically shrinking the time to recognizing a flaw and the ability to exploit it. So I think what challenge that creates for you folks watching today, the challenge that I think it creates for all of us is the time between finding a flaw and then being able to patch it. And I think someone, one of the headlines was, is patching now needs a stopwatch, not a calendar. From a standpoint of literally we are chasing patching so quickly and there's some additional challenges here that we'll talk about associated with that.

Shahin Pirooz: Well the perfect example is the PHP flaw that came out. You know, PHP is on almost every website in the market today.

Brian Moody: Over 70%.

Shahin Pirooz: Yeah, it's everywhere. It's prevalent. And this isn't a new flaw that got identified. It's a flaw back from 2024. And that flaw was identified because analysts were using AI tools to identify vulnerabilities in the platform. Now, finding a vulnerability and taking advantage of it, exploiting it, are 2 totally different things. And in this case, there's all these alarms that went off, and there was 2,900 organizations that got hit rapidly with this flaw, but there were only 300 real attacks that happened, or successful logins that happened as a result of this thing.  

So, when we look at these alarms and the end of the world is nigh kind of fearmongering that is happening in the industry to sell technology, yes, AI is accelerating things. Yes, bad actors are able to do things. Those 2,900 attempts happened in minutes, not days or weeks or months. The implication, though, is not all of them were successful, and it's because the exploits are harder to do than simply finding a vulnerability. So we always look at things— when you're looking at a vulnerability, a vulnerability by itself, a CVE, means nothing. But is it exploitable? And is it exploitable is layered by multiple things. Number one, Is there an easy way to get access to it? Is it externally facing, or does somebody have to break into your network to take advantage of it, or walk into your network as Brian was describing? Is it behind multiple layers that they have to bypass to get to it, or is it sitting on the surface and easy to access? Is it going through layers of the application itself, not just the network, before they get to the exploit, so they have to take steps in before they can go.  

All of those things add complexity to an attacker's ability to exploit something, and if it's not automated, a bad actor isn't going to waste their time. The thing that we should be scared of is AI is allowing us to automate these things faster, both on the detection side and on the exploit side. However, just because there's an exploit available from 2024 doesn't mean everybody's got to run to the hills screaming and yelling. But you do need to understand what you have in your network and how to patch it. I think somebody I read likened it to the Y2K issue. Y2K was potentially a major problem, but all the IT folks everywhere got together, did what they needed to do, and it turned out to be much ado about nothing.

Brian Moody: Right. Well, a lot of work got done.

Shahin Pirooz: Right? A ton of work got done.

Brian Moody: A ton of work got done to limit the impact of what that might have been.

Shahin Pirooz: Same thing with PHP. There's a lot of work that happened by a lot of security operations folks as soon as the PHP alarms went off. We did our— we definitely trumpeted our part to let people know you gotta go do this, and a lot of people moved, and that potentially is why the impact was as small as it was.

Brian Moody: Well, I think one of the huge things about the PHP thing was is, as you said, we didn't see really, if you looked at the number of environments that were affected or that had the flaw, how many actual breaches occurred were tiny. Tiny. It was almost minimal. I think the bigger focus is on what happened inside the organizations from a standpoint of understanding, oh crap, we have this. But then the bigger question was, is Are we running this? And how many—

Shahin Pirooz: Or do we have this is the bigger question.

Brian Moody: So dive into that a little bit is, do we have this? So not even being able to know, how do you patch something that you don't know if you don't have it? So I think the realization for a lot of organizations was, is we might not know what we have. Are we even running this?

Shahin Pirooz: Yeah, I think the really important part of doing vulnerability detection and scanning and understanding what you have, the asset discovery in general. I've always said, without an understanding of what you have, how do you protect it? It's impossible to protect something you don't know about. Most of the solutions in the market today really are doing surface scans on the operating system and whatever applications advertise themselves, but fingerprinting is something that falls short in most environments. And it's a critical thing that has to happen when you're scanning the network and doing an asset discovery, fingerprinting those systems to understand what's running on them and what is accessible, then gives you the ability to say, based on those fingerprints, I'm going to go and do a CVE analysis against the vulnerability database, National Vulnerability Database, and then attempt to exploit those vulnerabilities to see if they're exploitable.  

Those steps have to happen in order, and if fingerprinting is missing, you're going to scan a Linux server for vulnerabilities for Linux, but you won't scan it for what web server is running on it and what framework is underneath it. If that web server has PHP running on it, or if it's a JavaScript-based solution, or whatever the case may be, you won't know those details unless you're properly fingerprinting the platform.

Brian Moody: Well, and I think to extend that even further is it's not just about our servers. So here's the other aspect of think how many of you host, right? So is your hosting company running the PHP, right? So it doesn't just impact your environment, it extends out to your vendors and to your hosting companies and the folks that may be running that similar infrastructure.

Shahin Pirooz: We have many of our customers that are using our external security posture management solution to speak to their web hosting vendors about flaws they've identified with our tools that that hoster needs to address.

Brian Moody: So I think really around the PHP, when you step in and look at this, AI, clearly the world we live in today is AI is kind of advancing discovery around code that exists in our world today. And, I kind of said, you know, that we talk here and this topic is, you know, the breach is coming through the front door. I think another question here is, what's hidden in your walls? What already exists within your organization? Because what's interesting is, like you said, you go back years and years with respect to this PHP issue. It goes back to some very old code that's been running for a very long time. AI is now helping us find flaws that exist in code that has passed many a security test.

Shahin Pirooz: Oh yeah. And the other thing, so remember, the National Vulnerability Database is only updated when a vulnerability is identified and submitted and recorded. So if nobody found that vulnerability historically, it's not in the NVD. Yeah. So, no matter how many scans you do, it won't show up as vulnerable until something like this happens and it gets into the NVD. And now we have a fingerprint we can go after. But that fingerprinting doesn't fail you because understanding that you have PHP in your environment was the first step in the solution, not if that version of PHP was vulnerable or not, but yes, there's PHP running in my environment. Right. The— I would say there's the other factor that you talked about is we have people hiding in our house. It's worse than that. We have people who have the keys to the front door and they can come and go as they please. It's not just that they're — they're not hiding. They walk in when we're at work and we're not paying attention.

Brian Moody: So I would say for Black October, strike one, right? Let's dig into the ConnectWise kind of the backup issue a little bit, because that created kind of a little bit of a different issue from a standpoint of how they got access. So this wasn't a username and password access. This was application access.

Shahin Pirooz: Yeah, so in every environment we give application context and application access, if you will, from registering this application to be able to do certain things. In the context of backup, we're usually giving the backup technologies access to all of our data to protect it, but we don't give them more credentials than needed. We're giving them, you know, backup operator-level credentials so that they can back up the data. It still gives them visibility to the data, and those credentials are dangerous if they get into the hands of a bad actor. But in this case, the bad actor actually elevated privilege of the backup account to a global administrator, and there is zero notification monitoring of the app's registrations to say that they got elevated credentials because it's not a normal credential.

And we keep saying, every agent, every identity, every user, everybody needs to have their own identity, and that identity needs to be tracked consistently, needs to monitor access level, needs to monitor the controls associated when elevation of privilege happens, impossible logins. All the things you do for a user, you should be doing for any kind of app registration, and that's really the missing piece in the industry today. And what ended up happening was the bad actors got the key for this app registration, and they were— because it's a multi-tenant platform and it backs up multiple customers, that key was across tons of customers, not one customer. And that's the impact in an MSP environment.  

As an MSP, when you do a multi-tenant solution that you apply across everybody, if that multi-tenant solution is just rolling out across everybody with the same key and the same creds, bad actor only needs that one key. And that's the risk of the benefit, the risk side of the benefit of having a multi-tenant solution that rolls easily is that the access is also easy. And what we see consistently is that, you know, the very follow-on question to what about MFA? Did MFA not work? There's no MFA on app registration.

Brian Moody: You stole my next question. Sorry. That's what I was gonna round back to is, Wait, MFA will protect us, right?

Shahin Pirooz: There is no MFA on application registration. So those credentials sit there and they're a key, and key leakage is a major problem. And, you know, we'll talk about it when we talk about LiteLLM because that was a key leakage scenario. And key leakage, once somebody gets a key, because all app registration is, is a key. And when they have that key, they're they're not authenticating in a traditional sense. They're just using those keys to get into—

Brian Moody: But wait a minute, I disabled the application, so I'm good, right?

Shahin Pirooz: The key's still there unless you disable the key. The key stays.

Brian Moody: Absolutely.

Shahin Pirooz: Somebody has the key to your front door.

Brian Moody: Right. So that's one of the other key pieces is people thought, okay, well, we'll disable the application and disable that function. They came in, elevated privilege, they have the key. Disable the application, fine. They have the key to the door.

Shahin Pirooz: And people were deleting the application. They were deleting the key, which loses all forensics, and you have no ability to go back and find what's going on. So disabling the key or changing the key is the only way you stop that, and then you don't lose forensics data.

Brian Moody: Well, one of the other pieces too is around this is I haven't used that application for years. So this was another piece as I was doing a little research on this is, even if that application's there, you had provided access or key to it. I haven't touched it in a long time. They walked right in with it.

Shahin Pirooz: Let's say we wanted to do a POC of an application that goes and scans our Active Directory to see who has bad accounts or not, and we decided it was too expensive. We didn't want to go with it. Guess what? We never disabled that app registration. It still sits there, and if the bad actor gets their hands on that key, they're logging in with the same credentials that you gave that application, and the majority of the time in POCs, people give global admin because it's easy.

Brian Moody: Yep. And they don't necessarily uninstall it. It's sitting there, and it's, you know, again, you've talked to about how we put some of the best locks on the front door, but in the same token, we extend those keys out to users, applications, agents. We just, you know, think about being in your neighborhood and then basically giving all of your neighbors a key to your house. And then you go back and go, well, who's got a key to my house?

Shahin Pirooz: My lock's all biometric. How'd they get in?

Brian Moody: So let's jump to LiteLLM. So for those of those that don't really know what that is, walk in a little bit to talk about what that is and really what the exposure was, because I think it points out a couple of interesting things, too.

Shahin Pirooz: Yeah, so LiteLLM is a gateway that allows you to store your keys for the frontier models that you're using. So you basically create a single gateway that you can connect to Claude and OpenAI and whatever, so that you're able to have your agents talk to the gateway, whatever you're writing code, you talk to the gateway, and it simplifies your ability to talk to all these different solutions. Great solution for being able to talk to multiple LLMs at the same time in a simple way. The scenario was that one of their developers— they were, first of all, kudos out to LiteLLM because they were very open about how they shared the breach and what happened, which helps us all understand how to address it and how to fix it. One of their developers made a mistake and left a key in the code, and that key was leaked and the bad actor used that key later to repoint the repository to their fork, which they embedded malicious code into.

Brian Moody: So now what we have is a poisoned—

Shahin Pirooz: Yeah. So now when somebody's downloading an update, the update's coming from the fork instead of from LiteLLM, and that fork is a poisoned well, effectively. So this is, again, a situation where monitoring key usage and making sure there's no key leakage in your code repositories and all that is critical, and it's really frickin' easy to turn on monitoring of that in your — if you use GitHub, GitHub has native tools that will block and prevent you from putting keys in your repositories, but people don't turn it on because they consider it an expensive feature. It's much more expensive to lose all your customers than to block a key.

Brian Moody: But the important point of this is, this is AI tooling. This passed all of the kind of AI evaluations and all the safety eval— it passed all the evaluations. So I believe some very, very smart man talked about this back in August, maybe, when we talked about vibecoding.

Shahin Pirooz: Yeah, and it's the same thing. The opportunity to poison the open source stuff that's out there is way too easy because the benefit of open source technologies is that you have a massive development environment. You have a ton of people who can contribute and make the technology better faster, and the large number of contributors makes things move faster. The negative to it is there's not a standard in terms of how they all code. There's not a consistent way. There is no engineering body that is monitoring it, and it's all left to the person who built the open source project to put some controls in place to monitor the thing. It's impossible to monitor thousands of developers contributing to an application. And should one of those developers contribute really good code for a long time to set themselves up as a solid player and then decide they're going to do something bad because they were basically honeypotting the solution, if you will, the implication is there's a lot of code out there that has risk to it, and how do you check it?  

You need to have scans run on those codes. There's plenty of tools in the market, again, that do open source application scanning, look for what dependencies there are, if there's leaks in it, look to see if the dependencies are missing that should be there, if there's things that are there that shouldn't be there. All of that is like available, and there's free tools, there's commercial tools, there's a lot of ways to address this. People aren't doing a good job of doing application-level security, which is, you know, as we've talked before, which is why we're putting a lot of energy into our application security posture management.

Brian Moody: Well, I think one of the keys that you've talked to all the time too is keep a human in the cycle.

Shahin Pirooz: Always.

Brian Moody: Right? And we've talked a little bit about this.

Shahin Pirooz: But in this scenario, that doesn't help here. It's like, if you think about it, you've downloaded LiteLLM, you've vetted it, you've made sure it does the things it's supposed to. It's running in your environment, it's been up and running, and through no mistake of yours, the manufacturer made a mistake, and now the bad actor is able to take advantage of that mistake, and unbeknownst to you, in the next update, malicious code comes down.

Brian Moody: So I guess that the tough question is, and we've seen this a couple times, I mean, we don't need to go back and rehash the whole CrowdStrike thing. I mean, that wasn't really a breach much more than a takedown, but it was not any fault of any of the folks that were running those infrastructures. So I guess the question is, how do you protect yourself from something like that?

Shahin Pirooz: The key thing is to make sure you're continuously doing proactive scans. For example, the only way to have identified this would be that the source repository that the updates were coming from changed.  

How many people really pay attention to where the source repository is? Is all of a sudden the repository changed to a fork instead of the production one? If somebody was monitoring that to say, wait a minute, something stinks here, we all of a sudden changed where we're getting the code, don't do it, you could put controls in place to address that. And that's a great thing to do for any products that you're getting from an open source platform because you're always getting the updates from a repository that's published publicly.  

And the implication of somebody taking and hijacking that and playing man-in-the-middle are high enough that just like you look for man-in-the-middle on your IP networks, you should be looking for man-in-the-middle on your code repositories. So if all of a sudden the repository address changes, an alarm should go off. And that requires application registration. That requires monitoring of where the traffic is going to, and that requires a baseline behavior to say, for the last year, we've been going to this URL whenever an update happens on the 3rd of the month, all of a sudden we're going to a new URL. Alarms should go off and somebody should inspect.

Brian Moody: So what security components might someone offer?

Shahin Pirooz: That's the application security posture stuff, and just standard network monitoring to see traffic.

Brian Moody: Right. So like we always do, let's pull this together. So we've talked about kind of some key things. I mean, if you think about the PHP thing, that's really flaws that come from code that we may be running and things that we need to do. If you step into and now look at the ConnectWise component, that's application type security. So that's—

Shahin Pirooz: Application IDs.

Brian Moody: That's application, that's so completely different. Now you look at the LiteLLM, and now we're getting to kind of the advantage of taking advantage of AI code. I mean, how are our MSPs and really how are our customers really keeping up on this kind of thing? And so, as you do so well, I'm sure you've got some takeaways.

Shahin Pirooz: We do. We have a takeaway for you. You'll be able to access this after this session.

Brian Moody: So if you start top to bottom, I mean, as we walk through this, what are the key takeaways from your perspective, your seat, how folks can address this.

Shahin Pirooz: There's 5 things that I think you should be really asking yourself. The first one is, do we run this thing? And you need to be able to answer that in hours. And the best way to do that is to keep a live inventory of all your internet-facing assets, your software and the vendors, and test it from time to time to make sure that the random library is not doing something different than what it does. And you're able to answer your team's question. So have your team answer the question, how many of these are we running? And your team should know.

Brian Moody: Well, this gets down to reaction time, ability to react. And then if you begin to talk patching, you know, how do you know what to patch if you don't know what you have? Right. So, and as we said, how many CISOs panicked when the PHP thing came out and they asked the question, hey, do we have this? And the answer was, I don't know.

Shahin Pirooz: Right.

Brian Moody: How much time transpired between getting to that and being able to respond to the fact that we might need to patch something or not.

Shahin Pirooz: [Coughing] Excuse me, I got something in my throat. Here, I'm gonna have you read the other 3.

Brian Moody: The second component that you've highlighted here is audit your third-party apps. Have the ability to really pull every app with access to your Microsoft 365, or your cloud tenant down and remove what you don't use. Make sure that you've got the ability to reduce permissions on what's left there, 'cause every app needs an owner. So very similar to what we've talked about in past soundbites from a standpoint of even agents need an owner. So there needs to be oversight from that perspective. Turn that back over to you.

Shahin Pirooz: Where'd you get to?

Brian Moody: Now we're on number 3.

Shahin Pirooz: The next thing is alerting on human identity changes. Any new credentials that are added to an application should trigger an alarm for somebody to inspect. [Coughing] Sorry, it's not working.

Brian Moody: So you can say we do this live. So alerting on non-human identity changes. Number 4, Get your secrets out of the open. I mean, from a standpoint of scan your repositories, rotate anything that's been exposed, and shorten the life of those tokens. Number 5, I mean, practice your response.  I mean, if you've never run a rehearsal, you don't know what you're able to do and the time with which you're able to accomplish those tasks. So practice your response, know what services that you have to restore first, you know, what each critical vendor that you need to get ahold of in order to be sure that your tenant isn't got leftover persistence or a supplier has a compromise. Anything additional here?  

Shahin Pirooz: No, those are those are the big 5. And there's other things on here which talk about the lesson learned from Y2K. So there's a little bit of detail on the one sheet.

Brian Moody: So we'll make this sheet available, obviously, like we do for you to access. But I think there's some great lessons today, and I think the Y2K piece comes into play as kind of a key component of there's work to be done behind this. And I think like the Y2K lesson shows us, we put effort in and have limited impact. So putting that effort into your environments to make sure we continue to do the work. Again, as we know in cybersecurity, there is no endgame. This is a battle that doesn't end. It's a constant fight. And I would say too that we're here to support from a WhiteDog perspective and our infrastructure capability. you in any way that we can. So with that, I hope to see you next month. We may not see Shahin here next month. We're gonna go to get him some medical assistance. And we look forward to talking to you again in December. Thanks for joining us.

Shahin Pirooz: Thanks, everyone.

‍