The Cyber5 Podcast

ERIC: Part of our challenge is to say, “Hey, we’re hacking you, but we’re here to help.” And I think sometimes it takes a week or a month or a couple of months for the organization that we’re attacking to realize we’re actually not your enemy.

LANDON: Welcome to the award-winning Cyber5 Podcast. Here, we discuss the most relevant cyber and physical security challenges facing enterprise businesses today. I’m your host, Landon Winkelvoss, co-founder of Nisos, the Managed Intelligence Company. 

Welcome to episode 84, everyone. We’re glad to have the CrossCountry Consulting Team with us today. Before we kind of kick off, let’s do a round of intros. Brian.

BRIAN: I’m Brian Chamberlain. I am the R&D lead and offensive testing, I don’t know, what is my role? You put me on the spot now, and I just don’t even know. –

GARY: Whatever you want it to be today, Brian.

ERIC: He’s the wizard of Red Team.

BRIAN: Yeah, I’ve been given many names, most of them unkind. No, I’m the offensive R&D lead and also I do threat simulation operations.

LANDON: Awesome. Eric.

ERIC: Hey, I’m the team lead for CrossCountry’s icebreaker team that does all kinds of offensive security testing and threat modeling.

GARY: And, I’m Gary Barnabo, the market integrator for Cross-Country’s Icebreaker Team. I work with the really cool guys on the podcast, Eric and Brian to help translate, and take their findings and insight to C-suite and board level business leaders. 

CrossCountry Consulting is a 12 year old, full service, business and technology consulting firm of which cyber and privacy that is a key service line and offering for the firm. CrossCountry serves clients across the United States and increasingly around the world. 

Our cyber privacy service offerings really focus in three main areas. One is strategy and transformation, helping CISOs, CIOs, Chief Risk Officers, Chief Audit Executives, transform their security programs and capabilities to stay ahead of threats. The second is around the technical capabilities that we deliver like Gladd Security, application security, security architecture, and engineering, and then last but not least is our ice breaker mini brand or offensive security unit that Eric leads and that Brian’s the head of our R&D for where we emulate and simulate adversaries to share how advanced rights could cause real catastrophic damage to our clients.

LANDON: When it comes to their reactive and proactive side, there’s certainly no two services that are more actionable, you know, in the security landscape than adversary simulation, emulation, and certainly threat intelligence as well. 

And, that’s kind of what we’re gonna be talking about today. So, you know, kind of let’s set the table stakes real quick. In this context, what’s the difference between adversary simulation and emulation? I think that’s an important nuance to kind of really kick off and kind of get the discussion started.

ERIC: These definitions have sort of been solidifying over the past couple of years. And, I think where we’ve come is emulation is, you’re emulating a specific threat actor and a specific campaign. So, North Korea, Lazarus Group attacks such and such and they used this type of persistence technique. There’s this kind of payload. They used this kind of lateral movement, and you’re essentially replaying those exact steps. 

So, it’s an exact reproduction of an actual recorded event. That has some advantages and disadvantages, right? You can script it. You can replay a lot of these. You can score it easily. You can say, “Hey, we detected 47% of this entire attack stream.” And then, you go back and you make some fixes, and this time we detected 67%, and you can measure your progress. So, that’s some of the advantages. 

Some of the disadvantages are you got to think about what you’re really being measured against are the attacks that were detected and caught and documented. So, when you reach 100%, all you’re really 100% protected against is those attacks that have already been, they’re yesterday’s attacks. And, they’re yesterday’s attacks that were caught. So, simulation now is more, allows a little more improvisation. Hey, this technique didn’t work, but like a real adversary, I’m not gonna stop and cry and go home. I’m gonna adapt to something else. Let’s try something else. Let’s try something else. And, you’re resilient, and you persevere until you try to reach that goal. 

So, it requires a tremendous research and development effort. And, keeping tabs on what are the latest techniques that the security controls are using and how do we get around those controls? How do we evade detection from those controls? At the end of it, you have, you don’t really have a score. You can kind of score them, but it’s not as precise. It’s more like you have a story, not a score. And, that story can be really impactful, to say we started with nothing, we spearfished through and we ended up with all of the PII, or with the intellectual property or with the financial data.

LANDON: Let’s get in the technical weeds here for a couple of minutes. Give me a couple of examples of what he just kind of just described.

BRIAN I think a really good example is if you look at like, let’s say we’re doing a fishing campaign with a payload. Now, we can say, “All right we are going to be, we’re gonna emulate, you know, Conti Group using Bumblebee.” Right? And, we could actually pull down a copy of bumblebee. We could, you know, defang it, add our own stuff to it if we wanted. We could use exactly the same payload, right. They’re gonna use an ISO, and, they’re gonna use a link file in that ISO. That link file is gonna call Run DLL 32. And, call their, you know, whatever dot DLL. And, it’s gonna run, and it’s gonna drop other files. I think, really popular right now is they’re dropping like a signed AVAST.exe file somewhere with a DLL sideload for that. And, that’s gonna load, you know, practical bug strike. We’ll say, I don’t know if that’s exactly right, but that chain of events has definitely happened a number of times in the last year by somebody. 

Now, we can go through and do each of those things exactly, right. But, there’s a number of problems with that like Eric said. So, maybe what we would do for a threat simulation is say, “Okay, well, we’re going to use ISOs.” ISOs are hugely, hugely popular, and they’re popular because they’re successful. They’re difficult to deal with right now. And, maybe we’ll also use a third party signed exe, but instead of using the AVAST one, which has been well published and known about, we’re gonna find our own. And, instead of using some malware that’s well known, we’re gonna use our own malware that doesn’t get detected.” 

So, now, what we’re forcing the customer to do is instead of detecting, you know, we’re instead of creating a YARA rule for Bumblebee, or detecting on a specific file name, or a specific command line, we’re forcing them to look at a higher level of, okay, well, we want to figure out how we stop ISOs, or detect ISOs being mounted by users from the internet. That is a fundamental core element that we would do in a threat simulation that maybe we’re gonna do a little bit differently, or we may deliver it a little bit differently. We may do, like I said, instead of looking for a specific type of DLL side loading attack, we’re gonna force you to look at unsigned DOLs in user space. 

And, we’re looking more for a higher level detection strategy, and really forcing you to kind of challenge how you do things. However, like Eric said, it does become a little bit harder to rate. And, also, and we’ll talk about this a little more, I think, but one of the things you said is kind of the relevance between the threat intelligence that is now true. And, what we do, which is sometimes a little bit niche. And, that can be a little bit of a problem, where so many threat actors are doing these things like PowerShell one-liners, still. I haven’t run a PowerShell one-liner in four years. And, that can cause a little bit of friction. 

And so, being able to do some kind of atomic level testing of actual, you know, pulling a payload out of a FIN7 attack, running it in your environment and seeing how effectively you would have detected it. It can be a very effective and cost effective way of making sure that you’re resilient against some of these, especially commodity attacks.

LANDON: So I guess walk through what execution actually looks like from this perspective. How you weave in threat intelligence into this simulation, right? ‘Cause, if I understand what you’re saying correctly if, you know, a particular domain, an IP, and a malware string are actually run by a FIN7 or, name your actor group. The emulation part of that would seem to be challenging ’cause you can switch up that domain and IP literally in seconds. Right?

So, now all of the sudden that, quote, IOC is almost irrelevant, right? And so now, then you gotta, now we’re in the simulation type of game, so how does execution actually look like in that regards, how do you bring those things together and actually add the value to the clients? And do that in a cost-effective way that brings the teams together?

BRIAN: On the one hand, I think there are higher level concepts that can be gained from consuming threat intelligence for us and for the customer beyond, now the nice thing about IOCs in threat intelligence, right, IPs, domains. 

They’re immediately relevant and they’re 100% true positive. So they’re hugely cost effective. Right? Read a report, copy and paste, check your logs. The problem is threat actors are changing constantly and also, there’s kind of, what I like to say is infinite number of threat actors along an infinite timeline. 

If you, for you as a company, you, there’s no, kind of, limit to the amount of attacks you’re going to be attacked or the amount of threat actors that are going to attack you. And so, the payoff for doing that constantly, and we’ve seen this, right? This is basically how antivirus worked for years and years and years, and the reason that antivirus was a joke for years and years and years. Because it was wholly reliant on these kind of static IOCs, right? We’re going to detect X type of malware, and we’re going to detect only this one version of X type of malware because it has this one string in it. 

And so, what we’re seeing more and more of, and a really key piece is, defensive technologies have gotten better by focusing less on specific kind of YARA based IOCs, which are still relevant because they’re cost effective, but more on cureistic, right, we’re calling these APIs in this order. And things like that, and that’s where we come into play, we’re, it doesn’t matter that I’m running X or Y malware, you know, we run a custom implant, and that’s great and that evades all that stuff. But I still have to get it running into memory. I still have to enumerate your domain. I still have to do, you know, lateral movement, find your PII, X fill that PII. 

And all of these things are what we really want to push our customers toward, is the, it’s harder to do, it’s less immediate value, but if you can stop HTAs from deploying in your network, you just shut down an entire threat element. You just shut down an entire campaign for many of these threat actors. Right now HTAs aren’t that common because so many people did that, but, ISOs are the next one, right? As soon as people start figuring out ways to concretely control ISOs across their user base, which in some cases is really hard, but if you can do it, you just eliminated like, 90% of published initial access methods that don’t involve X Flick, right? 

Macros are the same way, places that controlled Macros just stopped worrying about a lot of phishing attacks. And that’s really what we’re trying to push our companies to do, our customers to do, but, and that’s what threat simulations allow us to do is, use our own custom tools, bypass those EDRs, bypass kind of, a lot of the crutches that a lot of these companies use, right? ‘Cause we see this a lot, companies are taking cybersecurity seriously, but they’re taking it seriously by plugging in tools and if those tools fail, they don’t have anything, they don’t know what else to do. 

Or, if those tools succeed but the threat actor is, you know, four steps deep in the network, they don’t know how to do instant response enough to actually backtrack and kick that threat actor out. And that’s another big part of what we’re doing is threat simulation is evoking that instant response, working through that whole process with a fake threat actor, but a fake threat actor that understands real threat actors, and is acting like them. And generating artifacts like them. And I think that’s really important and it allows people to kind of get off that crutch of IOCs, off of, you know, default detections, and really build robust, difficult to bypass defenses.

LANDON: Eric, how hard is detecting behavior? And how has the, how have your customers, have they gotten better at this, or is this still a huge challenge within enterprise?

ERIC: You know, part of adversary simulation, a big part of it, is training infinite response. You know, as Brian alluded to, we had a recent occurrence where, they detected, there was one detection by an EDR of one file and someone went in and deleted the file and said, “Okay, the alert went away, I’m all done here.” And there wasn’t really a procedure, or any real thought about hey, there might be a bigger problem here, we might want to do a deeper investigation. 

So, I think it starts with an incident response like plan, and procedures, and knowing, okay, this is an incident that we need to investigate further, and we need to determine the full scope of it before we can say that we’ve cleaned it up. So, yeah, some of the, you know, indicators you were asking about, like you can do things like, geographic locations, where is an IP, where is a certain person logging in from, are they logging in from Korea one day and the U.S. the next day, and France another day, you know, those kinds of things can be indicators, times that you log in, there’s unusual activity indicators that you can put in. But, yeah, incident response training is part of what we do.

LANDON: Let’s talk about how we try to make this relevant to a Chief Financial Officer and Gary, I’m sure you have quite a bit to say on this from this perspective, right? Security generally is an administrative function, just like with IT and HR, usually rolls up to a Chief Financial Officer in a typical company, it’s probably a little bit different sometimes in technology companies but, by and large, you know, manufacturing, retail, insurance, you know, healthcare, that’s usually how this kind of plays out. 

I would probably venture to say that everything, and I could be wrong about this, I think it’s getting better, but 5 years ago, if you said exactly what Eric and Brian said to a Chief Financial Officer, I mean they are getting really uncomfortable really quick ’cause they just do not understand. I think that’s changing a little bit. How do you quantify and qualify adversary simulation for a Chief Financial Officer?

GARY:  ‘Cause, a just play on your sort of premise, that the tide is maybe turning in terms of a CFO or a C suite acumen and understanding of cyber. And what I think about it is, look, 20 years ago we started having conversations that anybody in the C suite, regardless of the role, had to be converse in technology, just with where the world is going. But I think we’re getting to a point now in the 2020s where it’s a little bit of the same story around cyber. 

Everybody’s converse on technology, now there’s gotta be a basic expectation that if you are in a corporate leadership position, you’re an officer of the firm, if you’re a CFO who’s gonna be a top 3, top 5 leader, executive, at a corporation, you’re not expected to be fluent in cyber, but you have to be conversant in it, enough to get by, enough to ask good questions, enough to be able to make good judgements, enough to be able to interrogate a system and say why are you asking for this much in your budget? What’s the return on investment that I’m getting? It’s gonna take time for the CFO community to get there, but I think some of the top CFOs are starting to embrace this, of “hey, it’s part of the management and leadership agenda of the corporation, it’s something I’ve gotta lean into as a corporate officer and leader in the enterprise.”

But look to your point about how do you quantify it, I think one of the first things that we’ll often talk about with the CFO or the C suite leaders is look, just look at the cost of executing one of these adversary simulations that Eric and Brian have described, versus the cost of any sort of a breach, it’s gonna be 10 X less, it’s gonna be 100 X less, all right? 

So the ROI, the value, benefit minus cost, that you get from a 150K, 250K, you know, really rigorous red team, is significantly less than any sort of consequence that you’re gonna be facing financially, but, from even a small scale attack or breach. 

And I think that message sometimes resonates, right? It’s sort of a worthwhile investment for what it may be able to prevent. Now, to me the second thing, I’ll play on something that Brian was talking about, about this almost fetish of installing controls and tools. This explosion of tools, we’ll say look, you’ve been on this path of putting a lot of things, fancy things on your networks, this is how you can really see if those are generating value for you. Do these tools work? Do the processes and people and procedures that we wrap around all those tools and technologies work? And give the CFO confidence that okay, that outlay I have been spending on security for the past several years, is actually doing some good for the company, or learn that maybe it isn’t, I really gotta sit down and work with the CIL, the CISL, and their teams, to see if there’s maybe some changes in our spending priorities that we should make as it relates to the tool stack. 

And I think too, there’re still, there’s still a tendency for CFOs to say hey look, this isn’t affecting my corner of the world, this isn’t affecting the finance function of the organization, so we’ll sit down and say well actually, it is, you hold, you’re responsible for some of the most valuable data and information in your corporation. M and A data can be highly valuable to an attacker. Anything around long range financial plans, near term financial plans, budgets, can be highly valuable to an attacker. 

Now just, just look at some of the technology that’s used in finance functions today as well. Those things have vulnerabilities. If you can’t operate payment systems, pay your vendors, take payments from your customers, you’re going to be in a world of hurt. So we can devise scenarios for these adversary simulations that really get at the heart of the functions that are central to the CFOs.

LANDON: What I hear you saying is that oftentimes, these types of simulations often become highly political in organizations, would you agree with that?

ERIC: 100%

GARY: I think Eric could probably talk to a few, a few examples at Ohio Wealth.

LANDON: That’s where I’m going with this, I mean I would love to hear those examples, right? And those examples I’d like to really extrapolate like, the where, like the business actual functions, like where the rubber met the road with the business I think that that’s a nuance that still you know, very much, nuanced in our industry.

ERIC: Sometimes we’re brought in by someone who’s not representing the CISO or the CISO’s team, and it could be executive, or the board, or the audit department, but we’re brought in by some other party to sort of evaluate the information security program as a whole. And that can lead to bad feelings, it can lead to hostility, and it can lead to an adversarial relationship between us and the, the organization that we’re testing, and I think that’s part of our challenge is to say “hey, we’re hacking you, but we’re here to help.” 

And I think sometimes it takes a week, or a month, or a couple of months, for the organization that we’re attacking to realize we’re actually not your enemy, we’re not here to make you look bad, and spit in your face, and embarrass you. We’re actually here to find those problems and help them to get better. And I think this takes a lot of finesse in how you describe the problems and humility on our part that, you know there’s a lot of ego in in the security community and we have to kind of leave that at the door and say look, everyone’s got problems, we’ve seen this before 1,000 times, let us help you to fix it.

LANDON: How do you have, tell that story but not make it look like a liability.

ERIC: Yeah, carefully, you know, if we’re coming in through general counsel or coming in through audit, like those, whatever we find it’s going to be collected by that organization and used. So we just have to be as transparent as possible like hey, we found this, you’re gonna get some hassle from these people. But the end goal is we want to be more secure as a result, so you’re gonna be hassled but let’s not make that hassle the focus. 

The focus is let’s all get better together. There’s different mindsets of people when they ask for an offensive security test, and some people just want to check a box, and say yes I’m good and move on to the next thing, and other people actually want to use this as a tool to get better. And we’re trying to get people from the first column over to the second column. ‘Cause they don’t always start in the second column.

BRIAN: But I, I will say just to add on to that that it’s been surprising, we have had some almost openly hostile relationships and, pretty consistently once the report is in, once we’ve talked them through what we did, why we did it, and what it means, you know, especially with the attack simulations which are goal-oriented, we’re, it’s very easy to couch findings in terms of business impact, and we get really positive responses from the top tier, the C level people, who see, like, oh my goodness, this would have put me on the news. It didn’t. This is great. And, you know, and a lot of times the IC teams, you know we work closely with them, so things can get remediated quickly and cleanly and that helps a lot too.

GARY: And what we also do from time to time is we talked about sort of, this truism that’s out there now, being a learning organization, it’s said in all these contexts but it’s so true in cyber and that’s, that’s the power of these exercises, right? Is it promotes and forces this continual learning. 

And we try to encourage  our clients, our customers, our partners, to embrace this idea of this, this is about learning, this is about getting better. As Eric said, obviously it’s easier for some, it’s harder for others. But sometimes we’ll say ah there’s a real resistance to this kind of learning in an organization, and actually is, that’s red flags for us, we say okay, these kind of clients that we really want to invest and commit to, over the long run, or is there a brittleness there, a reluctance to really grow and be introspective and learn and improve, that might make these folks sort of less natural, more interesting partners for us.

LANDON: Have you guys seen any trend, where when a company is in growth mode, and really with enterprise that means a lot of M and A, that they tend to spend on security more robustly, whereas if somebody’s in profitability mode, a lot of times they’re more on cost cutting and trying to do things cheaper and you don’t see the spend on security, have you seen any trend from that regard?

BRIAN: Well it seems like the opposite, right? Like, I think that when people are growing they’re focusing on growing. But the companies that have good security are the ones that are like stable and trying to keep things safe, it seems like to me.

ERIC: I agree with that. You know, the companies that are in growth mode, they just don’t have as big of a security budget as a large company does, large established company. So their ability to, you know, they’re more cost conscious, I find ’cause they’re just a smaller company, have less money to spend.

GARY: At the risk of beating the horse dead here, you know CrossCountry Consulting works in the private equity space to a fairly significant degree and when we talk to PE firms, and our portfolio companies about cyber, we do find that it’s often an afterthought. I think that’s perfectly natural, right? To the point that Eric and Brian made, these are companies that have received funding, they’re in hyper-growth mode, they need to be accountable for every dollar, and they’re not in a position where they’re looking at cyber as sort of a core competency or a core requirement in business. 

I think what is encouraging is that private equity space is starting to say ah, we need to make sure we have some basic level cyber in place across our portfolio companies, right? To ask some really good questions. But that doesn’t mean, suddenly, that the budget floodgates are going to open and cyber’s going to be a top 3, top 5, or necessarily even a top 10 issue for these companies.

ERIC: You know, you actually just reminded me of something you know, I was hearing about this private equity company that owned 50 portfolio companies and of those 50, I think 11 had ransomware incidents in the past year, so it was a pervasive problem, these tiny companies getting hit with ransomware over and over and over again, but it’s just a matter of them figuring out, how are they going to spend this security, how are they going to find the budget to hire an adversary simulation team, it’s a challenge for them.

LANDON: That’s fascinating, we could always have another podcast, you know on this very topic, just because that’s what your experiences have been and then, they’re completely accurate I’ve even seen on the flip, where, take a $4 billion company gets in growth mode, they start acquiring, they realize IT can’t integrate at the speed that security can protect, so they’ve just spend massively on security, right, at all costs, then of course, then they get into, okay they’ve made all the acquisitions, now they need to cut costs, and then they start, you know, reducing security, so I, I’d be fascinated to trade was stories to see what, collect more data from that perspective. 

Remember doing some adversary simulations, and the directive was, we don’t care about APTs, we only care about fraud actors. Emulate, or simulate, fraud actors. Have you come across a similar, some similar types of discussions, what do you kind of, is that rational, is that a rational thought from that perspective? How do you go about that?

ERIC: Yeah, there have been a couple, I can think of a couple of cases where the focus was fraud, but they’re coming to us, and we have cybersecurity experience, so we’re trying to kind of blend those two things together. 

One example was a job I did a few years ago for a large mobile phone provider, and one of, they encountered a ton of fraud, and they lose a lot of money to it every year, and one of the big problems is SIM swap attacks. And so the garden variety version is you go into a cell phone store and you say hey, I lost my phone, can you port my number to this new phone that I bought, and you go in with a fake ID and they, basically you steal somebody else’s phone number. 

And then you can use their phone, you know, as a second factor of authentication to get into various applications. But what I was thinking was, can we do this at scale. So, combining these two realms together could we get into the back end database of the mobile phone provider, and commit SIM swapping at the scale of millions, or tens, or hundreds of millions of devices at once. 

And so, we just did an adversary simulation and spearphishing, and moving through the network and got access to the customer service rep, you know the person that answers the phone and actually processes this transaction, we got access to their account, to log into the backend application, and basically got enough information where we bought one phone with one number, bought another phone with another number, with a different person, and basically we were able to prove that we could do this on a couple of devices we did. 

But there’s nothing special about our devices, you could do this you know, 100 million times. And SIM swap everything. So that, that’s kind of an example of taking a fraud concept and sort of applying an adversary simulation mindset to it, to increase the impact.

LANDON: What were the controls put in place after that?

ERIC: I don’t know. That was one, we don’t always get to hear you know, what-

LANDON: Yeah.

ERIC: What, you know, from the red team sometimes you get to hear the next time, and sometimes you don’t. And in that case we didn’t get to hear back.

BRIAN: He kept access and we use those now as burner phones.

ERIC: Right.

LANDON: Isn’t that sometimes frustrating when you don’t understand, when you don’t see what happened on the back end?

ERIC: Yeah, sometimes, you know, with being an adversary simulation team, they kind of view you as, okay, you, we don’t want to give these guys too many hints.

LANDON: Sure.

ERIC: And so if we tell them too much they’ll be cheating for next year.

LANDON: Right.

ERIC: And so sometimes they like to keep things close to the vest.

LANDON: You said something that queued a question, and that’s really around what a win looks like when you’re doing these types, and I think we can all agree that, you know being more secure is of course what the win is. Frequently, you talked about though, you mentioned a key point of like how you do this at scale. 

Question then is, how do you win, but also do it at scale, where it’s not just a once a year type thing, I mean you know this, sometimes enterprises are remediating for the next six months, right? I mean from a test, you know, if you’re doing a expansive test, couple hundred thousand bucks, you know, over a month to two months, I mean that is tremendous lag time to remediate, when you have to get in, bring in IT, and bring in the developers, and bring in all these different business units, or stakeholders if you will, to remediate, how do you do this at scale?

ERIC: You can only do adversary simulation so often. If you do it every month, it becomes overwhelming. I think it does take time for them to recover, and come up with a game plan. And not just whack-a-mole responses, let’s not just change this password, default password over here, and lock down permissions in this server over here, but thinking about what are the problems, the underlying problems that led to this in the first place, they have some sort of device hardening procedure that locks down a device before you enter the network every single time. 

So those kind of like longer-term solutions kind of take time to implement. You know, we typically do this, probably at the most every 6 months. Something this deep. If it’s more on the pen testing side, maybe every three months, but, it does take time for them to, to respond in a thoughtful way. And I don’t know how much that could be accelerated, you know, transforming a whole security program does take time.

BRIAN: And I will say too, like this type of testing is different because it avoids the kind of whack-a-mole, with pen testing, and pen testing is hugely valuable, but it is, we’ve found these technical things, we fixed these technical things. 

And a lot of what we’re doing is looking for process improvements, things that improve your entire company at scale and they’re a little harder to find, they’re a little harder to remediate, but when they are remediated they’re long lasting and impactful. And, and the same thing with instant response type stuff, right? If you have good detection and instant response procedures, all the whack-a-mole stuff, while still important, becomes less impactful when abused. 

Doing this type of testing at scale is not necessary, because you’re looking at instead a few impactful, significant changes that last, you know a long time, forever.

LANDON: I don’t know how you do anything at scale, I don’t know about your experience, y’alls experience, when after you do and you’re debriefing, sometimes there are literally 30 to 40 people that come to these calls to listen, ’cause like that’s how many stakeholders it needs to get things done, I don’t know how you do anything at scale, to be honest.

It’s certainly a dance that is challenging, and certainly takes, you know, great professionals like yourselves to not only be the technical nuance to that but also like the business, you know the business almost, you know intermediary if you will.

ERIC: I think it’s successful, step one is like when we can tell an impactful story that changes how they think about security. Sometimes organizations have more, like the old castle wall mentality of like, well no one could ever get past our outer perimeter. So that’s what we’re going to focus on is strengthening the outer perimeter, but not really a defense in depth approach. 

So when we, when we tell a story that says hey, eventually with enough effort, and enough time, someone can get past that, and then what? That story kind of resonates with people ’cause we say okay we started with nothing, and we walked away with the crown jewels. 

And I think the other thing is sort of building that relationship and creating, helping create that culture at the organization that it’s not finger pointing and blame, and whose fault is this? But, it’s more of this collaborative, let’s all work together to get better you know, forget the blame, forget, you know, forget all that, let’s make it better.

GARY: So much of cyber is about technology and metrics, today, and what we’re really trying to do with these high end adversary simulations is make it about people and stories. You need the tech and the metrics but you also need the people and the stories. 

All right, this is a competition with an enemy, that enemy is somewhere down the road, a real human being, with emotional intelligence, and creativity, and a mental model, and a thought pattern, going against you, that’s what our team tries to bring. And in some ways I think of it, now listen, this service, adversary simulation, is sort of a luxury good. 

It’s not gonna be for everybody. It costs a little bit but you get really good, hands on, white glove, top notch engagement and service, and interaction, that’s so different than okay I’m gonna buy a noisy pen test, or I’m gonna go buy another couple of shiny tools and technologies that I saw on a billboard in a metro station or at the airport or at the latest vendor conference that I went to. I mean there’s space for all of that in the community, right, we all appreciate that, but we’re trying to provide that, we gotta have the people and the stories, and that really high end, high touch, let’s strategically move the needle on your program, and really understand in a visceral way what some of these bad guys are gonna do.

LANDON: I completely agree with everything you just said Gary. Full stop. You’re speaking truth to power. But now let’s say what you just said there, let’s take that context in a boardroom. 

I am a board of director, I am, you know, CEO of another company, and I put back to you in this question, what are the metrics that you, that I, that, what you just described, in this simulation that you just did, how do I put the metrics or quantify that against my peers, or what else is out there in the industry. ‘Cause I, I literally get that, I used to get that question every single boardroom discussion that I ever presented in, and it was, I’m not gonna lie it was not an easy answer.

GARY: Yeah well, look the first thing is you gotta be careful with the benchmarking, and the comparisons, right? Every organization is different, everyone’s got a unique attack surface, a unique risk profile, a unique, distinctive way that threats are gonna go, so we treat this all the time in consulting, give me the benchmarks, give me the scores against my peers, tell me how I stack up against that other big bank these are natural questions and we’re happy to address them, but it’s really about being a little bit introspective, and saying what really matters to your business and what are real bad guys going to do against your business, so it’s not what is this about comparing you to you, over time, as you go through these exercises, are you seeing the improvements coming up, year over year, from the work that Eric and Brian and the team are doing? 

And to a degree, who cares how you stack up against your peers, because again it’s really about are you protecting what matters to your distinctive business, rather than chasing some vision of what an industry benchmark or best practice is. Now are you gonna say that to a director in a board meeting? Dispute the premise of their question and say ah, we’re not gonna give you benchmarks? I don’t know, but I think that’s kind of the honest answer that I like to give to my clients. I don’t know, Eric, Brian, you probably have some thoughts on more concrete metrics that could potentially help.

ERIC: Yeah, I mean it’s hard, you know, we talk about threat emulation, or, adversary emulation, and that can give you a score, but the score is flawed in some ways, because it’s measuring yesterday’s attacks. So I think we have enough experience to say, okay, you guys are about average in line with your peers, you guys are a little above, you guys are way above or way behind, I think we can give that level of feedback, but trying to score it more precisely than that is-

LANDON: Completely flawed.

ERIC: Yeah, it’d be too subjective. We can give you an idea based on our experience that we’ve done, tests at these peers, and we can say you’re a little better than most, or a little worse than most, but giving you a number score would be a little disingenuous, I think. And anyone who’s willing to do it is probably not being 100% sincere.

BRIAN: But also, there is a kind of alternate view of this, and that is, it doesn’t matter, because, cyber security, compared to many other business functions, is unique because it is actually, purely, it’s purely adversarial.

Not in like a, are we making more money way, but like, they’re gonna hurt us if we’re wrong way. And so, in a lot of ways, you’re not comparing to your peers you’re comparing to the adversary. And that does help, especially with the number of high profile attacks over the last couple of years, it helps a lot to be able to frame our conversations in terms of that, right? 

You are probably going to be ransomwared by anybody that decides to hit you, is a relevant metric. You know, like, and so those kinds of things I think do help a lot when you look at it not in terms of competitions but, with, with other businesses but in terms of competition with bad guys, then it becomes a little more relevant, still hard to quantify, certainly. 

And metrics across cyber security are hard, right? Like I, feet on the blue team, how do you rate, how good your blue team is? The number of incidents they closed? Like? The whole thing is just hard.

LANDON: The mean time to response, or mean time to alert, I mean-

BRIAN: I’m a big fan of mean time to detection versus mean time to response, but you have to be mature enough to be measuring that in the first place, which many organizations are not.

LANDON: Gary, Brian, Eric you guys are super squared away, love what you guys are doing at CrossCountry Consulting, thank you for joining this conversation live.