- What does it mean to be a sophisticated nation-state hacking unit, and imagine if Russia had the same capabilities as Western democracies. What is the difference between Russian nation-state hacking groups, particularly the SVR and the GRU? We'll revisit the 2015 and 2016 attacks that cut power from Ukraine and look at some of the tactics showing increasing Russian sophistication, mainly that we were seeing different teams of hackers conducting different parts of the computer network exploitation kill chain. So where does it all end, and how do we properly defend ourselves? That's next on the fourth episode of "Know Your Adversary™." - I've been doing cybery things for a bit over 10 years at this point. At the time of the 2016 Ukraine attacks, I was actually in the US Department of Energy still. - This is Joe Slowik who is currently a senior manager at the security company Gigamon. He was on the front lines assisting the research in the 2015 and 2016 Ukrainian energy sector attacks. - And it wasn't until I had moved over out of an incident responder role within USDOE and gotten over to the private sector at a company called Dragos which focuses om industrial control systems that I really started to dig into the 2016 Ukraine power event. So to provide background on that, this was actually the second electric sector attack that took place in Ukraine in the span of two years, the first coming in December, 2015, and then almost exactly a year later, another event in 2016, and whereas the 2015 incident was fairly manual in how it was executed with lots of malicious operator interaction with control systems to induce a outage at the electric distribution level, 2016 was a little bit more interesting in that there was a specially crafted piece of malware referred to as Indestroyer by ESET and called Crashoverride by Dragos that automated a lot of the interaction with the control systems in the victim environment that allowed it to potentially scale a lot wider than what we actually saw in the incident. - So let's paint some context before we get into the weeds 'cause this gets pretty technical and complicated. On December 23rd, 2015, hackers compromised information systems of three energy distribution centers in Ukraine and temporarily disrupted electricity to Ukraine. This was the first known cyber attack on a power grid. For simplicity's sake, in energy, manufacturing or utility companies, there's an IT environment, which is mostly a Windows network, residing on a corporate function. Usually think marketing and sales and finance. Then there's the OT or operational technology environment that has names like Industrial Control Systems or SCADA. The OT environment is where manufacturing is conducted, so oil and gas, nuclear power, plastic containers, you name it. These are often separate units within the organizations that use different technologies, speak different technical languages, and frankly have very different cultures. Big technology like Honeywell, Rockwell, Siemens, these are all major operational technology players. How these attacks generally occur is a pretty simple strategy to gain corporate IT access and navigate to the OT environment. The hacker steals credentials, deceives you through phishing, or exploits you in a vulnerable system or some combination of all three. In the end, he's landing on the corporate IT environment, and they will have a foothold on a workstation. So in physical security parlance, they're in the front door of your house. To get the keys to the safe, now the hacker needs to get off the workstation and become a more elevated, privileged administrator in the IT environment and move laterally to the ICS environment and essentially act as a ICS administrator to remotely switch off the substations. This is not a trivial thing to do, but that's exactly what these hackers did to shut down the power in 2015, and then again in 2016, the same group of actors did the same thing, but with more sophistication. - What's interesting about the 2016 event is that a lot of this information wasn't immediately available in December, 2016, January, 2017 although Ukrainian authorities were pretty quick to point the finger at Russia as being responsible, and then it wasn't until we got into late spring and then into early summer of 2017 that ESET, and then Dragos, published public reporting identifying the malware in question, and then some initial indicators that the event had ties to an actor referred to in commercial cyber threat intelligence circles as Sandworm. Sandworm is an entity that has quite the history to it. There's even a book about them written by Andy Greenberg who's a reporter over at "Wired," and what's interesting about Sandworm, in addition to their longevity, two things. One, they are associated with a number of disruptive or even outright destructive cyber events such as the TV5 Le Monde incident in France in 2014 I believe. It might've been early 2015. They were closely linked with the 2015 Ukraine event, and then it seemed, at least at first without having too much evidence, that they were likely linked to 2016's Ukraine power event, and then finally getting into the 2017 Petya incident. The other element about Sandworm that's interesting is that the US, UK, and other government entities have linked that cryptonym Sandworm quite explicitly with Russian military intelligence, specifically the Center for Special Technologies, Unit 74455, and that's a level of attribution that we just don't typically see that often within the cyber landscape. - As Joe said, the Sandworm team is known as Unit 74455 within the Russian military unit GRU. For context to listeners who don't breathe cyber security every day, the 2015 attack on Le Monde came close to destroying a French TV network, and the 2017 NotPetya incident caused outages across many countries and businesses including France, Germany, Italy, Poland, the United States, the UK, and Russia. So bottom line is you can start to see the GRU's often buy numerous cyber attacks that have a destructive angle to them which is completely on purpose. After all, it is military unit. - So with all that background set up, further evidence emerged as we started to get into late 2017, early 2018 from ESET that started to tie, at a technical level, tools that were used as part of the 2016 Ukraine incident with subsequent activity that was linked directly to Sandworm in the form of some continuity in malware, a construction between families known as black energy, gray energy, the backdoor or remote access tool that was used as part of the 2016 Ukraine event, and then some follow-on tooling that ESET discovered that they named XRML. Not 100% sure if that's the correct way of pronouncing that word as it's not a real word, but anyway. And that really started to tie multiple threads together of identifying the 2016 Ukraine events at least in terms of their initial access and intrusion operations directly to Sandworm as an organization and as a collection of technical capabilities. - So to make this easy to understand, what Joe is saying is that private and public sector technical research links malware code to the same set of actors in the GRU. There's a lot of naming nomenclature that probably doesn't make a lot of sense to people. Black energy, gray energy are simply private sector naming conventions for malware research, malware that is designed to do certain functions like calls of denial of service to electric companies. What happens behind the scenes in government buildings and even outsourced to contractors is that developers spend hundreds of thousands, if not millions of dollars, of research and development for malware and exploits that take advantage of vulnerabilities in operating systems software and even hardware. Those malware and exploit tools that have to be delivered to a target often send via spear phishing email to a user which then gets executed and weaponized. Depending on how good the hacking unit is that these tools get flagged by antivirus or security tools, they frankly need to start over. So good hackers have to really cover their tracks well, or like the GRU, they don't care about being caught and just update the code base into different variants that do different actions upon different commands. And that's what happened between the 2015 power outage and the 2016 power outage in Ukraine. - If you start looking at the actual intrusion data and talking with some of the individuals in Ukraine that responded to the event that there is an interesting inflection point as part of the 2016 attack scenario where it looks like you had a set of behaviors getting up to and including the installation of that custom backdoor, the Indestroyer backdoor, which has the technical overlaps with XRML and Black Energy 3 and some other tools that are strongly associated with Sandworm operations. Well, and in some cases, exclusively associated with Sandworm operations, but then, it seemed like things changed almost as if that access was then handed over to a second team that operated in a distinct fashion that, instead of using the same intrusion methodologies, the same lateral movement methodologies that the Sandworm-linked activity has associated with started to do things slightly differently within the control system environment in the Ukrainian substation that was impacted. - Let's talk about what's going on here as this is important. In a good computer network exploitation unit, you have numerous levels of operation that should not be traced back to one another. So you have different networks, different people, sometimes different code bases, all different and segmented. Let's talk about these divisions of labor. You have the exploit and malware developers, the content creators who deliver the spear phish emails, the operators who deploy and administer the exploits, operators that move around the network and steal the data, even operators that administer the misattributable infrastructure and call back command and control servers. While few computer network exploitation units are good enough to keep all these functions completely separate, that's what we start to see in the 2016 Ukraine attack. The operators that gained access to the IT network were different than the operators who took over the industrial control systems network that controlled the substations. - And so that leads us to a very interesting observation that certainly based on several government attribution statements, Sandworm as an entity is linked to Unit 74455, but then there's an open question. Is there just one team, or just one entity operating out of this organization in Russian military intelligence because it seemed at least that in the 2016 power event and possibly even in the 2015 power event that there might've been a separate industrial control systems-specific team also involved. Maybe these are the tells of a specific operator that just happens to have some control system expertise, or maybe there is an entirely separate organization which at Dragos was tracked under the name Electrum that is responsible for carrying the baton forward from Sandworm in order to execute the ICS-specific phases of the given intrusion scenario. And that's where things get really interesting because now we can start seeing greater degrees of complexity that, instead of just having this one organization, this monolithic entity Sandworm underneath overall GRU auspices, is the possibility emerges of having distinct teams that are operating as part of Russian-linked cyber disruption operations that have different specialties, and while that's kind of academically interesting to a certain extent, it's also practically worrying as well because it seems to indicate a level of maturity and standardization in operations where we have an entity that's invested in having this sort of bureaucracy around cyber disruptive operations. - This division of labor for computer network exploitation units is nothing new for Western-allied powers. However, now our adversaries like the GRU are catching up. - For example, as we start expanding out from just the 2016 Ukraine event to the 2015 Ukraine event, we see the development of very specific tools such as a malicious firmware update that results in the bricking of devices within the control center environment, specifically serial-to-Internet converters that would inhibit, well, by bricking them, inhibits the ability to control the operating environment for the distribution substations. That's a fairly specialized capability, and are we really expecting the same group of four to six or maybe 10 individuals that all of this expertise resides in just this one shop, or can we see exploitation shop, a development shop, a follow-on execution and deployment shop, and then possibly even specialized teams under there? I think we really need to start looking at these organizations as being more than just a single team of individuals and rather as specialized interlocking teams that have different specializations that they bring to the table and working together in order to execute what are increasingly complex and non-trivial sorts of intrusion and attack scenarios. - Curious how security professionals make these technical linkages between different roles and responsibilities of nation-state hacking teams? - So, first team to second team linkage is fairly easy especially once we're able to get some greater access to forensic artifacts, log data, and some conversations with the responders in Ukraine that had done the incident response and the initial forensic analysis, and you can see, again, sort of an inflection point between these operations that appear very well-linked to Sandworm behaviors and TTPs, tactics, techniques, procedures up to the point of the deployment of the remote access tool, and then, there's a completely separate set of behaviors which you could make the argument that different environment because we start talking about the migration from the enterprise or office IT environment into the industrial environment at this point, but even things just in terms of the way specific commands were being entered, different sets of credentials that were being leveraged for lateral movement. It just strongly resembled or indicated that there was a, at minimum, a different individual if not a completely separate team with a different method of operating that had taken over that point in the intrusion, but the fact that you see the lead-up and that likely handover indicates a link between the two. And then, it became a moot point when UK's National Cybersecurity Center as well as the US government in subsequent sanctions and other statements just publicly named GRU did this in 2019, 2020 which sort of confirmed links that were previously made purely based on technical evidence. - As always, there's operational mistakes made with hackers that are either junior grade or don't care about getting caught which ultimately helped with attribution to help determine this sophistication. - So whereas group one, which we linked to Sandworm, used a pattern of life that's very similar to what we've seen with Sandworm operations in that same time period, leveraging IP-only communications so no domain names and utilizing servers that were also being leveraged for services like DHT nodes, Torah services, and whatnot. So really strongly indicating compromised legitimate, or at least not malicious infrastructure being used for command and control including for the remote access tool that was linked to other Sandworm-associated malware, but then as we start diving into the control system environment intrusion, there were instead of seeing compiled binaries and whatnot at least until the deployment of Indestroyer, we instead observed things like scripting objects, batch scripts, PowerShell scripts, and completely different callback infrastructure that is strongly associated with adversary-controlled, adversary-operated infrastructure as opposed to compromised third-party infrastructure, so completely separate ways of behaving. - Let's paint some context to be clear. The first-group GRU attackers that were charged with gaining the initial access to the Ukrainian Windows corporate domain in 2016 were reusing malware from infrastructure from previous campaigns and even using compromised third-party infrastructure. So imagine stealing credentials from a legitimate third-party private server owned by somebody else. This can be challenging to get third parties to turn over logs unless you have a subpoena. That's why they use this kind of stolen infrastructure, and as we've talked about in previous episodes, it's not challenging to call these companies and ask them politely for an image of a server, but this takes time, and if the actor doesn't care to hide their tracks, they probably can get done what they need to get done quick enough where rolling back their attribution won't make a difference, but the bottom line is it was clear the actors had experience during these initial access operations. However, the command and control infrastructure for the industrial control systems portion of the attack was a bit more automated and uncharacteristically sloppy. - It is interesting though that you bring up do we see any operational mistakes by these entities? So with the first group associated with Sandworm, not so much. Everything looks pretty dang good in terms of getting initial access, shoring things up, remaining relatively quiet, but then as we start getting into the control system space and phase of operations, things start getting really noisy. We start seeing things like really widespread credential guessing and enumeration, fairly noisy scanning activity within the environment. You could make the argument that control system environment visibility is fairly poor, and so these poor OPSEC sorts of behaviors aren't as big of a deal at this point, but it still represents a certain degree of brazenness or sloppiness on the part of the adversary and stands apart from what seems to be a fairly stealthy operation at the IT or enterprise environment level, but then, not only do we see those sorts of maybe mistakes or just not caring, but as we got into the actual deployment of the control system-specific payloads, start seeing some errors manifest themselves and some pretty functionally significant ones. For example, one of the payloads that was designed to manipulate breakers in the substation transmission yard, the way in which it was developed was not appropriate or not correct with the actual implementation of the communication standard that is used to control those devices. So based upon some analysis and worked with a couple of industrial equipment vendors and some other academics on this to test this out, but it doesn't appear that the tool would have worked as intended, that they would have sent commands to try to open breakers and force them open for a long period of time, but because there were errors in the precise sequencing of how those commands are being sent, it doesn't appear that that would have worked. Then in another case, and this gets into the protection-focused or the potentially destructive portion of the scenario, there was a denial of service payload targeting something called a protective relay in the transmission environment that protects against certain dangerous or damaging conditions in the day-to-day operation of the utility environment. The denial of service payload seeks to basically remove all that protection logic which could put the transmission substation at risk. Well, the problem is is that, while the attackers seem to code the denial of service condition incorrectly, they hard-coded the IP addresses of the four protective relays that they were aiming for within the environment, and the way in which they implemented the socket creation to deliver the payload, they reversed the endiness of the IP addresses so that they were all read in backwards when trying to build the actual UDP socket in order to connect to the relay, so the communication never reached them, and that's a pretty boneheaded error, or at least it seems so at first glance, and leads to a couple of questions. One, do they ever test this out in the first place? Two, was this operation potentially rushed at the control-specific stage? - Well, this is deeply technical and amazing. The bottom line is the tool development being used by the GRU Team Two against the ICS environment was sloppy. When efficient computer network exploitation units are working, there's a strenuous testing process of tools to ensure they work, and they avoid detection by antivirus or security tools. Clearly, that was not done here, and it looks like they didn't even care. - For example, one theory is that the entity in question, Team Two in this case, was given a mandate to execute this effect to roughly match the timeframe of the 2015 attack sorta to thumb noses at the Ukrainian officials that, yeah, we can do this, and you can even predict or see us coming, and thus, they had an artificial timeline that they were working with, and as a result, needed to deliver things in a hurry. Unfortunately, we don't really know these sorts of things, but it really does tie back into the sort of institutional division of labor and the possibility of having artificial, bureaucratically-enforced deadlines that then force the hands of a development team to rush a capability out before it was thoroughly tested and proven to work within even a test bed, let alone in the victim environment. - As a quick aside, this is rather comical. For anyone who's worked in government before, they've been through the mistakes made by the Russian hackers, mistakes implemented by bureaucracy from people who had no idea what they were talking about, particularly technical requirements needed to perform a denial of service against an electrical grid. As we've talked about before in previous episodes, we look at the callback infrastructure in the tool kits for designs. - So for attribution purposes, it does become clear looking at the structure of the binaries themselves and getting down into apparent coding and development styles or patterns that there is an apparent distinction between the developer behind the Team Two tool set, including the remote access tool, and the Team Two industrial tooling, the Crashoverride Indestroyer malware, and while we can't get enough fidelity to tie that back to a specific developer, we can at least leverage those distinctions to further solidify the separation between a Team One initial intrusion IT access team and a Team Two industrial effects disruption team. - So what's the point here, and how about attacks against the US energy sector? Any affiliations with GRU or SVR from that front? - This is where, to the extent where we can try to attribute these sorts of activities to specific entities and patterns of life become significant because there is a extensive history of Russian-linked threat actors operating in industrial and critical infrastructure environments. So for example, in addition to the 2015 and 2016 Ukraine events which then linked to Sandworm and GRU to varying degrees and potentially multiple teams within GRU operations, there also been other intrusions like the Dragonfly campaign which is not firmly attributed to any specific entity other than really being able to say quite clearly that it's not GRU, it's probably not SVR, likely is linked to the FSB, but there's no really firm evidence publicly available on that note except for some government leaks and just technical evidence based on infrastructure and coding capabilities that seem to align it more firmly with that camp. Now, why that's important though is this group has successfully intruded into pretty sensitive environments. The US power sector, Western European manufacturing sectors, oil and gas organizations, yet there is no evidence or indication that this group has ever sought to or move along just for being able to gain access to these environments to launching some sort of disruptive attack whereas with Sandworm, we have not only the proof of being able to access environments, but also the willingness to execute disruptive or even destructive operations from 2014 through at least 2017. - Russia has always used Ukraine as the geopolitical playground to test out military capabilities ranging from real weapons like missiles and tanks to covert action like the little green men in Crimea to software weapons such as cyber attacks and disinformation operations. If we are able to detect the GRU conducting operations in the US, does that mean that they are conducting staging operations for something later to come, or is it SVR operations where they're worried about low and slow intelligence collection to steal data so they can attribute US citizens, detect spies, and frankly spy on their own people in the US? - That seems to be fair. There's a little bit of oversimplification there, but generally speaking, that seems to be a fair way of dividing things out, and yeah, I would caution as well that we can say collection, but collection towards what purpose, and we can start getting into discussions on things like operational preparation of the environment which is always a thorny term When you start talking about cyber intrusions and potential cyber attack scenarios, but generally speaking, whereas GRU-linked activity especially to 74455 or Sandworm is closely associated with disruptive operations. Other activity does not possess the same link or the same history of being associated with, like you said, things that go boom or could go boom. - Why do power companies' security professionals care between distinctions between the GRU and SVR, for example? - Two preliminary comments on that first. This degree of attribution is typically hard, but it's not impossible, and where it is possible, I think it is worthwhile to pursue that because based on the conversation we just had, the difference in intentions or likely intentions that we've observed between groups when getting into, not just it's Russia, but rather it is GRU-linked, SVR-linked, FSB-linked, or something along those lines, we start seeing meaningful distinctions there in attack possibilities and attack goals, and based on that, the first thing that I would advise CSOs and others, especially operating within critical infrastructure verticals, is visibility is key because being able to differentiate between these actors is meaningless if you can't even spot them in your own network. So obviously, that's the first critical step. - This gets down to cybersecurity basics regardless if you run information security for a power company or a retailer. You can't detect what you can't see, and to see a nation-state hacking team in your logs requires a level of logging at the enterprise level and the industrial control systems level. But further, when you see all this available telemetry to build controls, it's about understanding what to do next and how to action the alerts within the organization to defeat the adversary. If you see the GRU who you know is tied to disruptive operations, you need to be able to react faster than someone like the SVR who's just collecting intelligence. - Then it becomes a question of understanding, at least on a behavior-specific level, what sort of activities, what sort of actions these entities are typically associated with so that entities- I mean, certainly any intrusion into a critical infrastructure environment is serious, but some can be more serious than others, and I think we saw this recently with some reporting from Dragos where they identified an activity group that they refer to as Kamasite which is linked to Sandworm operations, but appears to serve as a initial access team that, if you can identify that set of behaviors or activity related to this access team that may be linked to Sandworm operations, then that's something that becomes a priority given the possibility that follow-on actions can pass on to, or be used for, a team that is associated with critical infrastructure disruption operations. Whereas while a intrusion from a SVR is still serious, at the same time, it reflects less immediacy in terms of remediation and kicking this adversary out because there's not a history of movement from intrusion to disruption. It may be the ultimate plan of such a group to develop such access that, in event of war or some other extreme scenario, this access can then be weaponized and used to execute some sort of disruptive attack, but there's no evidence just yet that links such intrusions to direct disruption operations, and so the timeliness of a response becomes a bit different. - Did GRU and SVR collaborate together? - We have not seen that. That's not to say that's not a possibility although given the sometimes touchy relationships with entities or whatever, there's no publicly available or widely-known instances of those distinct teams potentially coordinating or cooperating in operations, and we could look at something like the 2016 operations around the US presidential election where you had separate entities, one linked to SVR, one linked to GRU, in this case Field Post 26165, Fancy Bear APT28, were both in the Democratic National Committee network simultaneously independently and not coordinating with one another at all. Obviously, that was five years ago now, so maybe things have changed, but at least the evidence that is available strongly suggests that these groups do not coordinate operations with one another as opposed to these organizations potentially leveraging each other's expertise and access in order to further their own missions. - Did GRU and SVR have different political targets? - So that's an interesting question. So for the most part, we can make a reasonable claim that GRU entities, especially the disruptive-minded entities like 74455 seem to be more active in identified conflict zones like Ukraine whereas SVR seems to be more engaged in non-conflict area operations like targeting Western Europe and North America. Having said that though and getting into the saying I love to use that all generalizations are stupid, there are notable exceptions to both of those, especially on the GRU side where we have observed and there was reporting from the US government subsequently confirmed through third-party technical analysis of GRU entity-linked intrusion attempts into the Western European and North American electric sector in late 2019 through at least the mid-2020. So again, general lines of responsibility seem to break out into general themes, but there are obvious exceptions to the point where it looks like these organizations operate independently of one another and at the same time are potentially operating in the same areas quite frequently. - And what's the feature or utility of attributing these types of attacks to an individual and holding them accountable? - I think that's the best case scenario for the private security industry that as we start getting into things like identifying how a command and control infrastructure is managed and maintained and even getting into second-order effects like identifying not just what the C2 infrastructure is, but where that C2 infrastructure is administered from, and subsequent communication flows really allows for powerful statements that can link to ultimate sources of activity. That's not to say that that's an easy thing to do and requires some significant collection capabilities, but where possible definitely has a lot of value in being able to really get into the who's responsible for these sorts of activities. - Why is so many levels of attribution so important in this new age of geopolitical conflict where cyber attacks are used more frequently as a means of attack than heavier weapons like tanks and missiles? - Yeah, you could make the argument that, well, something bad happened. I just wanna get up and running as quickly as possible because I'm losing money by the minute or by the second, so let's just focus on getting things back up and going. Well, I think if we look at the 2016 event as an example, that could be a potentially dangerous situation because based on a thorough analysis of how the attack was staged and what some of the possibilities could have been even though the adversary kind of messed things up, that attack was designed to remove protection prior to the restoration of operations that could have led to a dangerous impact. So being able to perform the appropriate level of root cause analysis as well as potentially identifying how certain actions could link to a known adversary could be the difference between restoring operations at a known, good, known safe state, or restoring operations that leads into a potentially destructive or dangerous scenario. So having that information can be quite key. Now, certainly getting that sort of information to that level of fidelity in near-real time is not easy, but by being able to link certain behaviors and certain past observations on adversary intentions means that defenders and decision-makers at least become armed with understanding that these sorts of scenarios can exist and need to be taken into account when defending networks and potentially restoring networks after some sort of incident, and we ignore them, not just to our peril, but potentially with the possibility of enabling even worse scenarios than just having a IT-related outage for a period of time than actually allowing for potentially damaging scenarios to manifest. - So to recap, the primary takeaways for attribution against this level of nation-state are clear. Intelligence about adversaries need to be leveraged very pointedly with enterprise to have a clear understanding of how it's going to affect a business. That's always the most important, but it's also critical to understand the geopolitical circumstances and underpinnings why nation-states, or even cyber crime groups as we're seeing in the Colonial Pipeline ransomware events, are prowling around industrial control system environments. It's either to monetize the event, create a disruptive event for geopolitical gain, or gain general intelligence collection. It's important to understand the context, tactics, techniques, and procedures behind these motivations to appropriately react and defend networks against this type of complexity. Many thanks to Joe Slowik for joining this episode of "Know Your Adversary™." Thank you for listening to "Know Your Adversary™." Every other week, we will bring you a new cyber crime attribution investigation that is representative of the work of Nisos operators past, present, and future. If you have any good stories to pitch, please reach out as no two investigations are the same and simultaneously fascinating how digital clues come together to bring context to crimes that victimize enterprise. For more information, please visit www.nisos.com. Thank you for listening.