Jump to content

Solar Winds Data Breach


rmgill

Recommended Posts

So on the 13th there was an announcement that Solar Winds's code itself had been compromised. This is posted in the Military Current Events because this is FAR worse than the OPM breach of more than 4 years ago. This means that malign third parties thus have root level access of networks of thousands of US companies AND Government networks. They have had this access since around April of this year. 

This is in this CERT announcement. 
Emergency Directive 21-01
https://cyber.dhs.gov/ed/21-01/

This is VERY Bad. Solar Winds, as a software package that runs on a server one places inside one's network is able to crawl one's networking devices so as to generate topological maps of your network architecture. I've used Solar Winds derived maps for solving networking problems that arise when you have a complex network and poor historical knowledge for various segments. 

SolarWinds-Network-Topology-Mapper-Netwo

The problem is that the mapping/analytics system needs to have passwords for your environment so as to crawl it and identify what ports are connected to what device. This means that the software effectively has credentials for your network that allow a profuse level of movement through it. The malware was inserted into the Solar Winds software at their code repository level, so it was compiled and sent out as part of the normal payload of updates one could apply to one's Solar Winds install instances. This mean that it would run, as part of solar winds and which would be white listed from your usual network security scans for malicious activity across your network. Interestingly, FireEye was who caught this and their platform managed to catch the data payload. 

The product list gives one an idea of what sorts of systems that may have been compromised by this rather large breach 

From the Solar Winds Security advisory:
https://www.solarwinds.com/securityadvisory
 

 

Quote

 

Known affected products: Orion Platform versions 2019.4 HF 5, 2020.2 with no hotfix installed, or with 2020.2 HF 1, including:

Application Centric Monitor (ACM)

Database Performance Analyzer 
Integration Module* (DPAIM*)

Enterprise Operations Console (EOC)

High Availability (HA)

IP Address Manager (IPAM)

Log Analyzer (LA)

Network Automation Manager (NAM)

Network Configuration Manager (NCM)

Network Operations Manager (NOM)

User Device Tracker (UDT)

Network Performance Monitor (NPM)

NetFlow Traffic Analyzer (NTA)

Server & Application Monitor (SAM)

Server Configuration Monitor (SCM)

Storage Resource Monitor (SRM)

Virtualization Manager (VMAN)

VoIP & Network Quality Manager (VNQM)

Web Performance Monitor (WPM)

 

 

 

 

Edited by rmgill
tweaks
Link to comment
Share on other sites

  • Replies 167
  • Created
  • Last Reply

Top Posters In This Topic

https://andygreenphd.com/2020/12/15/quick-thoughts-about-the-solarwinds-breach/
Quick thoughts about the SolarWinds breach

 
Over the weekend, SolarWinds announced a significant breach.  SolarWinds is a suite of tools that allows firms to manage and monitor their systems and networks.  SolarWinds says it provides services to:
 
  • More than 425 of the US Fortune 500
  • All ten of the top ten US telecommunications companies
  • All five branches of the US Military
  • The US Pentagon, State Department, NASA, NSA, Postal Service, NOAA, Department of Justice, US Secret Service, the Federal Reserve Bank, US Department of Defense, US CDC, and the Office of the President of the United States
  • All five of the top five US accounting firms
  • Hundreds of universities and colleges worldwide

FireEye discovered evidence of the SolarWinds breach while investigating its own recent breach.  FireEye is a SolarWinds client and notified them along with law enforcement once they ascertained the nature of the breach.

In simple terms, attackers used a “supply chain” attack to place their malware within one of the support files SolarWinds pushes out to customers as part of its overall patching and updating process for their software.  Attackers were able to access SolarWinds’ code repository, add the malware to it, and just wait for the software to get pushed out to SolarWinds customers everywhere.  Even worse, because the attackers were able to add their malware directly to SolarWinds’ existing code base, the malware was digitally “signed,” which is a means of ensuring authenticity.  So, the software was implicitly trusted by all customers as they installed SolarWinds updates on their networks.

The attackers designed the malware to stay silent for two weeks post-installation.  After that quiet period, the malware would contact a command-and-control (C2) server to receive instructions.  The malware had the potential to download and install software, profile the system it was running on, reboot the host system, and disable system services.  Because attackers embedded the malware within SolarWinds itself, the malicious traffic masqueraded as legitimate activity associated with a trusted application.  In short, the malware operated in plain sight and under the cover of a trusted application.  Once the attackers were able to gain a foothold in the victim’s network, they were able to engage in traditional activities to target other systems in the victim network, exfiltrate data, and establish a persistent foothold in the network for long-term activity.

The potential scope of this breach is breathtaking, based on the partial client list above.  On Sunday evening, the United States Cybersecurity and Infrastructure Agency (CISA) issued Emergency Directive 22-01 regarding the SolarWinds breach.  That directive instructed all federal agencies under its purview to:

  • forensically image all systems running known compromised versions of SolarWinds Orion and analyze for new system or user accounts
  • analyze network traffic to look for indicators of compromise (IoC) 
  • immediately disconnect any system running known compromised versions of SolarWinds Orion, and keep them offline pending further CISA guidance

We are very early in this incident.  We know that based on the partial customer list above, the potential for the size and scope of this breach is breathtaking.  All firms running this software must assume they’ve been breached at this point and take appropriate action.  Even though the CISA directive applies only to some federal agencies, it is trustworthy guidance that all firms running known compromised versions of the SolarWinds Orion software should follow.  However, dealing with SolarWinds Orion is only one part of the problem.  Firms must also take steps to determine whether an attacker has created a persistent “backdoor” foothold in their network and then take appropriate steps to remove it as soon as possible.  

At this point, “we don’t know what we don’t know.”  But one thing is sure – there are many incident response personnel working to assess the damage to their networks and systems, and we won’t know for a while (if ever) the extent of the actual damage.

Link to comment
Share on other sites

The breach is bad enough, but the way how Solar Winds is handling it sounds even worse. From an IT guy I know:

Quote

They were not forthcoming with any kind of detail...  

The CISA also says not to trust what solarwinds called a patch that was supposed to fix the problem...

Add some insider trading while [the bad news] was ... not released to the public...

I guess the management of the company considers the company finished and tried to dump stock while they could (which might end them up in federal prison, if I understand the penalties for insider trading right). So why should any other stock holder have faith in their fate, and the management's ability to handle it?

Link to comment
Share on other sites

9 hours ago, Stuart Galbraith said:

Thanks for the explanation Ryan.

Im guessing this is going to prove all but impossible to remove? Unless you presumably do a system update to a point previous to the first known hack? Is that even possible?

No, step 1 is turn off your solar winds instance. 
Step 2 is segregate yourself from the internet. 
Step 3 is re-image/install OS and firmware for each device on a bench with new password sets and swap out each of your devices on your network. 

You can't re-password your installed base while online as a compromised device could have a sub-process that might send out the new credentials. 

Link to comment
Share on other sites

On 12/19/2020 at 4:14 AM, Ssnake said:

I guess the management of the company considers the company finished and tried to dump stock while they could (which might end them up in federal prison, if I understand the penalties for insider trading right).

How energetically the SEC goes after inside trading depends on politics as much as anything else.

Link to comment
Share on other sites

https://www.reuters.com/article/global-cyber-solarwinds/hackers-at-center-of-sprawling-spy-campaign-turned-solarwinds-dominance-against-it-idUSKBN28P2N8

Quote

Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123”

Quote

Others - including Kyle Hanslovan, the cofounder of Maryland-based cybersecurity company Huntress - noticed that, days after SolarWinds realized their software had been compromised, the malicious updates were still available for download.

 

 

Link to comment
Share on other sites

https://krebsonsecurity.com/2020/12/vmware-flaw-a-vector-in-solarwinds-breach/

Quote

 

“MFA threat modeling generally doesn’t include a complete system compromise of an OWA server,” Ars’ Dan Goodin wrote. “The level of access the hacker achieved was enough to neuter just about any defense.”

Several media outlets, including The New York Times and The Washington Post, have cited anonymous government sources saying the group behind the SolarWinds hacks was known as APT29 or “Cozy Bear,” an advanced threat group believed to be part of the Russian Federal Security Service (FSB).

 

 

https://media.defense.gov/2020/Dec/17/2002554125/-1/-1/0/AUTHENTICATION_MECHANISMS_CSA_U_OO_198854_20.PDF

VMware can't win for losing, of late. 

Quote

It is critical when running products that perform authentication that the server and all the services that depend on it are properly configured for secure operation and integration. Otherwise, SAML tokens could be forged, granting access to numerous resources. Microsoft Active Directory Federation Services (ADFS)®4is aidentity federation technology used to federate identities with Active Directory (AD)®5, Azure Active Directory (AAD)®6, and other identity providerssuch as VMware Identity Manager. By abusing the federated authentication, the actors are not exploiting a vulnerability in ADFS, AD, or AAD, but rather abusing the trust established across the integrated components. Due to the popularity of ADFS, numerous actors target ADFSas well as other identity providers trusted by ADFS (T1199),to gain access to cloud servicessuch as Microsoft Office 365.

 

Link to comment
Share on other sites

Here is one hypothesis I have found on Reddit (much condensed for clarity, plus some speculation on my part);

step 1; exploit vulnerabilities in VMware Workspace One Access and VMware Identity Manager servers, create malicious SAML tokens

step 2; use malicious SAML tokens to access privileged account info for all managed systems in SolarWinds servers

step 3; use stolen privileged account credentials for Windows Server machines to fully compromise Active Directory domains and ADFS (if present)

step 4; push malicious GPOs and/or updates from AD domain controllers/WSUS/SCCM servers to all Windows servers.

step 5; laugh maniacally while petting Persian cat.

 

Link to comment
Share on other sites

https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html

Quote

 

SolarWinds.Orion.Core.BusinessLayer.dll is a SolarWinds digitally-signed component of the Orion software framework that contains a backdoor that communicates via HTTP to third party servers. We are tracking the trojanized version of this SolarWinds Orion plug-in as SUNBURST.

After an initial dormant period of up to two weeks, it retrieves and executes commands, called “Jobs”, that include the ability to transfer files, execute files, profile the system, reboot the machine, and disable system services. The malware masquerades its network traffic as the Orion Improvement Program (OIP) protocol and stores reconnaissance results within legitimate plugin configuration files allowing it to blend in with legitimate SolarWinds activity. The backdoor uses multiple obfuscated blocklists to identify forensic and anti-virus tools running as processes, services, and drivers.

 

Quote

 

Multiple trojanzied updates were digitally signed from March - May 2020 and posted to the SolarWinds updates website, including:

  • hxxps://downloads.solarwinds[.]com/solarwinds/CatalogResources/Core/2019.4/2019.4.5220.20574/SolarWinds-Core-v2019.4.5220-Hotfix5.msp

The trojanized update file is a standard Windows Installer Patch file that includes compressed resources associated with the update, including the trojanized SolarWinds.Orion.Core.BusinessLayer.dll component. Once the update is installed, the malicious DLL will be loaded by the legitimate SolarWinds.BusinessLayerHost.exe or SolarWinds.BusinessLayerHostx64.exe (depending on system configuration). After a dormant period of up to two weeks, the malware will attempt to resolve a subdomain of avsvmcloud[.]com. The DNS response will return a CNAME record that points to a Command and Control (C2) domain. The C2 traffic to the malicious domains is designed to mimic normal SolarWinds API communications. The list of known malicious infrastructure is available on FireEye’s GitHub page.

 

 

Link to comment
Share on other sites

It will be interesting, when the AARs are compiled, to see how quickly the Tier 1 ISPs and the DNS infrastructure responded. 

Note that VMware is getting good mentions for full disclosure, ditto Microsoft, vice SolarWinds. 

Tweets like these indicate that this is an extinction-level event for SolarWinds;

morristweet.png

Link to comment
Share on other sites

2 hours ago, Ivanhoe said:

Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123”

...and which point one has to ask if the solwarwinds management wasn't entirely and criminally complicit in this. It starts with weak passwords to begin with, and remaining passive about the removal of vulnerabilities. If they didn't see the problem and didn't immediately fire whoever set up that server for being criminally stoopid, they probably wanted it to be that way. You can't possibly be that incompetent in cyber security matters when you're running a cyber security software company.

Link to comment
Share on other sites

7 minutes ago, Ssnake said:

...and which point one has to ask if the solwarwinds management wasn't entirely and criminally complicit in this. It starts with weak passwords to begin with, and remaining passive about the removal of vulnerabilities. If they didn't see the problem and didn't immediately fire whoever set up that server for being criminally stoopid, they probably wanted it to be that way. You can't possibly be that incompetent in cyber security matters when you're running a cyber security software company.

This debacle is an illustration of how the people part of the problem is always the problem. The transition from monitoring to managing requires privilege elevation.

IMHO, SolarWinds is not a cyber security company, they are a network tools company that segued into centralized sysadmin tools. Essentially a software house. The elevated privilege required for centralized sysadmin demands fundamentally secure culture, policies, and practices. Few software houses have those, and IMHO most aren't even capable of developing those. 

people => culture => practices

Expect more debacles unless the global economy begins holding C-levels* fully accountable. Imagine how many organizations had a CIO telling the executive staff something along the lines of "We can cut IT Department salary spend by 10% if we manage all our servers using SolarWind software."

* Especially within the .gov sector; the dominant technologies, to a large extent, are dominant because .gov ensured their financial well-being though universal adoption. 

 

Link to comment
Share on other sites

23 minutes ago, Ivanhoe said:

IMHO, SolarWinds is not a cyber security company, they are a network tools company that segued into centralized sysadmin tools.

Given their customers, that distinction is in practice meaningless. A network tool that must, in order to do its job, have high level credentials attached to manage a complex network is by definition a network intrusion tool (as has just been demonstrated, duh). If you build network intrusion tools, you're a cybersecurity company whether you like it and acknowledge that, or not. If they didn't understand their role, they had no place to be where they were. That everybody used their tool without evaluating the company, their practices, corporate culture and attitudes is about as revealing about our general approach towards cybersecurity as that hysterical photo from the Hawaiian early warning center with the password post-its behind the commander's pose of a stalwart defender of the US.

Or, they did understand their role perfectly well.

 

So we have to choose between two possible explanations, that not only Solarwinds management was incompetent but also their techies and their customers, or that the management was malicious. To me, Occam's razor is in a balance right now. Both scenarios require significant willing suspension of disbelief.

 

I'm running a software development company myself. We're using distributed development. The biggest existential threat to the company would be that the source code would be stolen, or compromised. So we're taking appropriate measures to protect ourselves against that possibility as much as we can without becoming totally paranoid. I know that there's a dozen foreign port scans routinely testing our servers every day. It's a no-brainer to get your servers closed as tight as you possibly can.

Running a company means that you need to understand your risks just as well as your opportunities. I can't do much about the possibility of an insider job. At some point you have to trust the people around you (while we're at it, how much can you trust your router, or your compiler to begin with). But it's also clear that you don't trust blindly, and when you learn about a vulnerability (especially if an outsider is kind enough to alert you about something he just discovered), there is no excuse for inaction. "solarwinds123" is the equivalent to pin the sign "kick me" to your own jacket before stepping into a highschool hallway.

Link to comment
Share on other sites

TMK, no mention yet of SolarWinds' practices concerning vulnerability assessments/pentests. We can hope that in the future, C-levels will require 3rd party C&A for upstream vendors. Reality, though, will probably be more of the same. 

As for SolarWinds' executive staff perfidy, I'll apply Hanlon's Razor until the FBI issues arrest warrants. 

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...