Don't RegreSSH: An Anti-Pavlovian Approach to Celebrity Vulns

Before Crowdstrike caused the world to melt down for a few days, the talk of the security town was a recent OpenSSH vulnerability (CVE-2024-6387). Dubbed by its celebrity name regreSSHion, it is a Remote Code Execution vulnerability in some versions of OpenSSH discovered by the Qualys Threat Research Unit on July 1, 2024. Specifically, versions of OpenSSH compiled against the glibc library, which is to say “probably most of them”, were impacted. This spawned more than a few articles, as the world collectively panicked about its potential impact.

The vulnerability has to do with inducing a race condition in OpenSSH, and while some outlets reported it as “Critical”, if we are gonna be pedantic, it’s actually only High with a numeric CVSSv3.1 score of 8.1. The vulnerability was named with the clever SSH pun in the word “regression”¹, because it is a regression of a 2006 vulnerability that was caused by the same bug. That ancient vulnerability was patched, but an update in 2020 reintroduced it. For those unfamiliar this kind of functional/security step backwards is often called a “regression”. This means that relatively old versions of OpenSSH (pre 4.5p1) and more recent versions (post 8.6p1) are vulnerable, while a large swath of middle versions are not vulnerable (more on that in a bit).

Given the recent near miss of a widespread backdoor in OpenSSH through the xz compression library and the general consensus that OpenSSH could probably be considered “critical infrastructure”, people’s ears perk up when they hear about vulnerabilities like this. But before your cathexis² about this vulnerability becomes overwhelming, consider my thoughts as one of Bitsight’s leading Khans of Cognition.³

First it’s no surprise that OpenSSH is widely used. Others have reported some raw numbers, and we can do that too (32M unique IPv4 addresses have responded that they are hosting an OpenSSH server throughout 2024 so far). This means that ~5.7% of organizations that we track have OpenSSH on an external facing server. If only to do something slightly self serving, take a gander below at an interactive globe I made like it’s first grade show-and-tell.

Figure 1 Global distribution of all versions of OpenSSHh. Feel free to click and drag to rotate the globe and scroll to zoom in. Note the density will shimmer and shine as you do, as the bins will renormalize the totals to the current view.

At first, Figure 1 probably makes you think “Big deal, you made a population density map”. But on closer inspection, there is something interesting happening in China. In particular, despite having a massive population density we don’t see that many OpenSSH servers. This is likely due to the infamous “great firewall of China”, which ensures that there are but a few ways in and out of the Chinese Internet, and that those ways can be easily monitored by the CCP.

So let’s start with the regular song and dance about what versions we see and where we can see them. First let’s look at the distribution of versions for OpenSSH servers that we managed to snag information on. As you can see, most are running OpenSSH 7.X and 8.X that are not vulnerable to regreSSHion (approximately 78%). These are safely non-regressed versions, so no need for those orgs to “hop to it”, to get things fixed. Most of the ones that are vulnerable are the later version, with really small percentages in the lower version levels down there.⁴

Figure 2 Breakdown of OpenSSH versions found globally in 2024. The bars down there are tiny but they are there. Mouse over and they’ll display in the interactive.

Now, at this point we might move on to the typical step of “which industries are most affected?” or “where are there unusually high geographic concentrations of regreSSHion?”, but that would give us roughly some of the same statistics that were in the xz backdoor post we did a couple of months ago. So let’s do something a bit different.

One thing that we are keeping an eye on at Bitsight is how organizations are handling “digital transformation”. In particular how much they leverage technology to operate their business. One blunt measure of this is “detected internet assets per employee”. Interestingly this measure spans multiple orders of magnitude, with some organizations having at most 1 asset per 10k employees, while others seem to have thousands of internet facing assets per employee.

When we examine this in the context of regreSSHion we see that organizations that have a higher asset/employee density do indeed have a higher likelihood of having the regression vulnerability.

Figure 3 Asset density and likelihood of regreSSHion. Each dot is placed at the 0.1% quantile. That means 0.1% of orgs have that value of asset density or higher. The probability line was created using a logistic regression model against the log of the asset density at an organization.

I like this particular question, because the result really could have gone a number of different ways, all interesting in their own right. Reasonable alternative hypothesis might have been “More asset density means that orgs are more tech forward and therefore less likely to expose OpenSSH” or even “There’s no real difference”. Instead we get this slightly higher probability for that higher density.

Looking at this vuln a few weeks out from its disclosure we can better assess the risk it poses to organizations. While everyone was worried about this particular vulnerability it wasn’t totally clear that this was actually exploitable. The .txt writeup provided by Qualys was long, and somewhat discursive. They described the race condition was triggered on virtual machines with stable network connections, requiring non-default settings, and depending on the version, takes anywhere from a few hours to weeks to trigger. They do not provide a proof of concept.

While Qualys didn’t provide proof-of-concept code, some github repositories purported to have one. However, according to the folks at Qualys it is non-functional. In another case another purported exploit was malicious. I’d also expect that if there was evidence of exploitation of this one in the wild, we’d pretty quickly see its addition to the KEV Catalog (which we have not as of this writing). So at first blush, organizations were likely panicking to get this patched, but… maybe they shouldn’t have?

Prioritization is a tricky thing, but what we can do is ask “how many organizations had other open, critical or high severity vulnerabilities besides regreSSHion?”. The answer is 21% of organizations that were vulnerable to regreSSHion have other externally facing, open high or critical severity vulnerabilities. Moreover, 17% of them had open vulnerabilities that were on the KEV. Those vulnerabilities pose much higher risk than a celebrity vulnerability with a non-existent proof of concept.

We can break down that percentage by the version of OpenSSH folks are running.

Figure 4 Percentage of organizations with other high or critical vulnerabilities, aggregated by the most recent version of OpenSSH running within that organization.

So, generally organizations that have up to date versions of OpenSSH (well at least one up to date version) tend to have lower rates of other critical and high severity vulnerabilities. But it still bottoms out around 20%! These organizations should probably be focused on those vulnerabilities rather than the celebrity vulnerability of the minute.

Given we are still recovering from Crowdstrike and learning more about what exactly happened, it seems a little quaint now that we are talking about an open source vuln in a widely used product that is probably not posing a huge risk for organizations. We have to perhaps take a more data driven approach to understanding the threat when these things appear. And this probably goes for all those security influencers as well. Some were stoking the flames, but others got it right from the beginning.

So let this blog be a bit of catharsis⁵, and ease your mind that you maybe don’t have to pull all nighters every time some researcher comes out with a fancy branded vulnerability. Step back, read the tea leaves, and check if there are maybe other dangers facing your organization. Maybe take the time to fix those other open vulnerabilities, before you let regreSSHion skip the line.

 

¹All my statistics friends will need this clarification.
²Most easily defined as the opposite of catharsis.
³Thought Leader
⁴A quick footnote on OpenSSH security support. The OpenSSH developers don’t maintain a list of supported OpenSSH versions. Rather they claim that OpenSSH is simply an implementation of SSH protocol, and therefore any concept of support is nonsensical as long as the protocol itself doesn’t change. They rely on downstream developers to “support” various previous versions. In their eyes, you should always be on the latest.
⁵Best definition is the opposite of cathexis.