2024 Bitsight Ratings Algorithm Update: Purpose, Observations, and Impact

As our 2024 Rating Algorithm Update (RAU) goes live on July 10, 2024, we wanted to share some research that validates this update and reinforces the importance of the RAU process. As we noted in our announcement blog, after RAU 2024, remediated Patching Cadence findings will impact the Bitsight Rating for 90 days after the last vulnerable observation instead of 300 days.

Key takeaways

• RAU 2024 will change the lifetime of Patching Cadence findings from 300 days to 90 days. 
• Patching Cadence remains a strong indicator for undesirable cybersecurity outcomes. Statistic shows a 90-day lifetime for Patching Cadence in RAU 2024 is as well-correlated with breach incidents as the former 300-day lifetime. Poorer grades for the risk vector lead to progressively higher likelihoods of breach.
• With RAU 2024, Bitsight Ratings become more responsive to Bitsight customers’ efforts to improve their software patching behavior. 

A brief review of RAU objectives

At Bitsight, we perform periodic RAUs to revise and upgrade the core Bitsight Ratings product, aligning methodology and data with cybersecurity developments.

The key objectives of an RAU are to:

• Adapt ratings to better reflect the ever-evolving cybersecurity landscape.
• Enhance the utility of the ratings by:
  • Improving their statistical correlation with security incidents.
  • Making them more explainable, accurate, and actionable.
  • Refining the associated user experience.

Observations about Patching Cadence

In 2022, Marsh McClennan, the world’s largest insurance broker, conducted an independent analysis to quantify the relationship between the Bitsight Security Rating and 13 risk vectors using real-world cybersecurity incident data from 365,000 organizations. This research showed a significant correlation between Bitsight’s analytics and real-world cybersecurity outcomes. Patching Cadence stood out in this research as the risk vector with the highest correlation to real-world negative outcomes.

Based in part on these findings, we increased the impact of the Patching Cadence risk vector in RAU 2023. This significantly improved the correlation between the rating and security incidents. The downside of that change is that the 300 day lifetime for Patching Cadence findings made the rating less responsive to more recent remediations performed by companies. For RAU 2024, we sought to improve the responsiveness of the rating without impacting the correlation to security incidents.

As we stated in our blog announcing RAU 2024:

“In our research, we found that most of the predictive power of the Patching Cadence risk vector comes from vulnerabilities seen in the more recent past. We found that 90 days was a sufficient amount of time to consider observed vulnerabilities. Collectively, these improvements allow us to achieve a high confidence level in our understanding of vulnerability risk, enabling us to reduce the lifetime of Patching Cadence findings and create a more responsive measure of an organization’s remediation time.” 

So, in summary, the RAU 2024 update to Patching Cadence makes this important risk vector more responsive to recent changes in a company’s patching behavior.

A deeper look at the data

For any cybersecurity measurement, a sufficiently long observation lifetime is a must, due to the inherent latency between cause and effect. In other words, an observation must impact the rating for long enough to capture meaningful correlations with its downstream effect(s). With RAU 2024, our goal was to optimize the balance between correlation with security incidents and the time that vulnerability observations impact the rating. We found that we could maintain a significant correlation to breach even if we reduced the lifetime to 90 days.

In this case, we evaluated a 90-day lifetime against the prior 300-day duration for Patching Cadence using two different measurements:

1. The Rank-Biserial Correlation (RBC) measures total correlation with breach incidents.
2. Relative probability of breach as a function of risk vector grade provides insight into the incremental risk of breach with worsening performance.

The plots below compare the RBC values for RAU 2023 and RAU 2024 for the Patching Cadence risk vector and the overall security rating. Although we see slight differences between the RBC values, the overlap of the confidence intervals show that the RAU 2023 and RAU 2024 RBC values are statistically consistent.

Figure 1 Comparison of Rank-Biserial Correlations to Breach for Patching Cadence and the Security Rating¹:
Error bars indicate the 95% confidence interval.

In order to understand the meaning of the grading system for Patching Cadence, it is also valuable to study relative probabilities for bad outcomes across Patching Cadence letter grades, as presented in Figure 2. The chart below shows how much a grade other than A in Patching Cadence increases the likelihood of a breach. For example, under RAU 2024, a company with a Patching Cadence grade of B is 2.8 times more likely to experience a breach than a company with a grade of A. 

Figure 2 Comparative Breach Probability by Patching Cadence Grade: 
For each column, the relative probability is normalized by the respective A-grade value.

The RAU 2024 data presents behavior for breach susceptibility that is similar to what was previously reported (cf. Marsh McLennan Analysis). Most importantly, poor grades continue to show a strong correlation to breach incidents.

Conclusion

As seen from the data presented above, revisions to the Patching Cadence risk vector in RAU 2024 preserve its value as an indicator for undesirable cybersecurity outcomes, with differentiation of risk increments across the grade levels. At the same time, a 90-day observation lifetime in lieu of a 300-day span will be more responsive to Bitsight customers’ efforts to improve their software patching behavior. 

A preview of RAU 2024 has been available to users since April 9, 2024. RAU 2024 is slated to become effective on July 10, 2024.

---

¹ It merits clarification that Bitsight risk vectors are typically assigned letter grades (A-D and F). Underlying these letter grades are numerical feature scores that assess security performance over a continuum of numerical values. Since the Rank-Biserial Correlation measures statistical correlation between continuous values and a binary outcome, the feature score, as opposed to the letter grade, for Patching Cadence is the appropriate metric to consider.