In Search of Useful Models

I was in graduate school when I first heard the well-known quote by statistician George Box: “Essentially, all models are wrong, but some are useful."

As I was composing this post, I found some excellent discussions via Stack Exchange, Schaun Wheeler, and Thaddeus Tarpey with respect to Box’s assertions. My synopsis of these discussions is that most models are effectively approximations. As approximations, models can and do yield errors; however, within the proper context, these approximations provide us a useful framework to simplify and reason through complex systems or processes. Ultimately, the true test of any model is its usefulness.

BMI - Poor Precision, High Utility

I recently underwent a routine health check. As part of the examination, the doctor computed my Body Mass Index (BMI) based on my height and weight. According to the NIH, BMI is a model for body fat levels and is a “good gauge” for the health risks associated with excessive weight.

My BMI registered at the top end of the overweight scale. The “top end” is another way to say that, according to the BMI model, I was nearly obese. When I saw my position on the scale I said to myself, “This is ridiculous.” I immediately recalled the articles I had read about athletes like Shaquille O'Neal having a BMI in the obese range. Weaknesses in the BMI are pretty well understood. This simple model is clearly wrong- isn’t it?

For those who have seen me, it is pretty obvious that I am not obese. However, my encounter with the BMI did yield a useful result; the high number lead to a conversation with my doctor, internal contemplation of my dietary and exercise habits, and changes that lead to improved health. In addition, I now have a baseline from which I can judge progress.

Models for Security Risk Assessment

Over the past few years, I have had many conversations with security risk professionals about risk assessment models commonly espoused, and have observed the following:

1. We have a lot of models, frameworks, and methodologies (ex. ISO 27000, NIST 800-53, OCTAVE, 20 Critical Security Controls, COBIT, CMS, FAIR).
2. We haven’t settled or coalesced as an industry around any particular one or set of frameworks.
3. Many risk professionals are frustrated with points 1 and 2.

Although all of the models above have weaknesses, I would argue that, given the proper context, they can still prove useful. Organizations can use them to drive conversations and shape strategies for risk identification and management. The question is which model to use and in which context? According to statistician J. Michael Steele, good models sharpen the vision, provoke questions, and “let other people in on the conversation.”

We are a community rich in models, each model with varying degrees of approximation and shortcomings. Our challenge is to find, adapt, and embrace models that will ultimately sharpen vision, provoke better questions, and let other people in on the conversation.

The question that I would like to pose to the community at large is what is the equivalent of the security BMI for your organizations? What do you use to help establish a benchmark from which your organization can start questioning assumptions and make meaningful changes?