In Search of Useful Models

Stephen Boyer | October 1, 2013 | tag: Security Risk Management

useful-models-for-security-risk-measurmentI was in graduate school when I first heard the well-known quote by statistician George Box: “Essentially, all models are wrong, but some are useful."

As I was composing this post, I found some excellent discussions via Stack Exchange, Schaun Wheeler, and Thaddeus Tarpey with respect to Box’s assertions. My synopsis of these discussions is that most models are effectively approximations. As approximations, models can and do yield errors; however, within the proper context, these approximations provide us a useful framework to simplify and reason through complex systems or processes. Ultimately, the true test of any model is its usefulness.

BMI - Poor Precision, High Utility

I recently underwent a routine health check. As part of the examination, the doctor computed my Body Mass Index (BMI) based on my height and weight. According to the NIH, BMI is a model for body fat levels and is a “good gauge” for the health risks associated with excessive weight.

My BMI registered at the top end of the overweight scale. The “top end” is another way to say that, according to the BMI model, I was nearly obese. When I saw my position on the scale I said to myself, “This is ridiculous.” I immediately recalled the articles I had read about athletes like Shaquille O'Neal having a BMI in the obese range. Weaknesses in the BMI are pretty well understood.  This simple model is clearly wrong- isn’t it?

For those who have seen me, it is pretty obvious that I am not obese. However, my encounter with the BMI did yield a useful result; the high number lead to a conversation with my doctor, internal contemplation of my dietary and exercise habits, and changes that lead to improved health. In addition, I now have a baseline from which I can judge progress.

Models for Security Risk Assessment

Over the past few years, I have had many conversations with security risk professionals about risk assessment models commonly espoused, and have observed the following:

  1. We have a lot of models, frameworks, and methodologies (ex. ISO 27000, NIST 800-53, OCTAVE, 20 Critical Security Controls, COBIT, CMS, FAIR).
  2. We haven’t settled or coalesced as an industry around any particular one or set of frameworks.
  3. Many risk professionals are frustrated with points 1 and 2.

Although all of the models above have weaknesses, I would argue that, given the proper context, they can still prove useful. Organizations can use them to drive conversations and shape strategies for risk identification and management. The question is which model to use and in which context? According to statistician J. Michael Steele, good models sharpen the vision, provoke questions, and “let other people in on the conversation.”

We are a community rich in models, each model with varying degrees of approximation and shortcomings. Our challenge is to find, adapt, and embrace models that will ultimately sharpen vision, provoke better questions, and let other people in on the conversation.

The question that I would like to pose to the community at large is what is the equivalent of the security BMI for your organizations? What do you use to help establish a benchmark from which your organization can start questioning assumptions and make meaningful changes?

Suggested Posts

The BitSight and Moody's Partnership: A New Era For Cybersecurity

Cybersecurity is one of the biggest threats to global commerce in the 21st century.

By providing data-driven insights into cybersecurity, we can empower the marketplace to make better, risk-informed decisions and create a more secure...

READ MORE »

4 Critical Success Factors for Effective Security Risk Management

With the average cost of a data breach in the U.S. reaching nearly $8.6 million, your organization can’t afford to ignore cybersecurity risk. Indeed, the need for security risk management is greater than ever. When cyber risk is managed...

READ MORE »

IoT Cybersecurity: How Your Organization Can Tame the Wild West

From sensors on the factory floor to those that guide autonomous vehicles, the Internet of Things (IoT) is transforming how we live and work. Over the coming years, IoT will continue to change our world, with the number of connected...

READ MORE »

Get the Weekly Cybersecurity Newsletter.