Lessons From the BPP: Frequent Measurement Yields Invaluable Insights

Understanding and quantifying the state of an organization's security posture at any given point-in-time can be an elusive objective regardless of the model or framework employed. Infrastructure, configurations, and threats evolve rapidly making it difficult to assess and quantify the organization’s risk posture. Faced with resource constraints, risk managers often face the trade offs between frequency and scope.

Several months ago I attended an excellent talk by MIT professor Roberto Rigobon where he presented his collaborative work with professor Alberto Cavallo on the Billion Prices Project (BPP). Although their research focused on economic indicators, I argue that the results offer an important lesson for risk management.

Rigobon and Cavallo’s research effort has yielded a system that collects and tracks prices on over 5 million consumer items across the globe on a daily basis. The scope of the collection is limited to prices available online obtained through an automatable process. They then used this data to formulate daily inflation indices, along with other indicators about pricing behavior.

According to the project description, the "indexes are designed to provide real-time information on major inflation trends, not to forecast official inflation announcements."

The team explicitly stated that the project’s intent was “not to forecast” but to “provide real-time information.” Rigobon was adamant that the project was not trying to be predictive; they were simply providing insight by measuring quickly on a massive scale.

Today the Bureau of Labor Statistics Consumer Price Index (CPI) is the principal metric that economists have at their disposal to measure price movements. For decades, the Bureau of Labor Statistics has obtained CPI data on a monthly basis by directing its data collectors to "visit or call thousands of retail stores, service establishments, rental units, and doctors' offices, all over the United States, to obtain information on the prices of the thousands of items." 80,000 items in total make up the monthly sample, and they aren’t constrained to include only prices found online.

The figure below charts the BPP versus the CPI over the last 5 years.

Comparison of BPP data to CPI data

It is an intriguing juxtaposition: a daily sample rate for BPP versus a monthly sample rate for CPI; 5 million data samples for BPP versus 80,000 for CPI; data collected from only online sources with a small team (BPP) versus an army of data collectors making visits and calls (CPI). I found it equally intriguing that BPP not only closely matched CPI, but that it is a leading indicator for it. By measuring quickly on a large scale, BPP is regularly ahead of what is published in the monthly CPI report.

There is a valuable lesson to be learned here for security risk assessment. Measuring quickly in an automated fashion on even a subset of the data can provide tremendous insight. The models don’t necessarily have to be predictive, nor do all of the measurements need to be exhaustive. Trends and baselines can be established from which key indicators can be used to drive informed decision making. Whether the application is online retail pricing or risk assessment, frequent, intelligent and automated measurements can sometimes outperform an army.