A January 2013 Corporate Counsel article explores the potential of data modeling to support compliance with the U.S. Foreign Corrupt Practices Act. It’s great to see lawyers write about data and analytics, though I fear the article sends the wrong message.
In What Computer Models Can - and Can’t - Do, authors Ryan McConnell (Baker & McKenzie partner), Dianne Ralston (Schlumberger Ltd. deputy GC), and Charlotte Simon (Baker & McKenzie associate) discuss the recent book The Signal and the Noise—Why So Many Predictions Fail But Some Don’t (by Nate Silver) and what it means for lawyers. The book “discusses sorting through empirical data and identifying signals that enable better decision making”.
The authors ask “can techniques developed in fields as diverse as weather forecasting and baseball management be applied to developing a risk-based compliance program?” and, more specifically, “how do compliance lawyers sort through the noise to create a valuable risk-based program?” [emphasis added] They examine several data sources that might support compliance and point out the “noise” problems with each:
- Lawyers could focus compliance training based on corporate job descriptions but the authors express concern about the accuracy of job descriptions and determining which pose the most risk.
- “[C]alls to a compliance department’s ethics hotline highlights other sampling issues", specifically that there may not be enough historic data or the calls may exclude some regions so you “will only get half the story.”
- A gift tracking system might highlight questionable spending but can never tell you the expense was “improper entertainment because it cannot reveal the intent”.
- The “new FCPA guidance notes that taking a risk-based approach is particularly critical with respect to due diligence procedures for assessing third-party relationships” but the authors see constraints in the “reliability of the data obtained from the third party to assess risk and, for initial due diligence purposes”.
The authors conclude
“The success of a risk-based model, however, will ultimately depend not on technology tools, but on the compliance lawyer’s ability to successfully analyze risk data and sort the signals from the noise. That lawyer must be adaptive, creative, and look beyond the data to see organizational and industry trends and risks. By helping us understand the limits of technology and how to use data, Nate Silver can make us all better compliance lawyers.”
Reading the article, you might conclude that lawyers should seek less data, not more. That would be discouraging, and I think wrong. I think a better conclusion is “get more and better data and improve the hypotheses and analyses”.
In my view, lawyers should work with corporate colleagues to identify additional existing data repositories or collect additional data and then to improve the models. The limitations of modeling are data quality and quantity, creativity and rigor in modeling, and validating model outputs. Tools are not a limit: a host of both legacy and newer “Big Data” software support many types of models.
To be sure, models have limits. We must, however, take a step lawyers frequently forget - specify the next best alternative. With compliance costs skyrocketing, will companies simply continue to hire more and more professionals? Aside from expense, will ever bigger armies of minders actually find enough compliance problems? The issue of alternatives here reminds me of the debate around predictive coding in e-discovery. Computerized document review has limits, but so does human review.
Data modeling likely applies to many areas of law practice. Many lawyers thus must grapple with the issues here. In my view, that means, for many, overcoming a fear of numbers and working with statisticians. As a profession, lawyers need to get smarter about preventive law. That means at least trying more modeling, at least enough to conclude it really does not work. I have seen no evidence that we are anywhere near that point.
[Background note: I have little compliance experience but extensive experience working with and drawing inferences from data, including stints as an econometrician and management consultant. ]