Strategic Legal Technology

Tom O’Connor and I recently wrote a joint blog post about concept search software for e-discovery. Subsequently, we received comments from Herb Roitblat of Orcatec, an expert in information management, data mining, statistics, and eDiscovery processes. I share his comments here. 

Tom posted at his docNative Paradigm Blog Herb’s comments on Xerox CategoriX and Musings on the Best Approach to EDD Search (29 Oct 2009) by Tom and Ron:

• Herb Roitblat on ED Searches (12 Nov 2009)
• Herb Roitblat On Search: Part 2 (23 Nov 2009)

I publish here, with permission, additional comments from Herb, who wrote these in response to a message I sent him with my “take aways” from his first comments.

Summary
My summary and interpretation of Herb’s comments below and in the posts at Tom’s blog is that while concept search is a useful tool for e-discovery, the selection of the specific “flavor” of concept search tool matters less than smart application of it. Tool selection needs to be case specific because a “bake-off” among concept search tools only tells you how well a tool does against a specific set of documents. Since it’s not economically feasible to use multiple tools per case, you need to make a reasonable tool selection at the outset of a the case. As important, you need a reasonable and defensible process (which means documenting tool selection and process). The reasonableness standard depends on the stakes of the case.

Herb and Ron Exchange by E-Mail

Ron: So it sounds like what you are saying is that the difference in e-discovery concept search tools is probably overwhelmed by differences in document sets and in process / control.

Herb: I agree with this, but it has to be said carefully. Clothing does not make the man and high-powered tools do not make the builder, but they do help a good builder do better work. No matter how good your tools are, if they are not used well, you get a questionable result.

Ron: Concept search is not a magic bullet but helps expand the universe of documents to consider because it finds docs with words you would not otherwise think of as search terms.

Herb: It helps you think, but it is not a substitute for thinking. It is, as you say, not a magic bullet, just an amplifier.

Ron: Concept search can also help speed review by clustering similar documents.

Herb: Concept search expands queries to return results that are the best match to the expanded query. Thus, the top results are those that best match the query term and its context. (See green search on Truevert.com for an example, search for meat and get organic meat, not Omaha Steaks). There the context is given by green documents.

Ron: I back away from my initial assertion of the need to use multiple tools. I argued that to spur thinking among EDD professionals. Upon further reflection, what I really meant to say is that lawyers should focus more on industrial processes and controls, statistics, and metrics than on software features.

Herb: That’s what I think.

Ron: So that means we have no magic bullets. The legal profession has hard work ahead to industrialize its processes.

Herb: It’s actually not that hard. You just have to be thoughtful about what you are doing. It is not even terribly burdensome if you are realistic about the levels of accuracy that you can really achieve (see below).

Ron: We still don’t seem to have an objective standard by which to judge if a process is ‘good enough’.

Herb: There are lots of ways of deciding whether a process is good enough and lawyers are used to making reasonableness judgments and arguing about them. What are the consequences of different types of errors (e.g., retrieving too many documents, retrieving too few)?

Scientists, by tradition, usually use a standard of .95 confidence. For example, if two treatments are different with 95% confidence, then we accept them as different. That does not tell us how different they are or that the difference is practically important or useful, only that the difference is statistically significant. Scientists often report higher confidence levels than that, but the minimum is usually .95. That tradition has worked well in science where subsequent research can correct the relatively few times when the difference does not really exist, but resulted from sampling (luck of the draw).

As an analogy, if you play slot machines, the things return only about 95 - 98% of the money that gets pumped into them, but that does not mean that some people don’t actually win large amounts. It happens sometimes. The luck of the draw usually returns less than you put in, but sometimes it returns more.

Back to good enough. Engineers typically use confidence levels to tell them how well to build a bridge. They consider the consequences of different kinds of failure (think of the Tacoma Narrows Bridge). NASA uses confidence levels to determine the quality of their systems. Where the consequences are severe, they require higher confidence.

In eDiscovery, we are familiar with proportionality arguments and the like for determining things like cost shifting. The same thing applies here. A bet the company litigation may merit a higher level of confidence than a run of the mill litigation. Different types of errors may be weighted differently depending on the consequences of that kind of error.

None of this is hard nor does it require very much mathematical background. I published some tables a while back showing how many documents you should sample if you want to achieve a certain level of confidence and you are willing to accept the possibility of missing a certain proportion of responsive documents.

As I think I’ve said, I think that another part of reasonableness is transparency. Be able to describe what you did. A scientific publication is intended to describe enough of the methodology so that another scientist can replicate the observations. I don’t think that you necessarily have to publish to the other side what you did, but you should be able to provide that information if required (think Victor Stanley).

Just when law firms were beginning to master business intelligence (BI), the market goes and changes the rules. Growing volumes of alternative fee arrangements raise the question of what’s next with BI? 

In the old normal, law firms tracked a key few metrics: billable hours, capacity utilization, leverage, realization, and rates. It all seemed so easy. Of course, it was not as simple as it looked and smart firms figured out that BI is a way to dig deeper to understand what levers to pull to juice profits.

As more work moves to AFA, how will firms analyze their businesses and financials? I wish I knew; I’m just beginning to think through the options. Inputs welcome!

I started thinking about this after reading Execs Want Focus On Goals, Not Just Metrics (Information week, 13 Nov 09), which describes how business increasingly monitors real-time performance via either dashboards or scorecards (the former summarize metrics, the latter compare the metrics to pre-determined targets). I’m not sure law firms need real-time data but monthly seems way too long to wait.

Then I read the blog post Performance Measurement – the sequel by James Dunning. He emphasizes the importance of forward- rather than backward-looking metrics as well as providing good general advice about metrics.

In the past, I’ve been impressed by Redwood Analytics approach to BI (Redwood is now a LexisNexis company). Redwood’s Redwood Analytics® Planning Application “is one of the first products in the legal industry that can connect day-to-day matter management with increasing client demands for alternative billing arrangements.” It sounds great but it’s not clear how it works (even when I toggle to “features” at the top of the page). Thomson Reuters Elite also offers BI software; a Google search of the site - site:www.elite.com “alternative fee arrangements” - yielded no hits; in the singular, the only hit is to a conference program.

I don’t yet have a good answer about the right forward looking metrics for a firm driven by AFA but I do know that management should be thinking hard about the answer as firms increasingly use AFA.

Economic hardship has forced general counsels to cut costs. Most large law firms have, in turn, laid-off lawyers and staff and acquiesced to demands for alternative fee arrangements. Smart firms are also trying to differentiate themselves. For example, many now aim to become end-to-end e-discovery providers. 

UPS Cuts Costs With E-Discovery Counsel (Corporate Counsel, 11 Nov 2009), describes King & Spalding’s “cradle-to-grave solution” for e-discovery at UPS. Separately, off-the-record reports suggest that many law firms want to provide an integrated, one-stop e-discovery solution for clients. This is a big change; previously, most firms were content merely to counsel clients on the law, rely on vendors to do the work, and supervise the overall process.

Firms likely will find it a challenge to be, on their own, the “go to party for complete e-discovery”. They will need to partner with one or more vendors. In January 2007 I wrote Coming E-Discovery Battle between Vendors and Firms. I noted that “law firms and EDD vendors may compete for e-discovery consulting business.” Now I can say the same about all of EDD. My opinion then re consulting applies equally to the rest of the EDRM model:

“Clients should consider carefully who offers the better set of skills and experiences. BigLaw brands may offer comfort, but some vendors have hired experienced lawyers and may offer the better bundle of skills and experiences.”

Law firms have the advantage in counseling clients on legal strategy and e-discovery issues (as I observed in my May 2007 white paper 4 Ways an eDiscovery Attorney Can Make Your Firm More Successful). When it comes to providing integrated e-discovery service, however, law firms have several disadvantages relative to vendors:

• Art vs. Industry. BigLaw has long asserted that “everything we do is art” and cannot be standardized. That mentality works against the industrial strength processes (e.g., rigorous metrics and QC) and disciplined project management that high-quality and cost-effective e-discovery service requires.
• Sub-scale. Most law firms simply do not have the necessary scale to flex up and down to manage the peaks and troughs of e-discovery processing / hosting and document review. Large vendors can better manage the fluctuations because they aggregate demand across multiple firms and clients. Scale also limits most firms’ ability to stay on the leading edge of EDD technology by developing proprietary technology and/or evaluating and running multiple third-party platforms.
• Declining Unit Pricing. E-discovery and document review is moving to unit pricing that is declining over time. Vendors don’t like falling prices but have mechanisms to cope. BigLaw partners, in contrast, have enough trouble moving beyond the billable hour much less lowering fees. Law firm DNA makes it hard to deal with the current EDD trends.
• Scarce Investment Capital. Exploding data volumes require ever more servers and software licenses. That means capital. Law firm investment has typically meant “any outlay that cannot be billed to to a client in the same month.” Law firms have never been well-capitalized and, in the current economic environment, loans are difficult to obtain and costly. So it’s not clear how firms will fund growing their EDD infrastructure and keeping it state of the art. Well-managed and well-capitalized vendors are accustomed to on-going investment to keep and grow business.

Nonetheless, I think that law firms will continue to play a critical role in managing clients’ EDD requirements. Their best path is to focus on their core legal strength and their deep relationship with clients. For the heavy lifting of e-discovery and document review, most firms will find it easiest and best to partner with vendors.

[I adapted this post from a similar one I wrote at Integreon.com, posted earlier this week.]

Last week I spoke at the Ark Group Conference on Alternative Fee Arrangements (AFA). I gave a short presentation called Unbundling Repetitive Aspects of Large Matters, my effort at de-mystifying and simplifying alternative fees. 

Inside and outside counsel would find AFA easier if, instead of thinking about entire matters, they thought about the components of big matters. By big matters I mean single large matters such as major litigation or an M&A deal or “portfolio” matters such as real estate transactions, sales contracts, or managing NDAs.

These big matters typically include high volume, repetitive elements that can be treated as fairly discrete activities and therefore costed and priced separately. Here are some examples of common “discrete activites:”

The key to achieve reasonable alternate fees for both clients and firms is to unbundle these and other high volume tasks and treat them as discrete activities. Doing so can lower the cost and improve the predictability. All that’s needed is to apply the appropriate selection of process, technology and human resources:

• Process
• Workflow analysis
• Metrics and QC
• Data analytics (EDD)
• Knowledge management
• Business intelligence
• Project management
• Technology
• Document assembly
• Conceptual review tools
• Repositories
• Contract management
• Human Resources
• Paralegals
• Contract lawyers
• Outsourced lawyers (onsite, onshore, offshore, multi-shore)

By unbundling - that is, separating matters into discrete “chunks of work” - and then applying the tools above, tracking costs and effort, and monitoring and repeating the process to refine estimates, clients and lawyers likely will find that they reduce cost and, as important, make cost more predictable. That in turn should make AFA much easier.

Am I missing something, or is AFA easier than meets the eye?

Since not everyone reads Twitter, I reproduce here a selection of my recent Tweets. 

RT @tcorcoran: “Demystifying Outsourcing 4 Corporate Counsel” http://bit.ly/r2E06 || Excellent LPO article.

RT @technotheory Exciting new Twitter feature: friend lists built into the site + open APIs http://sn.im/twilists || Taxonomies anyone?

Re-Tweets force me to come up with two-word editorial comments. Maybe Tweeting lawyers will learn to write clearly

Desperate for outlining software with each point tag-able, like database - like Ecco. eg, project punch list by room AND by trade

Trying http://fastflip.googlelabs….. Images are too small. Need option to re-size so I can read the screen.

Slaughters considers legal outsourcing (Freshfields, DLA, Links too) The Lawyer http://bit.ly/pT4vz || Like Wachtell in US considering LPO

ATL: BigLaw swaps associates for staff attorneys http://bit.ly/11sBP0 || Clients resist assoc rates for doc review6:53 PM Oct 6th from web

I no longer waste time + bits putting fax # in contacts. Hope Outlook 2010 has fields for Skype, IM, Twitter, other personal URLs

Outside Counsel Spending Projected to Drop by 4.3 Percent Next Year’ NLJ http://bit.ly/47kz1 || on top of 11% this year per BTI survey

BigLaw 1st yr assoc comp stuck at 145k - 160k. Demand way down, why so sticky? 2-tier system, like UAW, a solution? Protect incumbents?

Litigation Trend Report by @Fulbright: Offshore doc review doubles in US (to 9%) and triples in UK (to 20%). 49% satisfied; 6% not.

ACC to Unveil Law Firm Ratings per Corp Cnsl http://bit.ly/17jjKI || Market changer if GCs ding firms publicly

Reading ‘What the New Law Firm Looks Like’ Robin Muir http://bit.ly/1qmztv || Thoughtful piece on where BigLaw is and needs to go

One of my friends has a substance abuse problem - Twitter. You know who you are. My lips are sealed.10:21 PM Oct 22nd from web

Law firms suffer “institutional isomorphism” - converged way of working + beliefs to justify http://bit.ly/Cp2w8 #FTLawyers || Great concept

@StevenBLevy From my prior hands-on regression analysis experience, proving EDD cost limits court access would be hard indeed

RT @jordan_law21 Era of hyper-measurement http://bit.ly/2nv1hV Lawyers get ready for productivity metrics || But what will we measure?

Anyone know if Bloomberg Law is making any headway against Westlaw or Lexis? Any predictions?

RT @DougCornelius blog on Google’s New Privacy Dashboard http://bit.ly/UATfB
|| Do we have any privacy? 3:56 PM Nov 5th from web

RIM @BlackBerry Desktop Manager 5.0.1 sync fails. 5.0 worked fine. Sprint Nextel support tells me to downgrade. Not answer I wanted.

BigLaw associate salaries are going down at some firms. Where is the bottom? How will we know when market equilibrates.

Twitter and Linkedin now integrated, using #in hash-tag so this Tweet shows at Linkedin. || Impact of two social media joining?

UPS Cuts Costs With E-Discovery Counsel http://bit.ly/3O16TG Corp Counsel mag || Law firm offers fixed fee per doc, UPS saves big. about 20 hours ago from web

LegalWeek last week published a survey of UK partner views on LPO. Partners seek new models but wary of LPOs opens “City partners increasingly believe the law firm model will have to be overhauled but remain sceptical that legal process outsourcing (LPO) will be a major part of the revolution.” First the findings, then comments on the survey, and finally the implications. 

Key Findings:

• GROWTH. Asked about LPO growth prospects, 13% see “major prospects for growth” and 34% “considerable”. The remaining 53% expect little or no LPO growth.
• QUALITY. Over 50% think LPO work quality “could be better", 7% found it poor, 14% “good", and 27% “ok”.
• SERVICE LEVELS ONSHORE v OFFSHORE. 36% think service levels in the UK are “much better” than offshore; 38% “better"; 19% no difference among locations.

Comments on the Survey Instrument (or why you may not want to trust it)

This survey strikes me is yet another legal market survey of questionable validity:

• LegalWeek provides no breakdown of who responded, so readers cannot determine if the sample represents the market.
• Qualitative answers such as “considerable growth” and “major prospects for growth” are too open-ended. Respondents might, for example, have had in mind anywhere from 20% to 100% for either answer. It should have asked using numbers, for example, “More than 40% growth” and “Between 20% and 39% growth”.
• Without knowing how many respondents have used LPO services, the results may merely reflect what respondents think other people in the market think.

Analysis Assuming the Survey Results are Valid

Let’s assume the results accurately reflect the large law firm market sentiment. In the old normal, meaning pre-economic crash, I would have said the mixed results could not last long. Large law firms in the US, UK, Canada, and Australia followed similar strategies: open multiple offices, merge, invest in similar technologies, build marketing operations, increase leverage, etc.

So if past is prologue and the new normal is like the old normal, then I think these results suggest LPO is tipping toward fairly wide spread adoption. Answers here could easily, with a bit of language tweaking, have been applied at one time to use of e-mail, law firm marketing, and doing e-discovery. As each of those new approaches started, lawyers were widely resistant and skeptical (and like, with LPO, typically without any first-hand experience). My rule of thumb: once a new thing gets to the point that major legal publications run just this type of survey, it almost inevitably means wide-spread adoption cannot be that far off.

Past may not be prologue. In my Integreon blog post last week, Law Firms Differentiate in a New Era I suggested we may see large law firms following different strategies. If so, then the survey could mean that LPO will only be adopted by some of the market. Then I would guess ½ of firms would adopt LPO based on the answers to the growth question and the fact that availability of onshore services will grow, which will satisfy those who think onshore offers better quality.

Legal process outsourcing (LPO) has been in the news a lot in the last few months. And there have been some good blog posts too. Here’s a listing. 

For a November 9th presentation on Legal Process Outsourcing: Opportunities and Challenges to the Palo Alto Area Bar Association (PDF of program) I assembled a list of recent legal press LPO coverage. I was surprised at the extent and nature of the coverage. That list appears at the end of this post.

I also want to call out a few blog posts of interest:

• Wearing my day job hat at Integreon, I posted today Law Firms Differentiate in a New Era at the Integreon blog. I suggest that law firms will follow different strategies in the future (picking up on Law Firm Business Models: A New Series at Adam Smith, Esq.) and that going public on initiatives brings benefit, illustrating with of examples two UK firms that recently publicized their legal outsourcing.
• Matthew Sullivan of RedBridge Strategy explains at his blog in Working with LPO Vendors: Relationship or Transaction? the pros and cons of forming a long term relationship with an outsourcing vendor. This is a good blog to read.
• Michael D. Bell of Fronterion, Inc. writes in his blog post Quantity vs. Quality that lawyers need to focus on quality and other factors in outsourcing as much as on cost savings. Another good blog.
• My colleague Matthew Bank’s writes in What’s in a Name? “Legal” + “Process” + “Outsourcing” that LPO means more than offshore, it’s all about process and outsourcing to any destination.

Legal Outsourcing in the News - Recent Articles, Jun - Oct 2009

• Legal process outsourcing is ‘here to stay’ (Law Society Gazette, October)
• “Out, out damn source” (The Lawyer, October)
• Xerox GC says LPO is a $4bil industry (Rees Morrison blog post, citing book, October)
• “Simmons goes live with outsourcing project” (The Lawyer, October)
• Gibson Dunn acknowledges going offshore for doc review (Denver Post, October)
• “Slaughters in talks over outsourcing plans” (The Lawyer, October)
• “Demystifying Outsourcing for Corporate Counsel” (The Legal Intelligencer, September)
• “Offshoring legal work.. at a lower cost requires even more stringent management, but the savings seen could prove increasingly persuasive..” (Managing Partner, September)
• “Will Increased Compliance Burdens Lead to Legal Process Outsourcing?” (NLJ, August)
• “ Offshoring firms: the day has come” (The Lawyers Weekly (Canada), August)
• “Aussie firms embrace LPO” (Lawyers Weekly (Australia), July)
• GC attitudes to LPO changing gradually (Int’l Bar Assoc article )
• Rio Tinto hires team of lawyers in India (Times Online, June)