Kay, MacEwan & Riskin on KM ROI

The responses for my KM Economics post (& questions) have been amazing. I'm including below a series of responses that I found particularly helpful. Warning, long post.

First up, Stuart Kay from Baker & McKenzie, Australia offers some wise cautions to using realization rates as a KM success measure. Stuart, as many of you will recall has written a number of academic & web pieces on KM ROI, including the classic LLRX article, Cost, Value and ROI for Knowledge Management in Law Firms. I'll add my support for all of the factors Stuart mentions, and add law firm collections & bad debts to his list.

With his permission, here are Stuart's thoughts on the matter:

Dear Steve,

In theory the measurement of write downs (pre-bill value less collections) might be an indicator of the success of KM efforts. However, without interviewing the partners and lawyers who worked on each matter, how do you know if the write down was due to:

• inefficiency in production (eg lack of precedents or process - which could be improved by KM);
• a poor initial estimation of fees resulting in the necessity to write down;
• an agreement with a dissatisfied client to reduce fees;
• inappropriate delegation to an inexperienced lawyer;
• a lawyer for some perverse reason known only to themselves not using a known existing K resource (eg precedent); or
• lawyer incompetence?

The labour involved in gathering the information across all the matters in the firm in which there was a write down would be prohibitive, and there is likely to be a significant error rate due to mis-reporting of the reasons for write down (eg what lawyer is going to admit to mis-quoting, dissatisfied clients or incompetence?).

If you simply benchmark KM against gross write downs, you may find yourself ruining the credibility of your KM effort as a result of a couple of poor hires a significant proportion of whose time is written down and over which you have no control or influence. The measure will not be accurate - it will be arbitrary, and you will essentially be gambling with your credibility.

Consequently, whilst I think the write down rate theoretically could be a valid measure of the ROI for KM, the practicalities of measuring it with the necessary degree of specificity will be, in most firms, absolutely prohibitive.

I hope this helps. I'm happy to discuss it in more detail if you wish.

Stuart Kay
Knowledge Manager
Baker & McKenzie, Australia

Next up, Adam Smith Esq. himself Bruce MacEwan, who has posted on this topic previously, answered each of my questions in turn:

The point is that an effective KM program minimizes writeoffs.

To answer your questions specifically:

1) Do firms consider this 'lost time' as part of their realization rate? (My firm does, but is this common?)

They better! Or they're seriously deficient in basic financial hygiene.

2) How significant is this gap? Is this a small factor in our evaluation of KM? or an important argument?

In my experience the reality of having to write off junior associates' time means that associates cost the firm money for at least the first year or two; to the extent that period can be shortened, it will justify a large investment in KM.

3) How can this be measured to get at KM's value? Are gross measures good enough, or must it be done at a matter level?

I think the gross measures are not only "good enough," but preferable to matter by matter--different practice areas are very differently "leveraged" in terms of associate/partner hours, and obviously a highly leveraged practice will show more benefit from KM than a less-levered one.

4) Does this not get back to the 'Leverage' driver? Good training (both formal and KM resource based), and building the skills of Associates, in order to reduce Partner time in delivering legal services.

Absolutely; see above.

5) What am I missing?

Not much!

Thanks for running this by me!
Cheers and best regards,
:Bruce

Response comparison: On first glance, the two responses may look to be going in opposite directions, and in a way they are. My take is that, while measurability may pose issues, the best approaches will factor in the more extreme ends of specificity. Either go very large and use realization rates at an Industry group or Practice Area level (watching for Stuart's caveats), OR, work down to sampling (cherry picking?) the most applicable types of Matters that avoid the issues Stuart mentions.

I think one of the fundamental points missed often is that KM is not always a drag on revenue. This is just an assumption, and a wrong one in my mind. As mentioned in my original post, and now having talked with a number of KM practitioners - in the trenches, the general feeling is that KM can have a significant effect on billing efficiency. Also the fact that KM is geared towards Associates in law firms. If Partners never lever down the work, the only way to make more money is to work longer hours. I guess this all comes down to the fact that KM is not ONLY an investment for the future (as David Maister & Gerry Riskin point out), but also an investment for now.

Finally I'd like to finish things off with this, the most practical advice I received, and it came via Gerry Riskin. I asked Gerry, "Do you think that I'm missing something because I feel there's value in looking for finite measures to track these processes. It's not so much the buy-in factor or making a business case for it - my firm's been at this for 6+ years. For me, it's more about guiding the process towards success, and finding some measures that let us know if things are going in the right direction."

His response:

As for "guiding the process towards success", I believe that you are correct in looking for metrics (the objective measurements) but I believe that there is a subjective side as well.

When Dan Mahoney created the Dupont model, he dramatically reduced the cost of certain legal procedures by requiring that various law firms serving Dupont share information and precedents. There was a dramatic reduction in per case costs that he could quantify. Similarly Dupont required the use of document generation software that dramatically accelerated the rate at which newer lawyers could acquire capabilities. This very positive step was likely difficult to measure in dollars but what competitive advantage goes to those firms who accelerate the progress of lawyers through the learning curve.

So, Steven, I would say continue to look for and use metrics but keep gathering the anecdotes and archive and publish them.

Is this helpful*?

It was to me Gerry :-), and now I hope it will be for everyone else too. Thank-you to everyone who responded to my questions or sent me their thoughts.