Recently delisted content aggregator Work Loss Data Institute (WLDI) published a write-up authored by Tom Ferrell Denniston (ODG, Account Executive), Dr. Stephen Norwood (ODG, Editor-in-Chief), and Phil Denniston (ODG, President). The write up was an attempt to “set the record straight” about the National Guidelines Clearinghouse’s (NGC’s) decision to remove the ODG content from its website. In order to participate in the NGC, guidelines must meet the Institute of Medicine’s (IOM) standards for trustworthy guidelines (click here for more information on IOM’s standards). Here are some takeaways from WLDI’s write-up:
- WLDI misrepresented both ReedGroup and the ACOEM Practice Guidelines on several occasions
- WLDI degraded the State of Colorado’s Medical Treatment Guidelines, calling their inclusion in the NGC “ironic”
- WLDI’s update cycles run contrary to IOM’s criteria for the development of true Evidence-based Medical guidelines (EBM)
- WLDI asserts that ODG is now a “Data Driven” guideline
WLDI appears to be on a campaign to degrade reputable, not-for-profit public-serving institutions such as the NGC, AHRQ, and HHS. The reason? WLDI is trying to draw attention away from its failure to meet the criteria that defines EBM (click here for my prior post about specific areas cited by the NGC). This is clearly in poor form, and any reasonable person closely following the series of events which have occurred since WorkComp Central first reported ODG’s removal from the NGC on June 23, 2016 would concur. Ultimately, ODG’s post-removal behavior is not really the heart of the matter. The vital element worth debating is that WLDI’s guidelines (ODG) are legislated/mandated as the standard of care for injured workers in a number of States’ workers’ compensation systems. The acceptance of sub-standard guidelines anywhere in our work comp system IS THE CRUX OF THE CONVERSATION.
Let me set aside the gross inaccuracies about ReedGroup and the state of Colorado for the time being. I will address these inaccuracies at a later date. For now, let’s take a look at how this all impacts worker health. Look closely:
The write-up (and a 26-page PowerPoint deck released by WLDI, download here) boasts that “ODG continuously updates a large number of topics and recommendations” and “ODG Low Back Chapter alone has close to 500 Procedure Summary entries,updated monthly”.
IOM requirements for creating quality, trustworthy EBM guidelines are very clear. Recommendations are not based on single studies, but rather a preponderance of evidence. The process of (1) creating the clinical questions to be answered, (2) gathering all of the evidence, (3) documenting every search term in every database, (4) creating the evidence tables, (5) scoring the articles, (6) drafting the recommendations, (7) incorporating review and guidance from multi-disciplinary panels, (8) allowing stakeholder / external review and incorporating this feedback to create final guidance isNOT a process that can take place on a monthly basis. Nor should it. The fact that WLDI is updating ODG on a monthly basis begs the following question – How can ODG do the proper diligence needed to adhere to all 8 steps outlined above in such a truncated time frame? The fact is they cannot. The foundation of evidence-based medicine lies in the meticulous, calculated, measured process outlined above.
Deciding to dig deeper, I conducted a little research and came across WLDI/ODG’s methodology document. See page 15, section 8. WLDI’s methodology is clearly defined. The materials listed as evidence include:
Ranking by Type of Evidence:
- Systematic Review/Meta-Analysis
- Controlled Trial – Randomized (RCT) or Controlled
- Cohort Study – Prospective or Retrospective
- Case Series
- Unstructured Review
- Nationally Recognized Treatment Guideline (from guidelines.gov)
- State Treatment Guideline
- Other Treatment Guideline
- Conference Proceedings/Presentation Slides
- Case Reports and Descriptions
Ranking by Quality within Type of Evidence:
- High Quality
- Medium Quality
- Low Quality
None of the materials under the “OTHER” header being reviewed by ODG wouldqualify as “high quality” and do not meet literature selection inclusion criteria for IOM, GRADE, AGREE, and AMSTAR. Over 50% of what they list as evidence is clearly immaterial. I suppose if one bases an update to a guideline on “10. Conference Proceedings/Presentation Slides” monthly revisions are not unreasonable. The guidance produced as a result of an update based on “Conference Proceedings/Presentation Slides”, however, is not even close to evidence-based by standard, and thus subjects workers to harm and shareholders to undue liability and risk. This “guidance” would also be nearly impossible to defend if someone were to challenge its basis.
The write up also highlights that “ODG Becomes a Data Driven Guideline” and therefore “does not fit the mold” that NGC has created for EBM. I agree that ODG doesNOT fit the EBM mold (due to its failure to meet the criteria), but I’m left wondering whose standards to trust – standards created by the Institute of Medicine or standards created by the Institute of ODG? The assertion that ODG is of higher quality because it incorporates outcomes claims data does not conform to any quality measure that I know of and is highly illogical. For example, a specific surgery may be performed at a high frequency for a large number of patients and thus is represented in “the data”; one cannot interpret the data to say that the procedure is of long term benefit to the patient and therefore should be recommended for injured workers. This is just ONE reason why workers’ compensation data should NOT drive treatment decisions.
WLDI also states, “Guidelines can study all of the literature, rank the reviews and/or RCT’s, adhere to the strictest standards of EBM, but most patients, providers, and payers simply want to know what treatments work and get better outcomes versus others.” I translate this to say WLDI feels that the science and the process is of little importance; all that matters is reduced utilization and subsequent savings. Of all of the studies published on the ODG content/tools, not one study has affirmed that workers are receiving better medical care as a result of the use of their content/tools. As a family member of a currently injured worker in the throes of the workers’ compensation system, the methodology behind the guidelines matters a GREAT deal. To base medical care on claims data is the epitome of putting the cart before the horse and is entirely misguided. Improvement in outcomes – healthcare and fiscal – are only appropriate when supported by scientific evidence scrutinized using atrustworthy, transparent, reproducible EBM methodology.
To date, WLDI has accused 3 not-for-profit federally administered public agencies of misrepresenting the truth, published a back-dated press release on their website, and distributed an outlandish write-up on LinkedIn about the de-listing being due to “formatting” issues. They have degraded very reputable institutions like the State of Colorado’s Department of Labor and Employment Division of Workers’ Compensation.
Why would anyone place any sort of stock/confidence in an organization that so effortlessly contours reality for financial gain? Should we trust the IOM or the IOODG? The choice is yours. Make the right one.
ReedGroup Sr. Vice President, Joe Guerriero posted a comment correcting WLDI’s assertions regarding the ACOEM Practice Guidelines’ participation in the NDC. The minute the comment post appeared the original WLDI write up was removed from LinkedIn. For the record, ACOEM will be submitting for inclusion in the NGC with full confidence that all IOM criteria will be met. (The original URL is: https://www.linkedin.com/pulse/odg-national-guideline-clearinghouse-real-story-denniston?trk=hb_ntf_MEGAPHONE_ARTICLE_POST)
For context and editorial integrity, here is the full unedited post as published by WLDI on August 2nd, 2016:
“ODG and the National Guideline Clearinghouse: the Real Story
Tom Farrell, Senior Texas Account Executive, ODG
Stephen Norwood, MD, Editor-in-Chief, ODG
Phil Denniston, President, ODG
On June 22, 2016, while speaking before a NAMSAP audience (the National Alliance of Medicare Set-Aside Professionals), Mary Nix, the federal administrator in charge of the National Guideline Clearinghouse (NGC), hosted online at Guideline.gov, announced that NGC was dropping ODG under its revised inclusion criteria. Coincidentally, on June 16, 2016, ODG had prepared a withdrawal letter to Ms. Nix, saying that being published on Guideline.gov is not a good fit for ODG. According to the letter, commercial guidelines are successful at improving outcomes because they are comprehensive and up to date, but the new NGC criteria do not support those requirements.
What brought ODG and NGC to the same conclusion at the same time?
ODG Was the First Workers’ Comp Guideline to be Included in the NGC
The previous NGC criteria required the medical treatment guidelines be evidence-based and be updated at least every five years. The ODG medical treatment guideline ODG Treatment in Workers’ Comp was first launched in 2003, and by the end of 2004 it was approved by the Agency for Healthcare Research & Quality (AHRQ) for inclusion in the NGC. For many years it was the only guideline used in worker’s comp to be accepted in NGC. The ACOEM Practice Guidelines were not accepted into NGC until late in the same decade. The Reed Medical Disability Advisor (MDA), which became MDGuidelines, has never been accepted into NGC.
The process for including ODG in NGC involved the NGC contractor ECRI having access to the online version of ODG, and preparing abstracts of the ODG content for posting on NGC, after approval by ODG. Over time, the ODG updating process became more and more frequent, with the objective of providing access to the most current evidence on the treatments that are proven to help injured workers recover and return to work. States adopting ODG, such as Texas, even specify that the “current” edition of ODG is the one required under the adoption rules. In addition, over time ODG became more and more comprehensive. If treatments or diagnostic procedures are missing from a guideline, the guideline cannot play its role of ensuring that injured workers’ get access to the right treatment and minimize friction in the system. As a result of the increased timeliness and comprehensiveness of ODG, it became more and more challenging for the federal contractor ECRI to do its job of abstracting ODG for NGC.
NGC Implements New Criteria
New NGC standards took effect on June 1, 2014, and require more detail. Those submitting guidelines must now explain how they selected studies for their evidence-based review, including the number of studies identified, the number of studies evaluated, a summary of inclusion and exclusion criteria, and the evidence from the selected studies must be synthesized in a detailed description or evidence table. Rather than evidence tables, ODG has Procedure Summary tables, where each entry follows a standard format, with a recommendation statement, a summary and discussion of the body of evidence, with links to the highest quality studies, that are rated and summarized, and in many cases a Criteria for use, and an evaluation of Risks versus benefits. The ODG Low Back Chapter alone has close to 500 Procedure Summary entries, updated monthly. ODG had all of the information that would appear in an evidence table, but not in the format desired by NGC. The ODG format has been proven successful in the marketplace, and ODG was not willing to change its format to meet the needs of the NGC.
During 2016 an impasse was developing over whether NGC would accept ODG under their new criteria. At the same time, NGC said in correspondence to ODG that they were mindful that their model for guidelines may not be a good fit for ODG, because ODG continuously updates a large number of topics and recommendations. Accurately fitting ODG into the NGC format was becoming more of a challenge. In fact, the most recent versions of ODG contained on NGC were dated 2013, which is a problem since states are requiring the current version of ODG. The NGC policy for including guidelines grandfathered under the old criteria, was to leave them in for five years, at which time they would be removed. However, because ODG had submitted updated guidelines, the 2013 versions of ODG were removed. This is in contrast to the ACOEM Practice Guidelines, where the versions on NGC are dated 2011. However, the ACOEM guidelines have not been removed because there are no updates. This is counterproductive that guidelines which are more up-to-date are penalized. Recently Reed Group, owner of the ACOEM Practice Guidelines, has announced they will not allow their updated guidelines to be included in NGC; making the ACOEM Practice Guidelines of NGC as good as a five year old textbook, based on five year old evidence. One wonders if ACOEM Practice Guidelines will remain on NGC now that they’re in the fifth year.
The Guideline Marketplace Goes in a Different Direction
During the process with NGC in 2016, ODG studied the guideline marketplace. The conclusion was that NGC has moved in the direction of hosting academic guidelines, primarily guidelines produced by medical specialty societies who give the guidelines away to support the interests of their members. Their updates tend to be measured in years (3-5), which fits the NGC submission process much better. ODG is considerably more rigorous, with the update process in continuous operation, and was the only independent for-profit guideline left in NGC. Given the changes, ODG decided it is no longer a good fit. All of the leading commercial medical treatment guidelines, including Hearst, Milliman, MCG, Zynx, McKesson, Interqual, Dynamed, UpToDate, and others, now also decline to participate in NGC. Like ODG, these guidelines measure success by proven patient outcomes, benefiting both injured workers and employers. NGC is a nice resource for combing the universe of free PDF-type guidelines from specialty societies and academic institutions, but not the place for dynamic Software as a Service subscription models with continuous review and updates. ODG decided that we appreciate working with them in the past, and wish them success in the future, but without ODG participation.
Ms. Nix’s decision to make the announcement about ODG at a NAMSAP meeting was questionable, since CMS, which controls what Medicare Set Asides get approved, has recommended two guidelines in their Medicare Set Aside instruction manual, Milliman and ODG, both for profit guidelines, and neither of which participating in NGC.
Examples of Guidelines that Remain in NGC
As stated above, the primary guidelines actually used to make healthcare decisions in the U.S. are not included in NGC. These are all published by for profit organizations. You can search the NGC by guideline publisher to identify those publishers that are for profit. Since ODG was delisted, doing this on July 5, 2016, indicates that there is only one for profit guideline on NGC, the SOLUTIONS® wound care algorithm. 1994 (revised 2013 Sep). NGC:010274, published by ConvaTec. Obviously this guideline is highly specialized and could not be used as a general workers’ comp medical treatment guideline. In addition, ConvaTec is a manufacturer of wound care products, so this guideline is essentially a users’ guide to their products, which raises additional questions about the NGC inclusion criteria.
When it comes to workers’ compensation, the newer chapters from the State of Colorado Division of Workers’ Comp have been accepted into NGC using the new criteria. This is ironic since these guidelines have a reputation for being very permissive, but not evidence based. The Colorado guidelines recommend lumbar fusion for degenerative disc disease. This is a major area of abuse in workers’ comp, and it is not recommended by ODG, nor by ACOEM. Colorado also recommends lumbar artificial disk, also found as not recommended in both ODG and ACOEM, due to risks and poor outcomes. Colorado had even recommended IDET, a procedure that almost no one does anymore because it is disproven. The Colorado guidelines are permissive because they were written for an employer-choice state, where there is less of a problem with unnecessary treatments. When other states have adopted Colorado, outcomes have not been good. For example, when Oklahoma had adopted Colorado, costs became among the highest in the U.S., then they passed legislation adopting ODG instead, and outcomes have improved substantially.
ODG Becomes a Data Driven Guideline
ODG does not fit the NGC mold for many reasons. In addition to proprietary formatting and user friendly software, ODG is increasingly reliant on data. ODG is unique as a predictive analytics company using the largest dataset available in workers’ comp. Guidelines can study all of the literature, rank the reviews and/or RCT’s, adhere to the strictest standards of EBM, but most patients, providers, and payers simply want to know what treatments work and get better outcomes versus others. ODG has the highest form of evidence because it contains both the studies/reviews and the actual data. Other guidelines preach to adhere to the strictest standards of EBM, yet there are no studies that demonstrate improved outcomes from using them. ODG is leading the paradigm shift from evidence-based medicine to data-driven medicine. The ODG UR Advisor is an example of this.
There are many studies showing that use of ODG results in successful outcomes. The latest study by Johns Hopkins is published in the ACOEM Journal.
 A New Method of Assessing the Impact of Evidence-Based Medicine on Claim Outcomes
Dan L. Hunt, DO, Jack Tower, MS, Ryan D. Artuso, PhD, Jeffrey A. White, MS,
Craig Bilinski, MS, James Rademacher, BA, Xuguang Tao, MD, PhD,
and Edward J. Bernacki, MD, MPH. JOEM _ Volume 58, Number 5, May 2016”