John Chaffin, M.D. Chairman of Cardiovascular Surgery and Transplant Surgeon

Introduction

INTEGRIS Heart Hospital (Baptist Medical Center) has had great coronary artery bypass results since 1990. We have used risk adjusted survival reports to help document improvements in processes and technology. Even with this sophisticated method, there are still important limitations. Just as in coronary stent procedures, risk adjustment models can be thrown off by 'referral bias' or a selective increase in ultra-high risk patients or by lack of power, which is an insufficient number of cases to arrive at a definite and stable survival estimate. Generally clinical registries are designed to partially overcome referral bias and can bar reporting of underpowered estimates. An academic partner such as the Duke Clinical Research Institute was very helpful in the design and technical application of risk adjustment models so as to avoid reporting invalid results. Commercial Internet based reporting systems rarely disclose their methods and frequently report underpowered outcomes.

How do you know your hospital is exceeding expectations?

Consumers need a reliable benchmark for making a comparison with other institutions. There are a number of public and private ratings systems to choose from such as the US News and World Report system, HealthGrades and more recently, Medicare reports. Obviously the quality of the comparison itself and the basis for the comparison is also very important. For instance, the US News and World Report emphasizes academic work under the assumption that doing research on a particular problem increases institutional competence and makes newer and better technology available to the patient. Numerous studies have shown that research capabilities do not necessarily transfer to clinical competence. The HealthGrades reporting system relies on an administrative database. These systems use Medicare billing data as their source and will therefore have data that is easy to obtain. The data covers all specialties and not just cardiology, and is available from all hospitals. For multiple reasons discussed in article number 6, it lacks suitable precision for detailed quality comparisons among providers. Medicare also has created a system of public reporting. It focuses on process measures more than outcomes and looks to see if checklists for supporting treatments were completed. However, improving process measures results in only a small increase in absolute survival rates. Finally, there are private systems usually developed by specialty organizations like the Society for Thoracic Surgery and the National Cardiology Database. These are both clinically rich data registries but are not audited so the results are questionable. Most importantly they do not permit public reporting. In summary, the current benchmarks for consumers lack the necessary precision or the breadth of coverage to assist in truly informed decision making. INTEGRIS Heart Hospital promises to provide a mixture of process measures and risk adjusted outcomes that have enough breadth and depth to assure the public we are a quality driven institution focused on good outcomes.

What happens next across institutions and whether or not everyone will be self reporting clinical outcomes remains to be seen. The current commercial public reporting companies, despite using administrative databases, have done a good job of raising public and institutional awareness of the need for public reporting. When questioned about accuracy they argue that weak data is better than no data at all. It is conceded that billing data registries lack many of the important patient attributes like co-morbidities and basic heart function. These are vital data to report a full risk adjusted outcome. Without them, the final reports are frequently either falsely high or low. As Dr. Pieto, the clinical epidemiologist, famously argued, weak data is more dangerous than no data because it creates the illusion of knowledge. For these reasons INTEGRIS has chosen to use the clinical registry method to measure quality. We hope that others will follow so the consumer can have better information for their choices.

In using clinical registries, the crucial step is the risk adjustment formula. When designed properly they would contain most of the information that determines outcomes. This requires developing populations of patients with similar risk backgrounds and then using detailed clinical information to create a risk score. As an example, the risk score would rise as the age increased, or if the patient had serious kidney or lung disease in addition to heart disease. Current risk scores using carefully validated data may explain up to 80 - 85 percent of the variation in outcomes. Thus, they are generally reliable but even these systems have weaknesses. For example, these risk scores are based on the average amount of risk a patient's co-morbidity (like lung disease or heart failure) might cause. However, neither patients nor their associated illnesses are all always 'average.' Some patients have very bad complications that will require back up support from specialists like lung and kidney doctors. These patients generally will not be cared for in smaller programs but will be referred to tertiary referral centers with broad subspecialty backup like the INTEGRIS Heart Hospital. Since these patients don't have average risk, the risk adjustment model will not fully level the playing field for these types of patients. In future reports we will be reporting how many patients are referred from different programs and reporting results with and without these ultra high risk patients.

This discussion about risk adjustment and expected vs. observed outcomes is a very complex discussion. It might be less confusing for you if you access the videotape of how to build a risk model in assessing patient outcome.

Conclusion

Quality measurement is an evolving technology that has great relevance to the consumer. Well-done clinical registries should be the gold standard for institutional comparisons. Even the clinically rich, carefully audited databases have more trouble in measuring ultra high-risk patients. They do not quite level the playing field. Even large numbers of patients do not undo the systematic reductions in survival rates caused by referral bias. In the future we will report the impact these patients have on our results. Finally we will report risk adjusted outcomes only when we have adequate numbers of patients. Understanding outcomes reporting is challenging and has limits but is indispensable to our efforts to improve quality.

For more information on how to read and understand outcomes click here to view our 3 minute Outcomes Animation.

Back to Introduction...