Imaging centers create their own interface solutions
It has been nearly a decade since the US Food and Drug Administration (FDA) approved the use of computer-aided detection (CAD) software to improve mammogram interpretation. In that time, the technology has grown in popularity, with roughly a quarter of all imaging facilities using it today.
But a study published in the April 5 issue of The New England Journal of Medicine associated CAD with reduced accuracy of interpretation of screening mammograms and an increased rate of biopsy, without a clear improvement in the detection of invasive breast cancer.
Upon publication, the work created a wealth of controversy?and conversation?about not only the accuracy of CAD, but also the ideal ways to implement it and measure the results.
A Closer Look
Taking one of the most comprehensive looks at CAD for mammography since its acceptance by the industry, the study involved more than 429,000 mammograms and 2,351 cases of cancer that were detected at 43 Breast Cancer Surveillance Consortium (BCSC) facilities in three states.
The BCSC is a collaborative network of seven mammography registries, along with connections to tumor and pathology registries, that serves as a “research resource for studies designed to assess the delivery and quality of breast cancer screening and related patient outcomes in the United States.”
Starting in 1998, Joshua J. Fenton, MD, MPH, from the University of California, Davis, in Sacramento, and colleagues spent 5 years comparing the performance of individual radiologists before and after the use of CAD at seven of the 43 facilities that began using the technology. Facilities that did not implement CAD were used as controls.
Fenton’s team found that after implementation, the positive predictive value decreased from 4.1% to 3.2% and the rate of biopsy increased by 19.7%. The authors noted the increase in sensitivity?from 80.4% to 84.0%?was not statistically significant, nor was the change in the cancer-detection rate, which rose by a trivial margin: from 4.15 cases per 1,000 screening mammograms before implementation to 4.20 cases afterwards. The cancer-detection rate accounted for invasive breast cancers and ductal carcinomas in situ (DCIS). Overall data showed that the use of CAD was associated with significantly lower overall accuracy.
“The results from using CAD aren’t what you would hope for, based on the prospective studies that have been done,” said Fenton, who works in the Department of Family and Community Medicine, UC Davis Health System. “Our results raise the question of whether CAD is being used differently in practice than the way it is being used in those prospective studies. Radiologists in the community should look closely at those protocols and try to replicate them in their practice.”
Judy C. Dean, MD, who operates a private mammography center in Santa Barbara, Calif, believes one aspect of CAD’s performance is the difference between the rigid procedures enforced during research and those employed in everyday use. She also says that the radiologist’s approach to the technology contributes to the ultimate success or failure of CAD within a practice.
“A lot of radiologists view CAD only as an additional procedure, and they don’t really believe in its value. If that is the case, you are probably not going to use it properly,” Dean said. “If you are actually following the kind of protocol required in a study where you assess the case and write down whether you are bringing a patient back or not before turning on CAD, it won’t take 2 days before the CAD marks something you didn’t see. You are not a machine.”
Determining What To Measure
Part of the debate surrounding the paper Fenton coauthored is some radiologists’ perception that it discounted the increased DCIS findings by CAD, focusing instead on the identification of invasive cancers. “They decided that very early cancers were not important, but we believe if you find cancers early, it won’t subsequently lead to an invasive cancer,” said Stamatia Destounis, MD, radiologist at the Elizabeth Wende Breast Clinic, Rochester, NY. “So when I find a very early cancer, even DCIS, I feel good about it because we are saving our patient from developing an invasive cancer somewhere down the line. I’m saving years of her life?and they decided to ignore that.”
To determine for themselves the impact of CAD, the breast imagers at the Elizabeth Wende Breast Clinic began an in-house study in 1999. They reviewed a subset of cancers diagnosed at the clinic in previous years and quickly realized that CAD was helping find cancers and leading to earlier detection. As a result, the five radiologists in the facility began using CAD in 2000?but they also retained the practice of having two radiologists look at each exam.
“We still double-read all of our mammos, because when we reviewed the data from our own study of 20,000 patients, we realized that CAD actually helped us find an additional 7% of cancers. It was still better than two doctors reading,” said Destounis, who is also an assistant clinical professor at the University of Rochester Medical Center. “We actually use CAD as a triple look, and we have been happy with it because it pinpoints areas that are asymmetric or very subtle microcalcifications.”
According to Dean, the study does not focus enough on the benefits of discovering early cancers.
“[The study] presents only a very limited point of view. They looked retrospectively at the so-called ‘costs’ of using CAD, which is more recalls and, in their situation, more biopsies. But they did not look at what you get for that?you not only detect more cancers, but you also detect smaller cancers, and that is very significant,” she said, noting that not all imagers experience a greater number of biopsies when using CAD. “They said they saw a slight increase in the number of cancer detections, but they didn’t tell us anything about size and stage, so they really haven’t told the whole story. Because if you are able to push back size and stage, it may be worth doing more recalls?and in fact, it probably is.”
Dean coauthored a study about the effectiveness of CAD in the July 2006 issue of the American Journal of Roentgenology that found that the technology is beneficial in both noninvasive and invasive cancers.
“If you look specifically at nonpalpable, invasive breast cancers, of the cancers that are detected because of using CAD, they are less than half the size,” said Dean, whose data showed that the median size of nonpalpable invasive cancers for CAD-aided diagnoses was 5 mm, as compared with 10.5 mm for cases detected by the radiologist regardless of CAD assistance. “If we find the cancer so early that the women may not even require chemotherapy, that is a huge savings not only in terms of dollars and cost to the health care system, but also to the patient,” she said.
For Fenton, the published results were not intended to minimize the importance of DCIS, but rather to measure CAD’s ability to find the cancers that have the most immediate impact on a patient’s mortality.
“In my view, we should focus on the size and stage of the invasive tumors, because the mortality benefit of mammography probably derives from stage-shifting of invasive breast cancers,” he said. “Our findings raise the question of whether this technology is helping to find cancers we really don’t want to miss; when you look at all of the data together, it’s not clear that the routine use of CAD technology is associated with higher rates of detecting the most serious forms of breast cancer. Ideally, we would look at this technology in the context of a larger study?with a larger number of radiologists and a larger number of women?so we would have the statistical power to really clarify what types of cancers CAD is finding.”
Real-World Experience
The editorial accompanying the study addressed the impact of a radiologist’s experience level with CAD. Ferris M. Hall, MD, of the Department of Radiology, Beth Israel Deaconess Medical Center, Boston, wrote that “one possible flaw … was the failure to assess the time it takes to adjust to computer-aided detection.”
“That is absolutely true,” Dean said. “CAD marks every calcification and, when you first get it, you don’t really understand what those marks mean until you work some of them up.”
The average time of CAD use at facilities in the study was 18 months.
For Ken Ferry, president and CEO of iCAD, Nashua, NH, the study also sent the message that new CAD users need more than an initial training session conducted at the time of implementation. iCAD is a provider of mammography solutions, specifically the SecondLook product line, which accommodates both high-performance film and digital systems.
|
| Digital mammogram showing SecondLook Digital? CAD marks. |
“If it sounded one major action item for us, it is that we should probably be making more of an investment in training,” he said. “And to do it in a manner where doctors can log on to our Web site, go through some exams, and then immediately find out if they were correct or not.”
Experience with the CAD product is important, and to some, the radiologist’s level of experience reading mammograms and breast images is equally valuable.
“Experienced radiologists, because they see very similar images over and over every day, actually discount a lot of the CAD marks, because they recognize it as a lymph node or a fibroadenoma, for example,” Destounis said. “Younger radiologists, however, are more apt to be attentive to all the marks, because they don’t know which are probably false marks, and which ones require them to recall the patient.”
Fenton contends that existing evidence does not demonstrate that the impact of CAD differs by radiologist experience.
“In our own analyses, we found that CAD reduced accuracy across all levels of radiologist experience or volume,” he said. “[In our study, more than] 70% of the radiologists at facilities using CAD had more than 10 years of experience in mammography, and we adjusted for experience and reading volume in our analyses, so the results cannot be attributed to the inexperience of the radiologists at facilities using CAD.”
Regardless of the level of combined experience and technical skill in a user, it is likely that any product increasing the level of detection is by its very nature going to identify items that are not cancers.
“Even with the best algorithm and the best radiologist, the goal, of course, is to find a substantially higher number of cancers and to reduce as a percentage the biopsies that would be negative,” Ferry said. “It is never going to go to zero. You are flagging more things, but what you want to see is that the number of things you flag are accurately cancers, and that number should be far greater than the absolute increase in unnecessary biopsies.”
Old Versus New
The specific technology used in this study raises concerns for those who manufacture CAD algorithms, which today are primarily delivered through software-based products tied to digital mammography systems. The time period selected for the study?from 1998 to 2002?represents the very first CAD solutions. Earlier versions of CAD made more marks, many of which turned out to be nothing.
“The technology used [in the study] was of our former competitor, and it was the very first version of their product,” Ferry said. “We are talking about a product that puts out a new release of the software algorithm on an annual basis, which always improves the performance. The technology discussed in the study is possibly as much as five generations old. To me, that’s like trying to compare the performance of a laptop purchased today with one that was made in 1998.”
Clinicians in the study were also using film-based mammography that was converted to a digital format before being processed by CAD. This presents another opportunity for less-than-stellar performance, according to Ferry.
“A fairly high percentage of women will not perform as well [with film-based mammography equipment] as with digital mammography,” he added. “If you did the study today, combining a much higher-performing product and a radiologist who now has, at a minimum, 5 more years of experience using the technology, I think the results would be substantially different.”
While there is no argument that the newer technology makes fewer marks, Fenton believes it comes at the cost of sensitivity. He also thinks the performance of CAD on film-based mammography could be cause for concern.
“There is very limited data on whether CAD impacts performance differently on film versus digital machines, but a recent published study suggests similar performance impacts,” Fenton said. “If indeed CAD doesn’t work as well with film mammography, this should give us further pause, because most facilities are still using film mammography.”
Moving Forward
Even with the controversy and disagreement surrounding the study and its processes, there is some common ground to be found in the benefits drawn from studying CAD technology.
“We just don’t have that much data on this technology, and that is not data that is generalizable to actual practice,” Fenton said. “There is reason to be concerned that the way radiologists are using the technology in real practice may not be parallel to what was done in some of those smaller, prospective trials. I think our challenge now is to square the information from prospective studies with the data that we published.”
The dialogue created as a result of the published findings is part of the process of creating a better product and, ultimately, better patient care.
“Digital mammography equipment and CAD are the standard in breast cancer screening and that is not going to change,” Ferry said. “It is solely a function of how do we properly put a journal article like this in the proper context going forward. It is rarely, if at all, disputed that CAD finds more cancers earlier and, ultimately, that is what we are trying to do.”
Fenton disagrees. “Cancer detection is not the goal of mammography: reduction of breast cancer mortality is,” he said. “If CAD finds a lot of DCIS without boosting invasive breast cancer detection, that may or may not be a good thing.”
Still, Ferry encourages and welcomes more studies.
“Science is not perfect, and it is always something that we relentlessly pursue to make it better,” he said. “It will be better a year from now than it is today, and the more studies that are done that identify things that we can improve on are to everybody’s benefit.”
Dana Hinesly is a contributing writer for Medical Imaging. For more information, contact .
