Laurens V. Ackerman, MD, PhD (left), and Patricia Castel Skarulis, CIO, monitor the network at Rush-Presbyterian to ensure that all application are functioning properly and that network load is properly balanced.

The very ability of a hospital to deliver medical care depends in no small measure on the uninterrupted availability of diagnostic images.

“In a typical hospital, an inpatient undergoes examination by the radiology department 2.3 times per admission, according to surveys I’ve seen, which confirms that radiology is a touch point for the entire hospital, so much so that if electronically distributed and stored images and reports couldn’t be accessed for a day or longer, the hospital would come to a crashing halt,” says Laurens V. Ackerman, MD, PhD, section director of medical informatics in the Department of Diagnostic Radiology and Nuclear Medicine at Rush-Presbyterian-St Luke’s Medical Center in Chicago. “Even brief unavailability of images and reports can be detrimental-if not to the entire hospital, then certainly to the revenue stream of the radiology department.”

A desire to avoid such catastrophes has prompted Rush-Presbyterian-St Luke’s Medical Center to invest in various means of ensuring that its radiology informatics systems stay operational and, in worst-case scenarios, their contents remain easily salvageable.

“Our picture archiving and communications system (PACS) implementation employs various safeguards, including multiple redundancies such as a duplicate jukebox storage system,” says Kiley Rodgers, PACS administrator. “Our mirrored approach gives us greater peace of mind than we would otherwise enjoy.”

FOUR-WAY PROTECTION

Rush-Presbyterian-St Luke’s Medical Center is an academic health center that includes 809-bed Presbyterian-St Luke’s Hospital and 154-bed Johnston R. Bowman Health Center for the Elderly. It was in 1998 that Rush acquired its PACS, which today consists of more than 20 workstations.

“These workstations are either 1,600 x 1,200-dpi screens or 2,000 x 2,500-dpi screens, with the majority of them being four-viewer configurations,” says Ackerman. “In our ICUs and our emergency department, we have larger two-monitor workstations with 1,000 x 1,000-dpi screens.”

Feeding into the PACS are the modalities of CT, MRI, computed radiography (CR), nuclear medicine, and ultrasound (mammography has not yet been linked due to the unavailability of a workstation offering sufficiently high resolution to support that subspecialty). The workstations and modalities each are connected to the PACS via a 100-megabit local area network, which, in turn, is part of an intradepartmental 1-gigabit switched Ethernet network bridging radiology offices in the medical center’s inpatient and outpatient buildings.

“We opted for a switched Ethernet configuration because of its reputation for eliminating the problems associated with transmitting the same information to many different locations,” says Ackerman. “We also chose it because of its ability to increase network speed.”

According to Ackerman, images generated by a modality are routed first to a workstation dedicated to that particular device. From there, the images are passed to a short-term archive-a 1.5-terabyte redundant array of inexpensive disks (RAID) memory. This offers enough capacity to retain each newly acquired image and its comparisons for a span of 30 days. After 8 hours, or whenever the case is signed off at a workstation, the images are sent to two jukeboxes housed in two buildings separated by an expressway. Backup tapes are then created by one jukebox. When it is full, those tapes are removed and stored off-site at a different, secured location.

Recently, Rush has been developing an integration between PACS and the department’s radiology information system (RIS). This integration also includes the hospital information system (HIS).

“We’re creating an electronic medical record system that provides from anywhere in the hospital access to both patient charts and images,” says Patricia Castel Skarulis, vice president and chief information officer. “We call the system R-Care-short for Rush Care, which includes what we’re calling R-Charts and R-PACS. The R-Charts is an integrated longitudinal record. When you look at someone’s record on the computer screen, if there is a diagnostic image associated with it, there will appear on screen an icon that you can click to be automatically taken to the PACS archive. Once there, you can call up the images.”

All of these systems and components are safeguarded by four levels of protection. “Assume our network goes down,” Ackerman proposes. “A radiologist still will be able to read the cases at his workstation because the data up until the instant the network went down resides there at the workstation. This is considered our first level of protection.

“The second level of protection is the jukeboxes. Assume now that one of them goes down. In that event, the other jukebox will automatically take over. A feature of our system is a query mechanism that continuously polls the two jukeboxes to find out whether they are alive. If one does not respond because it has disappeared, the system automatically redirects data to the jukebox that remains alive.”

“Our RAID memory is considered another level of protection. It is housed on its own computer. If its disk goes down, it’s a simple matter of pulling one hard drive and replacing it with another. This can be done without shutting down the entire system.”

The last level of protection is the back-up tapes. “If we lose both our jukeboxes, we have off-site storage of the data that we can bring back and recover,” says Ackerman. “Getting the data back from tape may be a time-consuming process, but the main thing is that the data is there.”

DATA FINGERPRINTING

Or is it? The answer is yes, for at Rush protecting data entails more than making copies of images and records.

“Of concern to us is making sure saved data are valid and corruption-free,” says Ackerman. “The primary means for accomplishing this is the use of a program that performs check-sums of data any time files are moved across the network.”

Rodgers explains that check-sums, in essence, are digital fingerprints that give each data file a unique identity. If a file is corrupted, its fingerprint no longer looks as it did when the data file was first created. This technique of checking for mismatches between digital fingerprints is the basis for antivirus protection software.

“It’s a means of confirming that a data file is unchanged,” he says. “Basically, you send an image across the network and the system creates for it a number that can be verified at the receiving end. If the number that shows up at the receiving end does not match the number that the image was assigned at the beginning, then the system alerts you that a problem exists.”

Ackerman indicates that check-sums are conducted for every image, beginning with the creation of the image at the modality. “Take CR, for instance,” he says. “At the CR workstation, the technologist sees the images and then transmits those images via a DICOM connection. The DICOM itself does a check-sum to guarantee that what goes across the network is good data.”

After that, the data undergo another round of check-sums before being sent into storage, Ackerman adds.

A secondary mechanism for ensuring the validity and integrity of data transfer is network monitoring. Here, Rush uses a special software application designed to frequently sample the status of applications and devices on the network, Skarulis explains.

“Alarms are triggered any time an application is not in proper working order, or if even small errors are beginning to appear on the network,” she says. “The software also allows us to observe the load on every machine. This has proven very helpful in moving applications around to be able to balance the load and thereby undo or, ideally, prevent bottlenecks.

“It’s imperative to be able to know why the network is slowing down and on which part of the network the slowdown is occurring. Previously, we could act based only on hunches about what we thought was happening across the network. Now, however, thanks to this software, we can immediately pinpoint sources of problems, such as a modality that’s gone haywire and is beginning to flood the system with messages.”

Not all problems associated with data integrity are software and hardware related. More often, they are the result of improperly trained staff, Ackerman cautions. “Data integrity sometimes involves basic things, like making sure the right patient name and demographics go with the correct image,” he says. “To address this, we rely on the DICOM Worklist feature, which automatically downloads patient demographics from our HIS and takes it directly to the imaging device. When the image is produced, the demographics appear on the image header without the technologist first having to type them in.

Security, HIPAA-Style

Another aspect of protecting data at Chicago’s Rush-Presbyterian-St Luke’s Medical Center revolves around making sure that electronic images and reports are secure, and that only authorized personnel have access to them.

This is particularly important for Rush because the hospital distributes radiology data to far-flung locations on- and off-campus via the Internet.

“One way we keep data secure on the Web is to use wavelet compression techniques,” says Laurens V. Ackerman, MD, PhD, section director of medical informatics in the Department of Diagnostic Radiology and Nuclear Medicine. “The algorithm behind wavelet compression has millions of variations and thus is very difficult to replicate, which has the effect of making the images protected as if they were encrypted.”

As an extra measure, the Web site to which users connect in order to gain access to PACS images is itself protected by 128-bit secure socket layer connection technology. Additionally, a firewall has been erected to thwart unauthorized access. To gain access through the firewall, users must carry with them a pocket device that serves as a key.

“This device gives each user a personalized numerical code to enter along with his regular password,” says Ackerman. “The numbers of his code are changed every 60 seconds in a pattern predetermined by and in coordination with our main computer. This allows the user a window of just 1 minute in which to log on. Then, once he is cleared through the firewall, the programs and files he is authorized to work with remain available to him until he logs out. The moment he logs out, those programs and files become inaccessible and stay that way until he logs back in with the new personalized numerical code his pocket device gives him.”

Adds Patricia Castel Skarulis, CIO, “This way, people can have access only if they work at Rush and we know who they are. Otherwise, they’re locked out of the system, since we control the key numbers on a minute-by-minute basis.” n
?-Rich Smith

“All of our machines, with the exception of one CT and one MRI, utilize DICOM Worklist. However, having DICOM Worklist is not an absolute guarantee of data integrity. For example, technologists inexperienced with it sometimes make the mistake of manually typing over some of the DICOM Worklist-provided data.”

GARBAGE IN, GARBAGE OUT

Data protection has long been a concern at Rush. Some years back, a story broke in the local press detailing how a group of Chicago public libraries were hurt by badly corrupted data in the backups to their electronic card catalogs.

“There were 17 libraries on the city’s North Shore that were affected,” Ackerman recalls. “These libraries were maintaining their card catalogs on a shared central computer. One day, the computer crashed and they lost their database containing those catalogs. They attempted to recover the lost data by going to the backup tapes. To their horror, they discovered that the data contained on the tapes was garbage. After everyone calmed down, they remembered that they had a second level of backup. So, they went to that one, loaded it in, and, again, garbage.

“Apparently, the library’s information technology people responsible for maintaining the computer were unaware that what they had been routinely backing up onto those tapes was corrupted data files. And afterward, unfortunately, the libraries had no hope of recovering the lost data. The electronically maintained card catalogs were gone. The only way to get out of the mess was to painstakingly reconstruct the database by hand: if they didn’t have an old-fashioned physical card somewhere that they could use to key in or scan off of, they had to go to every shelf and compile the information by pulling out and opening up each book, one at a time, to compile a brand-new catalog of titles, author names, and so forth.

“When we at Rush heard about the problem at the public libraries, it made us wonder just how safe radiology’s electronic images and records were. We didn’t want what happened to the libraries to happen to us. After all, a radiology department revolves around its images and reports.”

Once it was decided that radiology electronic images and reports needed to be protected, the department considered strategies for accomplishing that goal, says Jerry P. Petasnick, MD, chief of radiology.

“The plan we eventually developed included a requirement that we periodically test the safeguards so that, if there were holes in the protections or if elements flatly were not working, we would have opportunities to correct them before a real disaster could occur,” he says. “For example, when we add a new computer, we accumulate some test data on it and then purposely obliterate them so we can attempt to restore them from the file documentation and backed-up tapes. This way, we’ll know whether the system really can be brought back after a disaster.”

Skarulis says creating safeguards also required the involvement of her department, Information Technology (IT).

“The primary thing we did in IT was help design the network infrastructure to make sure it would be able to support the traffic that would be generated by PACS and its mirrored storage system,” she says. “We also assisted radiology from laying the fiber-optics to connecting the machines.”

The Price of Security

Jerry P. Petasnick, MD, chief of radiology, considered data integrity a high priority in PACS planning.

One last piece of the data protection effort at Rush is just being put in place. That, says Ackerman, is the hiring of an administrator to oversee the PACS and RIS databases who reports to the PACS administrator.

“We’ve determined that it is imperative to have someone responsible for resolving the kinds of database problems that can still develop, even in a well-protected system like ours,” Ackerman says. “For example, if technologists and clerical staff are keying in a patient’s name with five different spellings, you’ve got an indication that DICOM Worklist is being bypassed. But who is going to be responsible for making sure that those who are doing the bypassing are identified and provided the necessary remedial instruction so that the problem doesn’t occur again? The answer is probably no one if you don’t have a person assigned to the role of database manager.”

Skarulis suggests that any hospital contemplating the creation of data safeguards similar to those in place at Rush had best be prepared for not-inconsequential extra costs. “It does require a certain investment of capital to protect your data-anywhere from $20,000 for one terabyte of extra RAID on up to $100,000 for duplicate archiving,” says Skarulis. However, she thinks the challenge of convincing CFOs to allocate necessary funding for that purpose is lessening, thanks to requirements of the still-to-take-effect federal Health Insurance Portability and Accountability Act (HIPAA).

“HIPAA can be used as an ally to convince management to authorize capital for more and better backups, since the requirements of HIPAA are such that it compels hospitals to evaluate and do something about their data-security vulnerabilities,” she says. “CFOs know they’re going to have to spend money because of HIPAA, so here is an opportunity to talk to them about the full scope of data security-related needs. The bottom line is that building into your system multiple redundancy is actually cost-effective. It is certainly less costly than having your entire hospital come to a dead stop for hours or days while you try to recover your PACS or RIS or both.”

Rich Smith is a contributing writer for Decisions in Axis Imaging News.