In order to validate the reproducibility of results given by the CT Based Left Atrial Appendage Case Plan Protocol (Wang et al. 2016), anonymous and unbiased high resolution CT data was obtained from all patients who had already undergone the WATCHMAN implantation procedure at University of California, San Diego. For each set of patient data, operators uploaded the data into DICOM software for 3D medical images and proceeded to follow the instructions outlined in the LAA Case Plan Protocol. Additionally, operators were not informed of the results from the actual implantation procedure, thereby ensuring that the protocol results from each case was not influenced by the success or failure of the actual implantation procedure.
4 different operators performed the Case Plan Protocol on the 4 sets of patient data for 3 run-throughs for a total of 12 run-throughs per patient. Each operator followed the protocol, recording the values of all measurements made for the run-through. The operators also recorded the difficulties and confusion they experienced, subsequent step in the protocol, and the cause of the difficulties and confusion. After completing the Case Plan Protocol, the operator determined whether the WATCHMAN device could be successfully implanted, and if so, which device size would be optimal.
After the protocol was utilized on all available patient data, the conclusions made by the operators were compared to the actual implantation results. It was expected that many of the case plans for patients would clearly match the actual implantation results in determining whether a device could be implanted. It was also expected that the size of successfully implanted devices determined by the operator should match the size of the actual device implanted in the patient. All discrepancies between operator predictions and actual implantation results were carefully analyzed and discussed with physicians to determine their causes and significance in order to identity where difficulties interfering with ease of operation may be occurring and ensure that time spent planning actual procedures could remain outside of the operating room.
In addition to comparing operator results to patient results, statistical analysis measuring precision was performed on all measurement data to ensure that there was consistency in measurements for each patient. The analysis included taking measured data at specific steps in the protocol for one run through the case plan and comparing it to the measured data at the same steps for the other runs by all other operators. With reproducibility as the most important factor, measurements should not have significant variation among operators and among multiple runs. Statistical aberrations in measurements were carefully examined and discussed with physicians to determine the cause of the aberration, its significance, and how it could have been prevented. See the Case Plan Work Flow for testing the reproducibility of results given by the CT Based Left Atrial Appendage Case Plan Protocol (Wang et al. 2016).
4 different operators performed the Case Plan Protocol on the 4 sets of patient data for 3 run-throughs for a total of 12 run-throughs per patient. Each operator followed the protocol, recording the values of all measurements made for the run-through. The operators also recorded the difficulties and confusion they experienced, subsequent step in the protocol, and the cause of the difficulties and confusion. After completing the Case Plan Protocol, the operator determined whether the WATCHMAN device could be successfully implanted, and if so, which device size would be optimal.
After the protocol was utilized on all available patient data, the conclusions made by the operators were compared to the actual implantation results. It was expected that many of the case plans for patients would clearly match the actual implantation results in determining whether a device could be implanted. It was also expected that the size of successfully implanted devices determined by the operator should match the size of the actual device implanted in the patient. All discrepancies between operator predictions and actual implantation results were carefully analyzed and discussed with physicians to determine their causes and significance in order to identity where difficulties interfering with ease of operation may be occurring and ensure that time spent planning actual procedures could remain outside of the operating room.
In addition to comparing operator results to patient results, statistical analysis measuring precision was performed on all measurement data to ensure that there was consistency in measurements for each patient. The analysis included taking measured data at specific steps in the protocol for one run through the case plan and comparing it to the measured data at the same steps for the other runs by all other operators. With reproducibility as the most important factor, measurements should not have significant variation among operators and among multiple runs. Statistical aberrations in measurements were carefully examined and discussed with physicians to determine the cause of the aberration, its significance, and how it could have been prevented. See the Case Plan Work Flow for testing the reproducibility of results given by the CT Based Left Atrial Appendage Case Plan Protocol (Wang et al. 2016).