The last two articles looked at the utility of the different FDA databases (Investigating the Performance of FDA Databases in Terms of Reporting Medical Device Problems 1 - The utility of the FDA databases); and the utility of the MAUDE database (Investigating the Performance of FDA Databases in Terms of Reporting Medical Device Problems 2 - The utility of MAUDE database).
In this article, we deeply investigate the comprehensiveness and quality content in the report sections of ‘event description’ and ‘manufacturer narrative’, in the MAUDE database, by studying three cases; namely, haemodialysis (1633 reports), defibrillator (1222 reports), and nebuliser (1210 reports). The reasons for choosing these case studies and the six different types of FDA reporting forms are explained in the previous article. The Inductive Content Analysis and Mean and Standard Deviation are used to analyse the report content data according to the 4065 reports in total. Data validation is achieved through triangulation and member checking. Data was gathered from different sources, namely, previous literature, direct analysis by the author, and direct analysis and checking with different participants (researchers, designers, engineers, and a graduate tutor), providing for triangulation. Member checking was used to check the codings and coding groups with the participants involved. This allows the participants to critically analyse the results and comment on them. The results were named codings.
1. Validated Coding Groups in the ‘Event Description’ and the ‘Manufacturer Narrative’
The contents in the ‘event description’ and the ‘manufacturer narrative’ were analysed using the content analysis method. Tables 1 and 2 show the results of the validated coding groups for the ‘event description’ (ten validated coding groups) and the ‘manufacturer narrative’ (eight validated coding groups). The individuals came to similar or identical conclusions about the groups of information the reports contained. Based on these groups, two aspects of all reports for the three medical devices were analysed: one aspect focused on identifying the number of distribution of groups addressed in the reports; the other focused on identifying the frequency of each group.
These groups were decided after analysing the contents of many reports which provided the most complete and comprehensive information in the ‘event description’ and the ‘manufacturer narrative’. Thus, there are a few repeated groups in the ‘event description’ and the ‘manufacturer narrative’. How much information was contained in a report was determined quantitatively by how many of the above groups they covered.
2. What is the distribution groups?
By counting the number of groups in the ‘event description’ and the ‘manufacturer narrative’ for all the three medical devices reports (4065 reports in total) with Excel 2014, it is possible to get an overall picture of the typical level of detail contained in the reports. The data was summarised by grouping the reports into different device problem categories, these categories where then further separated for each of the three medical devices. Covering as many groups as possible does not necessarily mean a report is good quality, but having complete information is one of the essential factors for a report to be good quality and comprehensive. Figures 1, 2, and 3 show the frequency distribution of the groups contained in the ‘event description’ section for haemodialysis (1633 reports), defibrillator (1222 reports), and nebuliser (1210 reports) between 2004 to 2014.
For Figure 1, we found that 11% of haemodialysis reports contained no useful information and 55% of reports contained only one or two groups. Whereas, only 2% of reports contained over seven groups. The mean number of groups was 2.5±1.7. For Figure 2, 51% of reports contained one or two groups for the defibrillator reports, 40% of reports contained three or four groups, but no reports contained more than six groups. The mean of groups for it was 2.8±1.5. For Figure 3, 1% of reports contained no useful information of the nebuliser reports, 77% of reports contained only one or two groups, and only 3% of reports contained five, six or seven groups. The mean was 2.0±1.3. Due to the different mean, the dispersion coefficient was used to identify the variation in the number of groups. Though deviation divided mean, we can see, the dispersion coefficient of the haemodialysis reports was 0.68, the dispersion coefficient of the defibrillator reports was 0.54, and the dispersion coefficient of the nebuliser reports was 0.65. From the analysis, we can see that defibrillator reports had the greatest amount of information on average and haemodialysis reports had the greatest variation in the number of groups (or quality of the reports), but very few contain comprehensive information.
Figures 4, 5, and 6 show the frequency distribution of the number of groups in the ‘manufacturer narrative’ for haemodialysis (1633 reports), defibrillator (1222 reports), and nebuliser (1210 reports) between 2004 to 2014.
From Figure 4, we can see that 15% of reports contained no useful information of the haemodialysis reports, 59% of reports contained one or two groups, but only 1% of reports contained seven or eight groups. The mean number of groups for it was 1.9±1.6. Figure 5, 8% reports contained no useful information, 32% of reports contained one or two groups, 35% of reports contained three or four groups, 26% of reports contained five or six groups, and only 1% of reports contained seven or eight groups. The mean number of groups for it was 2.8±1.5. For Figure 6 of the nebuliser reports, 23% of reports contained no useful information, 42% of reports contained one or two groups, but 30% of reports contained over five groups. The mean number of groups for it was 2.4±2.2. We can see that the dispersion coefficient of the haemodialysis reports was 0.84, the dispersion coefficient of the defibrillator reports was 0.53, and the dispersion coefficient of the nebuliser reports was 0.92. From the analysis, we can see that, again, reports on defibrillators contained more information than for the other two, but this time reports on nebulisers had the greatest variation in quality.
What is more, for the manufacturer narrative of defibrillator and nebuliser reports, there appear to be too distinct clusters, one of lower quality (fewer groups) and one of higher quality (more groups); however, this could be for an insufficient number of analysed reports and would need further analysis to test for statistical significance. This could be done with p-test; however, the results would be invalid because it is not appropriate to fit p-test to a pattern after acquiring data, and would acquire a new set of data. This is because it is always possible to find statistical significance in random fluctuations, so a hypothesis made from a pattern recognised in data after it has been acquired is very likely to come out as statistically significant whether it was the real effect or just fluctuations. Therefore, the hypothesis must be decided beforehand and data acquired blindly, then a test for statistical significance may be applied.
Compared with the data in the ‘event description’ and the ‘manufacturer narrative’ of these three medical devices, the amount of information contained in the ‘manufacturer narrative’ was greater than that in the ‘event description’; however, it is difficult to make a direct comparison as each section was judge on slightly different criteria for the groups.
Following the investigation of the distribution of the total number of the groups in the ‘event description’ and the ‘manufacturer narrative’, an investigation into the frequency of each validated coding group was carried out in Section 3.
3. How often does each validated coding group occur?
This study investigated which groups were covered more frequently. Firstly the report contents were highlighted if they matched one of the ‘event description’ groups (ten groups) or one of the ‘manufacturer narrative’ groups (eight groups), this was done for all the reports for the three medical devices reports (4065 reports in total). Each group in the ‘event description’ and the ‘manufacturer narrative’ was calculated with Excel 2014 from the analysis of the report contents. Figures 7 and 8 show the frequency of each individual group in the ‘event description’ and the ‘manufacturer narrative’, respectively. The validated coding groups are in reference to the list in Section 1 (see Tables 1 and 2).
Figure 7 shows, for the ‘event description’, most reports covered ED2 (A patient’s side effect(s)), ED3 (Treatment for a patient’s side effect(s)), ED4 (A patient’s health status after an event), and ED5 (A patient’s medical records/a patient’s health history).
Figure 8 shows the frequency of distribution in the ‘manufacturer narrative’, where it can be seen the majority of groups (except MN1 and MN2) that were mentioned reasonably frequently, these where MN3 (A device’s performance during an event), MN4 (A device’s status after an event), MN5 (Details of a device reporting process), MN 6 (A device’s history of use), MN7 (History of similar events with a same brand of device), and MN8 (A device’s evaluation after an event and results).
Compared with the ‘event description’ and the ‘manufacturer narrative’ we can see, the ‘event description’ focuses more on a patient’s health situation and what happened in the event, whereas the ‘manufacturer narrative’ focuses more on the device (how the device was used and device evaluation). This is not surprising given the names of the sections.
An additional analysis was carried out to investigate the ‘event types’ for each report. There were five types of the event recorded, namely, ‘malfunction’, ‘injury’, ‘death’, ‘other’, and ‘unknown’. Figure 9 shows, for all of three medical devices, ‘malfunction’ was the most common event type. ‘Injury’ events also frequently occurred during users’ interaction with medical devices. The frequency of ‘death’ was similar to the ‘other’ event type, and only rare reports do not offer any information about the ‘event types’.
4. Observations on the structure of the MAUDE report contents
To better understand the structural issues related to the report contents, identical coding groups have been highlighted in the same colours for the ‘event description’ and the ‘manufacturer narrative’ (Figure 10, which is constructed Tables 1 and 2).
The two key structural problems were as follows: (1) a substantial amount of information contained in the ‘event description’ was repeated in the ‘manufacturer narrative’ with identical words or similar wording but with the same meanings; and (2) a number of reports lacked clarity and logical flow in the expression of the groups. In the previous article, figure 1 explained that the importers and device user facilities need to send the ‘injury’ and the ‘malfunction’ reports to manufacturers, and manufacturers then send the reports to the FDA. So the causes of these issues might be that the report contents were from different reporters. The manufacturers may copy a part of the reports from importers, which may lead to the repeated information and a lack of logical flow.
The identification of the categories of the groups in the ‘event description’ and the ‘manufacturer narrative’ enhanced understanding of the structure of the report contents.
Through the analysis of the categories, the validated coding groups were categorised to act as a guide for reporters, thereby improving the structure of the report contents. Identical coding groups in Tables 1 and 2 were omitted and the rest were reclassified. Figure 11 shows the structure of the report contents in the ‘event description’ focused on the patient.
Figure 12 shows the structure of the report contents in the ‘manufacturer narrative’ focused on the device and its relevant history. It consists of two main parts: ‘device’ and ‘history’. The relevant groups were branched into these two main parts.
5 Discussion
The quality of the ‘event description’ and the ‘manufacturer narrative’ content was studied from two perspectives: (1) by counting the number of validated coding groups in the ‘event description’ and the ‘manufacturer narrative’ for all the medical device reports, so as to give an overall picture of the typical details contained in the reports; and (2) by investigating which validated coding groups were covered more frequently. Although covering as many validated coding groups as possible does not necessarily mean the report is good quality, having complete information is essential. The more frequently covered the groups can provide insight into the main goals of the ‘event description’ and the ‘manufacturer narrative’.
Better structured report contents, in some certain sense, helps reporters complete reporting forms more easily and comprehensively. This makes the report easier to read and understand. In this study, the structure of the report contents was analysed to understand whether the structure was clear. From the analysis, it was found the structure lacked clarity and logical flow, which may be caused by many reasons, such as poor database design, and data from multiple sources. However, due to limitations, such as time constraints, the causes of the issues were not investigated in this study. What is more, there was limited information about target users, as has already been discussed at the beginning of the first article, and due to the severe difficulty in investigating the target users, this study does not focus directly on tailoring the databases to distinct groups. In addition, the MAUDE database is a complicated and large system, and the reasons causing these problems are difficult to uncover. It was assessed that these issues might be caused by the target users, limited data, missing data, or coming from different sources.