Semantic dynamic systematic reviews of online learning applied to healthcare professionals: A reproducible research project
Taís de Campos Moreira
Lucas de Oliveira Teixeira
Jacson Barros
Joao Ricardo Nickenig Vissoci
Uhana Seifert Guimaraes Suga
Lucas Lentini H. de Oliveira
Ricardo Pietrobon
Although systematic reviews and meta-analysis in education are supposed to capture the best available evidence in the literature, the judgment regarding which articles are rated as either "the best evidence" or the most applicable to different context is left entirely to the author. We present a semantic approach to this problem where the data from individual randomized trials are encoded using a computational ontology. This data set is then processed using a statistical language and presented as a Web application that allows peers and the general public to choose the criteria that they judge to be more appropriate in aggregating study results. Given the dynamic nature of this approach, we expect that time to inclusion of trials in meta-analyses might be reduced, also allowing for personalization of results to contexts that are specific to individual readers.
Online education is a major driving mechanism in the lifelong learning path of healthcare professionals . Despite a large number of systematic reviews and meta-analyses focusing on multiple aspects of online learning applied to healthcare professionals, the speed at which online learning technologies evolve largely defies our ability to create and maintain our reviews up to date . In contrast with this need, the field still largely relies on publications modes that can hardly address the practitioners' needs .
Traditionally, educational practice has been driven by more opinion than evidence . It was only recently that the number of randomized experiments and the overall quality in the reporting of other study designs has allowed for the execution of systematic reviews and meta-analyses . In education applied to healthcare in specific, recent efforts by researchers such as David Cook and have recently led to information that can now guide education policy in ways that were previously not possible . A recent example is
Despite of a significant increase in the number and quality of systematic reviews in education applied to healthcare , the field is not without problems. First, the time lag between the completion of studies and their inclusion in systematic reviews is still very large . If an additional five years are added in order to these systematic reviews to be finally translated into practice guidelines and put into educational practice, we are likely looking at somewhere around 10 years between knowing that something works and then providing society with the benefit of that knowledge. Second, although meta-analysis are conducted with the intent of being reproducible by peers, this assumption can and should be put into question. For example, it has been demonstrated that the exact same set of articles in a meta-analysis can lead to fairly different conclusions depending on which articles are included based on a wide range of quality scales (juni1999hazards). However, in its current format, systematic reviews do not allow researchers, policy makers or any others to attempt alternative assumptions, presenting instead a monolythic, falsely objective view of the literature.
As an alternative to the problems previously outlined, in the past few years semantic technologies now allow for databases to be dynamically created and distributed [@allemang2011semantic]. Specifically, the Linked Open Data movement allows for research articles to not only be published with a number of data points on each article that greatly supersedes the traditional PubMed abstracts, but also to bring in information about these articles that were previously not available. A typical example is the potential to map individual articles to the geographic location where each study was conducted, therefore assisting readers in evaluating whether their results might or not be relevant to them.
In light of the current issues with the literature, the objective of this article is to demonstrate a semantic, dynamic framework for systematic reviews and meta-analyses in online learning applied to healthcare education, where results can be dynamically displayed and stratified.
We considered studies with the following criteria: online education studies in the health area that were included by @du2013web as well as by @cook2010time.
Titles and abstract of the retrieved articles were independently evaluated by 2 reviewers (LLO and USS) to confirm inclusion. Two reviewers (LLO and USS) independently evaluated full-text articles and determined study eligibility. Disagreements were solved by consensus and if disagreement persisted, they sought a third reviewer’s opinion (R.P).
Articles meeting the previously described criteria were ranked according to the GRADE scale for randomized controlled trials (Atkins et al., 2004). The GRADE scale is based on the foundations laid by CONSORT (Balshem et al., 2011; Ioannidis et al., 2004).
Two reviewers (LLO and USS) independently conducted data extraction. Characteristics of the studies collected included year of publication, authors, geographic region of first author, objective, profession, time learning, learning outcome, method of measuring the learning, success and attrition rates.
As a demonstration of the dynamic, semantic meta-analysis framework we used the systematic review by @du2013web as well as by @cook2010time. The systematic review by @du2013web, 2013 included 9 studies: @bloomfield2010effect,@chiu2009effectiveness, @fernandez2011effects, @gerdprasert2010development, @horiuchi2009evaluation, @lu2009effects, @makinen2006teaching, @smeekens2011successful,@mcmullan2011effect.
The systematic review by @cook2010time, 2010 included 20 studies: @papa1999effects, @bell2000self, @grundman2000controlled, @spickard2002learning, @dennis2003problem, @leong2003integrating, @mattheos2004effects, @spickard2004randomised, @cook2005web,@blackmore2006role, @schittek2005computer,, @cook2006impact, @friedl2006multimedia, @friedl2006comparative, @nicholson2006can, @cook2008adapting, @cook2008introducing, @kopp2008fostering, @tunuguntla2008computer, @cook2010time.
The extracted data are provided as a CSV (comma-separated values) under FigShare as well as a Google spreadsheet. Finally, as will be detailed below, the data are made available in a directly queriable format through our SPARQL endpoint, from which a direct analysis can also be performed.
The first step in the Unified Process for ONtology (UPON) is defining an informal use case to support the ontology modeling. The informal use case develop for this study is presented below:
- User go to the Web application and either browses or searches across systematic reviews and RCTs
- For each systematic review or study, the user can stratify results based on a set of fields from the ontology developed
- Results are presented in both a qualitative as well as, when available, a quantitative perspective
- For qualitative results a table and Venn diagram showing overlapping characteristics are displayed
- For quantitative results, OR with 95% CI, forest, funnel and radial plot are displayed
Our ontology engineering process is loosely based on UPON and Agile methodologies. Thus, our approach is use-case driven divided in incremental cycles. Each cycle focus in a specific use-case that relies on a stakeholder need. Briefly, we followed these steps:
- Defined an informal use case, specifically focusing on the end-product our users were expected to obtain from the application
- Outlined the main sections from each articles included in the Cook meta-analysis, including both qualitative and quantitative components
- Outlined a first version of the ontology and instantiated it with data
- Released a qualitative table outlining the main characteristics of each study
- Conducted a meta-analysis using by directly importing the RDF instances into the R language using the metafor package
- Presented the results to an educator and went back to the beginning of the cycle to address any issues
In developing an ontology to represent the terms used in educational meta-analyses, we re-used terms from the Clinical Trials Meta-Analysis Ontology (CTMA). Many terms in educational meta-analyses are similar to terms already defined in Dublin Core thus many terms are reused. The Table 1 presents the Dublin Core terms reused and their role in CTMA.
Table 1. Terms reused from Dublin Core.
| Dublin Core Term | Role in CTMA |
|---|---|
| identifier | PUBMED ID of the study |
| title | Title of the study |
| creator | First author of the study |
The CTMA is represented in Figure X.
The ontology were then combined with the RDF Data Cube to enrich the semantic representation of the resulting statistical data. The following example is a study represented using CTMA and RDF Data Cube:
eg:10536581 a qb:Observation, ctma:Study;
qb:dataSet eg:datasetDynamicMetaAnalysis ;
dct:identifier 10536581;
dct:title "The effects of immediate online feedback upon diagnostic performance";
dct:subject "";
dct:date "1999-01-01"^^xsd:date;
dct:creator "Frank J. Papa";
dct:spatial [
rdfs:label "Fort Worth, United States";
geo:lat 32.749542;
geo:long -97.36903;
rdf:type dct:Location;
];
ctma:objective "Test two hypotheses related to the validity and effectiveness of the CAI. Measure the additional time-to-task required to transform a testing session into a CAI session. Use any observed DDX improvements as the basis for making inferences regarding the possible transfer mechanisms (learning) underlying performance";
dct:audience "Medical Students";
dct:educationalLevel "Graduate";
ctma:followUpDays "";
ctma:arm 3;
ctma:sampleSizeIntervention 52;
ctma:timeLearningAvgIntervention 41;
ctma:timeLearningSdIntervention "NA";
ctma:sampleSizeControl 56;
ctma:timeLearningAvgControl 32;
ctma:timeLearningSdControl "NA";
ctma:armInterventionDescription "CAI Program with feedback";
ctma:armControlDescription "CAI Program. If the diagnosis was incorrect, the CAI program informed the Pre/Trt subjects of the correct diagnosis";
ctma:successIntervention "Validity of Case Vignettes as a Measure of DDX Ability - Post/Con group (mean=23.88, SD=4.23, n=33) signincantly outperformed the Pre/Con group (mean=21.66, SD=4.49, n=56), t=2.30, p=.02. Effectiveness of Feedback - Pre/Trt: mean=26.35, SD=4.58; Pte/Con: mean=21.66, SD=4.49; Post/Con: mean=23.88, SD=4.23" ;
ctma:successControl "";
ctma:firstOutcome [
ctma:outcomeDescription "Validity of Case Vignettes as a Measure of DDX Ability";
ctma:methodMeasureOutcome "";
rdf:type ctma:Outcome;
];
ctma:secondOutcome [
ctma:outcomeDescription "Effectiveness of Feedback";
ctma:methodMeasureOutcome "";
rdf:type ctma:Outcome;
];
ctma:thirdOutcome [
ctma:outcomeDescription "Efficiency of Feedback";
ctma:methodMeasureOutcome "CAI program recorded the diagnoses and the amount of time (in seconds) taken per subject on each vignette";
rdf:type ctma:Outcome;
];
ctma:attritionRatioIntervention 0;
ctma:attritionRatioControl 0;
ctma:gradeSequenceGeneration [
rdf:value 0;
rdf:type ctma:Grade;
];
ctma:gradeAllocationConcealment [
rdf:value 0;
rdf:type ctma:Grade;
];
ctma:gradeBlinding [
rdf:value 0;
rdf:type ctma:Grade;
];
ctma:gradeLosses [
rdf:value 0;
rdf:type ctma:Grade;
];
ctma:gradeIntentionTreat [
rdf:value 0;
rdf:type ctma:Grade;
] .
Finally, CTMA is licensed under Apache License, Version 2.0 and it is available under our SPARQL Endpoint.
The Citation Typing Ontology (CiTO) is a ontology for the characterization of citations. It allows the semantic representation of citations of articles in general.
We have also combined CTMA with CiTO. This was possible because all studies selected for the demonstration of our framework were already published. The following example represents the additions needed for CiTO.
Finally, in order to demonstrate the practical applicability of CiTO, we generate the references for this article using it.
All analyses were conducted using the R language (@R2013). Specifically, we accessed the data through the previously created SPARQL endpoint using the rrdf package(@rrdf2013) package for rdf manipulation. The data were then converted into a dataframe, which is the equivalent of a flat table. Subsequently, we used the metafor package for meta-analysis, forest, funnel and radial plot and the QCA and Venn diagram package for qualitative analysis.
The Web application were develop using Shiny framework and it is available in our server. Shiny is a web application framework for R.
In order to make this study reproducible, we followed a series of published recommendations. . These included
sparql endpoint sparql scripts pra acessar endpoint - exemplos de queries
videos, url site
Figure 1: Venn diagram
Figure 2: Forest plot
Figure 3: Funnel plot
Figure 4: Radial plot
csv google sheets
github Joao's paper figshare
comparison with OCRe
The Ontology of Clinical Research (OCRe) is rather complex and, apparently, not supported anymore. The Clinical Case Meta-Analyses Ontology (CTMA) develop is a simple ontology to represent data for clinical trials meta-analysis.
Furthermore, we have found that there is a lack of solutions to represent clinical trials data as linked data.
compliance consort included into ontology cito reason for citation for citing and cited articles