Although performance of rapid immunochromatographic tests (RITs) for dengue virus (DENV) serotypes 1, 2 and 3 is relatively settled, evidence on accuracy of RITs for DENV-4 are based on studies with small sample sizes and with discrepant results.
ObjectivesTo assess accuracy and inter-observer agreement of RITs targeting dengue nonstructural protein-1 (NS1) antigen - Dengue NS1-Bioeasy™, Dengue NS1 Ag Strip-Bio-Rad™, IVB Dengue Ag NS1-Orangelife™ and Dengue NS1-K130-Bioclin™ in DENV-4 samples.
MethodsStudy sample (n = 324) included adults presenting at an emergency unit in Rio de Janeiro, Brazil, with fever of ≤72 h and two or more dengue symptoms. A serum sample from each patient was tested by each RIT. A positive reverse-transcription polymerase chain reaction was considered as the reference standard for dengue diagnosis. The diagnostic parameters analyzed for each RIT were sensitivity, specificity, positive and negative predictive values, and likelihood ratios. Each RIT was read by homogeneous (two junior nurses) or heterogeneous (one junior nurse and one senior biologist) pairs. Agreement was estimated by simple kappa with 95% confidence interval, positive (Ppos) and negative (Pneg) proportion concordance and prevalence and bias adjusted kappa, rated from poor (k < 0.0) to almost perfect (0.8 < k < 1.0), and perfect (k = 1).
ResultsNS1 RITs for DENV-4 diagnosis showed high specificity (95.9%–99.4%), but low sensitivity (14.7%–45.4%). Bioeasy™ had the best performance, with a positive likelihood ratio of 26.0 (95% CI: 8.4;81.0). Inter-observer agreement was almost perfect for all evaluated RITs. Mismatches in confirmed dengue were more common for the Bioclin™ (Ppos 88.3–90.0 %) and Orangelife™ (Ppos 91.7–94.1 %) tests.
ConclusionsFor DENV-4, the tested RITs had high specificity, but lower sensitivity compared to published results for other serotypes. They should not be used for screening purposes. Different brands may have very different performances. This should be considered upon deciding of using RITs in DENV-4 outbreaks.
Dengue fever is the most ubiquitous arboviruse in developing countries. In the Americas, the mean incidence increased from 16.4/100,000 inhabitants in the 1980’s to 218.3/100,000 inhabitants in the 2000–2010 decade.1,2 Among the four known serotypes, dengue serotype 4 (DENV-4) was the first to be recognized in Brazil. It briefly circulated in the Amazon, Northeastern Brazil, during a local outbreak in 1982. In 2010, it reemerged in Roraima, North of Brazil, and from there it disseminated throughout the country.3
Early diagnosis is important to allow appropriate treatment.4 Clinical diagnosis following the World Health Organization’s (WHO) criteria is based on the presence of fever of up to seven days’ duration and at least two other symptoms among the following: nausea, vomiting, arthralgia, myalgia, retroorbital pain, leucopenia, or exanthema. Warning signs for disease severity include abdominal pain or tenderness; vomiting at least three times in 24 h; clinical fluid accumulation; bleeding from nose, gums or gastrointestinal system; lethargy or irritability.1 However, clinical diagnosis has low specificity, making laboratory diagnosis highly desirable.5
Confirmatory dengue laboratory diagnosis relies on dengue viral ribonucleic acid (RNA) detection, virus isolation in culture or paired IgM/IgG serology. Such diagnostic methods are time-consuming, expensive, requiring dedicated infrastructure and personnel. For these reasons, they may be impractical, especially during epidemics.6
Rapid immunochromatographic tests (RITs) require neither specific infrastructure nor technical expertise, may be used at point-of-care and can be an option for early diagnosis in low resource settings.5 RITs are becoming increasingly popular for the diagnosis of epidemic infectious diseases in emerging countries, where they may be performed both in clinics by health care providers or in community settings by minimally trained health workers.7
Agreement of test readings is an issue when these tests are used at point-of-care or in low resource settings. Few studies on dengue RITs have investigated this aspect considering the serotypes. A single study testing RIT SD Bioline Dengue Duo™ NS1/IgM/IgG in predominantly DENV-2 samples in Singapore reported excellent inter-observer agreement between blinded readers (kappa = 0.98 CI: 0.83–1.00), but the expertise level of the readers was not specified.8
Most studies on dengue RITs accuracy concern dengue serotypes 1, 2 and 3 and there is a growing literature on the performance of commercial rapid and conventional NS1 antigen tests, as extensively reviewed.9 DENV-4 is the latest emerging serotype in circulation. Most cases occur in developing countries, such as Brazil and India.10 For DENV-4, there are fewer studies and sample sizes are smaller and, thus, NS1 RIT performance remains controversial.11,12
A pioneer study based on 272 patients in early dengue clinical stages (all serotypes) carried out in Antilles-Guyana included 46 DENV-4 positive samples. Two commercial tests for dengue NS1 in serum were assessed, including an NS1 Ag strip (Bio-Rad™). Sensitivity was 82.6% (95% CI 68.6–92.8) and specificity was 100% at 15 min. Confirmation was based both on RT-PCR and virus isolation.11 A study carried out in Venezuela, however, reported 60% (12/20 patients) sensitivity for DENV-4 using the same strip test (Bio-Rad™) and identical reference standards. Alere Dengue Early NS112 Rapid Test™ was evaluated in DENV-4 samples from Malaysia and Vietnam and compared with RT-PCR or IgM/IgG/NS1 ELISA. Sensitivity was 62.5% in 16 adults and 33.3% in three children. Due to the small sample size, 95% confidence intervals were not provided, precluding external generalizability. Also, specificity could not be calculated.13
Bioeasy™ and Bio-Rad™ showed high sensitivities (94.1% and 91%, respectively) and perfect specificities (100%) when compared to IgM/NS1 ELISA and MAC ELISA as reference standards in 77 Brazilian patients. In this specific study, only four samples were DENV-4.14
More recently, a study evaluated a well-characterized panel of acute febrile patients´ sera from Peru, Honduras and Ecuador. Of the 200 DENV-positive samples, 62 were DENV-4 based on virus isolation. Much lower sensitivities of RITs to DENV-4, ranging from 42.1% (InBios™) to 58.1% (Bio-Rad™), were described.15 Given these inconsistent results of RITs performance regarding DENV-4, the aim of this study was to discuss the accuracy and agreement of NS1 RITs in use in Brazil during a DENV-4 real outbreak, with consecutively enrolled patients in the acute phase of the disease (up to three days of fever).
MethodsThis was a cross-sectional prospective investigation of patients during an acute febrile syndrome outbreak at an emergency care unit in Tijuca, Rio de Janeiro, Brazil, to assess the accuracy and agreement of the four dengue RITs. We have previously used a similar panel to study only Bioeasy™ for NS1 antigen and WHO clinical algorithms.16
Population and data collectionThe study sample was made up of adults, older than 18 years, who, during the 2013 dengue outbreak, spontaneously sought the emergency care unit located in a central district of the city. All fulfilled the inclusion criteria of acute febrile syndrome for up to 72 h, absence of identified focus of infectious and at least two or more symptoms of suspected dengue cases according to WHO.1
According to sample size calculation, 137 positive dengue patients would be necessary to estimate, with 95% confidence level, a specificity of 85% with an absolute error of 6%. Considering an expected prevalence of dengue of 40% among febrile patients during the outbreak, 342 febrile subjects would have to be evaluated.
Blood samples were collected at the emergency care unit and stored and processed at the Flavivirus Laboratory of the Oswaldo Cruz Institute, the regional reference center for DENV diagnosis and characterization.
Two blood specimens were collected for each patient. Prior asepsis was performed with 70% alcohol, followed by brachial venipuncture by a nurse with a 25 × 7 mm BD Vacutainer™ needle. Blood was drawn and centrifuged at the emergency care unit laboratory and transported to be processed and analyzed at the Flavivirus Laboratory of the Oswaldo Cruz Foundation. All patient samples remained frozen at −70 °C for subsequent characterization based on RT-PCR and Dengue NS1 ELISA.
DiagnosisReference tests and case definitionSamples were considered as dengue cases when they tested positive in Reverse-Transcription Polymerase Chain Reaction (RT-PCR), performed according to the protocol described by Lanciotti et al.17 This method is accurate for early diagnosis of dengue, with a sensitivity exceeding 80% within the initial three days of symptoms.18
Platelia™ Dengue NS1 Ag ELISA (Bio-Rad™ Laboratories, France) was performed for detection of NS1 antigen according to the manufacturer’s specifications.19
Samples yielding negative results in those two methods were considered non-dengue and tested for ZIKA virus with PCR.20
The reference tests were performed at the Flavivirus Laboratory, by a biologist blinded to the index test.
Immunologic markersPanbio™ dengue IgM Capture ELISA (Alere™, Minas Gerais, Brasil) and Dengue Virus IgG DxSelect™ ELISA (Focus Diagnostics, California, USA) were performed, respectively, for the qualitative detection of anti-DENV IgM and anti-DENV IgG antibodies, and define primary or secondary infection.21,22
Non-dengue patients were also tested for Zika virus with RT-PCR23 to investigate co-circulation of these viruses.
Index testsThe index tests were Dengue NS1-Bioeasy™ (Standard Diagnostic Inc, Korea),24 Dengue NS1 Ag Strip-Bio-Rad™ (Bio-Rad Laboratories Inc, France),25 IVB Dengue Ag NS1-Orangelife™ (Orangelife Comercio e Industria Ltda, Brazil),26 and Dengue NS1-K130-Bioclin™ (Quibasa Química Básica Ltda, Brazil).27
These tests were provided as rectangular cassettes with an orifice, through which blood, serum or plasma contacts an immunochromatographic strip, and a window to enable reading the result.24–27 The Bio-Rad™ test uses serum or plasma and additionally requires a pipette with a migration buffer and a test tube (not available in the kit) to promote the reaction in the immunochromatographic strip.25 When only the control line (C) is visible, the test is considered negative; if lines C and T are visible, the result is positive; and when no line or only the T line is visible, the test result is considered invalid.24 Each serum sample was tested by the four rapid assays.
For assessing agreement in test readings, the RITs were performed with serum, according to manufacturer’s instructions, and read by a single pair of readers. Pairs were defined according to the experience in performing these tests: a homogeneous pair A (composed by two junior nurses) and a heterogeneous pair B (one junior nurse and one senior biologist). Each individual reader in a pair read each sample once in the same strip or cassette, independently and blindly at 15 min. In the case of initially invalid results, readings were repeated at 30 min.
There was no time interval between the tests and the reference standard, as they were performed using the same serum sample.
Statistical analysisQuantitative variables were described as simple frequencies and dispersion measures, and their frequencies were compared by chi-square statistical test. The following diagnostic parameters were calculated based on the senior biologist readings: sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratio of the RIT with 95% confidence intervals (CI). Tests were compared for sensitivity and specificity according to the chi-square McNemar test.28
The agreement index between the two pairs of readers (A and B) for each RIT (Bio-Rad™, Orangelife™ and Bioclin™) was estimated by the simple kappa statistic (the probability of chance agreement is taken into account in the calculation of kappa), in addition to positive proportion concordance (Ppos) and negative (Pneg), and prevalence and bias adjusted kappa (PABAK).29 For all of them, 95% CI was provided. Simple kappa (k) and PABAK values were interpreted according to the Landis and Koch30 classification: poor (k < 0.0), slight (0.0 < k < 0.2), fair (0.2 < k < 0.4), moderate (0.4 < k < 0.6), substantial (0.6 < k < 0.8), almost perfect (0.8 < k < 1.0), and perfect (k = 1). MedCalc 12.7 and R 3.2.1. were used for analysis.31,32
Ethical approvalThis study was approved by the Research Ethics Committee of the Evandro Chagas Clinical Research Institute (CAAE 0066.0.009.000-11) and registered in UTN: U1111-1145-9451. All adult subjects provided written informed consent.
ResultsThis study collected 375 serum samples, of which 51 were excluded: 19 with no index test, three with no reference test, eight ELISA NS1 invalid and PCR negative, one DENV-1, one DENV-3, 13 ELISA NS1 positive but PCR negative, and six with more than three days of fever (Fig. 1). Seven tests with invalid results were re-read after 30 min, of which three (Orangelife™) were excluded from analyses (two dengue cases, one non-dengue). Three serum samples that were negative for dengue by RT-PCR and NS1 ELISA were positive for Zika virus.
Of the 324 serum samples included, 56.8% were from women. Patients’ ages ranged from 25.5 to 46 years (median 33 years) and presented with fever for a median duration of two days (95% IQR: 1–2). The most frequent symptoms were asthenia (95.6%), headache (92.6%), myalgia (90.9%), low back pain (78.1%) and arthralgia (72.7%), with no differences between dengue and non-dengue groups, and there was no evidence of clinical warning signs.
Dengue prevalence based on the reference tests was 46.9% (95% CI: 41.4 –52.5) (n = 152 cases). Among positive cases, 78.9% were secondary infections. All tests showed high specificity (95.9 %–99.4 %), but low sensitivity (14. %–45.4%), with several false-negative results (Table 1).
Accuracy of rapid immunochromatographic tests in 152 dengue cases and 172 non-dengue cases classified according to the reference test.
| Bioeasy™ | Orangelife™ a | Bioclin™ | Bio-Rad™ | |
|---|---|---|---|---|
| Sensitivity % (95% CI) | 45.4b (37.3–53.7) | 14.7 (9.4–21.4) | 21.1 (14.9–28.4) | 17.8b (12.0–24.8) |
| Specificity % (95% CI) | 98.3 c (95.0–99.6) | 97.7 (94.1–99.4) | 95.9 (91.8–98.3) | 99.4c (96.8–100) |
| PPV % (95% CI) | 95.8 (88.3–99.1) | 84.6 (65.1–95.6) | 82.1 (66.5–92.5) | 96.4 (81.7–99.9) |
| NPV % (95% CI) | 67.1 (60.9–72.8) | 56.6 (50.7–62.3) | 57.9 (51.9–63.7) | 57.8 (51.9–63.5) |
| LR+ (95% CI) | 26.0 (8.4–81.0) | 6.3 (2.2–17.8) | 5.2 (2.4–11.4) | 30.6 (4.2–222.2) |
| LR− (95% CI) | 0.56 (0.48–0.64) | 0.87 (0.81– .94) | 0.82 (0.75–0.90) | 0.84 (0.79–0.90) |
CI, Confidence Interval; PPV, Positive Predictive Value; NPV, Negative Predictive Value; LR, Likelihood Ratio.
Bioeasy™ showed the best accuracy with a positivity rate approximately 26-fold higher among those with actual dengue than in non-dengue (positive likelihood ratio: LR+ = 26.0, 95% CI: 8.4–81.0). Although the LR + of the Bio-Rad™ test was similar to that of the Bioeasy™ test, its 95% CI was larger (LR+ = 30.6, 95% CI: 4.2 to 222.2). Bio-Rad™ sensitivity (17.8%) was about one third lower than that found for Bioeasy™ test (45.4%). The high negative likelihood ratio of all tests evidenced their inadequate performance to exclude the diagnosis of dengue (Table 1).
Inter-observer agreement of test results was almost perfect for all tests in both pairs of readers, except for the 95% CI of the Bioclin™ test for the heterogeneous pair (N), where agreement ranged from substantial to almost perfect. Such results persisted when analyzing with PABAK (Table 2).
Inter-observer agreement of the interpretation of dengue rapid immunochromatographic testsa in 345 patients according to the pairs A and B of readers.
| Index (CI 95%) | BioRad™ | Orangelife™ | Bioclin™ | |
|---|---|---|---|---|
| Kappa | Group A | 0.98 (0.93–1.00) | 0.89 (0.79–1.00) | 0.93 (0.86–1.00) |
| Group B | 0.98 (0.93–1.00) | 0.95 (0.88–1.00) | 0.91 (0.83–0.99) | |
| PPos% | Group A | 98.1 (92.9–100) | 90.0 (78.9–100) | 94.3 (86.6–100) |
| Group B | 98.1 (93.0–100) | 95.7 (87.3–100) | 92.5 (83.6–100) | |
| PNeg% | Group A | 99.6 (98.6–100) | 98.6 (96.6–100) | 98.5 (96.4–100) |
| Group B | 99.6 (98.7–100) | 99.3 (97.9–100) | 99.1 (95.8–100) | |
| PABAK | Group A | 0.99 (0.93–1.00) | 0.95 (0.88–0.99) | 0.95 (0.88–0.99) |
| Group B | 0.99 (0.93–1.00) | 0.98 (0.91–1.00) | 0.94 (0.86–0.98) |
Pair A (homogeneous expertise) Pair B (heterogeneous expertise). Kappa, Simple Kappa; Ppos, Positive proportion concordance; Pneg, Negative proportion concordance; PABAK, bias adjusted kappa.
Disagreements were more common in the presence of dengue diagnosis for Orangelife™ test (Ppos ranging from 78.9 %–100 %) and Bioclin™ test (Ppos ranging from 83.6%–100%). Agreement in the absence of disease reached high proportions for the three tests assessed (ranging from 97%–100%) (Table 2).
DiscussionWhen performed up to the third day of fever, RITs targeting dengue NS1 antigen in DENV-4 had high specificity and lower sensitivity comparatively to published results for other serotypes.14,33,34 The Bioeasy™ showed distinctly better performance than the other three tests. Similar results have been reported in Colombia33 for Bio-Rad™. This study tested 167 individuals (104 cases of dengue, of which 42 with serotypes identified by RT-PCR), with only five DENV-4 patients.
Our findings contradict initial findings in the Antilles-Guyana study, where NS1 RIT sensitivity point estimates for DENV-4 were high and did not differ much from that of other dengue serotypes. Large intervals for 95% CI already suggested imprecision in estimates, probably due to DENV-4 small sample size (38 in 46 individuals).11
Literature on the performance of the NS1 rapid test Bioeasy™ for DENV-4 is scarce and studies in Brazil did not include14 or had small sample sizes for this serotype.34 We were also unable to find reports on performance of the Brazilian RITs Bioclin™ and Orangelife™.
Studies on the NS1 rapid test Bio-Rad™ for other serotypes have yielded sensitivities as high as 91%,33 98%14 for DENV 1-2, and 88% for DENV-3.34 However, in Singapore this same test showed a more conservative performance, with overall sensitivity of 78.9%. The sample test included 209 DENV 1-2 patients and one DENV-4. This lower performance could be associated with the fact that patients had predominantly secondary dengue infections.35
Previous studies on DENV-4 using other diagnostic tests also report low sensitivity and high specificity. A Brazilian study36 identified low sensitivity for DENV-4 Platelia™ NS1 ELISA, resulting in underreporting and delay in detection in routine control programs. Another study in the same country21 testing Panbio Dengue early ELISA™ detected, in a sample of 31 DENV-4 patients in an epidemic period, a 15% false-negative rate for NS1 Ag rapid assay.
Differences in study results, even those with adequate designs, may be related to disease duration, and, consequently, to low viremia, which hinders NS1 protein detection by the tests. In addition, both the type of infection (primary or secondary)22,33 and the serotype15,33,34 seem to interfere with NS1 protein detection.
As repeatedly pointed out in literature,37 RITs should be used to rule in dengue diagnosis but not to rule out. The actual utilization context of RITs recommends strict guidance on treatment flows and on the information provided by these tests.35 Paradoxically, past studies38 argued that the excellent confirmatory accuracy would recommend their use in screening of imported dengue cases at airports, based on 17 out of 22 PCR confirmed dengue cases also positive in the NS1 strip test.
All tests assessed had high inter-rater agreement, independently of the reader’s expertise. Disagreements occurred in the presence of dengue diagnosis and were slightly higher within the heterogeneous group of readers, probably because of their different level of training to perform laboratory tests. Although the tests are easy to perform and read, the Bio-Rad™ test requires some laboratory infra-structure (test tube, refrigeration at 8 to 16° C and reagent) hindering its use as a point-of-care test.
Despite its importance, agreement of test results is not often reported for RIT studies. Previously, four studies described moderate to almost perfect inter-observer agreement rates for commercial RIT Duo NS1/IgM/IgG39 or IgM/IgG8,33,40 tests. However, in three of them,33,40,41 samples showed no dominance for any specific DENV serotype. Only one of these studies reported raters’ expertise (heterogeneous pair, including a physician and a bacteriologist), confirming that RITs are easy to read.41
Limitations of our study include non-evaluation of Bioeasy™ NS1 rapid test’s agreement and predominance of secondary infection in our sample, which may have affected the performance of the tests.39 Circulation of Zika virus was investigated and three positive cases were found.34
Strong points include evaluating NS1 immunochromatographic tests produced in Brazil; using RT-PCR as the reference test in acute samples for DENV-4 to decrease classification bias; and including evaluation of reproducibility of the other three tests. This study also compared performance of commercially available RITs of four manufacturers in early diagnosis of DENV-4 in an epidemic scenario with a robust sample size and consecutively enrolled patients. The need for prospective recruitment studies in different endemic locations and representative of real-life contexts in high burden countries has been previously pointed out.4
All tests evaluated show better specificity than sensitivity, and could be recommended to rule in patients, but not for screening. This result of worst accuracy with DENV-4 in a large, prospective sample is consistent with inter-serotype accuracy variation, as shown previously in last decade evaluation of NS1 ELISA tests36,42 and Strip Test.43 Our study also suggested minimal influence of operator training on inter-observer variation of NS1 RIT results.4 A previous study confirmed that these results are comparable using blood or serum samples.44
Erroneous interpretations of diagnostic tests for dengue are not uncommon38 and studies on the performance and application of such tests remain controversial. The dissemination of the results of robust accuracy studies could support the proper use of rapid tests in clinical practice. Cost-effectiveness, implementation and outcome evaluations are also needed in order to assess the incorporation of those tests. DUO tests (using NS1 and IgM/IgG) have also been recently introduced45 which could have better performances in various contexts than NS1 RITs.7
We highlight the fact that although not fit for screening purposes, NS1 RITs may be used as confirmatory tests to rule in a dengue diagnosis. This gains much relevance in current scenarios of outbreaks with co-circulation of several flavivirus. However, performance of NS1 RITs in DENV-4 as screening tests tends to be poor and may vary widely. The wide variations in sensitivity between different brands needs to be considered upon deciding on the use of this test in DENV-4 outbreaks.
FundingThis study was supported by the Conselho Nacional de Desenvolvimento Científico (CNPq) grant numbers. 402068 / 2012-2. SRLP had a fellowship from CNPq [grant number 3107665 / 2016-1] and UNESA Produtividade em Pesquisa. VEM had a doctoral fellowship from Faperj [grant number 221.354 E_01/2016]. The funding source had no involvement in the collection, analysis and interpretation of data or in the writing of the report.
Authors contributionsSNB, SRLP, VEM contributed to the conception and design of the study, acquisition of data, analysis and interpretation of data, drafting the article. MABS – draft and critical revise on for importante contente. SRLP – analysys and interpretation of data, drafting the article. SNB, VEM acquisition of data, drafting the article. All authors approved the final version to be submitted.
Conflicts of interestThe authors declare no competing conflicts of interest.
Data availabilityARCA Fiocruz Repositório.
