Journal Information
Download PDF
More article options
Original Article
Open Access
Uncorrected Proof. Available online 26 December 2021
Diagnostic performance of the RSNA-proposed classification for COVID-19 pneumonia versus pre-pandemic controls
Cauã O. Rochaa,b,
Corresponding author

Corresponding author.
, Tássia A.D. Priostea, Carlo S. Faccina, Luciano Foladora, Mateus S. Tonettoa,b, Pedro G. Knijnikc, Natalia B. Mainardic, Rogério B. Borgesd, Tiago S. Garciaa,b
a Radiology Department, Hospital de Clínicas de Porto Alegre (HCPA), RS, Brazil
b Graduate Program in Pulmonary Sciences, Universidade Federal do Rio Grande do Sul, RS, Brazil
c School of Medicine, Universidade Federal do Rio Grande do Sul, RS, Brazil
d Biostatistics Unit, Graduate Research Group (GPPG), Hospital de Clínicas de Porto Alegre, RS, Brazil
Article information
Full Text
Download PDF
Figures (4)
Show moreShow less
Tables (5)
Table 1. Baseline variables and clinical data.
Table 2. Radiological Society of North America-Proposed Reporting Language (5).
Table 3. CT results and inter-reader agreement (Fleiss kappa).
Table 4. RSNA diagnostic performance.
Table 5. Predictive values.
Show moreShow less

To evaluate the diagnostic accuracy of the Radiological Society of North America (RSNA) classification system for coronavirus disease 2019 (COVID-19) pneumonia compared to pre-pandemic chest computed tomography (CT) scan images to mitigate the risk of bias regarding the reference standard.

Materials and methods

This was a retrospective, cross-sectional, diagnostic test accuracy study. Chest CT scans, carried out from May 1 to June 30, 2020, and from May 1 to July 17, 2017, were consecutively selected for the COVID-19 (positive reverse transcription-polymerase chain reaction [RT-PCR] for severe acute respiratory syndrome coronavirus 2 result) and control (pre-pandemic) groups, respectively. Four expert thoracic radiologists blindly interpreted each CT scan image. Sensitivity and specificity were calculated.


A total of 160 chest CT scan images were included: 79 in the COVID-19 group (56 [43.5–67] years old, 41 men) and 81 in the control group (62 [52–72] years old, 44 men). Typically, an estimated specificity of 98.5% (95% confidence interval [CI] 98.1%–98.4%) was obtained. For the indeterminate classification as a diagnostic threshold, an estimated sensitivity of 88.3% (95% CI 84.7%–91.7%) and a specificity of 79.0% (95% CI 74.5%–83.4%), with an area under the curve of 0.865 (95% CI 0.838–0.895), were obtained.


The RSNA classification system shows strong diagnostic accuracy for COVID-19 pneumonia, even against pre-pandemic controls. It can be an important aid in clinical decision-making, especially when a typical or indeterminate pattern is found, possibly advising retesting following an initial negative RT-PCR result and streamlining early management and isolation.

Chest CT
COVID-19 pneumonia
RSNA classification
Pre-pandemic controls
Full Text

Coronavirus disease 2019 (COVID-19), caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has spread globally, becoming an ongoing pandemic responsible for more than four million deaths worldwide and more than 550,000 casualties in Brazil, the second most affected country in the world to date.1 Consequently, computed tomography (CT) of the chest has been employed to assist in the diagnosis and assessment of potential complications and prognosis of patients with COVID-19.2,3 Since findings associated with COVID-19 pneumonia can be found in other lung diseases,4 some radiological societies have published guidelines with objective criteria to improve accuracy and reproducibility of chest CT findings, notably the Radiological Society of North America (RSNA) expert consensus,5 among others.6,7

The RSNA classification system for reporting COVID-19 pneumonia classified chest CT findings into four categories (negative, atypical, indeterminate, and typical; Table 2). Studies have shown its noteworthy diagnostic performance with specific diagnostic thresholds (indeterminate and typical), as well as moderate-to-substantial inter-reader agreement, mostly adopting the SARS-CoV-2 reverse transcription-polymerase chain reaction (RT-PCR) test as the reference diagnostic standard.8–17

According to published meta-analyses,18–20 the quality of early articles designed to assess the accuracy of chest CT in the diagnosis of pneumonia caused by SARS-CoV-2 was considered suboptimal. Limited adherence to reporting guidelines and a high risk of bias, especially regarding patient selection and the reference diagnostic standard, have been reported. A more recent meta-analysis confirmed a high risk of bias regarding the reference test, specifically because RT-PCR testing was not repeated in all patients with an initial negative result in some studies.17 Concern over chest CT diagnostic accuracy during overlapping seasonal flu caused by influenza has also been raised.17,20

To address these concerns, a single-center, retrospective, cross-sectional, diagnostic test accuracy study was conducted to evaluate the RSNA classification system for reporting COVID-19 pneumonia5 using chest CT images of patients with RT-PCR-confirmed SARS-CoV-2 infection and pre-pandemic samples, true negatives for COVID-19, as controls. The RSNA classification may be an important aid in clinical decision-making, encouraging retesting following an initial negative RT-PCR result and streamlining early management and isolation.

Material and Methods

This single-center, retrospective, cross-sectional study was approved by the local institutional review board. Informed consent was waived because no risk was presented to the participants. All relevant clinical information was obtained from our hospital's electronic medical records system.

Participant selectionCOVID-19 group

From May 1 through June 30, 2020, chest CT scan images of inpatients, from both wards and emergency departments, with confirmed COVID-19 by at least one positive RT-PCR result within seven days of the CT scan date were consecutively selected. This period was chosen to match the initial regional rising incidence period of COVID-1921 as well as the historic regional peak incidence periods of influenza pneumonia and acute respiratory distress syndrome.22

Control group

From May 1 through July 17, 2017, chest CT scan images of inpatients, as requested by emergency department physicians, that had at least one of the following text descriptors were consecutively selected:

  • 1.

    Clinical information: pneumonia, infection, respiratory, dyspnea, cough, fever, acute, flu, viral, hypoxemia, tachypnea, shortness of breath, bronchopneumonia (BCP), desaturation, viral;

  • 2.

    Radiological report: pneumonia, inflammatory, infectious, ground glass, viral.

Exclusion criteria

The following exclusion criteria were applied to the COVID-19 group only:

  • 1.

    RT-PCR results were indeterminate or unavailable (e.g., external analysis).

The following exclusion criteria were applied to both groups:

  • 1.

    Patient <18 years old;

  • 2.

    Technically limited/low-quality images (e.g., severe respiratory motion artifacts).

Index test and reference standard

Chest CT scans were routinely requested at the discretion of the attending physician considering the institutional protocol, pneumonia severity criteria, laboratory tests, and comorbidities.

RT-PCR-positive results were considered as the reference diagnostic standard for COVID-19 diagnosis. It was a two-site test, oropharyngeal and nasopharyngeal swab, with analysis performed by the institution's own laboratory.

The control group selection period, approximately two years prior to the emergence of COVID-19, was considered the reference standard for the absence of the disease. Search and selection were performed using a business intelligence software in our Radiology Information System/Picture Archiving and Communication System (RIS/PACS) database.

CT acquisition technique

Images were obtained using an eight-row multidetector CT (MDCT) scanner (BrightSpeed Edge, GE Medical Systems, USA), a 16-row MDCT scanner (Brilliance 16, Philips Healthcare, the Netherlands), or a 64-row MDCT scanner (Aquilion 64, Toshiba Medical Systems, Japan), with patients in the supine position and at full inspiration. All scans were volumetric acquisitions (slice thickness: 1.0–2.0 mm) and were reconstructed with a high-spatial-frequency algorithm. Images were stored and analyzed using PACS (IMPAX, Agfa HealthCare, Belgium). When used, iodinated nonionic intravenous contrast medium was injected into a peripheral vein at a dose of 1–2 mL/kg of body weight.

CT image reading and classification

Chest CT scan findings were anonymized for patient identification/information and scanning date and then codified and randomly sorted in a predefined sequence by our PACS operated by the authors who did not participate in the interpretation. For convenience, only the axial series (lung window) of each scan was used.

The readers were four thoracic radiologists (reader 1 [CSF], 15 years of experience; reader 2 [LF], 8 years of experience; reader 3 [TSG], 20 years of experience; and reader 4 [MST], 10 years of experience) who were blinded to each other and to patient identification, clinical/laboratory information, previous imaging tests, RT-PCR results, and scanning date.

All chest CT scans were consecutively interpreted once and individually by all four reviewers in a predefined sequence. Radiological findings were classified according to the four categories proposed by the RSNA consensus (Table 1).5 No final consensus or agreement between readers was sought.

Table 1.

Baseline variables and clinical data.

Baseline  All (n = 160)  COVID-19 (n = 79)  Control (n = 81) 
Age, years  59 (44-70)  56 (43.5-67)  62 (52-72) 
Male sex  85 (53.1)  41 (51.9)  44 (54.3) 
Respiratory/pneumonia signs and symptomsa  130 (81.2)  72 (91.1)  58 (71.6) 
Cancer  48 (29.6)  4 (4.9)  44 (54.3) 
Primary lung cancer  9 (5.6)  9 (11.1) 
Chronic lung diseases  33 (20.6)  12 (15.2)  21 (25.9) 
Smoking  68 (42.5)  22 (27.8)  46 (56.8) 
Obesity (BMI ≥ 30)  26 (16.2)  19 (24.1)  7 (8.6) 
Hypertension  76 (47.5)  41 (51.9)  35 (43.2) 
Coronary artery disease  10 (6.2)  5 (6.3)  5 (6.2) 
Congestive heart failure  13 (8.1)  4 (5.1)  9 (11.1) 
Diabetes  34 (21.2)  21 (26.6)  13 (16.0) 
Hematologic diseases  16 (10.0)  3 (3.8)  13 (16.0) 
Immunocompromised  17 (10.6)  7 (8.9)  10 (12.3) 

Respiratory/pneumonia signs and symptoms = cough, fever, chest pain, dyspnea, tachypnea, hypoxemia, sputum.


Comorbidities = cancer, chronic lung diseases, smoking, obesity (body mass index, BMI > 30), hypertension, coronary artery disease, congestive heart failure, diabetes, hematologic diseases, immunosuppression.

Table 2.

Radiological Society of North America-Proposed Reporting Language (5).

CT category  Imaging findings 
Typical  Peripheral, bilateral GGOa with or without consolidation or visible intralobular lines (“crazy-paving”)Multifocal GGO of rounded morphology with or without consolidation or visible intralobular lines (“crazy-paving”)Reverse halo sign or other findings of organizing pneumonia (seen later in the disease) 
Indeterminate  Absence of typical features and presence of:Multifocal, diffuse, perihilar, or unilateral GGO with or without consolidation, lacking a specific distribution and being nonrounded or nonperipheral.Few, very small GGO with a nonrounded and nonperipheral distribution 
Atypical  Absence of typical or indeterminate features and presence of:Isolated lobar or segmental consolidation without GGODiscrete small nodules (centrilobular; “tree in-bud”)Lung cavitationSmooth interlobular septal thickening with pleural effusion 
Negative  No features to suggest pneumonia. 

GGO = ground-glass opacity.

Sample size

The sample size was calculated to estimate the sensitivity and specificity of the classification proposed by the RSNA to diagnose SARS-CoV-2 pneumonia using the formulas described by Buderer (1996).23 For a 95% confidence interval (CI), a desired precision of 10%, and a sensitivity and specificity of 75%, a sample size of 145 participants was reached. The final sample size was 162 participants, with 10% added for possible losses.

Statistical analysis

Qualitative variables were expressed as absolute and relative frequencies (n [%]), and quantitative variables as median and first and third quartiles (median [Q1–Q3]). The distribution of quantitative variables was assessed by graphical analysis of the histogram and the quartile–quartile plot.

Associations between qualitative variables were assessed using the Pearson χ² test of independence with Yates’ continuity correction. Fleiss kappa statistic24 was used to evaluate inter-reader classification agreement among the four readers.

A generalized linear mixed-effects model with binomial distribution was used to estimate the predicted probability of positivity for COVID-19. The estimated probabilities were then used to calculate a receiver operating characteristic (ROC) curve, which computed the area under the curve (AUC), sensitivity, and specificity. The AUC was calculated using the Wilcoxon nonparametric approach by comparing the predicted probabilities of all discordant pairs of observations.25 Empirical bootstrap (with 10,000 replications) was used to evaluate the 95% CIs. Analyses were performed using the R software, version

ResultsParticipant selection and clinical baselineCOVID-19 group

Within the chosen period, 82 chest CT scan images met the COVID-19 group inclusion criteria, with 79 remaining after excluding three for technical limitations.

Control group

For the control group, 83 CT scan images met the inclusion criteria, with 81 remaining after excluding one underage patient at the time of scanning and one for technical limitations.

Clinical baseline

The male sex relative frequency and median patient age were 51.9% and 56 (43.5–67) years in the COVID-19 group and 54.3% and 62 (52–72) years in the control group, with p = 0.8819 and 0.0693, respectively. In the COVID-19 group, the median time between RT-PCR sample collection and chest CT scan was one day (0–2 days), whereas the median time between symptom onset and chest CT scan was eight days (5–16 days). Population baseline and clinical data are summarized in Table 1.

CT image reading and classification

The results of the CT scan classification for each reader are shown in Table 3.

Table 3.

CT results and inter-reader agreement (Fleiss kappa).

CT results  All (n = 160)  COVID-19 (n = 79)  Control (n = 81)  Fleiss kappa [95% CI] 
All categories        0.527 [0.490-0.564] 
Typical        0.648 [0.584-0.711] 
Reader 1  39  39 (49.4)  0 (0.0)   
Reader 2  33  31 (39.2)  2 (2.5)   
Reader 3  56  54 (68.4)  2 (2.5)   
Reader 4  42  41 (51.9)  1 (1.2)   
Indeterminate        0.383 [0.320-0.446] 
Reader 1  61  35 (44.3)  26 (32.1)   
Reader 2  33  29 (36.7)  4 (4.9)   
Reader 3  34  19 (24.1)  15 (18.5)   
Reader 4  49  31 (39.2)  18 (22.2)   
Atypical        0.565 [0.502-0.629] 
Reader 1  50  3 (3.8)  47 (58.0)   
Reader 2  54  11 (13.9)  43 (53.1)   
Reader 3  50  5 (6.3)  45 (55.6)   
Reader 4  32  2 (2.5)  30 (37.0)   
Negative        0.510 [0.447-0.573] 
Reader 1  10  2 (2.5)  8 (9.9)   
Reader 2  40  8 (10.1)  32 (39.5)   
Reader 3  20  1 (1.3)  19 (23.5)   
Reader 4  37  5 (6.3)  32 (39.5)   

Notes: Qualitative variables are expressed as absolute and relative frequencies (n [%]). Percentages (%) shown are relative to the number of the respective category assigned within each group (i.e., COVID-19 and control) by each reader. The reported Fleiss kappa values are for all readers when all categories are considered simultaneously, as well as for each category individually.

Statistical analysisDiagnostic performance

Diagnostic accuracy was evaluated according to different possible positivity criteria (Table 4):

  • 1.

    Typical classification (Fig. 2) as a positive test for COVID-19 pneumonia, with an estimated sensitivity of 52.2% (95% CI 47.9%–52.4%), a specificity of 98.5% (95% CI 98.1%–98.4%), an AUC of 0.781 (95% CI 0.731–0.785), a positive predictive value (PPV) of 97.3% (95% CI 94.2%–99.4%), and a negative predictive value (NPV) of 67.9% (95% CI 63.1%–72.0%).

  • 2.

    Indeterminate classification (Fig. 3) as a positive test for COVID-19 pneumonia, with an estimated sensitivity of 36.1% (95% CI 35.8%–38.9%), a specificity of 80.5% (95% CI 79.2%–80.7%), an AUC of 0.583 (95% CI 0.579–0.598), a PPV of 64.1% (95% CI 56.1%–71.2%), and an NPV of 56.2% (95% CI 51.6%–60.7%).

  • 3.

    Typical or indeterminate classification as a positive test for COVID-19 pneumonia, with an estimated sensitivity of 88.3% (95% CI 84.7%–91.7%), a specificity of 79.0% (95% CI 74.5%–83.4%), an AUC of 0.865 (95% CI 0.838–0.895), a PPV of 80.5% (95% CI 76.4%–85.0%), and an NPV of 87.4% (95% CI 83.4%–91.2%).

Table 4.

RSNA diagnostic performance.

For COVID-19 pneumonia  AUC [95% CI]Sensitivity (%) [95% CI]Specificity (%) [95% CI]
Typical  0.781 [0.731-0.785]  52.2 [47.9-52.4]  98.5 [98.1-98.4] 
Indeterminate  0.583 [0.579-0.598]  36.1 [35.8-38.9]  80.5 [79.2-80.7] 
Typical or Indeterminate  0.865 [0.838-0.895]  88.3 [84.7-91.7]  79.0 [74.5-83.4] 
For an Alternative Diagnosis  AUC [95% CI]Sensitivity (%) [95% CI]Specificity (%) [95% CI]
Atypical  0.742 [0.700-0.778]  50.8 [44.8-56.4]  93.4 [90.2-96.0] 
Negative  0.646 [0.598-0.688]  28.4 [23.6-53.8]  94.8 [71.7-97.2] 
Atypical or Negative  0.865 [0.838-0.895]  79.0 [74.5-83.4]  88.3 [84.7-91.7] 

AUC = area under the curve; COVID-19 = coronavirus disease 2019.

The ROC curve is shown in Fig. 1.

Fig. 1.

Receiver operating characteristic (ROC) curve for the possible coronavirus disease 2019 (COVID-19) pneumonia positivity criteria evaluated.

Fig. 2.

Typical CT imaging features for COVID-19 pneumonia. Axial unenhanced CT image of the lungs in a 61-year-old man with a positive RT-PCR showing bilateral, multifocal rounded and peripheral GGO.

Fig. 3.

Indeterminate CT imaging features for COVID-19 pneumonia. Axial contrast-enhanced chest CT image of the lungs in a 41-year-old female with a positive RT-PCR, showing bilateral widespread GGO with nonrounded morphology and no specific distribution with areas of consolidation.


Atypical (Fig. 4) and negative classifications were evaluated as criteria for an alternative diagnosis, returning specificities of 93.4% (95% CI 90.2%–96.0%) and 94.8% (95% CI 71.7%–97.2%), respectively.

Fig. 4.

Atypical CT imaging features for COVID-19 pneumonia. Axial unenhanced chest CT image of the lungs in a 37-year-old control patient showing tree-in-bud opacities and centrilobular nodules, caused by active tuberculosis.


Predictive values computed for each scenario are shown in Table 5.

Table 5.

Predictive values.

For COVID-19 Pneumonia     
Criteria  PPV (%) [95% CI]  NPV (%) [95% CI] 
Typical  97.3 [94.2-99.4]  67.9 [63.1-72.0] 
Indeterminate  64.1 [56.1-71.2]  56.2 [51.6-60.7] 
Typical or Indeterminate  80.5 [76.4-85.0]  87.4 [83.4-91.2] 
For an Alternative Diagnosis     
Criteria  PPV (%) [95% CI]  NPV (%) [95% CI] 
Atypical  88.7 [83.7-92.9]  65 [60.6-69.5] 
Negative  84.7 [66.7-91.4]  56.6 [53.0-60.9] 
Atypical or Negative  87.4 [83.4-91.2]  80.5 [76.4-85.0] 

Note: Table shows the predictive values of different diagnostic criteria of the RSNA Classification for COVID-19 pneumonia and for an alternative diagnosis.

Inter-reader agreement

Inter-reader overall agreement analysis, including all four categories, demonstrated a moderate Fleiss kappa value of 0.527 (95% CI 0.490–0.564). Agreement was highest for the typical classification, with a substantial Fleiss kappa value of 0.648 (95% CI 0.584–0.711), and lowest for the indeterminate classification, with a fair Fleiss kappa value of 0.383 (95% CI 0.320–0.446) (Table 3).


The RSNA-proposed chest CT imaging classification can provide excellent diagnostic accuracy for COVID-19 pneumonia, with high specificity and sensitivity, depending on the chosen diagnostic threshold. The classification also showed a noteworthy inter-reader agreement when tested against pre-pandemic controls affected by diverse respiratory diseases and treated at the emergency department, specifically during the regional peak incidence periods of influenza pneumonia and acute respiratory distress syndrome.

Our findings suggest that the typical pattern is highly associated with COVID-19 pneumonia, even when compared to pre-pandemic controls. Moreover, an atypical or negative pattern suggests an alternative diagnosis, but should be interpreted with caution, as it does not rule out COVID-19 diagnosis, nor the possibility of COVID-19 pneumonia. In addition, different possible diagnostic thresholds for COVID-19 pneumonia were explored, attaining one of high specificity and another of high sensitivity, with typical pattern only and typical plus indeterminate patterns combined, respectively.

Using pre-pandemic controls, that is, true negatives, results agree with the current body of evidence showing, as already mentioned, good correlation between the RSNA classification and the RT-PCR results, as well as a consistent inter-reader agreement. Moreover, two possible diagnostic thresholds that could be interpreted and used according to specific clinical context needs were analyzed.

We ascertained the role of RSNA-proposed CT patterns for COVID-19 pneumonia diagnosis, especially the strong association between the typical pattern and COVID-19 pneumonia and between the atypical pattern and an alternative diagnosis. Moreover, we propose that the two possible diagnostic thresholds may aid in clinical decision-making considering their advantages in specific contexts, primarily combined with the reference test, but also particularly helpful when the reference test is scarcely or not readily available. The typical pattern was highly specific and thus strongly suggestive of COVID-19 pneumonia diagnosis against a differential diagnosis that could demand a different management. Both typical and indeterminate patterns for a positive test were highly sensitive, thus considering retesting following an initial negative RT-PCR result as well as to streamline early management and isolation until a definitive diagnosis can be made.

Our study has limitations, including the usual profile of the hospitalized patient population at our institution, whose normally high prevalence of admissions due to chronic conditions (e.g., heart failure, cancer) may be related to the statistically significant difference in the presence of pneumonia/respiratory signs and symptoms between the two groups. Another possible reason is the undervaluation of said symptoms in the pre-pandemic context, especially when of mild or vague nature, or even when linked with chronic conditions. Nevertheless, differential diagnosis is not restricted to infections of obvious presentation or of infectious etiology,4 which in our opinion justifies the inclusion of patients without explicitly reported signs/symptoms. We also recognize that there is concern that the control patients and settings do not ideally match the review question, but we understand that it is an unavoidable issue given the case-control selection nature of the study. The limit of seven days between RT-PCR sampling and CT scanning for the COVID-19 group may also be of concern, but it was necessary as CT scans were reserved for selected cases and often not immediately ordered in agreement with the institutional protocol and the principle of justification in a broad perspective, as RT-PCR testing was readily available and rapidly processed in-house. Finally, a fair Fleiss kappa-value was found for the indeterminate classification, possibly related to the intrinsic unspecificity of its imaging findings, magnified by the use of four readers; however, further analysis is necessary for clarification.

In conclusion, when tested against pre-pandemic controls, the RSNA classification system for reporting COVID-19 pneumonia showed prominent diagnostic performance, in agreement with the current literature, with potentially high specificity and sensitivity provided by its different diagnostic thresholds. Thus, we believe it can be an important aid in clinical decision-making, especially when a typical or indeterminate pattern is found, considering retesting following an initial negative RT-PCR and streamlining early management and isolation.

Center for Systems Science and Engineering. COVID-19 dashboard by the Center for Systems and Engineering at Johns Hopkins University. Available at: [accessed August 10, 2021].
GD Rubin, CJ Ryerson, LB Haramati, et al.
The role of chest imaging in patient management during the covid-19 pandemic: a multinational consensus statement from the fleischner society.
Radiology, 296 (2020), pp. 172-180
WHO Guidance Note.
Use of chest imaging in COVID-19: a rapid advice guide.
M Parekh, A Donuru, R Balasubramanya, S. Kapur.
Review of the chest CT differential diagnosis of ground-glass opacities in the COVID era.
Radiology, (2020), pp. E289-E302
S Simpson, FU Kay, S Abbara, et al.
Radiological Society of North America expert consensus document on reporting chest CT findings related to COVID-19: endorsed by the Society of Thoracic Radiology, the American College of Radiology, and RSNA.
Radiol Cardiothorac Imaging, 2 (2020),
M Prokop, W Van Everdingen, T Van Rees Vellinga, et al.
CO-RADS: a categorical CT assessment scheme for patients suspected of having COVID-19 - definition and evaluation.
Radiology, 296 (2020), pp. E97-104
A. Johnstone.
Thoracic imaging in COVID-19 infection: guidance for the reporting radiologist.
D Byrne, SBO Neill, NL Müller, et al.
RSNA expert consensus statement on reporting chest CT findings related to COVID-19: interobserver agreement between chest radiologists.
Can Assoc Radiol J, 72 (2021), pp. 159-166
MO Hadied, PY Patel, P Cormier, et al.
Interobserver and intraobserver variability in the CT assessment of COVID-19 based on RSNA consensus classification categories.
Acad Radiol, 27 (2020), pp. 1499-1506
SB O’ Neill, D Byrne, NL Müller, et al.
Radiological Society of North America (RSNA) expert consensus statement related to chest CT findings in COVID-19 versus CO-RADS: comparison of reporting system performance among chest radiologists and end-user preference.
Can Assoc Radiol J, (2020),
S Inui, R Kurokawa, Y Nakai, et al.
Comparison of chest CT grading systems in COVID-19 pneumonia.
Radiol Cardiothorac Imaging, 2 (2020),
A Som, M Lang, T Yeung, et al.
Implementation of the Radiological Society of North America expert consensus guidelines on reporting chest CT findings related to COVID-19: a multireader performance study.
Radiol Cardiothorac Imaging, 2 (2020),
RD Grando, VB Brentano, AP Zanardo, et al.
Clinical usefulness of tomographic standards for COVID-19 pneumonia diagnosis: experience from a Brazilian reference center.
Brazilian J Infect Dis, 24 (2020), pp. 524-533
F Ciccarese, F Coppola, D Spinelli, et al.
Diagnostic accuracy of North America Expert Consensus Statement on reporting CT findings in patients suspected of having COVID-19 infection: an italian single-center experience.
Radiol Cardiothorac Imaging, 2 (2020),
TMH de Jaegere, J Krdzalic, BACM Fasen, RM Kwee.
Radiological Society of North America chest CT classification system for reporting COVID-19 pneumonia: interobserver variability and correlation with reverse-transcription polymerase chain reaction.
Radiol Cardiothorac Imaging, 2 (2020),
PNVP Barbosa, AGV Bitencourt, GD de Miranda, MEA Almeida, R. Chojniak.
Chest CT accuracy in the diagnosis of SARS-CoV-2 infection: initial experience in a cancer center.
Radiol Bras, 53 (2020), pp. 211-215
RM Kwee, HJA Adams, TC. Kwee.
Diagnostic performance of CO-RADS and the RSNA classification system in evaluating COVID-19 at chest CT: a meta-analysis.
Radiol Cardiothorac Imaging, 3 (2021),
D Suchá, RW van Hamersvelt, AF van den Hoven, PA de Jong, HM. Verkooijen.
Suboptimal quality and high risk of bias in diagnostic test accuracy studies at chest radiography and CT in the acute setting of the COVID-19 pandemic: a systematic review.
Radiol Cardiothorac Imaging, 2 (2020),
HJA Adams, TC Kwee, D Yakar, MD Hope, RM. Kwee.
Systematic review and meta-analysis on the value of chest CT in the diagnosis of coronavirus disease (COVID-19): Sol Scientiae, Illustra Nos.
Am J Roentgenol, 215 (2020), pp. 1342-1350
MD Mair, M Hussain, S Siddiqui, et al.
A systematic review and meta-analysis comparing the diagnostic accuracy of initial RT-PCR and CT scan in suspected COVID-19 patients.
Secretaria Estadual da Saúde. SES/RS - Coronavirus. Governo Do Estado Do Rio Grande Do Sul, Brazil. Available at: /[accessed November 18, 2021].
Secretaria Estadual da Saúde.
Informe epidemiológico da vigilância de influenza, Governo Do Estado Do Rio Grande Do Sul, (2018),
NM Fenn Buderer.
Statistical methodology: I. Incorporating the prevalence of disease into the sample size calculation for sensitivity and specificity.
Acad Emerg Med, 3 (1996), pp. 895-900
JL. Fleiss.
Measuring nominal scale agreement among many raters.
Psychol Bull, 76 (1971), pp. 378-382
H Liu, T. Wu.
Estimating the area under a receiver operating characteristic (ROC) curve for repeated measures design.
J Stat Softw, 8 (2003), pp. 1-18
Core Development Team R.
R Found Stat Comput, (2020),

This study was financed in part by the Fundo de Incentivo à Pesquisa (FIPE)/HCPA.

The authors declare no conflict of interest associated with this manuscript.

Copyright © 2021. Sociedade Brasileira de Infectologia
The Brazilian Journal of Infectious Diseases

Subscribe to our newsletter

Article options