1naresh<pre>Array
(
    [urn:ac.highwire.org:guest:identity] => Array
        (
            [runtime-id] => urn:ac.highwire.org:guest:identity
            [type] => guest
            [service-id] => ajnr-ac.highwire.org
            [access-type] => OpenAccess
            [privilege] => Array
                (
                    [urn:ac.highwire.org:guest:privilege] => Array
                        (
                            [runtime-id] => urn:ac.highwire.org:guest:privilege
                            [type] => privilege-set
                            [privilege-set] => GUEST
                        )

                )

            [credentials] => Array
                (
                    [method] => guest
                )

        )

)
1naresh<pre>Array
(
    [urn:ac.highwire.org:guest:identity] => Array
        (
            [runtime-id] => urn:ac.highwire.org:guest:identity
            [type] => guest
            [service-id] => ajnr-ac.highwire.org
            [access-type] => Controlled
            [privilege] => Array
                (
                    [urn:ac.highwire.org:guest:privilege] => Array
                        (
                            [runtime-id] => urn:ac.highwire.org:guest:privilege
                            [type] => privilege-set
                            [privilege-set] => GUEST
                        )

                )

            [credentials] => Array
                (
                    [method] => guest
                )

        )

)
Noninvasive Determination of IDH and 1p19q Status of Lower-grade Gliomas Using MRI Radiomics: A Systematic Review
=================================================================================================================

* A.P. Bhandari
* R. Liong
* J. Koppen
* S.V. Murthy
* A. Lasocki

## Abstract

**BACKGROUND:** Determination of *isocitrate dehydrogenase* (*IDH*) status and, if *IDH*-mutant, assessing 1p19q codeletion are an important component of diagnosis of World Health Organization grades II/III or lower-grade gliomas. This has led to research into noninvasively correlating imaging features (“radiomics”) with genetic status.

**PURPOSE:** Our aim was to perform a diagnostic test accuracy systematic review for classifying *IDH* and 1p19q status using MR imaging radiomics, to provide future directions for integration into clinical radiology.

**DATA SOURCES:** Ovid (MEDLINE), Scopus, and the Web of Science were searched in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Diagnostic Test Accuracy guidelines.

**STUDY SELECTION:** Fourteen journal articles were selected that included 1655 lower-grade gliomas classified by their *IDH* and/or 1p19q status from MR imaging radiomic features.

**DATA ANALYSIS:** For each article, the classification of *IDH* and/or 1p19q status using MR imaging radiomics was evaluated using the area under curve or descriptive statistics. Quality assessment was performed with the Quality Assessment of Diagnostic Accuracy Studies 2 tool and the radiomics quality score.

**DATA SYNTHESIS:** The best classifier of *IDH* status was with conventional radiomics in combination with convolutional neural network–derived features (area under the curve  = 0.95, 94.4% sensitivity, 86.7% specificity). Optimal classification of 1p19q status occurred with texture-based radiomics (area under the curve  = 0.96, 90% sensitivity, 89% specificity).

**LIMITATIONS:** A meta-analysis showed high heterogeneity due to the uniqueness of radiomic pipelines.

**CONCLUSIONS:** Radiogenomics is a potential alternative to standard invasive biopsy techniques for determination of *IDH* and 1p19q status in lower-grade gliomas but requires translational research for clinical uptake.

## ABBREVIATIONS:

AI
:   artificial intelligence
AUC
:   area under the curve
CNN
:   convolutional neural network
*IDH*
:   *isocitrate dehydrogenase*
*IDH-*mut
:   *IDH*-mutant
LGG
:   lower-grade gliomas
ML
:   machine learning
PRISMA-DTA
:   Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Diagnostic Test Accuracy
QUADAS-2
:   Quality Assessment of Diagnostic Accuracy Studies 2
RQS
:   radiomics quality score
SVM
:   support vector machine
VASARI
:   Visually Accessible Rembrandt Images
WHO
:   World Health Organization

Lower-grade gliomas (LGG), World Health Organization (WHO) grades II/III, are diffusely infiltrative tumors of the CNS. With time, these tumors typically progress to glioblastoma (WHO grade IV), which has a median survival of only 12–18 months despite treatment.1 A growing understanding of the prognostic and therapeutic importance of molecular markers has led to their incorporation into the 2016 WHO classification, and they now constitute a key component of the diagnosis of LGG.2 The 2 key markers of LGG are *isocitrate dehydrogenase* (*IDH*), with tumors classified as either *IDH*-mutant (*IDH*-mut) or *IDH*-wild-type, and 1p19q, with 1p19q-codeletion representing a combined loss of both the short arm of chromosome 1 and the long arm of chromosome 19.

Determining *IDH* and 1p19q status is invasive, requiring a tissue specimen via stereotactic biopsy or definitive resection, with the associated operative risks3 and possibility of sampling error. While the possibility of sampling error is perhaps of greatest relevance to the determination of tumor grade,4 it is also relevant to the determination of tumor genetic status.5,6 For example, *IDH* sequencing may be falsely negative if there are few glioma cells within the sample,5 and intratumoral genetic heterogeneity can occur.6 These considerations have led to research into characterizing *IDH* and 1p19q status by imaging, known as “radiogenomics” or “imaging genomics.” The most specific visual MR imaging feature is the “T2-FLAIR mismatch sign,” which has been shown to predict an *IDH*-mut 1p19q-codeletion gliomas with 100% specificity and high interobserver correlation (κ = 0.38–0.88).7⇓-9 Other useful features include the presence of calcification (suggestive of a 1p19q-codeletion glioma)8,9 and homogeneous signal (likely 1p19q-intact).10 While some features such as >50% T2-FLAIR mismatch and the presence of calcification have high interobserver correlation, other features are limited by greater variability in interpretation. Furthermore, a substantial proportion (29%–37%) of gliomas do not exhibit these features, limiting sensitivity.8

Artificial intelligence (AI) is emerging as a solution to the limitations of conventional visual assessment. AI techniques may identify features hidden to the naked eye by extracting data from images and relating them to outcomes. Given the inherent signal and volume heterogeneity of gliomas, a perceived signature or pattern may be modelled to genetic, clinical, and biochemical outcomes.11 Features can be learned from the image or predefined. The field of radiomics involves the extraction of predefined features such as shape, intensity, and texture from a segmented (tumor) volume of interest.12 This is opposed to deep learning–derived features, which are identified without human predefinition. Radiomic features can be correlated with genetic status through a subset of AI known as machine learning (ML). The ML algorithm is trained to a clinical outcome via a training dataset and validated using a testing/validation dataset. Extracted radiomic features undergo selection and can then be related to molecular markers such as *IDH* and 1p19q, providing a more objective method of radiogenomic correlation.

Radiomic analysis has several advantages compared with human observers, including the ability to rapidly assess multiple imaging features, less interobserver variability,13 and potentially higher sensitivity and specificity. The aim of this article was to perform a systemic review of the use of MR imaging radiomics for the classification of *IDH* and 1p19q status in LGG.

## MATERIALS AND METHODS

Search methodology and study synthesis were performed in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Diagnostic Test Accuracy (PRISMA-DTA) checklist.14 The search was performed on the Web of Science, Ovid (MEDLINE), and Scopus on April 18, 2020. Online Table 1 summarizes the search strategy. Search terms were developed from the PICO framework and Medical Subject Headings, which included terms relating to radiomics or radiogenomics, gliomas, and *IDH*/1p19q status. The PRISMA flowchart is available in Online Fig 1.

### Study Selection

Studies were included if they were original research articles relating radiomic features to *IDH* and/or 1p19q status in LGG (WHO grades II/III) with pathologic confirmation. Studies were excluded under the following circumstances: 1) They investigated the effects of radiogenomic pipelines on factors that affect imaging quality rather than assessing diagnostic potential or 2) they included imaging modalities other than MR imaging because recent literature has not shown superior outcomes.15 There was no restriction on study date.

The references were imported from the Web of Science, Ovid (MEDLINE), and Scopus into EndNote (Version X9; [https://www.endnote.com/product-details/](https://www.endnote.com/product-details/)). Duplicates were removed using the “Find Duplicates” function in EndNote and manual review of the reference list. Two independent authors (A.P.B. and J.K.) screened the titles and abstracts for eligibility. The full texts were then screened. When questions arose regarding inclusion of articles, these were resolved through discussion between both authors responsible for data extraction (experience: A.P.B., medical doctor with a master’s degree in medical imaging analysis, and J.K., medical doctor with 4 years’ clinical experience). Ties were to be reviewed together with the senior author, but none were encountered.

### Data Collection and Analysis

The primary outcome was the classification of *IDH* and/or 1p19q status by MR imaging radiomics. This was based on the receiver operating or precision recall curve and associated sensitivity (%), specificity (%), and area under the curve (AUC) if available. The AUC is presented as a value between 0.5 and 1, with 1 representing perfect classification (and 100% sensitivity and specificity). For studies that did not include ML in the pipeline, descriptive statistics (for example, mean and SD with *t* testing) were also included. Only significant findings for descriptive statistics were reported or the highest AUC for ML classifiers, given that some studies related numerous radiomic features to genetic status (*IDH* and/or 1p19q) or reported a considerable number of ML classifiers. If training and validation set data were reported, only the validation set was used. Secondary outcome measures were related to pipeline features and included the number of lesions, imaging sequences and segmentation method, features and their selection method, ML classifier, genetic status, and WHO tumor grade. A meta-analysis using random effects16 was performed on AUC values with 95% confidence intervals when available in MedCalc (MedCalc Software). A Higgins I2 index of heterogeneity was reported, in which 0% represents no heterogeneity and 100% represents maximum heterogeneity.

### Quality Assessment

Quality assessment was performed using the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool and the radiomics quality score (RQS).12 The QUADAS-2 scoring system was developed to assess bias and the applicability of diagnostic-accuracy studies.17 The RQS is specific to radiomics and is based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis initiative, which examines domains of application for predictive models.18 Application of RQS and QUADAS-2 was performed by discussion between A.P.B. and J.K. A κ statistic19 was considered for the RQS, similar to that used in previous studies;20 however, for quantitatively-defined criteria, it was determined that resolution by discussion would be superior.21

## RESULTS

The initial search obtained 610 articles; 431 articles were from Ovid (MEDLINE); 111, from Scopus; and 68, from the Web of Science. After duplicates were removed, a total of 532 articles remained. The articles were screened by title and abstract, and 18 remained. Full texts were reviewed, and 14 articles22-35 fit the review question and inclusion criteria. The publication dates of the 14 included studies22⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓-35 ranged from 2017 to 2020. A total of 1655 LGG were analyzed. Online Table 2 summarizes the pipeline features for each study.

All segmentations incorporated manual components except for 2 studies, both of which used convolutional neural network (CNN)-based segmentation.28,32 Standard imaging sequences included pre- and postcontrast T1WI, T2WI, and FLAIR. ADC,23,25,30 cerebral blood flow/volume,28,30 DTI,29 and exponential ADC30 were used as adjuncts in some studies. Radiomic features were extracted most commonly by programs developed in-house on the Matlab software platform (MathWorks).23,25,27 AlexNet ([https://www.mygreatlearning.com/blog/alexnet-the-first-cnn-to-win-image-net/](https://www.mygreatlearning.com/blog/alexnet-the-first-cnn-to-win-image-net/)) was used in 1 study for deep learning–derived features in the highest discriminating pipeline.27 The most common method of feature selection was support vector machine (SVM)–recursive feature elimination,25,30,33,34 followed by a Student *t* test.27-29 All categories of radiomic features were used. Two studies did not use ML.23,24 Most studies assessed WHO grade II and III LGG,22⇓⇓⇓⇓⇓⇓⇓⇓-30,32⇓-34 apart from one that assessed only WHO grade II LGG.31 Table 1 demonstrates the derived aims and key findings of studies that examined the *IDH* status of LGG, while Table 2 summarizes studies examining the 1p19q status of *IDH*-mut LGG. Figures 1 and 2 provide the associated forest plots for studies assessing *IDH* and 1p19q, respectively. Further details are provided in the online material. A meta-analysis on *IDH* status was performed on 5 studies22,26,29,30,34 that had sufficient data with a pooled value of 0.827 (95% CI , 0.760–0.894; I2 = 88.55%). For 1p19q status, a meta-analysis was performed on 4 studies22,26,34,35 that had sufficient data with a pooled value of 0.872 (95% CI, 0.789–0.954; I2 = 86.19%).

View this table:
[Table 1:](http://www.ajnr.org/content/42/1/94/T1)

Table 1: 
Derived aims and key findings of studies comparing *IDH*-mut and *IDH* wild-type LGG

View this table:
[Table 2:](http://www.ajnr.org/content/42/1/94/T2)

Table 2: 
Derived aims and key findings of studies examining 1p19q status of *IDH*-mut LGG

![FIG 1.](http://www.ajnr.org/http://ajnr-stage2.stage.highwire.org/content/ajnr/42/1/94/F1.medium.gif)

[FIG 1.](http://www.ajnr.org/content/42/1/94/F1)

FIG 1. 
*IDH* status forest plot of included studies with an AUC.

![FIG 2.](http://www.ajnr.org/http://ajnr-stage2.stage.highwire.org/content/ajnr/42/1/94/F2.medium.gif)

[FIG 2.](http://www.ajnr.org/content/42/1/94/F2)

FIG 2. 
1p19q status forest plot of included studies with an AUC.

The QUADAS-2 score showed low bias and high applicability (see Online Fig 2 for individual studies). The radiomic-specific RQS average score was low, with a mean of 10 (range, 2–14). On average, the RQS was 29% (range, 6%–39%) of the highest possible score. There were no studies that reported on cost-effectiveness, imaging used on phantom models, a prospectively validated radiomic signature in an appropriate clinical trial or performed clinical utility statistics (beyond just discussion of uses).21 Further details are provided in Online Table 3.

## DISCUSSION

The systemic literature review found that the highest classifier for *IDH* status was conventional radiomics with CNN deep learning–derived features, which achieved an AUC = 0.95 (sensitivity of 94.4%, specificity of 86.7%).27 For classification of 1p19q status, conventional texture-based radiomics was optimal, with an AUC = 0.96 (sensitivity of 90%, specificity of 89%).34

Segmentation had manual components in both studies28,32 and was generally performed by trained personnel and approved by neuroradiologists or neurosurgeons. Manual segmentation is time-consuming, resource-intensive and introduces interobserver variability. Automation of segmentation is being actively progressed by the Brain Tumour Segmentation Challenge, and ongoing improvements have the potential to address the limitations of manual segmentation and thus improve the accuracy and efficiency of radiomic methods.36⇓-38 For the whole tumor, the 2018 winning team achieved a Sørensen–Dice coefficient of 0.88, in which a value of 1 represents perfect consistency between manual (ground truth) and automated segmentation.39

For *IDH* status, the literature indicates that a standard sequence image acquisition, use of texture-based features (most common being gray-level co-occurrence matrix,23-25,27-29,34,40 followed by the gray-level run-length matrix25,27⇓-30,34,40) with deep learning–derived features, and an SVM machine learning model may result in an optimal radiomic pipeline. One study classified solely using texture-based radiomic features and achieved an AUC = 0.79.34 Integration of deep learning with radiomic features did not increase the AUC in 1 study22 but produced the highest AUC = 0.95 in another study.27 Features derived from qualitative visual inspection (Visually Accessible Rembrandt Images; VASARI) did not increase the AUC compared with just radiomic features.34 Four studies examined multiparametric imaging.23,25,29,30 The entropy (randomness of voxel intensities) feature derived from ADC images was significantly different between *IDH*-mut and *IDH* wild-type LGG,23 suggesting that heterogeneity of ADC values may be helpful in predicting *IDH* status. Nevertheless, while integration of diffusion/perfusion imaging showed improved classification in 3 studies,25,29,30 ultimately it was not superior to using standard sequences with a different radiogenomic pipeline.28

For 1p19q status, the literature indicates that standard image sequences, use of texture-based features (the most common being grey-level run length matrix26,28,34,35 followed by gray-level co-occurrence matrix26,28,34), and a linear SVM machine learning model may result in an optimal radiomic pipeline. The highest AUC = 0.96 was achieved solely using texture-based radiomic features.34 Clinical and imaging-feature (such as age, sex, and the presence of bleeding or enhancement) integration did not improve the classification performance,38 nor did solely examining visually-created features.34 Deep learning feature integration with radiomic features increased classification performance; however, solely examining deep features was superior.22 The best-performing ML model classifier was achieved by a linear SVM.28

For studies included in the meta-analysis, there was high heterogeneity, given the variation in the unique elements of each radiomic pipeline. Heterogeneity is inevitable with any meta-analysis; however, acceptable levels may be a Higgins I2 of 0%–40%.41 The meta-analysis found 88.55% and 86.19% heterogeneity for *IDH* and 1p19q status, respectively. Although the QUADAS-2 showed low bias and high applicability, the radiomic-specific RQS assessment showed an overall inadequate clinical applicability of studies, identifying issues, including a lack of cost-effectiveness analysis, clinical utility statistics, or prospective validation. This is consistent with other neuro-oncologic radiomic studies in the literature.21 The RQS has some limitations, however. For example, greater emphasis is placed on the image-acquisition parameters12 than on the image-normalization process (making the voxel, section thickness, and matrix size similar among MR imaging scans), despite the latter being important for optimal translation into multi-institutional contexts. Of note, a perceived advantage of the AI algorithms is greater objectivity and thus a more consistent diagnosis, but this has yet to be convincingly proved in the literature.42

Classification of LGG for *IDH* status followed by further classification of 1p19q status (when *IDH*-mut) will have multiplicative effects. There was a sensitivity of 94.4% and specificity of 86.7%27 for *IDH* status, with a sensitivity of 90% and specificity of 89%34 for 1p19q. Thus, by using multiplication, we can find the maximum literature prediction of 1p19q status in an *IDH*-mut LGG to have a sensitivity of 85.0% (94.4% × 90%) and a specificity of 77.2% (86.7% × 89%). The conventional radiogenomic pipelines assume that the features assessed are independent, though they are not. For example, to take an example from the visual-feature literature, ill-defined tumor margins have been correlated with *IDH* wild-type LGG,43 but if the tumor is *IDH*-mut, it is more likely 1p19q-codeletion.10 There is also uncertainty regarding the interaction between radiomic and conventional visual MR imaging features. For example, if the T2-FLAIR mismatch sign is present, the literature would suggest that this can predict an *IDH*-mut 1p19q-intact glioma with greater confidence than radiomics.8,44 Yet, when these conventional features with the greatest predictive value are absent, one could expect that radiomics would predict the genotype better than other conventional MR imaging features. Thus, optimal classification may be achieved using a combination of conventional and radiomic features.

Acceptance of AI into clinical practice remains an issue. Much of the literature on integration is opinion-based,45⇓-48 and research related to understanding challenges is in its early stages.49⇓-51 Acceptance by patients also remains an issue; a recent study by Palmisciano et al52 found that only 66.3% of patients found it acceptable for AI to be used during imaging interpration.53 Issues raised by patients include distrust, lack of knowledge, a lack of personal interaction, questions about the efficacy of the AI algorithm, and the importance of being properly informed of its uses.54 Similar relevant issues were identified by a computer science literature review55 on human-AI interaction, such as task allocation, lack of knowledge and/or trust, incorrect use due to confusion, and integration issues due to a potentially radically different work practice.

Future directions for integration into the clinical sphere may come in the form of examining the nonmedical sphere,55 given successful implementation in other fields such as failure detection in truck engines and welding robots.56 One specific issue is that some AI programs used were developed in-house and may not be readily available to other institutions; important next steps include comparisons between programs and subsequent validation on larger external cohorts. There is also a lack of clinical trials assessing the integration of radiomic analysis into clinical practice,44 which was confirmed on our RQS assessment. Guidelines have recently been developed to address these issues, which may provide a framework for integration. For example, Microsoft has recently released a set of 18 general principles for integration into systems, such as explaining to the user (clinician) what the AI algorithm can do, how well it can be done and making it clear how it is performed.57 A thinking paradigm that may solve this is treating radiomic analysis as a new intervention or drug and applying ideas from existing protocols such as Phase I–IV clinical trials.58 The Food and Drug Administration has also recently released guidelines for AI integration into health care systems.59 Given that radiomic analysis is rapidly progressing and combining AI with standard radiologist assessment may show superior outcomes, there needs to be greater effort to translate findings into an interpretable format for clinical radiology.

## CONCLUSIONS

The greatest classifier of *IDH* status in LGG was achieved with conventional radiomics in combination with convolutional neural network–derived features, providing a sensitivity of 94.4% and specificity of 86.7% (AUC = 0.95). Optimal classification of 1p19q status occurred using texture-based radiomics, with a sensitivity of 90% and a specificity of 89% (AUC = 0.96). The literature is limited by the use of manual segmentation, suboptimal study design, and the lack of translational work to integrate radiogenomic analysis into clinical practice.

## Footnotes

*   Disclosures: Arian Lasocki—*RELATED*: *Grant*: Peter MacCallum Cancer Foundation, *Comments*: Arian Lasocki was supported by a Peter MacCallum Cancer Foundation Discovery Partner Fellowship, providing clinical backfill to allow dedicated research time.* *Money paid to the institution.

Indicates open access to non-subscribers at [www.ajnr.org](http://www.ajnr.org)

## References

1.  1.Stark AM, van de Bergh J, Hedderich J, et al. Glioblastoma: clinical characteristics, prognostic factors and survival in 492 patients. Clin Neurol Neurosurg 2012;114:840–45 doi:10.1016/j.clineuro.2012.01.026 pmid:22377333
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.clineuro.2012.01.026&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=22377333&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

2.  2.Louis DN, Perry A, Reifenberger G, et al. The 2016 World Health Organization Classification of Tumors of the Central Nervous System: a summary. Acta Neuropathol 2016;131:803–20 doi:10.1007/s00401-016-1545-1 pmid:27157931
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00401-016-1545-1&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=27157931&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

3.  3.Akay A, Rüksen M, Islekel S. Magnetic resonance imaging-guided stereotactic biopsy: a review of 83 cases with outcomes. Asian J Neurosurg 2019;14:90–95 doi:10.4103/ajns.AJNS\_81\_17 pmid:30937016
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access\_num=10.4103/ajns.AJNS_81_17&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=30937016&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

4.  4.Lasocki A, Tsui A, Tacey MA, et al. MRI grading versus histology: predicting survival of World Health Organization grade II-IV astrocytomas. AJNR Am J Neuroradiol 2015;36:77–83 doi:10.3174/ajnr.A4077 pmid:25104288
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo3OiIzNi8xLzc3IjtzOjQ6ImF0b20iO3M6MTg6Ii9ham5yLzQyLzEvOTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

5.  5.Horbinski C. What do we know about IDH 1/2 mutations so far, and how do we use it? Acta Neuropathol 2013;125:621–36 doi:10.1007/s00401-013-1106-9 pmid:23512379
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00401-013-1106-9&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=23512379&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000317923400001&link_type=ISI) 

6.  6.Preusser M, Wöhrer A, Stary S, et al. Value and limitations of immunohistochemistry and gene sequencing for detection of the IDH 1-R132H mutation in diffuse glioma biopsy specimens. J Neuropathol Exp Neurol 2011;70:715–23 doi:10.1097/NEN.0b013e31822713f0 pmid:21760534
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1097/NEN.0b013e31822713f0&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=21760534&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

7.  7.Broen MPG, Smits M, Wijnenga MM, et al. The T2-FLAIR mismatch sign as an imaging marker for non-enhancing IDH-mutant, 1p/19q-intact lower-grade glioma: a validation study. Neuro Oncol 2018;20:1393–99 doi:10.1093/neuonc/noy048 pmid:29590424
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1093/neuonc/noy048&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=29590424&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

8.  8.Lasocki A, Gaillard F, Gorelik A, et al. MRI features can predict 1p/19q status in intracranial gliomas. AJNR Am J Neuroradiol 2018;39:687–92 doi:10.3174/ajnr.A5572 pmid:29519793
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIzOS80LzY4NyI7czo0OiJhdG9tIjtzOjE4OiIvYWpuci80Mi8xLzk0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

9.  9.Patel SH, Poisson LM, Brat DJ, et al. T2-FLAIR mismatch, an imaging biomarker for IDH and 1p/19q status in lower-grade gliomas: a TCGA/TCIA Project. Clin Cancer Res 2017;23:6078–85 doi:10.1158/1078-0432.CCR-17-0560 pmid:28751449
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImNsaW5jYW5yZXMiO3M6NToicmVzaWQiO3M6MTA6IjIzLzIwLzYwNzgiO3M6NDoiYXRvbSI7czoxODoiL2FqbnIvNDIvMS85NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

10. 10.Johnson DR, Diehn FE, Giannini C, et al. Genetically defined oligodendroglioma is characterized by indistinct tumor borders at MRI. AJNR Am J Neuroradiol 2017;38:678–84 doi:10.3174/ajnr.A5070 pmid:28126746
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo4OiIzOC80LzY3OCI7czo0OiJhdG9tIjtzOjE4OiIvYWpuci80Mi8xLzk0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

11. 11.Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures: they are data. Radiology 2016;278:563–77 doi:10.1148/radiol.2015151169 pmid:26579733
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1148/radiol.2015151169&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=26579733&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

12. 12.Lambin P, Leijenaar RT, Deist TM, et al. Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol 2017;14:749–62 doi:10.1038/nrclinonc.2017.141 pmid:28975929
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1038/nrclinonc.2017.141&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=28975929&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

13. 13.van den Bent MJ. Interobserver variation of the histopathological diagnosis in clinical trials on glioma: a clinician's perspective. Acta Neuropathol 2010;120:297–304 doi:10.1007/s00401-010-0725-7 pmid:20644945
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00401-010-0725-7&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=20644945&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000281038700004&link_type=ISI) 

14. 14.McInnes MD, Moher D, Thombs BD, et al; PRISMA-DTA Group. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: the PRISMA-DTA Statement. JAMA 2018;319:388–96 doi:10.1001/jama.2017.19163 pmid:29362800
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1001/jama.2017.19163&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=29362800&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

15. 15.Matsui Y, Maruyama T, Nitta M, et al. Prediction of lower-grade glioma molecular subtypes using deep learning. J Neurooncol 2020;146:321–27 doi:10.1007/s11060-019-03376-9 pmid:31865510
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s11060-019-03376-9&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31865510&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

16. 16.Zhou XH, McClish DK, Obuchowski NA, et al. Statistical Methods in Diagnostic Medicine. John Wiley & Sons; 2011
    
    

17. 17.Whiting PF, Rutjes AW, Westwood ME, et al; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155:529–36 doi:10.7326/0003-4819-155-8-201110180-00009 pmid:22007046
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1059/0003-4819-155-8-201110180-00009&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=22007046&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000296066300018&link_type=ISI) 

18. 18.Collins GS, Reitsma JB, Altman DG, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med 2015;162:735–36 doi:10.7326/L15-5093-2 pmid:25984857
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.7326/L15-5093&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=25984857&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

19. 19.McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22:276–82 doi:10.11613/BM.2012.031 pmid:23092060
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.11613/BM.2012.031&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=23092060&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000309525200005&link_type=ISI) 

20. 20.Ursprung S, Beer L, Bruining A, et al. Radiomics of computed tomography and magnetic resonance imaging in renal cell carcinoma-a systematic review and meta-analysis. Eur Radiol 2020;30:3558–66 doi:10.1007/s00330-020-06666-3 pmid:32060715
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00330-020-06666-3&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=32060715&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

21. 21.Park JE, Kim HS, Kim D, et al. A systematic review reporting quality of radiomics research in neuro-oncology: toward clinical utility and quality improvement using high-dimensional imaging features. BMC Cancer 2020;20:29 doi:10.1186/s12885-019-6504-5 pmid:31924170
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1186/s12885-019-6504-5&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31924170&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

22. 22.Fukuma R, Yanagisawa T, Kinoshita M, et al. Prediction of IDH and TERT promoter mutations in low-grade glioma from magnetic resonance images using a convolutional neural network. Sci Rep 2019;9:20311 doi:10.1038/s41598-019-56767-3 pmid:31889117
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1038/s41598-019-56767-3&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31889117&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

23. 23.Gihr GA, Horvath-Rizea D, Hekeler E, et al. Histogram analysis of diffusion weighted imaging in low-grade gliomas: in vivo characterization of tumor architecture and corresponding neuropathology. Front Oncol 2020;10:206 doi:10.3389/fonc.2020.00206 pmid:32158691
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.3389/fonc.2020.00206&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=32158691&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

24. 24.Jakola AS, Zhang YH, Skjulsvik AJ, et al. Quantitative texture analysis in the prediction of IDH status in low-grade gliomas. Clin Neurol Neurosurg 2018;164:114–20 doi:10.1016/j.clineuro.2017.12.007 pmid:29220731
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.clineuro.2017.12.007&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=29220731&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

25. 25.Kim M, Jung SY, Park JE, et al. Diffusion- and perfusion-weighted MRI radiomics model may predict isocitrate dehydrogenase (IDH) mutation and tumor aggressiveness in diffuse lower grade glioma. Eur Radiol 2020;30:2142–51 doi:10.1007/s00330-019-06548-3 pmid:31828414
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00330-019-06548-3&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31828414&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

26. 26.Kocak B, Durmaz ES, Ates E, et al. Radiogenomics of lower-grade gliomas: machine learning-based MRI texture analysis for predicting 1p/19q codeletion status. Eur Radiol 2020;30:877–86 doi:10.1007/s00330-019-06492-2 pmid:31691122
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00330-019-06492-2&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31691122&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

27. 27.Li Z, Wang Y, Yu J, et al. Deep Learning-Based Radiomics (DLR) and its usage in noninvasive IDH 1 prediction for low grade glioma. Sci Rep 2017;7:5467 doi:10.1038/s41598-017-05848-2 pmid:28710497
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1038/s41598-017-05848-2&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=28710497&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

28. 28.Lu CF, Hsu FT, Hsieh KLC, et al. Machine learning–based radiomics for molecular subtyping of gliomas. Clin Cancer Res 2018;24:4429–36 doi:10.1158/1078-0432.CCR-17-3445 pmid:29789422
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImNsaW5jYW5yZXMiO3M6NToicmVzaWQiO3M6MTA6IjI0LzE4LzQ0MjkiO3M6NDoiYXRvbSI7czoxODoiL2FqbnIvNDIvMS85NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

29. 29.Park CJ, Choi YS, Park YW, et al. Diffusion tensor imaging radiomics in lower-grade glioma: improving subtyping of isocitrate dehydrogenase mutation status. Neuroradiology 2020;62:319–26 doi:10.1007/s00234-019-02312-y pmid:31820065
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00234-019-02312-y&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31820065&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

30. 30.Ren Y, Zhang X, Rui W, et al. Noninvasive prediction of IDH 1 mutation and ATRX expression loss in low-grade gliomas using multiparametric MR radiomic features. J Magn Reson Imaging 2019;49:808–17 doi:10.1002/jmri.26240 pmid:30194745
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1002/jmri.26240&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=30194745&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

31. 31.Shofty B, Artzi M, Ben Bashat D, et al. MRI radiomics analysis of molecular alterations in low-grade gliomas. Int J Comput Assist Radiol Surg 2018;13:563–71 doi:10.1007/s11548-017-1691-5 pmid:29270916
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s11548-017-1691-5&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=29270916&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

32. 32.Yu J, Shi Z, Lian Y, et al. Noninvasive IDH 1 mutation estimation based on a quantitative radiomics approach for grade II glioma. Eur Radiol 2017;27:3509–22 doi:10.1007/s00330-016-4653-3 pmid:28004160
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00330-016-4653-3&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=28004160&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

33. 33.Zhang X, Tian Q, Wang L, et al. Radiomics strategy for molecular subtype stratification of lower-grade glioma: detecting IDH and TP53 mutations based on multimodal MRI. J Magn Reson Imaging 2018;48:916–26 doi:10.1002/jmri.25960 pmid:29394005
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1002/jmri.25960&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=29394005&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

34. 34.Zhou H, Vallieres M, Bai H, et al. MRI features predict survival and molecular markers in diffuse lower-grade gliomas. Neuro Oncol 2017;19:862–70 doi:10.1093/neuonc/now256 pmid:28339588
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1093/neuonc/now256&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=28339588&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

35. 35.Han Y, Wang W, Yang Y, et al. Amide proton transfer imaging in predicting isocitrate dehydrogenase 1 mutation status of grade II/III gliomas based on support vector machine. Front Neurosci 2020;14:144 doi:10.3389/fnins.2020.00144 pmid:32153362
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.3389/fnins.2020.00144&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=32153362&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

36. 36.Bakas S, Akbari H, Sotiras A, et al. Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci Data 2017;4:170117 doi:10.1038/sdata.2017.117 pmid:28872634
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1038/sdata.2017.117&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=28872634&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

37. 37.Bakas S, Reyes M, Jakab A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. 2018 [https://arxiv.org/abs/1811.02629](https://arxiv.org/abs/1811.02629). Accessed April 18, 2020
    
    

38. 38.Menze BH, Jakab A, Bauer S, et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Trans Med Imaging 2015;34:1993–2024 doi:10.1109/TMI.2014.2377694 pmid:25494501
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1109/TMI.2014.2377694&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=25494501&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

39. 39.Myronenko A. 3D MRI brain tumor segmentation using autoencoder regularization. International MICCAI Brainlesion Workshop: Springer; 2018:311–20
    
    

40. 40.Zhang Z, Xiao J, Wu S, et al. Deep convolutional radiomic features on diffusion tensor images for classification of glioma grades. J Digit Imaging.2020;33:826–37 doi:10.1007/s10278-020-00322-4 pmid:32040669
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s10278-020-00322-4&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=32040669&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

41. 41.Higgins JP, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons; 2019
    
    

42. 42.van der Voort SR, Incekara F, Wijnenga MM, et al. Predicting the 1p/19q codeletion status of presumed low-grade glioma with an externally validated machine learning algorithm. Clin Cancer Res 2019;25:7455–62 doi:10.1158/1078-0432.CCR-19-1127 pmid:31548344
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImNsaW5jYW5yZXMiO3M6NToicmVzaWQiO3M6MTA6IjI1LzI0Lzc0NTUiO3M6NDoiYXRvbSI7czoxODoiL2FqbnIvNDIvMS85NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

43. 43.Delfanti RL, Piccioni DE, Handwerker J, et al. Imaging correlates for the 2016 update on WHO classification of grade II/III gliomas: implications for IDH, 1p/19q and ATRX status. J Neurooncol 2017;135:601–09 doi:10.1007/s11060-017-2613-7 pmid:28871469
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s11060-017-2613-7&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=28871469&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

44. 44.Zhou Q, Cao YH, Chen ZH. Lack of evidence and criteria to evaluate artificial intelligence and radiomics tools to be implemented in clinical settings. Eur J Nucl Med Mol Imaging 2019;46:2812–13 doi:10.1007/s00259-019-04493-3 pmid:31696246
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00259-019-04493-3&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31696246&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

45. 45.Lew C. The promise of clinical AI: an adaptive future. Applied Radiol 2018;47:3–6 [https://www.appliedradiology.com/articles/the-promise-of-clinical-ai-an-adaptive-future#:∼:text=The%20Promise%20of%20Clinical%20AI%3A%20An%20Adaptive%20Future.,predictive%20analytics%20for%20population%20health%20management%2C%20and%20beyond](https://www.appliedradiology.com/articles/the-promise-of-clinical-ai-an-adaptive-future)[.](http://. ) Accessed April 18, 2020
    
    

46. 46.Sana M. Machine learning and artificial intelligence in radiology. J Am Coll Radiol 2018;15:1139–42 doi:10.1016/j.jacr.2017.11.015 pmid:29275917
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.jacr.2017.11.015&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=29275917&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

47. 47.Shader RI. Some thoughts about artificial intelligence. Clin Ther 2019;41:1401–03 doi:10.1016/j.clinthera.2019.06.016 pmid:31345558
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.clinthera.2019.06.016&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31345558&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

48. 48.Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019;25:44–56 doi:10.1038/s41591-018-0300-7 pmid:30617339
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1038/s41591-018-0300-7&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=30617339&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

49. 49.European Society of Radiology. Impact of artificial intelligence on radiology: a EuroAIM survey among members of the European Society of Radiology. Insights imaging 2019;10:105 doi:10.1186/s13244-019-0798-3 pmid:31673823
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1186/s13244-019-0798-3&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31673823&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

50. 50.van Hoek J, Huber A, Leichtle A, et al. A survey on the future of radiology among radiologists, medical students and surgeons: students and surgeons tend to be more skeptical about artificial intelligence and radiologists may fear that other disciplines take over. Eur J Radiol 2019;121:108742 doi:10.1016/j.ejrad.2019.108742 pmid:31734640
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.ejrad.2019.108742&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31734640&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

51. 51.Waymel Q, Badr S, Demondion X, et al. Impact of the rise of artificial intelligence in radiology: what do radiologists think? Diagn Interv Imaging 2019;100:327–36 doi:10.1016/j.diii.2019.03.015 pmid:31072803
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.diii.2019.03.015&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=31072803&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

52. 52.Palmisciano P, Jamjoom AA, Taylor D, et al. Attitudes of patients and their relatives toward artificial intelligence in neurosurgery. World Neurosurg 2020;138:e627–33 doi:10.1016/j.wneu.2020.03.029 pmid:32179185
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.wneu.2020.03.029&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=32179185&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

53. 53.Likert R. A technique for the measurement of attitudes. Archives of Psychology 1932;22:55
    
    

54. 54.Ongena YP, Haan M, Yakar D, et al. Patients’ views on the implementation of artificial intelligence in radiology: development and validation of a standardized questionnaire. Eur Radiol 2020;30:1033–40 doi:10.1007/s00330-019-06486-0 pmid:31705254
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s00330-019-06486-0&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

55. 55.Janssen CP, Donker SF, Brumby DP, et al. History and future of human-automation interaction. International Journal of Human-Computer Studies 2019;131:99–107 doi:10.1016/j.ijhcs.2019.05.006
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.ijhcs.2019.05.006&link_type=DOI) 

56. 56.IBM Services. Beyond the hype: a guide to understanding and successfully implementing artificial intelligence within your business. [https://www.ibm.com/downloads/cas/8ZDXNKQ4.](https://www.ibm.com/downloads/cas/8ZDXNKQ4. ) 2018. Accessed April 18, 2020
    
    

57. 57.Amershi S, Weld D, Vorvoreanu M, et al. Guidelines for human-AI interaction. In: Proceedings of the Cultural Homestay International Conference on Human Factors in Computing System, Glasgow, UK; May 4–9, 2019
    
    

58. 58.Friedman LM. Furberg CD, DeMets Dl, et al. Fundamentals of Clinical Trials. Springer-Verlag; 2010
    
    

59. 59.He J, Baxter SL, Xu J, et al. The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019;25:30–36 doi:10.1038/s41591-018-0307-0 pmid:30617336
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1038/s41591-018-0307-0&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=30617336&link_type=MED&atom=%2Fajnr%2F42%2F1%2F94.atom) 

*   Received May 2, 2020.
*   Accepted after revision August 17, 2020.


*   © 2021 by American Journal of Neuroradiology