Diagnosis of Recanalization of the Intracranial Artery Has Poor Inter-Rater Reliability

M. Bar; R. Mikulik; T. Jonszta; A. Krajina; M. Roubec; D. Skoloudik; V. Prochazka

doi:10.3174/ajnr.A2896

Abstract

BACKGROUND AND PURPOSE: Recanalization is the important outcome measure for acute stroke therapy. Several methods of recanalization assessment are used in clinical practice, but few studies have addressed their reliability. We, therefore, sought to assess interobserver reliability of the diagnosis of intracranial artery recanalization following intervention by using TIMI criteria.

MATERIALS AND METHODS: The digital angiography scans of all patients with acute ischemic stroke during 2009 undergoing DSA and endovascular procedures at Ostrava University Hospital were assessed in the study. Images were retrospectively evaluated for intracranial artery recanalization on the TIMI scale by 2 experienced neuroradiologists who were blinded to clinical findings and to each other.

RESULTS: The angiography scans of 43 patients (16 females; age, 70.5 ± 14 years; median baseline NIHSS score, 15 [IQR, 11–18]) were retrospectively evaluated in our study. At 3 months, 27% of patients had mRS scores ≤ 2 and mortality was 18%. Two radiologists diagnosed TIMI grades as follows: TIMI 0, 16%, and 16%; TIMI 1, 21%, and 8%; TIMI 2a, 32% and 29%; TIMI 2b, 13% and 16%; TIMI 3, 18, and 31%. Interobserver agreement for recanalization was weighted κ = 0.4 (95% CI, 0.2–0.6).

CONCLUSIONS: The diagnosis of recanalization after interventional procedures was found to have poor interobserver agreement between 2 experienced neuroradiologists. TIMI criteria, therefore, do not permit reliable comparison of the efficacy of recanalization therapy among different studies.

ABBREVIATIONS:

CI: confidence interval
IQR: interquartile range
IVT: intravenous thrombolysis
mRS: modified Rankin Scale
TICI: thrombolysis in cerebral infarction
TIMI: thrombolysis in myocardial infarction

Occlusion of cervical or intracranial arteries is the most common cause of acute ischemic stroke.¹ Recanalization improves outcome after stroke and is, therefore, considered an important surrogate outcome measure for the efficacy of therapeutic intervention in patients with acute ischemic stroke.^1⇓⇓–4 However, there is, at present, no consensus on how intracranial artery recanalization should be evaluated. Various methods of assessment used in clinical practice include TICI and TIMI scales and their variants, but only limited data are available concerning their reliability.^5⇓–7 The aim of the present study was to evaluate interobserver reliability of assessment by using TIMI criteria for intracranial artery recanalization following intervention.

Materials and Methods

All patients with acute ischemic stroke during 2009 undergoing DSA and endovascular procedures at Ostrava University Hospital were enrolled in the study. Eligibility criteria for the endovascular procedure were as follows: time of ≤8 hours from stroke onset, moderate or severe neurologic deficit (NIHSS score, >8), occlusion of a major intracranial artery on CT angiography, and failure of intravenous thrombolysis (nonrecanalization on transcranial Doppler or no clinical improvement after 30 minutes of intravenous thrombolysis infusion).

We recorded and evaluated demographic and clinical data: age, sex, time from symptom onset to the endovascular procedure, baseline NIHSS score, type of recanalization procedure, results of nonenhanced CT and CT angiography scans before the procedure and nonenhanced CT scans after the procedure, and mRS at 3 months after stroke.

DSA images were recorded according to a standard protocol. The baseline and final DSA images were retrospectively evaluated by 2 neuroradiologists for the site of intracranial artery occlusion and for the presence or absence of arterial recanalization. Both neuroradiologists were experienced in performing mechanical thrombectomy procedures, but they did not pass a certified training in TIMI (TICI) classification before the study.

The 2 neuroradiologists were blinded to the clinical findings and to each other. Recanalization of the intracranial artery was assessed by using modified TIMI classification³ (Table 1).

View this table:

Table 1:

Modified TIMI classification³

Data Analyses and Statistical Analyses

Interobserver agreement was assessed by using linear weighted κ statistics. Linear weighting here denotes that the same weights were assigned to the difference between adjacent TIMI categories (eg, TIMI 0 versus 1 or TIMI 2 versus 3). Agreement was considered poor with κ <0.4, good with 0.41–0.75, and excellent with >0.75.

Results

A total of 43 patients were considered candidates for intervention. Of these, 1 patient was excluded because no recanalization therapy was performed after the initial angiography. A further 4 patients were excluded as a result of errors in transferring images to the radiologist located in another hospital via the PACS. The scans of the remaining 38 patients were analyzed. Detailed patient characteristics are shown in Table 2. Eleven (27%) patients had mRS ≤ 2 at 3 months.

View this table:

Table 2:

Patient characteristics

Two radiologists diagnosed TIMI grades as follows: TIMI 0 16% and 16%; TIMI 1, 21% and 8%; TIMI 2a, 32% and 29%; TIMI 2b, 13% and 16%; and TIMI 3, 18% and 31%. Interobserver agreement for recanalization was weighted κ = 0.4 (95% CI, 0.2–0.6) (Table 3).

View this table:

Table 3:

Interobserver results of recanalization on a modified TIMI scale: cross-tabulation

For patients with MCA versus non-MCA stroke, the weighted κ was 0.39 (95% CI, 0.2–0.6) and 0.45 (95% CI, 0.2–0.9), respectively.

Discussion

In this study, we report that there was poor agreement in the assessment of recanalization on the TIMI scale by 2 experienced radiologists. Our results, therefore, challenge the reproducibility of recanalization assessment after interventional procedures. As a consequence, it is likely to be difficult or even impossible to compare results based on TIMI scores between different clinical trials, recanalization devices, or case series. Furthermore, our results could explain, in part, why the association between recanalization and outcome differs substantially among different studies.^{2⇓⇓⇓–6}

Few studies have addressed the reproducibility of recanalization assessment following intervention. The Interventional Management of Stroke II (IMS II) study³ reported that 41% of evaluations of recanalization by TIMI were discordant, mostly due to over-rating of the TIMI score by the clinical site and under-rating by the core laboratory. In the IMS I study, TIMI and arterial occlusive lesion scores had only modest agreement. It has become common practice to compare revascularization trials with historical controls. For these comparisons to be valid, we must use the same revascularization end points.⁸

Several factors could explain the low reproducibility of TIMI scores between different observers. First, the TIMI scale was initially developed for the assessment of reperfusion in patients with myocardial infarction; it is possible that the scale is not well-suited for the assessment of recanalization in stroke. Second, neuroradiologists are insufficiently trained in TIMI grading. Third, radiologists who do not themselves perform the procedure might be less confident in interpreting the images—this feature could explain the most discrepant readings (eg, TIMI 3 or TIMI 2a versus 0, which occurred in 2 cases). Finally, the utility of TIMI could differ between different intracranial territories, though our data suggest that the inconsistency of interobserver agreement did not differ for MCA and non-MCA territories.

A major limitation of this study is its retrospective design; potentially this could have affected the quality and standardization of the images. Our study was also performed at a single center with a limited number of patients.

Conclusions

Interobserver agreement in the evaluation of intracranial artery recanalization following endovascular treatment in our study was poor. A more reliable tool for the assessment of recanalization in stroke is needed. The process of evaluation of recanalization must be unified, and discrepancies in the recanalization rate should be decreased by assessment images in a core laboratory. Training programs for improving the evaluation of recanalization should be established.

Acknowledgments

We thank Dana Salounova for assistance with statistics.

Footnotes

Disclosures: Michal Bar—RELATED: Support for Travel to Meetings for the Study or Other Purposes: Pfizer, Comments: registration fee to European Stroke Conference in Hamburg, 2011. Antonin Krajina—UNRELATED: Grants/Grants Pending: Czech Healthcare Ministry Grant Agency.
Paper previously presented at: European Stroke Conference, May 24–27, 2011; Hamburg, Germany.

Indicates open access to non-subscribers at www.ajnr.org

References

1.↵
1. Rha JH,
2. Saver JL
. The impact of recanalization on ischemic stroke outcome: a meta- analysis. Stroke 2007;38:967–73
Abstract/FREE Full Text
2.↵
1. Smith WS,
2. Sung G,
3. Starkman S,
4. et al
. Safety and efficacy of mechanical embolectomy in acute ischemic stroke: results of the MERCI trial. Stroke 2005;36:1432–38
Abstract/FREE Full Text
3.↵
The IMS II Trial Investigators. The Interventional Management of Stroke (IMS) II study. Stroke 2007;38:2127–35. Epub 2007 May 24
Abstract/FREE Full Text
4.↵
The Penumbra Pivotal Stroke Trial Investigators. The Penumbra Pivotal Stroke trial: safety and effectiveness of a new generation of mechanical devices for clot removal in intracranial large vessel occlusive disease. Stroke 2009;40:2761–68. Epub 2009 Jul 9
Abstract/FREE Full Text
5.↵
1. Tomsick T
. TIMI, TIBI, TICI: I came, I saw, I got confused. AJNR Am J Neuroradiol 2007;28:382–84
FREE Full Text
6.↵
1. Schellinger PD,
2. Hacke W
. Recanalization devices should be restricted to clinical trials: pro (kind of). Stroke 2010;41:191–93
FREE Full Text
7.↵
1. Qureshi AI
. New grading system for angiographic evaluation of arterial occlusions and recanalization response to intra-arterial thrombolysis in acute ischemic stroke. Neurosurgery 2001;50:1405–15
8.↵
1. Katri P,
2. Neff J,
3. Broderick J,
4. et al
. Revascularization end points in stroke interventional trials: recanalization versus reperfusion in IMS-I. Stroke. 2005;36:2400–03
Abstract/FREE Full Text

Received June 23, 2011.
Accepted after revision August 19, 2011.

In this issue

Download PDF

Email Article

Citation Tools

Purchase

Cited By...

More in this TOC Section

Show more INTERVENTIONAL

[1] 1.↵
Rha JH,
Saver JL
. The impact of recanalization on ischemic stroke outcome: a meta- analysis. Stroke 2007;38:967–73
Abstract/FREE Full Text

[2] Rha JH,

[3] Saver JL

[4] 2.↵
Smith WS,
Sung G,
Starkman S,
et al
. Safety and efficacy of mechanical embolectomy in acute ischemic stroke: results of the MERCI trial. Stroke 2005;36:1432–38
Abstract/FREE Full Text

[5] Smith WS,

[6] Sung G,

[7] Starkman S,

[8] et al

[9] 3.↵
The IMS II Trial Investigators. The Interventional Management of Stroke (IMS) II study. Stroke 2007;38:2127–35. Epub 2007 May 24
Abstract/FREE Full Text

[10] 4.↵
The Penumbra Pivotal Stroke Trial Investigators. The Penumbra Pivotal Stroke trial: safety and effectiveness of a new generation of mechanical devices for clot removal in intracranial large vessel occlusive disease. Stroke 2009;40:2761–68. Epub 2009 Jul 9
Abstract/FREE Full Text

[11] 5.↵
Tomsick T
. TIMI, TIBI, TICI: I came, I saw, I got confused. AJNR Am J Neuroradiol 2007;28:382–84
FREE Full Text

[12] Tomsick T

[13] 6.↵
Schellinger PD,
Hacke W
. Recanalization devices should be restricted to clinical trials: pro (kind of). Stroke 2010;41:191–93
FREE Full Text

[14] Schellinger PD,

[15] Hacke W

[16] 7.↵
Qureshi AI
. New grading system for angiographic evaluation of arterial occlusions and recanalization response to intra-arterial thrombolysis in acute ischemic stroke. Neurosurgery 2001;50:1405–15

[17] Qureshi AI

[18] 8.↵
Katri P,
Neff J,
Broderick J,
et al
. Revascularization end points in stroke interventional trials: recanalization versus reperfusion in IMS-I. Stroke. 2005;36:2400–03
Abstract/FREE Full Text

[19] Katri P,

[20] Neff J,

[21] Broderick J,

[22] et al

Main menu

User menu

Search

American Journal of Neuroradiology

Diagnosis of Recanalization of the Intracranial Artery Has Poor Inter-Rater Reliability

Abstract

ABBREVIATIONS: