Clinical relevance of the effects of robotic rehabilitation for upper limb recovery after stroke in randomized studies: a systematic review with meta-analysis

Abstract

Introduction: Recent randomized clinical trials (RCTs) provide evidence on the effects of robot-assisted training (RAT) for upper limb impairments in stroke subjects; however, evidence on the clinical relevance of these differences is lacking. This study aimed to perform a systematic review with meta-analyses of RCTs on clinical relevance, expressed as minimal clinically important difference (MCID), of RAT to improve independence in activities of daily living, arm function, and impairments in patients with stroke.
Methods: Four databases were searched. RCTs investigating RAT aimed at recovering motor and functional skills of the upper limb in adult post-stroke patients were included. MCID values were retrieved from specific databases. Two independent reviewers performed screening, data extraction, and assessment of methodological quality. Meta-analyses for both statistical significance and clinical relevance were performed. Clinical relevance was expressed as a standardized MCID overall score (SMOS) for each outcome measure, calculated as the difference between mean outcome measures in experimental and control groups divided by corresponding MCID, when available.
Results: Eighty-five studies were included. Conventional meta-analyses showed that RAT, compared to control, had significant effects in the domains of activities of daily living, dexterity, arm function, and strength, but not on pain. Meta-analyses for clinical relevance reported non-clinically relevant differences between groups for all domains.
Conclusion: RAT produces some significant improvements for the upper limb, but these differences are not clinically relevant when compared to other therapies. Improvements in using the RAT in clinical practice may not be more clinically relevant than other therapies for stroke patients.

Introduction

Randomized clinical trials (RCTs) are intervention studies that apply an experimental design to establish causal relationships between interventions and outcomes. In an RCT, two or more groups are compared, and differences among them are reported in terms of statistical significance or estimating the effect size. However, small outcome changes that are statistically significant may lack clinical relevance. Statistical significance means that two events, represented by two variables, occur jointly more often than what can be anticipated only by the effect of the chance (1), possibly – but not necessarily – because they are causally related. The effect size is used as an indicator of the magnitude of the effect of an independent variable on the dependent one.

However, to judge whether one treatment is more effective than another, simply knowing whether a difference exists is not enough; we need to know how big the difference is (2).

From a clinical perspective, it is more useful to report the results of an RCT in terms of their clinical relevance, that is, changes that patients acknowledge as beneficial or harmful (3).

The perception of relevant changes can be estimated within or across individuals or groups (4); while the Minimal Important Change (MIC) refers to the smallest change in health status perceivable by patients compared to a baseline, the minimal clinically important difference (MCID) represents the smallest difference between groups that is considered by patients to be clinically relevant (5).

Upper limb impairments are frequent after a stroke, causing activity limitations, participation restrictions, and poor quality of life (6). Previous studies showed that robot-assisted training (RAT), compared to other therapies, is associated with statistically significant improvement in upper limb motor function recovery after stroke (7,8). On the other hand, more recent findings reported no statistically significant differences between RAT and conventional therapy (9-11).

Despite a growing body of evidence on RAT effects, the clinical relevance of this treatment approach has never been synthesized within systematic reviews. Therefore, this systematic review aimed to assess the clinical relevance of RAT, expressed as MCID, in improving activities of daily living, arm function, and upper limb impairments in patients recovering after a stroke, compared to control treatments.

Methods

Study design

The study protocol was a priori registered on the PROSPERO database (Number: CRD42023387467). This systematic review was reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (12) (Supplemental Material 1).

Data sources and searches

To identify eligible studies, systematic literature searches were conducted on MEDLINE (via PubMed), EMBASE, CINAHL, and PEDro databases from their inception through February 2023. The following search strings were used for MEDLINE, EMBASE, and CINHAL: (stroke OR CVA OR cerebrovascular accident) AND (upper limb OR arm OR upper extremity OR hand) AND (robot* OR robotics OR electro-mechanic OR exoskeleton device) AND (RCT OR randomized controlled trial OR clinical trial OR controlled trial). For PEDro searches, the following keywords were used: stroke AND robotics, with “clinical trial” as a filter. Moreover, we checked the reference list of the included and related articles to identify other relevant publications (cross-referencing).

Search strategy for clinical relevance (MCID)

For each assessment tool used in the RCTs retrieved, the value of MCID was searched in the Rehabilitation Measures Database, provided by the National Institute on Disability, Independent Living, and the Rehabilitation Research Shirley Ryan AbilityLab (https://www.sralab.org/rehabilitation-measures, last access, March 2023) and Stroke Engine (https://strokengine.ca/en/, last access: March 2023) using the term “assessment” and applying a search restriction to the “stroke severity” field.

When the MCID value of a measurement instrument was not available, we carried out further research on PubMed, Google Scholar, and EMBASE, from their inception through February 2023, using the following keywords: ((MCID OR “Minimal clinically important difference”) AND stroke AND “name of the outcome measure”).

MCID values calculated in post-stroke populations were preferred; when they were unavailable, the MCID values estimated on different patient populations were also considered. When different MCID values were obtained with the same method on the same population, the lowest MCID value criterion was chosen.

Study selection and eligibility

Articles published in English or Italian were included in this systematic review if they reported findings from RCTs assessing the effects of RAT on motor and functional skills of the upper limb in adult post-stroke patients. Conference proceedings and studies that administered RAT to participants in the control group were excluded. No time restriction was applied.

Two authors (AU and SV) independently screened and assessed the studies retrieved for inclusion with the support of dedicated software (Rayyan®, Qatar Computing Research Institute. Qatar; www.rayyan.ai). First, duplicated records were removed; then, AU and SV excluded non-pertinent articles based on title and abstract; finally, the full texts were retrieved and assessed for definitive inclusion. Reasons for exclusion during the full-text reading were given for each study excluded. Any disagreements regarding inclusion were resolved through the involvement of a third researcher (MP).

Assessment of the risk of bias

The methodological quality of the trials included was assessed with the PEDro score (13), which was extrapolated from the PEDro database. When the PEDro score was unavailable, it was calculated independently by two authors (LP, MP), with the involvement of a third researcher (SV) in case of disagreement. This score includes eleven items; each scored as 0 (criterion not satisfied or unreported) or 1 point (criterion satisfied); because the eligibility criterion is not included in the final score, the total score ranges from 0 (worse methodological quality) to 10 points (highest methodological quality).

FIGURE 1 -. Interpretation of the meta-analysis for clinical relevance.

Data extraction

Data were extrapolated independently by two authors (SV and AU), involving a third author (MP) in case of disagreement. The following data were extracted: author, year of publication, outcome measures with related statistics (i.e., sample size, mean, standard deviation (SD), median, inter-quartile range, minimum and maximum values, both at baseline and post-treatment), type of intervention in the experimental and control groups, duration of the study and the timing of the first follow-up. In the case of cross-over trials, the data from the first period only (i.e., before the crossing-over) was used.

Statistical analysis

Quantitative data synthesis was performed; in the meta-analyses, articles for which the MCIDs of the outcome measures used were not available were excluded.

Preliminarily, manipulation of the original outcome measures was sometimes required, as higher scores indicated better performance in some tools and worse performance in others. Thus, the scores were reversed in the latter to express all the results in the same direction of improving performance with increasing scores.

To obtain a traditional meta-analysis, the findings of each study were expressed in terms of standardized mean difference (SMD), with the corresponding 95% confidence interval (CI), and combined using both fixed effects and random effects models (Restricted Estimator Maximum Likelihood method).

The procedure to assess the clinical relevance of each study and to obtain an estimate of the pooled effect can be described as follows. In any given article, when an MCID value was available for the main outcome measure, the clinical relevance of the original findings was expressed as a standardized MCID overall score (SMOS), calculated as the difference between the mean of the outcome measures in the experimental and control groups divided by the corresponding MCID. The 95% CI associated with the SMOS point estimate was calculated using a pooled SD normalized by the MCID of the corresponding outcome measure. From the individual studies, the pooled estimate of the SMOS and the corresponding 95% CI were calculated, again using fixed and random effects models. For both the individual study and the pooled estimates, the interpretation of the SMOS is as follows (Figure 1): a) when the SMOS and the lower limit of its 95% CI are one or greater, the advantage offered by the experimental intervention over the control would reach or exceed the MCID and, therefore, the result would credibly be of clinical relevance; b) when the SMOS is one or greater but the lower limit of its 95% CI is not, the advantage offered by the experimental intervention would not reach an acceptable level of certainty in terms of clinical relevance; (c) when the SMOS point estimate is below 1, the experimental intervention would not offer any clinically relevant advantage. The interpretation of SMOS point estimate negative values and 95% CI would be similar but in favor of the control intervention. A detailed description of the calculations used to study clinical relevance with examples is provided in Supplementary Material 2.

All the results obtained were summarized in tables with forest plots.¹² Data analysis was performed with R, a free software environment for statistics, using the “meta” package ¹³; in particular, the “metacont” function was used for the analysis of continuous data.

Statistical heterogeneity between eligible studies was assessed using the I² statistic and the Cochran Q test (χ²) (14). The I² statistic was interpreted according to the following guide(14): 0-40% = not important statistical heterogeneity; 30-60% = moderate heterogeneity; 50-75% = substantial heterogeneity; 75-100% = considerable heterogeneity.

In case of heterogeneity, subgroup analyses were performed, grouping studies based on involved upper extremity segments, classified as proximal, distal, or both, and based on the robotic types, coded as unilateral, bilateral, or both. Additional sub-analyses were conducted by grouping studies based on the therapy content of control groups. Finally, a meta-regression using the duration of interventions as an independent factor was performed.

Results

Study selection

After removing duplicates, 5,302 articles were obtained. After reviewing the abstract and title, 5,190 studies were excluded for failing to meet the inclusion criteria, leaving 112 articles for full-text screening. After excluding 28 other articles, eighty-five RCTs were included in the systematic review. A list of the included articles is reported in Supplemental Material 3. Supplemental Material 4 reports a list of the studies excluded after full-text reading (during the study selection and cross-reference phases) with reasons for exclusion.

Since not all MCID values were found for all outcome measures or their subscales, seven RCTs were excluded, and 78 studies were included in the meta-analyses. The study selection process is shown in Figure 2.

MCID selection

MCID values of all measures were found except nine. All retrieved MCID values were referred to stroke populations, except for six outcome measures: QuickDASH (Disability of the Arm, Shoulder, and Hand) and Numeric Rating Pain Scale (NPRS) were referred to musculoskeletal disorders, Medical Research Council, Ashworth Scale, and Modified Ashworth Scale to neurological population other than stroke and hand-grip strength to the heterogeneous population.

FIGURE 2 -. Flow chart of the study procedure.

For the analysis, outcome measures were pooled according to six domains (activities of daily living, arm function, muscle strength, dexterity, muscle tone, and pain). MCID values of scales assessing quality of life were not found. Domains, MCID values, and related references are reported in Supplemental Material 5. The classification of outcomes is based on the review by Mehrholz et al. (7). The three original outcomes were expanded within the domain of impairments, including dexterity, muscle tone, and pain, since additional outcomes were found in the included trials.

Characteristics of the included studies

Detailed characteristics of the included studies are reported in Table 1. In the systematic review, 3452 patients with stroke were included, with a mean±SD number per study of 40.6±40.8 (range: 11, 224). The experimental treatments included a high variability of devices, while the most commonly used control intervention was defined as “conventional physiotherapy” or equivalent terms (n = 60), followed by occupational therapy (n = 11). The included studies used 77 different outcome measures, the most commonly used of which was the Fugl-Meyer Assessment. The treatment duration was 45.2±25.8 days (range: 12, 180).

TABLE 1 -. Characteristics of the included studies

Study	N	Experimental group	Control group	Outcome	Treatment duration*
Abdullah et al. 2011	20	Name not available	Conventional physiotherapy	Likert scale	70
Aisen et al. 1997	20	MIT - MANUS	Conventional physiotherapy	MSS, FIM, FMA (UE)	14
Ang et al. 2014	14	Haptic Knob	Conventional physiotherapy	FMA (UE)	12
Aprile et al. 2020	224	Motore Humanware, Amadeo Tyromotion, Pablo Tyromotion	Conventional physiotherapy	NRPS, FMA (UE), BI, MI	42
Bayindir et al. 2022	32	HandTutor	Conventional physiotherapy	FMA (UE), BBT, NHPT	35
Brokaw et al. 2014	11	ARMin III HandSOME	Conventional physiotherapy	ARAT	30
Budhota et al. 2021	44	H-Man	Conventional physiotherapy	Grip, FMA (UE), ARAT	42
Burgar et al. 2011	35	MIME	Conventional physiotherapy	Ashworth, WMFT (FAS), FIM, FMA, MP	60
Bustamante Valles et al. 2016	20	TheraDrive, NESS H200, NESS L300	Conventional physiotherapy and occupational therapy	BBT, FMA (UE)	49
Calabrò et al. 2019	50	Amadeo	Conventional physiotherapy	FMA (UE), NHPT	56
Cameirao et al. 2011	16	Rehabilitation Gaming System	Occupational therapy or non-specific interactive games	FMA (H/W), FMA (Arm), BI, FMA (UE), CAHAI, MI	84
Carpinella et al. 2020	40	Braccio di Ferro	Conventional physiotherapy	FIM, FMA (UE)	28
Chen et al. 2021	20	Name not available	Conventional physiotherapy	FMA (H/W), FMA (Arm), FMA (UE), ARAT, BI	28
Chen et al. 2022	31	The Hand of hope	Conventional physiotherapy	MAL (QOM), MAL (AOU), WMFT (TIME), WMFT (FAS), FMA (UE), ARAT	28
Chinembiri et al. 2020	45	The Fourier M2 machine	Occupational therapy	BI, FMA (UE), FMA (Arm)	42
Connelly et al. 2010	16	PneuGlove	Conventional physiotherapy	FMA (H/W), FMA (UE), BBT, Pinch, Grip	42
Conroy et al. 2011	41	InMotion 2.0	Intensive conventional arm exercise program	SIS (Mobility), SIS (ADL), SIS (Hand), FMA (UE), WMFT	42
Coskunsu et al. 2022	24	Hand of hope	Conventional physiotherapy	MAL (QOM), MAL (AOU), ARAT	21
Daly et al. 2005	12	InMotion2	Functional neuromuscular stimulation and motor learning	FMA	84
Daunoraviciene et al. 2018	34	Armeo Spring	Occupational therapy	HAD, FIM, ACE-R, FMA (UE)	84
De Araújo et al. 2011	12	Name not available	Conventional physiotherapy	MAS (Elbow), MAS (W/H), FMA (H/W), FMA (Arm), FMA (UE)	56
Dehem et al. 2019	45	REAplan1	Conventional physiotherapy	BBT, ABILHAND, Activlim, WMFT (FAS), SIS (ADL), FMA (UE)	63
Fasoli et al. 2004	56	MIT-MANUS	Conventional physiotherapy	MRC, FIM (Self-care), FIM, FMA (UE)	49
Fazekas et al. 2007	30	REHAROB	Bobath	Rivermead arm score, MAS (Elbow), FMA (Arm), ROM, FIM (Self Care)	20
Franceschini et al. 2019	48	InMotion2	Conventional physiotherapy	MAS (Shoulder), MAS (Elbow), FMA (UE), pROM	90
Frisoli et al. 2022	26	L-EXOS	Conventional physiotherapy	FMA (W/H), FMA (Arm), FMA (UE), MAS	42
Gandolfi et al. 2019	32	Armotion	Conventional physiotherapy + botulinum toxin	FMA (UE)	35
Grigoras et al. 2016	25	Hybrid glove	Conventional physiotherapy	SIS (ADL), FMA (H/W), BBT, FMA (UE), FMA (Arm)	21
Gueye et al. 2021	50	Armeo Spring	Standard rehabilitation program + conventional physiotherapy	MoCA, FIM, FMA (UE)	21
Hesse et al. 2005	44	Bi-Manu-Track	Conventional physiotherapy + electromyography electrical stimulation	MAS, MRC, FMA (UE)	42
Hesse et al. 2014	50	Bi-Manu-Treck, RehaDigit, Reha Slide, Reha Slide duo	Double sessions of individual arm therapy	BBT, MAS, MRC, ARAT, FMA (UE), BI	28
Housman et al. 2009	34	T-WREX	Conventional exercises	MAL (AOU), MAL (QOM), Grip, FMA (UE)	42
Hsieh et al. 2011	12	Bi-Manu-Track	Conventional physiotherapy	ABILHAND, MAL (QOM), MAL (AOU), MRC, FMA (UE)	30
Hsieh et al. 2012	36	Bi-Manu-Track	Intensive therapist-administered control therapy	FMA (H/W), FMA (Arm), FMA (UE)	30
Hsieh et al. 2014	32	Bi-Manu-Track	Conventional physiotherapy	MAL (AOU), MAL (QOM), WMFT (FAS), FMA (H/W), FMA (Arm), FMA (UE)	30
Hsieh et al. 2017	31	Bi- Manu- Track	Task-oriented approach	MRS, Grip, BBT, SIS (strength), SIS (ADL), SIS (Mobility), SIS (Hand), FMA (UE), FIM	28
Hsiu et al. 2019	43	Bi-Manu-Track	Usual care	FMA (Arm), MAL (QOM), MAL (AOU), FMA (UE)	28
Hsiu et al. 2021	34	TIGER	Occupational therapy + usual care	MAL (QOM), MAL (AOU), FMA (Arm), FMA (UE), BBT	63
Hwang et al. 2016	17	Amadeo	Conventional physiotherapy	MAS (Elbow), Pinch, FMA (Arm), FMA (H/W), Grip, SIS (ADL), NHPT	30
Iwamoto et al. 2019	12	Hybrid Assistive Limb - SJ	Occupational physiotherapy	MAL (AOU), MAL (QOM), Grip, MI, BI, FIM	28
Kim et al. 2017	38	InMotion ARM	Conventional physiotherapy	VAS, K-SDQ, pROM	30
Klamroth-Marganska 2014	73	ARMin	Conventional physiotherapy	FMA (UE), WMFT (FAS), WMFT (TIME), MAL (QOM), MAS, Grip	60
Kuo et al. 2022	18	TIGER	Task-specific motor training	MAL (QOM), MAL (AOU), FMA (UE), BBT	28
Kutner et al. 2010	18	Hand Mentor robotic system	Therapist-supervised repetitive task practice	SIS (Hand), SIS (Mobility), SIS (ADL)	21
Lee et al. 2016	58	Robot Neuro-X	Conventional physiotherapy	MAS, BI	14
Lee et al. 2018	30	REJOYCE	Occupational therapy	BI, FMA (UE)	60
Lee et al. 2021	29	Gloreha	Conventional physiotherapy	Grip, FMA (W/H), FMA (Arm), FMA (UE), BBT	42
Lencioni et al. 2021	32	Braccio di ferro	Usual care	FMA	nr
Liao et al. 2011	40	Bi-Manu-Track	Dose-matched active control therapy	ABILHAND, MAL (QOM), MAL (AOU), FIM, FMA (UE)	28
Lin et al. 2022	222	Arm1	Conventional physiotherapy	FMA (Arm), FMA (UE)	21
Lo et al. 2010	25	Name not available	Intensive conventional physiotherapy	Ashworth, FMA, SIS (ADL), WMFT	84
Lum et al. 2002	27	MIME	Neurodevelopmental therapy	FIM, BI	60
Lum et al. 2006	16	MIME	Conventional physiotherapy	FIM, MP	180
Masiero et al. 2006	20	NeReBot	Conventional physiotherapy	MRC (Elbow), FMA (H/W), FMA (Arm), FIM- mot, upMI	30
Masiero et al. 2007	35	NeReBot	Conventional physiotherapy	MRC (Deltoid), MRC (Biceps), FMA (H/W), FMA (Arm)	35
Masiero et al. 2011	21	NeReBot	Conventional physiotherapy	MAS, MRC (Biceps), MRC (Deltoid), FAT, FMA (H/W), FMA (Arm), BBT, FMA (UE), FIM-mot	90
Masiero et al. 2014	30	NeReBot	Conventional physiotherapy	MAS, MRC (Biceps), MRC (Deltoid), FAT, FMA (H/W), FMA (Arm), FMA (UE), BBT, FIM-mot	35
MC Cabe et al. 2015	25	InMotion2	Motor learning	FMA (Arm), FMA (H/W), FMA (Coordination)	35
Orihuela Espina et al. 2015	17	Amadeus Tyromotion	Occupational therapy	FMA (H/W), MI	75
Page et al. 2013	16	Myomo	Repetitive task-specific practice	SIS (Strength), COPM-P, COPM-S, SIS (ADL), FMA (UE), SIS (Mobility)	56
Rabadi et al. 2008	20	MIT-Manus	Occupational therapy + conventional physiotherapy	VAS, MAS, FMA (Arm), FMA (H/W), MSS (Arm), ARAT, MSS (Hand), MP	56
Ramos-Murguialday et al. 2019	28	Name not available	Random orthoses movements + conventional physiotherapy	GAS, FMA (H/W), FMA (Arm), MAS, MAL	30
Ranzani et al. 2020	33	ReHaptic Knob	Conventional physiotherapy	FMA (W/H), FMA (Arm), FMA (UE), BBT	28
Reinkensmeyer et al. 2012	26	Pneu-WREX I	Conventional tabletop therapy	FMA (UE), MAL (AOU), MAL (QOM), Grip, BBT	56
Rémy Néris et al. 2021	215	Armeo Spring	Conventional physiotherapy	FMA (UE), ARAT, FIM, SIS (Hand)	28
Rodgers et al. 2020	511	MIT-Manus, InMotion	Conventional physiotherapy	FMA (UE), ARAT, BI, SIS (ADL), SIS (Hand), SIS (Mobility), NRPS	90
Sale et al. 2013	20	Amadeo Robotic System	Occupational therapy	MRC, MAS, BBT, FMA (UE), MI	35
Sale et al. 2014	53	MIT-MANUS/InMotion2	Conventional physiotherapy	MAS (Elbow), MAS (Shoulder), FMA (UE), MI, pROM	42
Singh et al. 2022	27	Name not available	Conventional physiotherapy	MAS, BI, FMA (UE), FMA (W/H), FMA (Arm)	28
Straudi et al. 2019	40	Reo Therapy System	Conventional physiotherapy	FMA (UE)	21
Susanto et al. 2015	19	Name not available	Non-assisted finger training groups	FMA (H/W), FMA (Arm), FMA (UE), WMFT, ARAT	35
Takahashi et al. 2016	56	ReoGo	Conventional physiotherapy + self-training	MAL (QOM), MAL (AOU), FMA, WMFT	42
Takebayashi et al. 2022	78	ReoGo-J	Conventional physiotherapy	FMA (UE)	70
Taravati et al. 2022	45	ReoGo	Conventional physiotherapy	FMA (W/H), FMA (UE), FIM, Grip	28
Taveggia et al. 2016	27	Armeo Spring	Physical and rehabilitation medicine+ conventional physiotherapy	MAS, NRPS, MI, FIM	84
Terranova et al. 2021	51	InMotion	Conventional physiotherapy	FMA (UE), WMFT	84
Timmermans et al. 2014	22	Haptic Maste	Task-oriented non-robotic arm-hand training	FMA (UE), ARAT, MAL (AOU), MAL (QOM), MAL	56
Tomic et al. 2017	26	ArmAssist	Conventional arm training	WMFT FAS (Arm), WMFT-FAS, FMA (Arm), FMA (UE), BI	21
Vanoglio et al. 2016	30	Gloreha	Conventional hand rehabilitation	NHPT, Pinch, Grip, QuickDASH, MI	42
Villafane et al. 2017	32	Gloreha	Conventional physiotherapy + occupational therapy	MAS, NIHSS, VAS, QuickDASH, BI, MI	21
Volpe et al. 2000	56	MIT-Manus	Conventional physiotherapy	MSS (Hand), FMA (H/W), FMA (Arm), MSS (Arm), MP	30
Volpe et al. 2008	21	Mit-Manus	Intensive movement protocol	VAS, FMA (H/W), FMA (Arm), MAS, ARAT, SIS (Hand), MP	90
Wolf et al. 2015	99	Hand Mentor Pro	Home exercise program	FMA (H/W), FMA (Arm), FMA (UE), ARAT	56
Wu et al. 2012	28	Bi-Manu-Track	Therapist-based bilateral arm training	MAL (AOU), MAL (QOM), FMA (Arm), FMA (H/W), FMA (UE), SIS (Strength), SIS (ADL), SIS (Hand)	28
Yoo et al. 2013	22	ReoGo	Conventional physiotherapy	Grip, BI, BBT, WMFT	42

Risk of bias in included studies

The methodological quality of the included studies is reported in Table 2. The PEDro score was 6.1±1.3 (range: 2, 8). Items 2, 10, and 11 were the most frequently satisfied (98.8%, 97.6%, and 96.4%, respectively), while items 3, 5, and 6 were the least often satisfied (35.7%, 2.4%, and 0.0%, respectively).

Meta-analyses

The summary results of the meta-analyses are reported in Table 3. The forest plots considering the statistical significance for each domain are detailed in Supplemental Material 6, and those reporting the clinical relevance for each domain in Supplemental Material 7.

TABLE 2 -. Methodological quality of the included studies

Study	Item 1	Item 2	Item 3	Item 4	Item 5	Item6	Item 7	Item 8	Item 9	Item 10	Item 11	Total score
Abdullah et al. 2011	Y	Y	Y	Y	N	N	Y	Y	N	Y	N	6
Aisen et al. 1997	Y	N	N	Y	N	N	Y	Y	N	N	Y	4
Ang et al. 2014	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Aprile et al.2020	Y	Y	N	Y	N	N	Y	N	Y	Y	Y	6
Bayindir et al. 2022	N	Y	Y	Y	N	N	N	Y	N	Y	Y	6
Brokaw et al. 2013	Y	Y	N	N	N	N	Y	N	N	Y	Y	4
Budhota et al. 2021	Y	Y	Y	Y	Y	N	N	Y	Y	Y	Y	8
Burgar et al. 2011	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Bustamante Valles et al. 2016	Y	Y	N	N	N	N	N	N	N	Y	Y	3
Calabrò et al. 2019	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Cameirao et al. 2011	Y	Y	N	Y	N	N	Y	N	N	Y	Y	5
Carpinella et al. 2020	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Chen et al.2021	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Chen et al. 2022	Y	Y	N	Y	N	N	Y	N	N	Y	Y	5
Chinembiri et al. 2020	Y	Y	N	N	N	N	Y	Y	N	Y	Y	5
Connelly et al. 2010	Y	Y	N	Y	N	N	N	Y	N	Y	N	4
Conroy et al. 2011	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Coskunsu et al. 2022	Y	Y	N	Y	N	N	Y	N	N	Y	Y	5
Daly et al., 2005	Y	Y	N	Y	N	N	Y	Y	N	N	Y	5
Daunoraviciene et al. 2018	N	Y	N	Y	N	N	N	Y	N	Y	Y	5
De Araújo et al. 2011	N	Y	N	Y	N	N	N	Y	N	Y	Y	5
Dehem et al. 2019	Y	Y	Y	Y	N	N	Y	N	Y	Y	Y	7
Fasoli et al. 2004	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Fazekas et al. 2007	N	Y	N	N	N	N	N	Y	N	N	N	2
Franceschini et al. 2019	Y	Y	Y	Y	N	N	Y	N	N	Y	Y	6
Frisoli et al. 2022	Y	Y	N	Y	N	N	N	N	N	Y	Y	4
Gandolfi et al. 2019	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Grigoras et al. 2016	Y	Y	N	Y	N	N	N	Y	Y	Y	Y	6
Gueye et al. 2021	Y	Y	N	Y	N	N	N	Y	N	Y	Y	5
Hesse et al. 2005	Y	Y	Y	N	N	N	Y	Y	Y	Y	Y	7
Hesse et al. 2014	Y	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Housman et al. 2009	Y	Y	N	Y	N	N	Y	N	N	Y	Y	5
Hsieh et al. 2011	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Hsieh et al. 2012	N	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Hsieh et al. 2014	N	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Hsieh et al. 2017	N	Y	Y	Y	N	N	Y	Y	N	Y	Y	7
Hsiu et al. 2019	N	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Hsiu et al. 2021	Y	Y	Y	Y	N	N	Y	Y	N	Y	Y	7
Hwang et al. 2016	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Iwamoto et al. 2019	Y	Y	N	Y	N	N	N	N	N	Y	Y	4
Kim et al. 2017	Y	Y	Y	Y	N	N	N	Y	N	Y	Y	6
Klamroth Marganska et al. 2014	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Kuo et al. 2022	Y	Y	Y	Y	N	N	Y	Y	N	Y	Y	7
Kutner et al. 2010	N	Y	Y	Y	N	N	Y	N	N	Y	Y	6
Lee et al. 2016	Y	Y	N	Y	N	N	N	N	N	Y	Y	4
Lee et al. 2018	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Lee et al. 2021	Y	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Lencioni et al. 2021	Y	Y	N	Y	N	N	Y	N	N	Y	Y	5
Liao et al. 2011	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Lin et al. 2022	Y	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Lo et al. 2010	Y	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Lum et al. 2002	N	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Lum et al. 2006	N	Y	N	N	N	N	Y	N	N	Y	Y	4
Masiero et al. 2006	Y	Y	N	Y	N	N	Y	N	N	Y	Y	5
Masiero et al. 2007	Y	Y	Y	Y	N	N	Y	N	Y	Y	Y	7
Masiero et al. 2011	N	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Masiero et al. 2014	Y	Y	N	Y	N	N	Y	N	Y	Y	Y	6
MC Cabe et al. 2015	N	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Orihuela Espina et al. 2015	Y	Y	N	Y	N	N	N	Y	Y	Y	Y	6
Page et al. 2013	Y	Y	N	Y	N	N	Y	N	N	Y	Y	5
Rabadi et al. 2008	Y	Y	Y	N	N	N	Y	Y	N	Y	Y	5
Ramos-Murguialday et al. 2019	N	Y	N	N	N	N	Y	Y	N	Y	Y	5
Ranzani et al. 2020	Y	Y	N	Y	N	N	Y	N	Y	Y	Y	6
Reinkensmeyer et al. 2012	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Rémy Néris et al. 2021	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Rodgers et al. 2020	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Sale et al. 2013	N	Y	N	N	N	N	Y	Y	Y	Y	Y	6
Sale et al. 2014	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Singh et al. 2022	Y	Y	Y	Y	N	N	Y	N	N	Y	Y	6
Straudi et al. 2019	N	Y	Y	Y	N	N	Y	N	Y	Y	Y	7
Susanto et al. 2015	N	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Takahashi et al. 2016	Y	Y	N	N	N	N	Y	Y	N	Y	Y	5
Takebayashi et al. 2022	Y	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Taravati et al. 2022	Y	Y	N	Y	N	N	Y	N	N	Y	Y	5
Taveggia et al. 2016	N	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Terranova et al. 2021	Y	Y	N	Y	N	N	N	N	Y	Y	Y	5
Timmermans et al. 2014	Y	Y	Y	Y	N	N	Y	Y	Y	Y	Y	8
Tomic et al. 2017	Y	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Vanoglio et al. 2016	Y	Y	Y	Y	N	N	Y	Y	N	Y	Y	7
Villafane et al. 2017	N	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Volpe et al. 2000	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Volpe et al. 2008	Y	Y	N	Y	N	N	Y	Y	N	Y	Y	6
Wolf et al. 2015	Y	Y	N	Y	N	N	Y	Y	Y	Y	Y	7
Wu et al. 2012	N	Y	Y	Y	Y	N	Y	Y	N	Y	Y	8
Yoo et al. 2013	N	Y	N	Y	N	N	Y	N	N	Y	Y	5

In terms of statistical significance, the domains of activity of daily living (SMD = 0.29; 95% CI: 0.15; 0.43), dexterity (SMD = 0.19; 95% CI: 0.01; 0.37), arm function (SMD = 0.27; 95% CI: 0.17; 0.38), and strength (SMD = 0.44; 95% CI: 0.17; 0.71) showed a significant effect in favor of the experimental group. Conversely, a non-significant effect in favor of the experimental treatment was observed for the domain of pain (SMD = 0.53; 95% CI: −0.15; 1.20), while a non-significant effect in favor of the control group was achieved in the domain of muscle tone (SMD = −0.02; 95% CI: −0.26; 0.23).

In terms of clinical relevance, the forest plots reported non-clinically relevant findings for all the domains considered (SMOS < 1.00 for all meta-analyses).

Heterogeneity ranged from negligible to substantial for all analyses, ranging from 0% to 85% for the meta-analyses considering statistical significance and from 0% to 91% for those exploring clinical relevance.

TABLE 3 -. Summary of the meta-analyses

Domain		Statistical significance									Clinical relevance
	N	Fixed	95% CI	Random	95% CI	Q	Tau 2	I 2		N	Fixed	95% CI	Random	95% CI	Q	Tau 2	I 2
Activity of daily living		34	0.20	0.11, 0.30	0.29	0.15, 0.43	58.52	0.06	44		34	0.13	0.08, 0.18	0.17	0.08, 0.26	66.43	0.02	50
Dexterity		18	0.19	0.01, 0.37	0.19	0.01, 0.37	14.84	0.00	0		18	0.30	0.06, 0.53	0.30	0.06, 0.53	16.26	0.00	0
Arm function		73	0.26	0.19, 0.33	0.27	0.17, 0.38	140.18	0.09	49		73	0.30	0.23, 0.37	0.45	0.27, 0.64	204.74	0.27	65
Muscle Tone		19	0.07	−0.09, 0.22	−0.02	−0.26, 0.23	40.37	0.16	55		19	−0.08	−0.17, 0.02	0.01	−0.22, 0.23	54.37	0.11	67
Pain		7	0.12	−0.02, 0.26	0.53	−0.15, 1.20	39.71	0.72	85		7	0.31	0.11, 0.51	0.65	−0.13, 1.42	70.55	0.95	91
Strength		23	0.40	0.24, 0.56	0.44	0.17, 0.71	56.95	0.26	61		23	0.39	0.24, 0.54	0.48	0.18, 0.78	90.79	0.24	76

Subgroup analyses

When the specific interventions applied in the control groups were considered, the following subgroup analyses were performed: conventional physiotherapy, occupational therapy, conventional physiotherapy plus occupational therapy, intensive therapy, task-oriented approach, virtual reality, and others.

Heterogeneity showed a general reduction in subgroup analyses, although it remains substantial or considerable for some domains. However, the results were similar to those found in the general meta-analyses, both in terms of statistical significance and clinical relevance (Supplemental Materials 8 and 9). The meta-regression analyses using the duration of interventions as an independent factor are shown in Supplemental Material 10.

Discussion

This systematic review with meta-analysis showed that the RAT, compared to other physiotherapy, produces statistically significant improvements in activities of daily living, dexterity, arm function, and strength but not in muscle tone and pain in patients with stroke. However, none of these improvements were clinically relevant.

Our results showed that the RAT produced significant effects in improving daily living activity, dexterity, arm function, and strength but not in muscle tone (15). RAT uses active or actively assisted movements. This technique is used in clinical practice when the goal is to improve function, dexterity and strength. Instead, usually, passive techniques and stretching are used to reduce muscle tone (fundamentals on which RAT is not based) (16). Therefore, it could be reasonable that RAT is not able to reduce muscle tone more than other therapies. Several recent systematic reviews on RAT effects in patients with stroke have been published, reporting different conclusions. The majority of reviews report that RAT was effective in improving upper limb motor function recovery (or, at least, some of the function-related outcomes) when compared to other treatments (7,8,17). In contrast, other reviews reported no significant difference between RAT and conventional physiotherapy (9-11). Conflicting results among these studies are concerned with the study methodology and the different strategy analyses. However, the conclusions of these studies were only based on the statistical significance of the differences and not their clinical relevance. Hence, the findings of an RCT or meta-analysis might be significant but irrelevant in practice.

Our results showed that clinical relevance was never reached for all the outcome domains. Discussing clinical relevance is useful for those who must apply the results of clinical research in daily clinical practice. From the perspective of evidence-based practice, it is certainly crucial that the result of the clinical study is statistically significant (i.e., that the difference evaluated is not attributable to chance but to the real effectiveness of the treatment). Yet, it is even more crucial that a result is clinically relevant (i.e., that the effect size is of such a magnitude that it is useful for clinically improving the patient’s condition), implying that, even if true within the study, these results may not be so useful in determining a clinically significant change in the patient’s condition in clinical practice.

In the research field, authors of scientific articles are more likely to report and discuss the statistical significance of a result rather than its clinical relevance. Verhagen et al. (18) analyzed frequencies and proportions of reporting clinical relevance of RCTs published between 2000 and 2018 in six major physiotherapy peer-reviewed journals; over 40% of RCTs failed to report the clinical relevance of the results. However, the attention to clinical relevance has been increasing with time; many publications still fail to document and interpret study findings accordingly. Two previous studies systematically investigated this issue by analyzing RCT trials for low back pain (19,20), recommending reporting clinically important and statistically significant differences in meta-analyses to identify the real, meaningful effect. Van Tulder et al. (19) transformed means and standard deviations to a 0 to 100 scale and considered results clinically important when their magnitude was 20% or more for pain and 10% or more for functioning. Differently, Gianola et al. (20) compared between-group differences with the planned MCID reported in the sample size calculation, determining if the effect size reached clinical relevance. The authors classified clinical relevance as ‘achieved’ if the point estimate of the mean difference was equal or greater than the a priori planned MCID and ‘not achieved’ in the other case. In both studies, only a minority of RCT findings were statistically significant and clinically relevant. Unlike previous investigations, we performed analyses using an approach similar to the mean difference calculation (i.e., when the studies all assess the same domain but measure it using different scales). This approach allows us to (I) estimate the effect using a scale-free measure, (II) clarify the size of the effect, and (III) compare and summarize results when studies use different outcome scales (21).

Eighty-five studies were included in the systematic review, while 78 studies were included in the meta-analysis because none of the MCIDs for each outcome measure for the seven studies were found; indeed, we retrieved 27 out of 36 MCID values. The lack of known clinically important values (i.e., MCID values) may hinder researchers from reporting and interpreting their findings concerning clinical relevance. Future research that aims to determine MCIDs for core outcome measures is necessary. In addition, not all the MCIDs included in this review were related to stroke populations, which should be considered when results are interpreted. The MCID construct is not clearly defined, and different index categories could cause uncertainty in the results (22). For example, within-group methods might be used to calculate the MCID, providing more probably the value of the within-group MIC (23). Finally, grouping outcomes in domains could have sometimes been arbitrary since the construct of some measures included more than one domain (e.g., arm function and activities of daily living or dexterity).

An additional concern may be related to the content of the so-called “conventional physiotherapy.” Previous studies on physiotherapy interventions in neurological conditions, including stroke (24-26), reported inadequate descriptions of the interventions, especially when they are defined as “conventional physiotherapy.” Without strong evidence for the superiority of one treatment over others, conventional physiotherapy (and equivalent terms) may be related to a variety of practice approaches since they are considered complex interventions.

High heterogeneity was found for all analyses, despite sub-group analyses, according to upper extremity segments, robotic types, and content of control groups, not clarifying the heterogeneity. However, this finding was not unexpected because the studies retrieved differed in treatment dosages and applications and control groups’ treatment contents (27). In fact, potential factors influencing outcomes in stroke survivors are the dose and intensity of therapy (28). These variables have not been included in the analyses because they are underreported in the included studies. The quality of reporting of interventions used in RCTs could be improved following the Template for Intervention Description and Replication (TIDieR) (29) or the Consensus on Exercise Reporting Template (CERT) (30). However, the results of sub-group analyses in terms of both statistical significance and clinical relevance confirmed those obtained by overall analyses, also when a meta-regression was conducted using the duration of interventions as a covariate.

Finally, our results should be used with caution: considering that the MCID is influenced by the clinical conditions and the phases post-stroke of the subjects studied (31), including also MCID from populations other than stroke patients could have influenced the results of this study.

Conclusion

In conclusion, RAT produces significant clinical improvements for the upper limb in some domains (i.e., activity of daily living, dexterity, arm function, and strength), but these improvements are not clinically relevant when compared with other therapies. Using the RAT in clinical practice can produce improvement that may not be clinically relevant for patients with stroke. Future research in this field should consider the clinical relevance when interpreting results. Because a substantial heterogeneity remained unexplained for some analyses, pooled data should be interpreted with caution.

Other information

This article includes supplementary material

Corresponding author:

Leonardo Pellicciari

email: leonardo.pellicciari@gmail.com

Disclosures

Conflict of interest: The authors reported no conflict of interest

Financial support: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author’s contributor role: Conceptualization: AU, MP; Data curation: SV, AU; Formal Analysis: AU; Investigation: SV; Methodology: LP, MP; Project administration: SV; Resources: AU; Software: AU; Supervision: MP; Validation: MDB; Visualization: MDB; Writing – original draft: LP, MP; Writing – review & editing: all authors

Data availability statement: Data sharing not applicable: No new data were created or analyzed in this study