- Research article
- Open Access
An external validation study reporting poor correlation between the claims-based index for rheumatoid arthritis severity and the disease activity score
Arthritis Research & Therapyvolume 17, Article number: 83 (2015)
We conducted an external validation study to examine the correlation of a previously published claims-based index for rheumatoid arthritis severity (CIRAS) with disease activity score in 28 joints calculated by using C-reactive protein (DAS28-CRP) and the multi-dimensional health assessment questionnaire (MD-HAQ) physical function score.
Patients enrolled in the Brigham and Women’s Hospital Rheumatoid Arthritis Sequential Study (BRASS) and Medicare were identified and their data from these two sources were linked. For each patient, DAS28-CRP measurement and MD-HAQ physical function scores were extracted from BRASS, and CIRAS was calculated from Medicare claims for the period of 365 days prior to the DAS28-CRP measurement. Pearson correlation coefficient between CIRAS and DAS28-CRP as well as MD-HAQ physical function scores were calculated. Furthermore, we considered several additional pharmacy and medical claims-derived variables as predictors for DAS28-CRP in a multivariable linear regression model in order to assess improvement in the performance of the original CIRAS algorithm.
In total, 315 patients with enrollment in both BRASS and Medicare were included in this study. The majority (81%) of the cohort was female, and the mean age was 70 years. The correlation between CIRAS and DAS28-CRP was low (Pearson correlation coefficient = 0.07, P = 0.24). The correlation between the calculated CIRAS and MD-HAQ physical function scores was also found to be low (Pearson correlation coefficient = 0.08, P = 0.17). The linear regression model containing additional claims-derived variables yielded model R2 of 0.23, suggesting limited ability of this model to explain variation in DAS28-CRP.
In a cohort of Medicare-enrolled patients with established RA, CIRAS showed low correlation with DAS28-CRP as well as MD-HAQ physical function scores. Claims-based algorithms for disease activity should be rigorously tested in distinct populations in order to establish their generalizability before widespread adoption.
One of the most important confounders in observational studies of patients with rheumatoid arthritis (RA) is disease severity. Owing to large size and relative ease of access, health-care utilization databases have been increasingly used to study various treatment outcomes in RA [1-4]. However, clinical disease activity markers are not available in these databases, and hence studies conducted using these data are prone to residual confounding by disease severity. To address this problem, Ting et al.  developed an algorithm to create a claims-based index for rheumatoid arthritis severity (CIRAS) by using numerous variables from claims. In their original article, Ting et al.  used RA records-based index of severity (RARBIS), which was constructed by using ratings by a Delphi panel on potential markers of RA severity commonly found in medical charts, to demonstrate the validity of CIRAS and reported moderate correlation between the medical records and claims-based indices.
Despite being commonly used in RA observational research [6-8], CIRAS has not been validated against a clinical marker of RA severity until now. RA severity is a complex concept that depends on a combination of disease activity, physical function impairment, and physical damage to the joints. Clinically accepted measures for accurately determining RA severity that include all of these aspects are scarce. However, the disease activity score in 28 joints calculated by using C-reactive protein (DAS28-CRP)  is commonly used to evaluate treatment success and to guide treatment selection in patients with RA . Therefore, in the absence of a standard clinical measure for RA severity, we selected the disease activity measure DAS28-CRP to validate the claims-based severity measure CIRAS in this external validation study using data from the Brigham and Women’s Hospital Rheumatoid Arthritis Sequential Study (BRASS) linked to Medicare claims. Furthermore, we examined the correlation between the multi-dimensional health assessment questionnaire (MD-HAQ)  physical function scores and CIRAS.
The BRASS registry is a single-center, prospective, observational cohort of 1,350 patients with a rheumatologist-verified diagnosis of RA. For the subjects enrolled in this registry, data on patient-reported items, including demographics, lifestyle factors, medication use, and quality-of-life scales, as well as physician-reported items such as DAS28-CRP, extra-articular manifestations, and medication changes are collected during annual follow-up visits. For this study, we identified BRASS patients who were also enrolled in Medicare between 2006 and 2010, and linked their data from these two sources. Of these subjects, we further identified those with at least one valid DAS28-CRP measurement in BRASS after 365 days of continuous enrollment in Medicare. The algorithm proposed by Ting et al.  was implemented by using Medicare claims data in the period of 365 days immediately prior to the DAS28-CRP measurement date to calculate the CIRAS for these patients. Pearson correlation coefficients between the calculated CIRAS and DAS28-CRP were calculated. We also analyzed MD-HAQ physical function scores measured on the same day as DAS28-CRP for these patients and calculated Pearson correlation coefficients between the calculated CIRAS and MD-HAQ. Personal identifiers were removed from the dataset before the analysis to protect subject confidentiality. Patient informed consent was, therefore, not required. This study was approved by the Brigham and Women’s Hospital’s Institutional Review Board.
Furthermore, we identified several other potential predictors of RA severity, which were not part of the original CIRAS, from medical and pharmacy claims in a subset of patients who had Medicare part D enrollment for the 365-day period prior to the DAS28 measurement date in order to improve the algorithm for CIRAS. These variables included rheumatoid lung involvement, hand surgery, tuberculin test ordered, and anti-cyclic citrullinated peptide (CCP) test ordered, steroid use, opioid use, non-steroidal anti-inflammatory drug (NSAID) use, number of non-biologic disease-modifying anti-rheumatoid drugs (DMARDs) used, and biologic DMARD use. A multivariable linear regression model was built by using DAS28-CRP as the outcome and these claims-derived variables as predictors. Adjusted correlations between the predictors and the outcome were reported as partial R2 values. Full model R2 was reported as a measure of the overall performance of this model.
We located 368 patients who were enrolled in both BRASS and Medicare. We then excluded 53 patients who did not have at least one valid DAS28-CRP measured in BRASS after 365 days of continuous enrollment in Medicare, leaving 315 patients with sufficient baseline data for calculation of CIRAS. Of these 315 patients, the majority (81%) were females. The mean (standard deviation) age of the cohort was 70 (10) years. The median (interquartile range) DAS28-CRP and CIRAS were 3.3 (2.3 to 4.6) and 4.4 (3.7 to 5.1), respectively. Other patient characteristics used for CIRAS calculation are summarized in Table 1. The correlation between the calculated CIRAS and DAS28-CRP was found to be poor (Pearson correlation coefficient = 0.07, P = 0.24). The correlation between the calculated CIRAS and MD-HAQ physical function scores was also found to be low (Pearson correlation coefficient = 0.08, P = 0.17).
Furthermore, we identified a subgroup of 119 patients who had at least 1 year of Medicare part D enrollment immediately prior to the DAS28-CRP measurement date. The linear regression model containing additional claims-derived variables along with the variables originally proposed by Ting et al.  yielded model R2 of 0.23, suggesting limited ability of this model to explain variation in DAS28-CRP. Among some of the most influential predictors in this model were biologic DMARD use, opioid use, tuberculin test ordered, and number of non-biologic DMARDs used in the prior year (Table 2).
In this validation study using data from an external cohort of Medicare-enrolled patients with an established diagnosis of RA, the previously published algorithm to approximate RA severity by using claims-based variables had poor correlation with DAS28-CRP and MD-HAQ. Adding more variables derived from both medical and pharmacy claims as predictors in a linear regression model did not substantially improve the performance of this algorithm in predicting DAS28-CRP.
Several potential differences between this external validation study and the original study in which Ting et al.  developed CIRAS may explain the poor performance of CIRAS in this cohort. First, it must be noted that CIRAS was validated against a medical records-based RA severity index (RARBIS) in the original study and that the correlation between the two indices was found to be moderate (Spearman correlation coefficient = 0.51). RARBIS itself has been shown to correlate only moderately with DAS28 (Spearman correlation coefficient = 0.41) . The majority of the clinical parameters measured through RARBIS, including patients’ functional status, arthritis flares, x-ray results, and laboratory results, are not captured in claims and hence in CIRAS. Therefore, the poor performance of CIRAS against DAS28-CRP may simply reflect the inability to account for these important clinical parameters. Next, important differences between the current cohort and the CIRAS derivation cohort, including sizable gender differences (81% versus 9% females), differences in the disease activity, and differences in health-care utilization patterns, may help explain the poor performance of CIRAS in this validation cohort.
CIRAS has been used in observational studies of RA treatments in the past mainly to control for confounding by disease activity. Two prior studies used CIRAS as a covariate in their regression models for the outcome [6,7]. Another study used CIRAS as one of the variables for prediction of a disease risk score (infection score) and stratified analysis based on this disease risk score to account for measured confounding . Findings from our study show poor correlation between CIRAS and DAS28-CRP (RA activity measure, which often drives treatment selection) as well as MD-HAQ (patient physical function score, which may be indicative of frailty and hence may be an important confounder). These findings suggest that CIRAS may not accurately approximate disease activity or frailty in observational studies of RA treatments using insurance claims data. Given this poor correlation between CIRAS and important confounders unmeasured in health-care claims data, future research should be considered to critically evaluate the benefit of using CIRAS as a tool for confounding control.
Another important contribution of our study is that it highlights the importance of external validation of claims-based algorithms. Two prior studies have attempted to build algorithms predicting RA severity. Wolfe et al.  used data on the type and number of DMARDs used by the patients in the National Data Bank for Rheumatic Diseases to predict their RA severity and found suboptimal performance of these variables in predicting RA severity as measured by a patient activity scale. Baser et al.  used Veterans Health Administration claims to build a severity index for rheumatoid arthritis (SIFRA) and reported moderate correlations with the CIRAS. Before widespread adoption of these indices, broad testing is critical to determine their appropriateness in different databases.
Our study reported a low correlation between the previously proposed CIRAS and DAS28-CRP as well as MD-HAQ physical function scores, suggesting that CIRAS may not approximate RA disease activity or frailty reliably in observational cohorts. Claims-based algorithms for clinical disease activity should be rigorously tested in distinct populations in order to establish their generalizability.
Brigham and Women’s Hospital Rheumatoid Arthritis Sequential Study
claims-based index for rheumatoid arthritis severity
disease activity score in 28 joints calculated by using C-reactive protein
disease-modifying anti-rheumatic drug
multi-dimensional health assessment questionnaire
rheumatoid arthritis records-based index of severity
Solomon DH, Curtis JR, Saag KG, Lii J, Chen L, Harrold LR, et al. Cardiovascular risk in rheumatoid arthritis: comparing TNF-alpha blockade with nonbiologic DMARDs. Am J Med. 2013;126:730 e739–17.
Suissa S, Bernatsky S, Hudson M. Antirheumatic drug use and the risk of acute myocardial infarction. Arthritis Care Res. 2006;55:531–6.
Schneeweiss S, Solomon DH, Wang PS, Rassen J, Brookhart MA. Simultaneous assessment of short-term gastrointestinal benefits and cardiovascular risks of selective cyclooxygenase 2 inhibitors and nonselective nonsteroidal antiinflammatory drugs: an instrumental variable analysis. Arthritis Rheum. 2006;54:3390–8.
Solomon DH, Massarotti E, Garg R, Liu J, Canning C, Schneeweiss S. Association between disease-modifying antirheumatic drugs and diabetes risk in patients with rheumatoid arthritis and psoriasis. JAMA. 2011;305:2525–31.
Ting G, Schneeweiss S, Scranton R, Katz J, Weinblatt M, Young M, et al. Development of a health care utilisation data-based index for rheumatoid arthritis severity: a preliminary study. Arthritis Res Ther. 2008;10:R95.
Kim SY, Schneeweiss S, Liu J, Daniel GW, Chang C-L, Garneau K, et al. Risk of osteoporotic fracture in a large population-based cohort of patients with rheumatoid arthritis. Arthritis Res Ther. 2010;12:R154.
Johnston SS, Turpcu A, Shi N, Fowler R, Chu B-C, Alexander K. Risk of infections in rheumatoid arthritis patients switching from anti-TNF agents to rituximab, abatacept, or another anti-TNF agent, a retrospective administrative claims analysis. Semin Arthritis Rheum. 2013;43:39–47.
Curtis JR, Xie F, Chen L, Baddley JW, Beukelman T, Saag KG, et al. The comparative risk of serious infections among rheumatoid arthritis patients starting or switching biological agents. Ann Rheum Dis. 2011;70:1401–6.
Wells G, Becker J, Teng J, Dougados M, Schiff M, Smolen J, et al. Validation of the 28-joint Disease Activity Score (DAS28) and European League Against Rheumatism response criteria based on C-reactive protein against disease progression in patients with rheumatoid arthritis, and comparison with the DAS28 based on erythrocyte sedimentation rate. Ann Rheum Dis. 2009;68:954–60.
Goekoop-Ruiterman YP, de Vries-Bouwstra JK, Kerstens PJ, Nielen MM, Vos K, van Schaardenburg D, et al. DAS-driven therapy versus routine care in patients with recent-onset active rheumatoid arthritis. Ann Rheum Dis. 2010;69:65–9.
Pincus T. A multidimensional health assessment questionnaire (MDHAQ) for all patients with rheumatic diseases to complete at all visits in standard clinical care. Bull NYU Hosp Jt Dis. 2007;65:150–60.
Sato M, Schneeweiss S, Scranton R, Katz JN, Weinblatt ME, Avorn J, et al. The validity of a rheumatoid arthritis medical records-based index of severity compared with the DAS28. Arthritis Res Ther. 2006;8:R57.
Wolfe F, Michaud K, Simon T. Can severity be predicted by treatment variables in rheumatoid arthritis administrative data bases? J Rheumatol. 2006;33:1952–6.
Baser O, Du J, Xie L, Wang H, Dysinger AH, Wang L. Derivation of severity index for rheumatoid arthritis and its association with healthcare outcomes. J Med Econ. 2012;15:918–24.
This study was not funded by any institution.
DS is supported by National Institutes of Health (NIH) grants K24 AR055989, P60 AR047782, and R01 AR056215. He receives research grants from Amgen (Thousand Oaks, CA, USA) and Eli Lilly and Company (Indianapolis, IN, USA). He serves in unpaid roles on studies sponsored by Pfizer Inc. (New York, NY, USA), Novartis (Basel, Switzerland), Eli Lilly and Company, and Bristol-Myers Squibb (New York, NY, USA). He also receives royalties from UpToDate.com. SK is supported by NIH grant K23 AR059677. She received research support from Pfizer Inc. and tuition support for the Pharmacoepidemiology Program at the Harvard School of Public Health partially funded by the Pharmaceutical Research and Manufacturers of America foundation. RD reports owning Biogen Idec (Cambridge, MA, USA) stock due to spouse’s employment. MW has received consulting fees, speaking fees, and/or honoraria from MedImmune (Gaithersburg, MD, USA), Crescendo Bioscience (South San Francisco, CA, USA), and Bristol-Myers Squibb (less than $10,000 each) and has received research grant support from those companies. NS has received research grant support from MedImmune, Crescendo Bioscience, Amgen, AbbVie (North Chicago, IL, USA), and Genentech (South San Francisco, CA, USA).
RD participated in conceiving and designing the study and conducted data analysis. SK participated in conceiving and designing the study. DS participated in conceiving and designing the study and shared responsibility for data acquisition. MW and NS shared responsibility for data acquisition. All authors contributed equally in the interpretation of the results and preparation of the manuscript. All authors read and approved the final manuscript.