American College of Rheumatology Classification
Criteria for Sjo¨gren’s Syndrome: A Data-Driven,
Expert Consensus Approach in the Sjo¨gren’s
International Collaborative Clinical Alliance Cohort
S. C. SHIBOSKI,1 C. H. SHIBOSKI,1 L. A. CRISWELL,1 A. N. BAER,2 S. CHALLACOMBE,3
H. LANFRANCHI,4 M. SCHIØDT,5 H. UMEHARA,6 F. VIVINO,7 Y. ZHAO,8 Y. DONG,9 D. GREENSPAN,1
A. M. HEIDENREICH,4 P. HELIN,5 B. KIRKHAM,3 K. KITAGAWA,6 G. LARKIN,3 M. LI,9 T. LIETMAN,1
J. LINDEGAARD,10 N. MCNAMARA,1 K. SACK,1 P. SHIRLAW,3 S. SUGAI,6 C. VOLLENWEIDER,4
J. WHITCHER,1 A. WU,1 S. ZHANG,9 W. ZHANG,11 J. S. GREENSPAN,1 AND T. E. DANIELS,1
FOR THE SJO¨GREN’S INTERNATIONAL COLLABORATIVE CLINICAL ALLIANCE (SICCA)
RESEARCH GROUPS
This criteria set has been approved by the American College of Rheumatology (ACR) Board of Directors as Provisional. This
signifies that the criteria set has been quantitatively validated using patient data, but it has not undergone validation based
on an external data set. All ACR-approved criteria sets are expected to undergo intermittent updates.
The American College of Rheumatology is an independent, professional, medical and scientific society which does not guar-
antee, warrant, or endorse any commercial product or service.
Objective. We propose new classification criteria for Sjo¨gren’s syndrome (SS), which are needed considering the
emergence of biologic agents as potential treatments and their associated comorbidity. These criteria target individuals
with signs/symptoms suggestive of SS.
Methods. Criteria are based on expert opinion elicited using the nominal group technique and analyses of data from the
Sjo¨gren’s International Collaborative Clinical Alliance. Preliminary criteria validation included comparisons with
classifications based on the American–European Consensus Group (AECG) criteria, a model-based “gold standard”
obtained from latent class analysis (LCA) of data from a range of diagnostic tests, and a comparison with cases and
controls collected from sources external to the population used for criteria development.
Results. Validation results indicate high levels of sensitivity and specificity for the criteria. Case definition requires at
least 2 of the following 3: 1) positive serum anti-SSA and/or anti-SSB or (positive rheumatoid factor and antinuclear
antibody titer >1:320), 2) ocular staining score >3, or 3) presence of focal lymphocytic sialadenitis with a focus score >1
focus/4 mm2 in labial salivary gland biopsy samples. Observed agreement with the AECG criteria is high when these are
applied using all objective tests. However, AECG classification based on allowable substitutions of symptoms for objective
tests results in poor agreement with the proposed and LCA-derived classifications.
Conclusion. These classification criteria developed from registry data collected using standardized measures are based
on objective tests. Validation indicates improved classification performance relative to existing alternatives, making them
more suitable for application in situations where misclassification may present health risks.
INTRODUCTION
Sjo¨gren’s syndrome (SS) is a multisystem autoimmune
disease characterized by hypofunction of the salivary and
lacrimal glands. It is among the group of diseases overseen
by rheumatologists; however, its diagnosis and manage-
ment require 3 areas of specialty practice: rheumatology,
ophthalmology, and oral medicine. The multidisciplinary
Supported by the NIH (National Institute for Dental and
Craniofacial Research, National Eye Institute, and Office of
Research on Women’s Health; contract N01-DE32636).
1S. C. Shiboski, PhD, C. H. Shiboski, DDS, MPH, PhD, L. A.
Criswell, MD, MPH, DSc, D. Greenspan, BDS, DSc, ScD,
T. Lietman, MD, N. McNamara, OD, PhD, K. Sack, MD,
J. Whitcher, MD, MPH, A. Wu, DDS, J. S. Greenspan, BDS,
PhD, FRCPath, T. E. Daniels, DDS, MS: University of Cali-
fornia, San Francisco; 2A. N. Baer, MD: Johns Hopkins Uni-
versity, Baltimore, Maryland; 3S. Challacombe, BDS, PhD,
Arthritis Care & Research
Vol. 64, No. 4, April 2012, pp 475–487
DOI 10.1002/acr.21591
© 2012, American College of Rheumatology
SPECIAL ARTICLE
475
aspect of the disease represents a challenge for the defini-
tion and validation of classification criteria because there
is no single gold standard test for diagnosing SS, and it is
not feasible to use a single clinician’s diagnosis for the
case/control definition. The closest substitute is based
on expert assumptions about the characteristics of SS,
specifically that it: 1) is a systemic, multiorgan autoim-
mune disease, 2) has a chronic or progressive course, and
3) is characterized by, but not limited to, secretory dys-
function.
While there have been 11 classification or diagnostic
criteria published for SS since 1965 (1–11), none have
been endorsed by the American College of Rheumatol-
ogy (ACR) or the European League Against Rheumatism
(EULAR). The American–European Consensus Group
(AECG) criteria (11) have better specificity than their pre-
decessor (9), as they require evidence of autoimmunity
from positive anti-SSA/Ro and/or anti-SSB/La serology or
focal lymphocytic sialadenitis (FLS) with a focus score
(FS) �1 in a labial salivary gland (LSG) biopsy sample.
However, they have been criticized for including subjec-
tive tests (symptoms), physiologic measures that lack spec-
ificity, and alternate objective tests that are not diagnosti-
cally equivalent. For example, Schirmer’s test may be used
instead of the rose bengal ocular stain, even though they
differ in sensitivity and specificity (11). Further, the inclu-
sion of symptoms of dry mouth and/or eyes can lead to
misclassification of asymptomatic patients. In addition,
physiologic measures, such as unstimulated whole sali-
vary (UWS) flow, unanesthetized Schirmer’s test, and sal-
ivary scintigraphy, are useful for assessment of salivary or
tear function, but lack specificity for SS.
The need for new classification criteria is clear consid-
ering the current lack of standardization inherent to the
use of multiple older criteria in the field and the emer-
gence of biologic agents as potential treatments. Consider-
ing the potentially serious adverse effects and comorbidi-
ties of these agents, criteria used for enrollment into
clinical trials will need to be clear, easy to apply, and have
high specificity. They also must rely upon well-estab-
lished objective tests that are clearly associated with the
systemic/autoimmune, oral, and ocular characteristics of
the disease, and include alternate tests only when they are
diagnostically equivalent. Furthermore, it is desirable for
new classification criteria for SS to be endorsed by profes-
sional rheumatology organizations across the world (such
as the ACR and EULAR) to increase their credibility and
maximize standardization when enrolling participants
into clinical trials.
The Sjo¨gren’s International Collaborative Clinical Alli-
ance (SICCA) is funded by the National Institutes of Health
(12) to develop new classification criteria for SS, better
define the SS phenotype, and collect/store clinical data
and biospecimens to support future research. We propose
new classification criteria for SS, following the ACR guide-
lines (13) to the extent possible for a condition requiring
multiple clinical specialties for diagnosis. Below we de-
scribe our approach to criteria development and valida-
tion.
MATERIALS AND METHODS
We used a consensus methodology derived from the nom-
inal group technique (14) to: 1) define the target popula-
tion to whom the classification criteria should apply; 2)
identify the initial list of criteria components that have
face validity, would be measured as part of the SICCA
project, and could be considered in the classification
criteria (item generation and reduction); and 3) select
preliminary classification criteria. We then engaged in a
series of validation exercises. Our overall approach relied
on analyses of current SICCA data and consisted of 4
phases.
FDSRCS, FRCPath, FMedSci, B. Kirkham, MD, FRCP,
FRACP, G. Larkin, FRCoPath, P. Shirlaw, BDS, FDRCPS:
King’s College London, London, UK; 4H. Lanfranchi, DDS,
PhD, A. M. Heidenreich, MD, C. Vollenweider, MD: Univer-
sity of Buenos Aires, Buenos Aires, Argentina; 5M. Schiødt,
DDS, P. Helin, MD, PhD: Rigshospitalet, Copenhagen, Den-
mark; 6H. Umehara, MD, PhD, K. Kitagawa, MD, PhD, S.
Sugai, MD: Kanazawa Medical University, Ishikawa, Japan;
7F. Vivino, MD: Penn Presbyterian Medical Center and Uni-
versity of Pennsylvania, Philadelphia; 8Y. Zhao, MD: PUMC
Hospital, Beijing, China; 9Y. Dong, MD, M. Li, MD, S. Zhang,
MD: Peking Union Medical College Hospital, Beijing, China;
10J. Lindegaard, MD, PhD: Glostrup Hospital, Glostrup, Den-
mark; 11W. Zhang, MD: Chinese Academy of Medical Sci-
ence and Peking Union Medical College Hospital, Beijing,
China. Collaborators of the Sjo¨gren’s International Collab-
orative Clinical Alliance are shown in Appendix A.
Drs. S. C. Shiboski and C. H. Shiboski contributed equally
to this work.
Dr. Vivino has received consultant fees and/or honoraria
(less than $10,000 each) from Daiichi-Sankyo and Parion
Sciences. Dr. Wu owns stock and/or stock options in Isis
Pharmaceuticals and Schering-Plough. Dr. J. S. Greenspan
has received consultant fees (less than $10,000) from
GlaxoSmithKline.
Address correspondence to S. C. Shiboski, PhD, Uni-
versity of California, San Francisco, Department of Epi-
demiology and Biostatistics, Division of Biostatistics, 185
Berry Street, Lobby 5, Suite 5700, San Francisco, CA 94107.
E-mail: steve@biostat.ucsf.edu.
Submitted for publication March 31, 2011; accepted in
revised form December 20, 2011.
Significance & Innovations
● New classification criteria for Sjo¨gren’s syndrome
are needed to better support etiologic and genetic
research and therapeutic trials for this prevalent
autoimmune disease.
● Criteria used for enrollment into clinical trials
need to be clear, be easy to apply, and have high
specificity, considering the potentially serious ad-
verse effects and comorbidities of biologic agents.
● We propose classification criteria for Sjo¨gren’s
syndrome that are developed from registry data
collected using standardized instruments and di-
agnostic tests, and are based entirely on objective
measures.
476 Shiboski et al
Phase 1: expert panel member selection and item gen-
eration. Expert panel member selection. We first identi-
fied panel members who were experts in the relevant clin-
ical specialties (rheumatology, ophthalmology, or oral
medicine) and constituted a heterogeneous group with
respect to geographic area, sex, and seniority/level of ex-
pertise. The panel included 20 experts: 7 rheumatologists,
6 ophthalmologists, and 7 experts in oral medicine. Nine
members (45%) were from the US, and the rest were from
4 countries on 3 continents. All of the panel experts prac-
ticed their specialty within a university-affiliated medical
center. Sixteen (80%) were senior investigators at the pro-
fessor level, 20% were at the associate professor level, and
40% were women. All of the investigators had been se-
lected for their experience with SS within their clinical
specialty and for geographic representation.
Expert panel first face-to-face meeting: review of evi-
dence-based literature and item generation. In February
2004, the panel of experts gathered for a face-to-face meet-
ing moderated by a statistician (SCS) and an epidemiolo-
gist (CHS). The goal of this meeting was to obtain consen-
sus (at least 80%) on the target population to whom the
classification criteria would apply, and the initial list of
variables or criteria items that would be collected as part of
SICCA. The meeting began with presentation of a compre-
hensive literature review by one of the senior investigators
(TED) of the 11 previous classification and diagnostic cri-
teria for SS that had been published in the past 40 years,
none of which had been endorsed by the ACR or EULAR.
There was consensus among the panel that the criteria
should apply to the population of patients who may be
referred to a specialist because of signs and/or symptoms
possibly suggesting SS. Recruitment strategies and eligi-
bility criteria are described below. The rationale for select-
ing this target population is that a given patient would not
be evaluated for SS unless she/he had signs or symptoms
suggesting this diagnosis. There was also consensus that if
asked to select cases and controls for validation of new
classification criteria, panel members would use objective
tests (e.g., specific serum measures of autoimmunity, ocu-
lar staining reflecting lacrimal hypofunction, and LSG bi-
opsy reflecting FLS) that would likely be part of the new
classification criteria, leading to circularity. Therefore, it
was agreed that no diagnostic labels would be used for
enrollment, and that all of the participants would undergo
the same set of standardized objective tests and question-
naires capturing various signs and symptoms.
The panel agreed upon the examinations and tests used
to assess ocular and oral signs and symptoms, tear and
salivary function, LSG biopsy results, and various serum
measures of autoimmunity. The list created was based
both on published results and on the clinical experience of
panel members. There was discussion among the rheuma-
tologists regarding which extraglandular manifestations
possibly associated with SS should be captured, and con-
sensus was achieved regarding a list of signs/symptoms
that would be measured through a targeted rheumatologic
examination, review of systems, careful medical history,
and serologic laboratory measures. Similarly, the oral
medicine specialists agreed on a list of tests measuring
salivary function (both stimulated parotid and UWS flow
rates) and salivary gland expression of autoimmunity
through biopsy of LSG, examining them for the presence of
FLS, and measuring FS accordingly as described in detail
elsewhere (15). The ophthalmologists agreed on tests eval-
uating participants for the presence of keratoconjunctivitis
sicca (KCS). There was consensus that, while rose bengal
had been widely used for grading conjunctival and corneal
damage in patients with KCS, it is inherently toxic to
epithelial cells and very painful for patients. Therefore,
fluorescein was selected to grade the cornea and lissamine
green was selected to grade the bulbar conjunctiva. Effec-
tiveness for grading KCS is established for both (16). They
agreed on a standardized quantitative grading system that
would be easily reproducible and could be used in clinical
practice in the future (17). The ocular staining score (OSS)
is the sum of a 0–6 score for fluorescein staining of the
cornea and a 0–3 score for lissamine green staining of
both the nasal and temporal bulbar conjunctivae, yielding
a total score ranging from 0–12. Alternative established
tests for dryness used in prior criteria, such as tear breakup
time (TBUT) and unanesthetized Schirmer’s test, were also
included.
The final list of criteria items that was agreed upon by
the end of the first meeting included nearly all those pre-
viously reported in the relevant literature. It has been
described previously (12) and is available online at http://
sicca.ucsf.edu.
Phase 2: item reduction. Expert panel second face-to-
face meeting: review of preliminary SICCA data analyses.
Following 2 years of standardized data collection, includ-
ing the criteria components selected in phase 1, another
face-to-face panel meeting was convened in April 2006.
Data analysis summaries were presented to the group by
the epidemiologist and statistician who moderated the
initial meeting, and the panel was divided into small spe-
cialty-specific focus groups to review evidence-based re-
sults for each clinical specialty. The goals of these explor-
atory analyses included understanding the relationship
between variables representing the oral/salivary, ocular,
and systemic features of the disease; defining cutoff values
for particular tests that could be used as components of the
classification criteria; and assessing the value of tests that
could serve as surrogates for primary objective tests in
alternate criteria sets. Methods used included frequency
tables, binary regression, and classification trees. We also
used area-proportional Venn diagrams to visualize the
overlapping relationships of 3 variables simultaneously,
each representing one of the primary disease features (18).
Phase 3: candidate SS criteria. Expert panel third face-
to-face meeting: data-driven, consensus-based selection of
preliminary classification criteria for SS. The panel met in
May 2009 to decide on preliminary classification criteria.
Additional analyses were presented, including longitudi-
nal assessment of the stability of criteria components over
time (based on results from scheduled 2-year followup
visits that mirrored baseline assessments). Results from a
statistical classification based on latent class analysis
(LCA) (19) were presented (approach further described
New Classification Criteria for Sjo¨gren’s Syndrome 477
below). The data presented to the panel represented a
subset of participants (n � 1,107) consecutively enrolled
as of April 1, 2009.
Following presentation of results by the statistician, a
discussion among panel members was moderated by the
epidemiologist and statistician. The goal of the discussion
was for members to select, based on their understanding of
the data analyses presented and on their clinical experi-
ence, which objective test(s) they believed was/were the
most specific for SS within their own specialty. Further-
more, various preliminary classification criteria were dis-
cussed, and panel members were asked to select which
criteria they believed would best classify patients with SS.
Following this third face-to-face meeting, a report summa-
rizing the data analyses was circulated among the panel
members and a questionnaire was distributed by e-mail.
The questionnaire was designed to assess consensus
among the expert panel members regarding the prelimi-
nary classification criteria. It also measured the level of
consensus within each specialty regarding which criteria
component was thought to have the highest level of face
validity within that specialty. The response to each ques-
tion was on a scale of 1–5, with 1 indicating strong dis-
agreement and 5 indicating strong agreement.
SICCA registry cohort. The participants in the SICCA
cohort have been enrolled since 2004 in 5 collaborating
academically-based research groups, located in Argentina,
China, Denmark, Japan, and the US, and directed from the
University of California, San Francisco (12) (Table 1). Sub-
sequently, additional research groups joined the SICCA
project: in 2007 from the UK and in 2009 from India and 2
additional sites in the US.
To be eligible for the SICCA registry, participants must
be at least 21 years of age and have at least 1 of the
following: symptoms of dry eyes or dry mouth; a previous
suspicion or diagnosis of SS; elevated serum antinuclear
antibody (ANA) titer, positive rheumatoid factor (RF), or
anti-SSA and/or anti-SSB; bilateral parotid enlargement in
a clinical setting of SS; a recent increase in dental caries; or
diagnoses of rheumatoid arthritis (RA) or systemic lupus
erythematosus (SLE) and any of the above. The rationale
for these eligibility criteria is that only patients with such
characteristics would be evaluated for SS or considered for
enrollment in a clinical trial designed to evaluate a poten-
tial therapeutic agent for SS. Therefore, our classification
criteria target individuals with signs and symptoms that
may be suggestive of SS, not the general population.
Participants are recruited through local or national SS
patient support groups, health care providers, public me-
dia, and populations served by all 9 SICCA research
groups. Exclusion criteria include known diagnoses of
hepatitis C, human immunod