These materials were assembled
by the following participants in the workshop: Kristen
Arbutiski, Julie Ballin, Matt Bietz, Jeremy Birnholtz,
Tom Finholt, Darren Gergle, Erik Hofer, Gary Olson,
and Diane Sonnenwald.
Measures of collaboratory
All of the discussion groups, to varying degrees, began
consideration of collaboratory success by first attempting
to define a collaboratory.
Synthesizing across these efforts produces
the following description:
is a network-based facility and organizational entity
that spans distance, supports rich and recurring human
interaction oriented to a common research area, fosters
contact between researchers who are both known and
unknown to each other, and provides access to data
sources, artifacts and tools required to accomplish
Several features of this definition are helpful in thinking
about how to measure collaboratory success.
First, the emphasis on spanning distance suggests
that when measuring collaboratory success, the correct
comparison is with other forms of long-distance collaboration
– and not with collocated collaboration.
Second, the emphasis on contacts, recurring
interaction, and artifacts suggests that a critical
contribution of a collaboratory is to be a social
arena for research work – but an arena that
is not tied to a particular physical space.
And finally, the emphasis on research suggests
that the impact of collaboratories must be evaluated
in terms of products critical to the research endeavor,
such as creation of new knowledge and techniques.
following examples illustrate different kinds of measures
that may reflect success in collaboratory-based collaboration,
both in terms of process and product, when compared
to more traditional long-distance collaboration: frequency
and impact of discoveries; frequency and impact of
publications; diversity among collaborators (on multiple
dimensions); satisfaction of collaborators; frequency
of travel; number and prestige of honors accumulated
by collaborators; proportion of collaborations composed
of first-time collaborators; impact of the collaborations
on the training of young scientists; attracting newcomers
to the field, frequency and duration of delays; level
of interpersonal trust; extent of shared mental models;
degree of mutually consistent work practices; and
level of public interest.
proposed measures form the focus for three principal
research threads: a) exploration of the feasibility
and sensitivity of the success measures using retrospective
and prospective data from operational collaboratories
– in comparison with traditional long-distance
collaborations; b) exploration of the different collaboratory
types that have evolved (e.g., integrated versus aggregation
of separate applications) – and how these types
influence success; and c) exploration of how collaboratory
stakeholders, level of analysis, and time interact
to influence success.
An overarching concern was the problem of finding
a sufficient number of viable collaboratories to form
a sample for examining the impact of collaboratories.
here for detailed notes on measures)
Group processes in collaboratories
A way to view group processes in collaboratories is
to see these as independent variables that influence
the success measures summarized in the preceding section. This section, then, is divided into two parts. First, a list of candidate processes are
these candidate processes are analyzed in terms of
their importance to understanding collaboratory success.
The processes suggested in the discussion groups included:
a) awareness, meaning the availability and interpretation
of cues about what others are doing; b) task interdependence,
such as Van de Ven et al.'s (1976) scheme of pooled,
sequential, reciprocal, and team tasks; c) maintenance
of common ground; d) group dynamics (including trust,
leadership, conflict resolution, decision making,
roles, and goal setting); e) creation of routines
and procedures; f) communication, such as cultural
sensitivity and fluency; and g) interaction of task
and process characteristics and technology (e.g.,
scarcity of instrument time).
guide examination of the processes listed above, workshop
participants discussed building on an effort launched
by the Science of Collaboratories group at Michigan
to produce a framework for identifying variables of
This framework can be developed both retrospectively based
on data previously collected and prospectively for
new and ongoing collaboratories.
Examples of relevant contexts include face-to-face
collaboration within and across science and engineering
labs and distributed collaboration outside of a collaboratory.
additional suggested starting point was to consider
how the identified processes and variables might moderate
and mediate collaboratory success.
That is, from Baron and Kenny's (1986) description,
a moderator variable influences the strength or direction
of the relationship between an independent and a dependent
variable and a mediator variable explains the mechanism
behind the relationship.
In the case of collaboratories, for example,
there may be an interest in how a particular measure
of collaboratory use relates to a particular measure
of collaboratory success – where this relationship
may be moderated by some factors (e.g., task interdependence)
and mediated by others (e.g., awareness).
here for detailed notes on processes)
Organizational and community
consequences of collaboratory use
discussions of organizational and community consequences
of collaboratory use were focused along two lines:
examination of potential consequences, assuming widespread
collaboratory use; and specific collaboratory-based
interventions that might have beneficial outcomes
(e.g., expanding the pipeline for training of future
access – While some workshop participants noted that claims
for expanded access to science, through collaboratory
use, are probably exaggerated – there was still
hope that collaboratory development might create new
opportunities for broader participation in research
(particularly among institutions with fewer resources,
both domestically and internationally).
Specifically, collaboratories may provide a
way for faculty at Master's Colleges and Universities
I and II (i.e., from the Carnegie Classification –
institutions that are primarily oriented to undergraduates
and award 20 to 40 masters degrees per year) to maintain
active research involvement with colleagues at Doctoral/Research
Universities – Extensive (i.e., from the Carnegie
Classification – institutions with undergraduate
and graduate programs, typically awarding 50 or more
doctorates per year across 15 or more disciplines).
Also, collaboratories may provide a compelling
mechanism for outreach, particularly to K-12 audiences
(i.e., via science museums, classroom use and etc.).
transformation – Again, while some participants expressed skepticism
– many expected that widespread collaboratory
use would produce changes in the structure of scientific
Specifically, collaboratories were seen
as accelerating trends toward increased multi-disciplinarity
(although there were concerns about the scientific
value of multi-disciplinarity) and toward cross-institutional
there was an expectation that feedback might occur,
such that wider adoption of previously unorthodox
practices (e.g., electronic publication) and career
tracks (e.g., outside the tenure system) may also
produce incentives that reward collaboratory use –
which in turn, may accelerate transformation of the
status quo (e.g., by increasing visibility of junior
scientists or of researchers outside the scientific
kinds of science – A principal envisioned impact was the capability
to bring previously unrelated communities of researchers
together through the capacity of collaboratories to
combine data in new ways.
For example, in the Space Physics and
Aeronomy Collaboratory, observational and simulation
data can be viewed side by side, which has led to
new interactions between theoretical space physicists
and experimental space physicists.
production – In the discussion of success measures, increased
rates of publication and discovery were identified
as important outcomes of collaboratory use.
Discussions focused on the sensitivity of such
measures with particular attention to the lag between
adoption of innovative practices and impact in terms
of publication or citation patterns.
Comparisons to earlier innovations, like email and electronic
journals, suggested that even at mature points in
the penetration of these innovations – it was
still hard to see impact in terms of citation data.
However, broader access to the Internet does
seem to have played a role in the dramatic increase
of papers within one very special class (authors from
ten or more different countries).
Collaboratory impact might be most visible
in the case of similar “mega-collaborations.”
One proposal was to track the number of
different congressional districts represented among
authors on papers from collaboratory-based collaborations
versus conventional long-distance collaborations.
terms of next steps for research there was a reluctance
to pursue the connection between collaboratory use
and broad changes – both because of the immaturity
of collaboratory technology and because of the multi-causal
nature of large transformations (e.g., the trend toward
multi-institution authorship has been increasing for
several decades – pre-dating the Internet, collaboratories
and other more recent inventions).
However, there was enthusiasm for tightly scoped
interventions with high impact, both in terms of outcomes
and as demonstrations.
For example, in terms of Ph.D. production in
the United States, a small set of selective liberal
arts colleges have performed at a disproportionately
high level (Carrier & Davis-Van Atta, 1987) –
with corresponding disproportionate representation
among authors of highly-cited publications and among
members of the National Academy of Sciences.
The keys to this success are partially related to the admission
criteria of these institutions – but other important
factors include exposure to hands on research activity
and mentorship by faculty.
With the right mixture of support, collaboratories
might be used to produce the conditions currently
found only at these selective liberal arts colleges,
but across a wider array of institutions – with
corresponding increases in the number of students
choosing to pursue graduate degrees and research careers.
here for detailed notes on organizational and community
R.M., & Kenny, D.A. (1986). The moderator-mediator
variable distinction in social psychological research:
Conceptual, strategic, and statistical considerations.
Journal of Personality and Social Psychology,
S., & Davis-Van Atta, D. (1987).
Maintaining America's scientific productivity
Oberlin, OH: Oberlin College.
de Ven, A.H., Delbecq, A. L., & Koenig, R., Jr.
Determinants of coordination made within organizations.
American Sociological Review, 41(3),