Summary Prepared by Matthew Bietz, Darren Gergle,
and Gary Olson
The first question we addressed in the workshop
was: How would one judge a collaboration or a collaboratory to be
successful? While we were unable to develop a universal measure
of success for collaboratories, we were able to make progress on
developing a framework for measuring the success of collaboratories,
eliciting a number of possible measures of success, and pinpointing
open issues and problems.
We proposed a 3-dimensional framework for
thinking about measurement of collaboratories:
Dimension 2: Level of analysis
- social policy
short-term vs. long-term
measures of success
The measures used to
gauge success will vary on each of these three axes:
A social scientist may
consider a collaboratory successful because it produced new knowledge
about how people work, even if it did not advance the domain.
Similarly, a collaboratory
that leads to successful organizational outcomes may have a negative
impact on some individual careers.
The third dimension,
time, takes into account that second-order effects may be slower
A proposed dimension for the framework involved
a focus on the collaboratory as a socio-technical system or on the
collaboration process itself. In retrospect, this distinction
seems to overlap with the other dimensions. We will probably
focus more on socio-technical environment when looking at longer
time spans and larger levels of analysis. Shorter time spans
and individual levels of analysis lead to a focus on the micro-processes
Domain sits as a layer on top of this framework.
The measures of success may not be the same for AIDS researchers
and physicists. In addition, the definition of the axes may
be altered based on various domains. Some fields may have
a relatively homogeneous set of users, while others may have many
different sub-categories with different success criteria.
The time dimension may differ among domains, for example, on the
length of time for a study to reach publication.
Measures of Success
|Classes of Criteria
of the collaboratory tools
demos it working
| Original target
users use it with direct support
| Original target
users use it unaided
| Original target
users sustain its use
| New users try
| New users continue
to use it
| Users complain
when it is taken away (e.g., fails to work with new OS)
moves from research prototype to production system
project is refunded
| Software technology
|| Software developed
that is reused
| Lessons learned
about technical issues
| Lessons were
| Direct effects
on the science
demonstrations for new forms of scientific work
| New forms of
work are sustained
| Existing collaborations
work more smoothly or quickly
| More collaborations
| New collaborations
have greater geographic spread
| New discoveries
are made as a result
| A big discovery
made more quickly
| A conceptual
revolution in the science is enabled
| More jointly
| Jointly authored
papers produced more quickly
| More papers
published in general
| Findings are
shared more quickly (multiple pathways)
| Improved quality
of life for researchers (e.g., less travel)
| More high quality
research is conducted
| Those using
collaboratories take on major roles (lab director, journal editor,
president of society)
organized in different ways
| The artifacts
that are shared are richer
| Desirable cogntive
activities more frequent (analogies, anomaly detection, novel
| Less undesirable
are more satisfied
| Less burnout,
departures from field, etc.
| Change in the
mix of normal vs. revolutionary science
| New models
of science emerge (e.g., micro-participation)
discussions are accelerated, enriched
kind of barrier crossed (distance, discipline, instituion,…)
willingness to exchange early ideas
diversity of participation in research
success in competitive areas J(I.e., win more often)
activities (such as trips to Greenland)
to initiate a project, write a grant proposal
to invest in ftf interactions, knowing collaboratory can sustain
| Science careers
liklihood of tenure
diversity of scientists
| Effects on
learning, science education
are mentored using the collaboratory
is used in classroom instruction
learning paradigms emerge from collaboratory
are attracted to the field as a result of the collaboratory
time to PhD in science fields
in related fields learn from the collaboratory
colloquia & seminars extend the reach
to establish common ground
participation extends beyond R1 universities
diversity of those attracted to science careers
attracted from a wide range of nations
for other collaboratories
capabilities are demonstrated
software is built for the field inspired by it
are developed as a result of it
| Learning about
collaboratories in general
were learned about how to build collaboratories
| Effects on
funding, public perception
becomes more interested in the specific science
becomes more interested in science in general
literacy about science increases
more interested in participating
becomes more interested in science
initiatives in the specific area of science
initiatives for collaboratories in various areas of science
How can we know what to measure from the start
(e.g., good baseline information)?
How do we integrate qualitative and quantitative
data? When are each appropriate?
How do we measure secondary effects?
How does the evolution of the collaboratory
affect which measures are appropriate and when?
Measures will be different based on the kind
of collaboratory, is there a way to integrate?
How do we deal with the effects of time on
measures of success
Lag in the results of increases for things
Each collaboratory may progress at a different
timescale and level of granularity
How do you deal with successful uses of the
collaboratory that remain invisible.
E.g., Traditional success measures not always
useful in determining the success of the collaboratory (e.g., successful
collaboration, but the hypothesis under investigation fails)
Are different measures needed to differentiate
productivity at the levels of the individuals, groups and within the
entire community or larger culture as a whole?
How can less ambitious results and goals be
used in the measurement of success?
Utility of measures
Are these measures always indicative of good
Are the measures of success truly indicative
of scientific advancement?)
Is multi-disciplinary work better for science
Do collaboratories have better research coming
out because people can share and test hypotheses? Basis of design makes communication and results faster and
Is 'faster and cheaper' better?
Can new ideas be quelled early on due to wider
exposure garnered from collaboratories?
Is there a lost efficiency in collaborations
due to the duplication of effort, requirements to learn new terminology,
or new interaction patterns?
Ortega hypotheses. Science moves by key papers
and do collaboratories support this or just more widespread and lower
How do you use anecdotal evidence effectively?
(e.g., to nurture funding)
A distinction needs to be made between the
process of doing science and the productivity
of science. There may be important and interesting
changes that occur in the process that traditional measures do not
What are collaboratories?
"Technology-mediated scientific collaboration
at a distance."
A place where people can discuss common terms
People are brought together (either physically
or virtually) to solve common problems.
Solving distance issues and a place where
participants can share ideas more often.
Often involves people who don't know each
other personally, but are in the same field.
May form a virtual proximity.
A certain level of interaction is facilitated
between or among individuals or groups.
Provides way for detection and resolution
of central and peripheral problems.
Is a citation a collaboration? (e.g., putting
things into database)
Is there a requirement of interaction for
Can you have chains of interaction to form
Allows individuals to have collaboration without
common knowledge of the collaboration
E.g., Genbank has a community and model of
the way things get done, and can this then be collaborative
Necessary conditions for a collaboratory
Whether collaboratory activity and success could
include individual work done within the collaboratory.
How to articulate the necessary social environment
for a collaboratory.
Whether such things as supercomputing facilities
could be considered
What kind of distances must collaboratories
span (geographic, functional, and institutional)?
Tension between collaboratory use as the enabler
of collaborations vs. the use of facilities designed to enable collaboration.
Suggestions for what isn't a collaboratory
Not doing anything new.
No attempt to create synergy.
Not bringing new people into contact with
Might be an actual collaboration using Finholt's
demonstration of German use of tele-pointers and interaction in a
collaboration to discuss and have mutually constructed understanding.
Canned presentation without support for collaboration
of people (perhaps just a presentation).
E.g., a virtual seminar may be more towards
a collaboration than others.
Broadcast types of interactions (e.g., lectures
in distance learning).
However, it doesn't provide a means for return
As an overall goal, we need to develop theory
around collaborations and collaboratories with technology –
make a jump from explaining what goes on in collaboratories to theorizing
why it happens. As part of this, we envision four documents:
- Stock-taking: document what we know, inventory
of criteria of success, etc
what do we know works
run it on existing collaboratories to see
if they are matching the criteria (to check inventory and
to tell something about collaboratories)
possibly include known best practices
- Document on what data we need to collect
and how to collect it (and where): more focuses on prospective
Get coordination across projects to increase
sample size and get same measures
Maybe interview current collaboratory researchers
about what they wish they'd asked in previous surveys but
Problem identified/open issue: how to obtain
base rate information – how do we know that collaborations
would not occur without the collaboratory, we don't have a
lot of information about how people collaborate generally
(not necessarily under the umbrella of a collaboratory)
If starting in the beginning of a collaboratory:
Who is friends with whom (B. Wellman studies
– friendship matters in collaborations); diferrentiate
between knowing and friendship
Make an investment in unobtrusive measures
– this won't get us everything we want to know but we
can do more with them
Observation of fine grain process of collaboration
(collect all drafts of a co-authored paper and track them
across time, maybe through email or technology
We need to establish a baseline:
efficiency (time to publication)
grad student learning
- Design Requirements
Can we make suggestions for collaboratories
that are informed? Make
general recommendations for what works
Mapping all of it into requirements for
- Taxonomy: define a collaboratory –distinguish
between types (CFAR is different from SPARC, is different from
dimensions on which they differ (synchronous,
levels of commitment of participants/centrality
of collaboratory to scientific practice
Are they based on capital or human resources
– access to instrumetation or to other people?
Shared instrumentation, data or expertise?
How do these change across time?
Access control, roles in the collaboratory
What stage of the science does it support?
Back to the
Final Summary main page