Science of Collaboratories logo - link
An alliance to advance the understanding of collaboratories
Science of Collaboratories
Notes on Measures of Collaboratory Success


Summary Prepared by Matthew Bietz, Darren Gergle, and Gary Olson

The first question we addressed in the workshop was: How would one judge a collaboration or a collaboratory to be successful? While we were unable to develop a universal measure of success for collaboratories, we were able to make progress on developing a framework for measuring the success of collaboratories, eliciting a number of possible measures of success, and pinpointing open issues and problems.


We proposed a 3-dimensional framework for thinking about measurement of collaboratories:

Dimension 1:Stakeholders

  • technologists
  • domain
  • userssocial
  • scientists
  • sponsors/funders
  • corporations

Dimension 2: Level of analysis

  • individuals
  • groups
  • organizations
  • fields
  • social policy

Dimension 3:Time

short-term vs. long-term measures of success

The measures used to gauge success will vary on each of these three axes:

A social scientist may consider a collaboratory successful because it produced new knowledge about how people work, even if it did not advance the domain.

Similarly, a collaboratory that leads to successful organizational outcomes may have a negative impact on some individual careers. 

The third dimension, time, takes into account that second-order effects may be slower to appear.

A proposed dimension for the framework involved a focus on the collaboratory as a socio-technical system or on the collaboration process itself.  In retrospect, this distinction seems to overlap with the other dimensions.  We will probably focus more on socio-technical environment when looking at longer time spans and larger levels of analysis.  Shorter time spans and individual levels of analysis lead to a focus on the micro-processes of collaboration.

Domain sits as a layer on top of this framework.  The measures of success may not be the same for AIDS researchers and physicists.  In addition, the definition of the axes may be altered based on various domains.  Some fields may have a relatively homogeneous set of users, while others may have many different sub-categories with different success criteria.  The time dimension may differ among domains, for example, on the length of time for a study to reach publication.

Measures of Success

Classes of Criteria Specific Criteria
Use of the collaboratory tools Designer/Builder demos it working
Original target users use it with direct support
Original target users use it unaided
Original target users sustain its use
New users try it successfully
New users continue to use it
Users complain when it is taken away (e.g., fails to work with new OS)
Collaboratory moves from research prototype to production system
Collaboratory is sustained
Collaboratory project is refunded
Software technology Software developed that is reused
Lessons learned about technical issues
Lessons were published
Direct effects on the science Feasability demonstrations for new forms of scientific work
New forms of work are sustained
Existing collaborations work more smoothly or quickly
More collaborations are attempted
New collaborations are formed
Collaborations have greater geographic spread
New discoveries are made as a result
A big discovery is made
Discoveries made more quickly
A conceptual revolution in the science is enabled
More jointly authored papers
Jointly authored papers produced more quickly
More papers published in general
Findings are shared more quickly (multiple pathways)
Improved quality of life for researchers (e.g., less travel)
More high quality research is conducted
Those using collaboratories take on major roles (lab director, journal editor, president of society)
Researchers organized in different ways
The artifacts that are shared are richer
Desirable cogntive activities more frequent (analogies, anomaly detection, novel methods)
Less undesirable duplication
Resarchers are more satisfied
Less burnout, departures from field, etc.
Change in the mix of normal vs. revolutionary science
New models of science emerge (e.g., micro-participation)
Theoretical discussions are accelerated, enriched
Some kind of barrier crossed (distance, discipline, instituion,…)
Greater willingness to exchange early ideas
Greater diversity of participation in research
Greater success in competitive areas J(I.e., win more often)
Few distruptive activities (such as trips to Greenland)
Willing to initiate a project, write a grant proposal
Willing to invest in ftf interactions, knowing collaboratory can sustain
Science careers Higher liklihood of tenure
Faster to tenure
Greater diversity of scientists
Effects on learning, science education Students are mentored using the collaboratory
The collabortory is used in classroom instruction
New distance learning paradigms emerge from collaboratory
New students are attracted to the field as a result of the collaboratory
Faster time to PhD in science fields
Scientists in related fields learn from the collaboratory
Virtual colloquia & seminars extend the reach
Easier to establish common ground
Tacit knowledge shared
Scientific participation extends beyond R1 universities
Increased diversity of those attracted to science careers
Students attracted from a wide range of nations
Inspiration for other collaboratories New collaborative capabilities are demonstrated
Other software is built for the field inspired by it
New collaboratories are developed as a result of it
Learning about collaboratories in general Lessons were learned about how to build collaboratories
Lessons were published
Effects on funding, public perception The public becomes more interested in the specific science
The public becomes more interested in science in general
Public literacy about science increases
Public more interested in participating
Congress becomes more interested in science
New funding initiatives in the specific area of science
New funding initiatives for collaboratories in various areas of science

Open Issues

Measurement issues

How can we know what to measure from the start (e.g., good baseline information)?

How do we integrate qualitative and quantitative data? When are each appropriate?

How do we measure secondary effects?

How does the evolution of the collaboratory affect which measures are appropriate and when?

Measures will be different based on the kind of collaboratory, is there a way to integrate?

How do we deal with the effects of time on measures of success

Lag in the results of increases for things like citations.

Each collaboratory may progress at a different timescale and level of granularity

How do you deal with successful uses of the collaboratory that remain invisible.

E.g., Traditional success measures not always useful in determining the success of the collaboratory (e.g., successful collaboration, but the hypothesis under investigation fails)

Are different measures needed to differentiate productivity at the levels of the individuals, groups and within the entire community or larger culture as a whole?

How can less ambitious results and goals be used in the measurement of success?

Utility of measures

Are these measures always indicative of good things?

Are the measures of success truly indicative of scientific advancement?)

Larger questions

Is multi-disciplinary work better for science overall?

Do collaboratories have better research coming out because people can share and test hypotheses? Basis of design makes communication and results faster and easier.

Is 'faster and cheaper' better?

Can new ideas be quelled early on due to wider exposure garnered from collaboratories?

Is there a lost efficiency in collaborations due to the duplication of effort, requirements to learn new terminology, or new interaction patterns?

Ortega hypotheses. Science moves by key papers and do collaboratories support this or just more widespread and lower quality results?

How do you use anecdotal evidence effectively? (e.g., to nurture funding)

A distinction needs to be made between the process of doing science and the productivity of science. There may be important and interesting changes that occur in the process that traditional measures do not capture.

What are collaboratories?

"Technology-mediated scientific collaboration at a distance."

A place where people can discuss common terms and ideas.

People are brought together (either physically or virtually) to solve common problems.

Solving distance issues and a place where participants can share ideas more often.

Often involves people who don't know each other personally, but are in the same field.

May form a virtual proximity.

A certain level of interaction is facilitated between or among individuals or groups.

Provides way for detection and resolution of central and peripheral problems.

What is collaboration?

Is a citation a collaboration? (e.g., putting things into database)

Is there a requirement of interaction for collaboration?

Can you have chains of interaction to form collaboration?

Allows individuals to have collaboration without common knowledge of the collaboration

E.g., Genbank has a community and model of the way things get done, and can this then be collaborative

Necessary conditions for a collaboratory

Whether collaboratory activity and success could include individual work done within the collaboratory.

How to articulate the necessary social environment for a collaboratory.

Whether such things as supercomputing facilities could be considered

What kind of distances must collaboratories span (geographic, functional, and institutional)?

Tension between collaboratory use as the enabler of collaborations vs. the use of facilities designed to enable collaboration.

Suggestions for what isn't a collaboratory

Not doing anything new.

No attempt to create synergy.

Not bringing new people into contact with one another.

Might be an actual collaboration using Finholt's demonstration of German use of tele-pointers and interaction in a collaboration to discuss and have mutually constructed understanding.

Canned presentation without support for collaboration of people (perhaps just a presentation).

E.g., a virtual seminar may be more towards a collaboration than others.

Broadcast types of interactions (e.g., lectures in distance learning).

However, it doesn't provide a means for return interaction.

Next Steps

As an overall goal, we need to develop theory around collaborations and collaboratories with technology – make a jump from explaining what goes on in collaboratories to theorizing why it happens. As part of this, we envision four documents:

  1. Stock-taking: document what we know, inventory of criteria of success, etc
  2. what do we know works

    run it on existing collaboratories to see if they are matching the criteria (to check inventory and to tell something about collaboratories)

    possibly include known best practices

  3. Document on what data we need to collect and how to collect it (and where): more focuses on prospective piece
  4. Get coordination across projects to increase sample size and get same measures

    Maybe interview current collaboratory researchers about what they wish they'd asked in previous surveys but did not

    Problem identified/open issue: how to obtain base rate information – how do we know that collaborations would not occur without the collaboratory, we don't have a lot of information about how people collaborate generally (not necessarily under the umbrella of a collaboratory)

    If starting in the beginning of a collaboratory:

    Who is friends with whom (B. Wellman studies – friendship matters in collaborations); diferrentiate between knowing and friendship

    Make an investment in unobtrusive measures – this won't get us everything we want to know but we can do more with them

    Observation of fine grain process of collaboration (collect all drafts of a co-authored paper and track them across time, maybe through email or technology


    We need to establish a baseline:

    efficiency (time to publication)

    grad student learning



  5. Design Requirements

    Can we make suggestions for collaboratories that are informed? Make general recommendations for what works

    Mapping all of it into requirements for collaboratories

  6. Taxonomy: define a collaboratory –distinguish between types (CFAR is different from SPARC, is different from Nestor's stuff…)
  7. types

    dimensions on which they differ (synchronous, asynchronous, etc.)

    levels of commitment of participants/centrality of collaboratory to scientific practice

    Are they based on capital or human resources – access to instrumetation or to other people?

    Shared instrumentation, data or expertise?

    How do these change across time?

    Access control, roles in the collaboratory

    What stage of the science does it support?

Back to the Final Summary main page



  Home | About SOC | Workshops | Resources | News & Events  

University of Michigan Logo

University of Michigan

School of Information Logo

School of Information University of Michigan