Collaboratories at a Glance (C@G)
Presenters: Judy Olson, Nathan Bos, and Erik Dahl
Discussants: Jonathan Grudin, Deborah Agarwal, John Walsh
The Collaboratories at a Glance (C@G) is the "baseball card" view of collaboratories. It is an attempt to gather some standard facts and figures about a large number of collaboratories to allow an initial look into what the major types are, associated timelines, and some other major trends. We are following Stu Card's model of the major stages in the development of a technical field (Olson et al, 1993):
The C@G is an attempt to describe the "points in the field" that the SOC project is trying to dimensionalize, in order to discover behavioral clusters. The workshop participants were encouraged to help the SOC group assess if the right dimensions are being attended to.
- People make example systems to explore possibilities, which can be thought of as points in a field
- People reflect on the dimensions on which these points differ,
- Behavioral and feature level clusters start to become apparent,
- Then some laws can be drawn from the regularities seen in the clusters.
The major goals of the SOC C@G project, both theoretical and practical, are to:
- Understand the space of collaboratories by comparing and contrasting instances of collaboratories and distilling the successes and challenges facing different collaboratories. While looking into these collaboratories the researchers will pay particular attention to collecting a number of stories about what works and what doesn't work.
- Develop a theory of collaboratories as organizational entities. We believe that collaboratories are a new organizational form allowing people to take advantage of working remotely
- Develop practical prescriptions, so that when a new collaboratory comes along, we ask the right questions about the management, about the nature of the work, how long these people worked together, what is their pedigree, do they respect each other, etc. This will form the basis for making recommendations about both sets of technologies and social practices in order to be successful.
A secondary goal of C@G is to discover the range of collaboratories so that we can make wise choices of which collaboratories to investigate in-depth.
The method we used to develop these included first collecting a large set of collaboratory instances (at the time the set numbered about 83 possible candidates). We asked the workshop participants to suggest ones not yet on our list so that it can be more complete. The plan is to collect a basic set of information about all these collaboratories, and to note similarities and differences on both technical and social dimensions. This included an examination of the existing literature on what drives successes and challenges in organizations. From this material the researchers will begin to dimensionalize the space, and hypothesize causal relationships.
Early in the project we used as a guide a simple frame of describing the way people interact with each other, with remote instruments, and remote data sources Our early attempts at grouping the collaboratories followed these dimensions, each collaboratory providing access to a different mix of resources.
Noting that this was too simple a view, we engaged in extensive debate about which of these features really make a collaboratory and which ones do not. For example, some organizations called collaboratories have extensive use of remote instruments but not much interaction among the researchers. We decided not to put a hard boundary on the set of organizations we look at, but rather to be more inclusive at the beginning, and develop a more "hard-line" definition later. By being more inclusive, there may be some lessons learned from "incomplete" collaboratories that would be useful for more complete ones.
The current definition of a collaboratory has evolved somewhat from that produced at the first workshop. Breaking it up into essential elements, a collaboratory is
The Science of Collaboratories project is particularly focused on investigating science and engineering collaboratories, with the understanding that the lessons gleaned from this investigation will apply to other types of collaboratories or collaborations.
- an organizational entity
- that links a community of individuals
- working at a distance
- on common problems or tasks
- electronic tools that support
- rich and recurring human interaction and
- provides common access to resources, including information and instrumentation, needed to engage in the problems or tasks.
Types of collaboratories
The C@G group, by investigating a wide set of collaboratories, had found seven different kinds of collaboratories. The seven types fall into two groups - those with a research focus and those with a practice focus.
The ones with a research focus are:
Distributed Research Center,
Product Development, and
Community Data Systems.
Those with a practice focus are:
Virtual Community of Practice,
Virtual Learning Community, and
The Distributed Research Center type can be understood as functioning like a research center, but one that spans distances. These collaboratories are unified by a topic area of interest, and most often include a number of joint projects in that topic area. The communication (or access to resources) is primarily human-human contrasting with the human-machine communication prevalent in a Shared Instrument Collaboratory. These collaboratories also tend to not have a well-specified product as their focus, though long-term generalized goals might include a product. An example of this type of collaboratory would be the Alliance for Cell Signaling.
The Shared Instrument Collaboratory focuses on providing access to a remote, usually expensive, scientific instrument. This type of collaboratory is often supplemented with some other technology to support communication usually between remote scientists and instrument operators. Occasionally there is communication between researchers as well. An example of this type of collaboratory would be the Keck Observatory.
The Product Development Collaboratory focuses on the building of a product, usually a fairly tangible thing that might be an instrument, some infrastructure, a policy, a particular research methodology, etc. These types of collaboratories tend to be more time-bounded than others. A key challenge for Product Development collaboratories, is dealing with "drift" - a phenomena that occurs when people at individual sites make small changes in focus or goals that make perfect sense in their own setting, but that over time, the aggregation of all these changes lead to a project that is going in multiple directions. An example of a Product Development Collaboratory is Intermed Collaboratory. Some excellent evaluative material on Intermed has been written by Dr. Vimla Patel (Patel et al, 1999).
The Community Data System type tends to take the form of an information resource that is created, maintained, or improved by a distributed community. The information is most often semi-public and of wide interest to the community. The key challenge encountered by Community Data Systems is the difficulty that can arise in motivating people to contribute. The situations that develop around these collaboratories often take on the character of a social dilemma, where the incentive structure is very important in determining the success of the collaboratory. An example of this type of collaboratory would be the Zebrafish Information Network.
The Virtual Community of Practice is essentially a network of individuals who share a research area and communicate about it online. The communications tend to focus on sharing news of professional interest, advice, techniques, etc., and do not usually focus on joint projects. An example of this type of collaboratory would be the Ocean US.
The Virtual Learning Community has as its main focus the increasing of the knowledge of the participants. These collaboratories are not formed to do original research, but instead are more interested in providing in-services or professional development. An example of this type of collaboratory would be the Ecology Circuit Collaboration.
The Expert Consultation Collaboratory provides increased access to an expert or set of experts. The flow of information is mainly one-way rather than two-way as in a distributed center. An example of this type of collaboratory would be the EU TeleInViVo.
Most collaboratories are not purely one type; they have characteristics that span multiple categories. The SOC researchers chose one of the seven categories described above as a primary category for each of the collaboratories, but then also mentioned what other categories also applied to each collaboratory. The decision of how to categorize a collaboratory had much to do with which type best fit the collaboratory, or described what seemed to be its main goals or activities.
Clustering the collaboratories according to the major functions they serve allows for the comparison across collaboratories, with hopes of discovering significant connections between category type and other dimensions, such as what technologies were used, how money flow might have affected the work, if common successes and challenges were involved, etc. Our focus is currently on a small set of data elements, making the project a manageable one, with relatively complete data collected on 34 collaboratories to date.
The SOC group also using a standard diagramming convention to display resources, institutions, money flow, communication and others in order to glean other significant similarities and differences. An example of each of the major types is shown below.
Some of the early discoveries in mining the C@G involved looking at what technologies were in use by different projects. (Technical Capabilities of Collaboratories (PDF)) For example, there were a wide variety of different needs leading to different use of video technology. CFAR used off-the-shelf tools (NetMeeting), the Alliance for Cell Signaling used Polycom desk and room systems, whereas other projects had future plans to use the Access Grid for video conferencing. These different approaches have significantly different demands on resources, and meet different objectives. As this data is gathered, the SOC C@G group is making the connections between the use of particular technologies in particular situations with differing constraints, and paying attention to the successes and challenges that emerge from these situations so useful recommendations can be made for future efficient use of resources.
Other comparisons focused on noting that different Shared Instrument Collaboratories might handle instrument control in different ways. For example, in Bugscope, children can zoom and move the view on the instrument; in SPARC, because of safety issues, only an on-site operator can move the large radar. This of course greatly effects what system can be built into the collaboratory to support the needs of the users.
Other findings noted the different ways Community Data Systems are populated. In the Alliance for Cell Signaling, for example, data is managed as a vetted contribution by many members, whereas in the Visible Human project, one group provides the core dataset for many others to use.
SOC is making this data available openly, and encouraging others to fill in the C@G data elements for including in the set, making it a Community Data System. This will likely include the ability for users to enter collaboratories, as well as use our evaluation methods and tools (such as the in-depth interview protocol) on collaboratories users are participants in, in order to further the data reach of the SOC community. This will be a "collaboratory of collaboratories."
Jonathan Grudin with commentary from workshop participants
The timing of this data collection effort is excellent. The study of collaboratories is approaching a turning point in the maturation and routinization process. It is similar to the development of the Man-Machine Interaction field of study when it turned into the more widely known Human-Computer Interaction, and how ARPANET eventually turned into the INTERNET. Both of these are examples of situations where some group of people had the incentive to make the huge requisite effort to get the initial phases of a new technology (or way of thinking) up and running, so that the practice becomes commonplace. The study of collaboratories is much like these efforts, in that it is still in the compilation and analysis phase that requires a huge effort, but that it will not be long before collaboration technologies become widely adopted.
Many operating system (OS) developers are currently focused on collaboration support. These OSs are probably five years away from being deployed, so the developers are ripe for input into what they should include into the OSs. The workshop participants should communicate the results of this SOC project to these OS developers, in order to help them to do their job properly.
Being clear about the goals of the SOC effort is very important. There is a touch of madness in the scientific method, in that progress looks inevitable in hindsight. The development of the categories is difficult. When developing any taxonomy, typology or classification scheme, what is really the most important consideration is whether the distinctions being drawn will be useful. Including the secondary categories may or may not be useful. It could just make things cloudy when it comes to making assertions about a particular category. The point that ultimately matters is really whether or not the categories are useful, so the SOC project might want to constantly check against that. We cannot simply trust that things will all pay off in the end. But the best way to start is to start.
Collaboratories are moving targets. They are constantly evolving, and any effort to categorize these entities will have to reckon with that aspect. Collaboratories can change their focus, their domain of concern, their use, etc.
It is important to capture collaboratory failures. There is a real need to discover the shortcomings or failures and lack of adoptions, etc. Different facets of failure are important, such as the context of failure - when something does not work in one place, but does work somewhere else. Understanding this contextualized success is important, but not always something that gets the attention it needs.
Information about failures is an extremely difficult thing to gather, particularly through self-reporting. 'Best practices' are difficult enough to gather, but less successful practices are almost impossible to document, and they are just as important as 'best practices.' These failures are difficult to collect for a variety of reasons, including
1) You don't get funding for what didn't work,
2) People will often finger-point and bicker when they try and document why something didn't work,
3) It is depressing to dwell on failures, and groups instead tend to use their energies jumping into the next project.
In the SOC project, one of the intentions of doing the in-depth collaboratory investigation is to get at this deeper understanding of a collaboratory, and get to know what really happened, or is happening, beyond the publicity material.
It can also be important to learn how much effort went into the development of a collaboratory. This sort of information allows researchers and developers to think about where to put attention on making things easier to succeed, as well as where it simply was easy to succeed, what it is we want to replicate.
We should also collect information about whether the support and enthusiasm for the collaboratory persists. Surveys can capture a given level of enthusiasm, but the surveys don't accurately capture a good indication of "future use."
Deborah Agarwal with commentary from workshop participants.
The "research focus," or goal, of a collaboratory is an important element to consider when determining the various categories of collaboratories. The C@G should have more information in it that would allow researchers to gauge the success of the collaboratory. Questions she would like to be able to answer with the C@G include:
John Walsh with some commentary from workshop participants
- Did it go into use?
- Did it have the intended result?
- use in practice
- Who used it and why?
- How much did it augment versus change the way people interact?
- How was it promoted and funded?
- Did it persist beyond its development funding?
- What purpose did it end up serving?
The database being developed might not let the researchers at large answer the questions they want to ask. The SOC group should give some thought to how the C@G database itself can become a resource for discovery, for people to access and test ideas on. He asked for more detail in many of the measurements, so that, for example, one can learn not just that some technology was used, but how often was it used, and if it was only used minimally was that because users were not interested, or was it because the resource was scarce.
The C@G database could pay explicit attention to the social elements that play a role in the success of collaboratories. The social elements he mentioned included:
There could be further clarification about who the audience is for the C@G, as that determines what sort of questions the data might aim to answer, as well as how questions of success might be framed.
- What were the management and leadership issues?
- How are incentives handled in the collaboratory? What is considered an important contribution?
- How was the collaboratory integrated with the work ecology?
- How are groups defined within the collaboratory?
"What do we know about collaboration that we didn't know without looking at collaboratories?"
Olson, J., Card, S. K., Landauer, T. K., Olson, G. M., Malone, T., and Leggett, J. (1993) Computer supported cooperative work: Research issues for the '90s. Behavior and Information Technology. Pp. 115-129.
Patel V.L., Kaufman D.R., Allen V.G., Shortliffe E.H., Cimino J.J., & Greenes R.A. (in press, Summer 1999) Toward a framework for computer mediated collaborative design in medical informatics. Methods of Information in Medicine