Victoria Bellotti, Simon Buckingham Shum, Allan MacLean and Nick Hammond
Contents
The time and effort involved in systematic analytic approaches to HCI has always been a disincentive for applying them. Traditionally, the view has been that the costs involved are prohibitive other than for large scale or high risk implementations. However, we can argue that since users dislike having to learn the same thing again and again for each new release of a system, there is always commercial advantage to be gained from getting the interface and interactive style for core functionality right first time. This is especially true of innovations for which there are likely to be many future releases and upgrades.
There can be little doubt about the benefits of insight into the underlying principles involved in complex user and system characteristics and behaviour which can help us understand how to make an interface better, if not right first time [e.g. 16, 21, 29, 30]. Such an understanding can inform design before prototyping and user testing is possible, and can also highlight the reasons for problems which users might have. However, whilst these benefits appear attractive, the problems with the existing techniques must be overcome if they are to be enjoyed.
If HCI analytic techniques are to support innovative commercial design they must be able to operate in a multidisciplinary fashion to cope with the broad range of issues arising for today's increasingly diverse systems, including multimedia and collaborative applications. Systematic accounts of usability must now transcend the traditional disciplinary boundaries that separate, for example, system, user and design analysis. They must also be able to support non-routine design of more exploratory systems where the problem space is not yet understood.
In this paper, we first describe how HCI analytic techniques provided valuable insights when used in 'multidisciplinary modelling consultancy' for the exploratory design of a user interface in a digital media space development project. Secondly, we discuss a number of lessons for facilitating the transfer of such techniques to innovative design practice. Since the interaction between modellers, domain and practitioners was of central importance, we describe in some detail how the design context and modelling analyses were communicated.
These are ambitious and largely unprecedented aims for a single project. The route to applied, multidisciplinary theory has taken the AMODEUS collaborators through several different kinds of work. Initially, effort was devoted to establishing mutual understanding between user-, system- and design-analysts; next, the focus was on identifying common concepts and interdependency between very different domains of theory [4]; finally, we have progressed to studies evaluating the usability of individual techniques [9, 12], and exercises in which we provide HCI 'modelling consultancy' to development projects in industrial [10] and innovative design contexts.
The work reported here falls into the last class of study. We describe a large scale exercise, conducted by 15 researchers for up to a month each (in some cases more), with the aim of demonstrating the following:
This project team was located at EuroPARC in Cambridge, UK, along with two members of AMODEUS (co-authors of this paper). This meant that there were good opportunities for communication between members of the two projects. The AMODEUS researchers who were co-located with the designers were developing the QOC notation for design space analysis [4, 24]. This succinctly expresses as a semiformal network the key design questions being asked, alternative options being considered, and argumentation about them in the form of trade-offs among criteria. Such a representation shows how different design alternatives have addressed a common set of problems. As described shortly, design space analysis played a key role in supporting communication between modellers and designers.
We focused on the design of ECOM, EuroCODE's media space, a digital audio-visual (AV) communications and messaging system for use over local, high bandwidth and long distance public telecommunications networks. This media space had a well documented design history, based as it was upon at least two existing analogue media spaces, the EuroPARC RAVE system [20]; and the Toronto University CAVECAT system [25].
FIGURE 1: Example of QOC used to represent the designers' own rationale, prior to modelling. Criteria (C) are the bases for deciding among alternative Options (O). Questions (Q) identify issues which structure alternative Options. Solid Option-Criterion links are positive assessments, and dashed links are negative.
DYNAMIC FIGURE 1 (QuickTime Movie, about 10 mb). If you want to learn more about QOC design space analysis, click here to see a brief video overview from [27]
An ECOM design document was produced to communicate the design to the AMODEUS modellers. The contents of this document included:
We evaluated the comprehensibility and utility of the modelling and design space integration in several ways. We were especially interested to know if the ECOM design team felt that the modelling input would influence further design, and whether the design space provided a useful summary of that input. Specifically, we assessed the initial design space, the modelling-enriched version and the modelling reports, in the following manner:
The main ECOM interface designer (D1) used the initial QOC (summarising the team's design rationale) to explain his design to another team member (D2) [video recorded session].
The modelling-enriched QOC design space was separately presented to D1 and D2, together with the modelling reports which it indexed. The designers gave feedback on the usefulness of the modelling and QOC [a video recorded session plus a one day workshop].
Designer D2 used the modelling-enriched design space to help bring a new team member up to speed with important ECOM design issues [video recorded session].
Modellers presented some of their analyses to a number of media space designers at the CHI'94 conference. Presentations were followed by discussion on the potential of such approaches [3 hour meeting].
The next section provides examples of the lessons learned about specific aspects of the media space designs which were examined. In the following section we step back and describe some of the wider issues related to our experience of the process of applying multiple theory-based perspectives to a design problem.
The AMODEUS analyses provided several design insights for ECOM which seem to be generalisable to other computer mediated communication systems. In this section we outline three examples of how analysts tackled the design of mechanisms for displaying general availability for communication and setting specific accessibility levels.
CAVECAT's general and RAVE's person-specific access control mechanisms each have their advantages and disadvantages. In ECOM, it was decided that the best of both worlds might be achieved by combining these two kinds of access control, allowing users to select from four general availability levels and to choose person-specific exceptions for any level.
FIGURE 2: ECOM media space prototype user interface.
Figure 2 shows the interface to ECOM's person-specific access control mechanism. By hitting the exceptions button in the main dialogue box (top left), the user brings up the exceptions dialogue box (right hand side of the screen). They then select a user or group (members are displayed in the scrolling window), followed by con, a connection type (one of glance, snapshot, connect and message), and finally a state with which this connection exception is to be associated.
States range from always, through a series of door states through to never. Thus, when the user sets their general availability using one of the door icons in the top left of the figure, any specified exceptions will overrule the default permissions for that door state, for specified people.
The availability of other users can be seen at the time of selecting a connection to them. When making a connection the user selects the target's name from the user menu in the top dialogue box and the target's availability is shown by a door to the right of a small picture of them.
The architecture modellers and both of the cognitive modelling groups highlighted the potential for confusing ECOM's two concepts of general availability and person-specific accessibility. In CAVECAT, general availability is intended to correspond to the current state of the user. In RAVE, person-specific accessibility corresponds to the user's current permissions settings, i.e., for each other user, which AV connections they request will be accepted and established by the media space.
ECOM attempted to combine these two notions, but in doing so confounded them somewhat, resulting in what was an extremely complex and possibly unworkable control mechanism. The architecture modellers recommended that these concepts should be dealt with separately and the cognitive modellers argued that they arose from two different user concerns which should be supported separately in the interface.
The architecture modellers identified domain, task and system concepts to be dealt with by software handling the making and breaking of connections. They modelled different ECOM connection types and levels of user availability. Their view was that the system architecture would need to maintain a database of permissions lists of permitted connections for all users and a representation of each user's current availability. Each of these would best be handled by different system components.
A person's availability for communication depends on their current activities which may change from time to time. Whilst a user may be allowed to set a level of general availability, as in CAVECAT, this will not always correspond to their desire to be accessible (or inaccessible) to any specific user or group. ECOM gives the user the flexibility to define any set of connections to be permissible at any level. However, this means that the system architecture's representation of availability in ECOM may end up being arbitrarily related to accessibility settings and consequently somewhat meaningless.
The result is highly problematic. A user may forget what access permissions their door state availability levels correspond to. They might also find that they can or cannot connect to person X regardless of X's apparent door state. The system could show a door state which corresponds to the specific accessibility rather than the general availability level of X (this was, we later learned, the designer's intention); however, this defeats the idea of having separate representations for both availability levels and accessibility.
The cognitive modellers showed that ECOM made it hard to separate the task goal of making oneself more or less generally accessible (i.e., available) from that of choosing to be more or less accessible to a particular individual or group. The modellers proposed controlling general and specific accessibility levels separately. Furthermore, the user's current accessibility would be hard to keep track of due to the complexity of the possible exceptions. They recommended that the setting of person-specific exceptions should only be possible at one level of general availability.
FIGURE 3: Example of how the QOC design space was used to summarise modelling analyses of the initial design space. In this example, the analysis in Figure 1 has been added to by the PAC architecture modellers and the CTA and PUMs cognitive modellers.
Figure 3 shows an example of the way in which modelling input to ECOM's design, and, in this instance, to solving the problem of confusing general availability and person-specific accessibility, were added to the design space as shown in Figure 1. These analytic contributions were always indexed back into their source in a modelling report so that details of the modelling could be examined.
The second contribution we outline is that users must have feedback about their current accessibility. This is all the more important if there are to be any exceptions to accessibility settings. Both of the cognitive modelling analyses identified the feedback issue. These suggested that since a user's availability levels were not likely to correspond to permissions settings, they would be unable to remember and therefore unable reliably to predict at any given moment whether any other user could make a particular kind of connection to them. This was compounded by the fact that ECOM did not provide a simple way to find out what any other particular user's exceptions were. One had to select their name and then inspect each connection type to see what the door state settings were.
The third insight we describe came from formal systems modelling (FSM). The complexity of managing media space connections is a good example of an HCI domain warranting the application of formal methods, given their ability to support verification and failure modes analysis. The FSM group formally specified how entities in the system would be defined in terms of their membership of certain sets and the properties they inherited from the sets they belonged to. Thus, users could be members of groups, but users and groups could have different access permissions. The FSM analysis revealed the implementational requirement to resolve conflicting permissions settings.
If access permissions are to be applied to groups as well as users, problems arise if users are members of one or more groups with excepted permissions. The designer could opt for the system to apply just the most specific permissions relating to a given user. However, if the only exceptions relevant to a user related to two groups they belonged to, and those two groups had different permissions, the user would reach an impasse if they attempted a connection which they were simultaneously permitted and not permitted to make, due to their group membership.
To summarise, the contribution of the analysts in this design consultancy exercise was to use their approaches to identify usability and implementational flaws in the existing ECOM prototype and to propose a number of alternative solutions to overcome these weaknesses. The modellers' various contributions to the original design space were combined into a single modelling-enriched space, as shown in Figure 3. This space also contained figures (showing example design solutions) from the analytic AMODEUS reports and annotations providing background information such as application domain constraints or explanations of the contributions to help the reader, together with indexing links back to relevant pages in the modelling reports.
Both designers commented that the cognitively motivated design principles and illustrative redesigns of dialogues were helpful, and would be seriously considered for inclusion in future versions of the interface. Since the modelling exercise, the EuroCODE project decided (for reasons unrelated to AMODEUS) to pursue a simpler AV facility than the ECOM prototype we analysed. This means that to date we have not had the opportunity to track the extent to which the modelling recommendations are taken up. However, the issues raised by the designers (controlling multiparty access) will be a core feature of future media spaces, and the insights yielded by modelling on ECOM are not restricted to that particular system.
To judge the value of modelling, designers need to understand it. For example, the CTA modellers experimented with a dual-format report, in which a column of raw modelling notation on one side of the page was accompanied by another column of 'translation' for the designers to follow the analysis and recommendations. This provoked discussion on how much detail modellers should provide to back up their recommendations. ECOM's main designer suggested that it would be useful to have an intermediate-level CTA representation, accessible to a trained designer, to bridge between the raw modelling and analytic conclusions. This is in fact precisely what the CTA approach aims eventually to provide in the form of a tool which semi-automates the modelling process [26]. A trained human factors designer will be able to inspect a high-level representation of the cognitive model. Elsewhere, we report a study investigating the skills which the current prototype tool requires of its users [12].
Since modelling requires extra effort, any discussion of benefits must be balanced by the costs. It is extremely difficult to derive a useful measure of the cost/benefit trade-off in this exercise, since the modelling approaches are in many cases still research tools, being developed through exercises such as the ECOM collaboration. Balanced against modelling cost is the cost of generating the ECOM prototype which again is hard to assess, based as it was upon the design and use of other media spaces, and on design meetings involving four other members of the EuroCODE project.
We should also emphasise that we adopted a 'modelling consultancy' paradigm as an effective research strategy for investigating HCI modelling techniques, and not as the envisaged modus operandi for such techniques in the long term. Ultimately, we envisage modellers based at the same site as the designers, familiar with the design context, and able to provide direct input to the design team. This is the way that both analysts and designers (in this and other design teams with whom we have worked) say that they would like to work, and under such circumstances costs of modelling would drop significantly. In the real world, of course, such constraints are typical of large development projects, where different contributing stakeholders operate under time pressure and often with poor opportunities for communication [3, 18].
A realistic cost/benefit analysis for HCI modelling must therefore be based on modelling tools in use in realistic design contexts by trained practitioners. We have therefore begun to investigate the requirements for training practitioners in a number of these techniques (described further below).
In summary, we cannot yet verify that costs and benefits of HCI modelling were well balanced in this exercise. However, the issues analysed are central to media space design. We maintain that detailed understanding of the usability and implementational implications of design decisions at this early stage will have a pay-off for future systems evolving from the current generation of prototypes. A more detailed report of this modelling exercise is presented in [5].
As one might expect with such an ambitious exercise, the envisaged process did not run entirely smoothly. We are now in a better position to understand how this 'interface' can be improved for future exercises. The lessons we draw may assist others seeking to deliver analytic HCI techniques to practitioners. In a number of cases, the points may seem rather unremarkable; however, we suggest that they are comparable to the user-centred orientation which is taken for granted within the usability community--the principles are in some sense 'obvious', but instantiating them is rather more complex, and training others in them even more so. To date, very little attention has been paid to understanding how to make HCI modelling accessible to practitioners.
Somewhat ironically, we ourselves failed to anticipate the difficulty which our project partners would have with using the QOC notation to summarise their modelling, even after five years of exposure to it. In retrospect, we should have provided some practical training in using the technique, as we have done for designers over several years [23, 28]. This serves to emphasise that, just as with interactive systems, 'end-user-support' is essential for analytic HCI techniques. Practically applying an approach draws on very different skills to simply comprehending it in principle.
If we reflect on the 'sociology of method uptake,' it is undeniably the case that new methods and tools are accepted and rejected as much on the basis of the way in which they are presented, and by whom, as on their technical merit. A sense of ownership of the techniques needs to be engendered on the part of the designers. Practitioners should be respected as fellow professionals, invariably working in more highly constrained environments than researchers.
Firstly, given that AMODEUS is pan-European, much communication took place via documentation, email, QOC, or analytic notations. The ECOM design document and related background documents constituted around 250 pages of text and design space. However, the modellers still reported that they were missing background context information concerning design goals, domain and system constraints and characteristics, users, and tasks to be supported. Had the analysts been more closely linked with the designers and the design context, common experience and informal communication would probably have reduced these documentation problems. In the existing literature, most analytic modelling techniques underplay the importance of access to background information and of participating in the design context prior to formal analytic work.
With respect to the design space, the designers, although having never encountered the QOC notation before, had little trouble understanding the space when they were asked to verify its contents. When later presented with the modelling-enriched version, they were able to see how the modelling augmented their original rationale. In contrast, the AMODEUS members had not participated in the context and discussions which shaped the initial design space. Whilst they had seen many QOC papers and presentations, they found it very hard to make sense of the argumentation, and relied heavily on the ECOM design document to make sense of it.
A conclusion to draw from this is that for a terse notation like QOC to be accessible outside the original design team, supplementary detail is needed initially to give context to the analytic representation. Further research is required to determine the factors which make argumentation and other shared representations intelligible to their users, particularly in situations where projects restrict the opportunities for participation and appropriate contextual experience.
We have found, both with ECOM and in other modelling consultancy studies, that example sketches of redesigns are an extremely successful way to communicate model-based design insights. Having identified certain principles as critical, the modellers devised simple design examples to satisfy those properties. This provides something concrete to ground understanding and to reflect upon, but is not meant to dictate design--having understood the principle at stake, a designer can tailor an example for a specific solution.
Receptivity to HCI modelling will therefore be moderated by design culture--where a given 'culture' will attract people with particular skills and interests, and discourage others. One obstacle seems to be that the formalisms used by theorists presuppose certain process elements early in the lifecycle, such as a formal requirements specification or task analysis. These do not seem to have been features of the design of ECOM or other media spaces we have considered. A consequence of this 'culture clash' is that designers rooted in the exploratory, design-by-building culture have not been receptive to the idea of using more abstract system specification techniques, even with appropriate training. This was the general reaction at the meeting we held with media space designers at CHI'94, as well as with the ECOM designers. In contrast, work with a software engineering company who already use formal methods has met with much more positive feedback [10].
How is this gulf between cultures to be bridged? A key problem when different cultures meet is lack of mutual understanding. One of the driving forces behind our work within AMODEUS has been to find ways to communicate to practitioners what modellers seek to do, how, and for whom. We are acutely aware of the need to provide sufficient background modelling information for designers before attempting detailed discussions. One encapsulation of this background material is a set of executive summaries and short worked examples for each technique, written for designers with whom we are collaborating, and for other interested parties [14].
Of course, education needs to be two-way, and we are working to understand how HCI modelling techniques can be tailored to the demands of current design practice. Ultimately, we want to see the fruits of research being exploited by practitioners, not just an elite group comprising those who have developed analytic techniques.
Consequently, we are challenging the conventional notion within the software engineering community that formal methods are only useful if used within a structured development context from the beginning of a project, through refinement, to implementation. This seriously restricts the class of project in which formal methods can be used, cutting out exploratory design projects. The ECOM modelling exercise exemplifies a strategy of selective formal specification of particularly complex HCI problems in order to maximise the cost/benefit trade-off.
Selective modelling is also the envisaged strategy of use for cognitive modelling approaches seeking to develop software tools to support rapid re-modelling [7, 26]. Similarly, when training designers to capture and reflect on their rationale using QOC, we emphasise the different ways in which it may fit into their design context, and the importance of using it strategically to maximise the gain from the effort.
In summary, the power and influence of design culture on attitudes to modelling
techniques should not be underestimated. In our role as modelling consultants,
we explored new ways to communicate modelling to practitioners (by integrating
modelling with QOC, and concretising abstract modelling principles). In the
longer term, with a view to seeing the techniques in use by designers
themselves, we are seeking to adapt the application and delivery of
theory-based modelling techniques to the contexts in which they must be
used.
CONCLUSION
There has been considerable debate in recent years over whether the HCI science
base has anything to contribute to real design. In the modelling-design
exercise reported here, we have demonstrated how formal modelling techniques
grounded in cognitive science and software engineering theory can inform
innovative and exploratory user interface design such as that of a digital
media space. Moreover, we have explored some of the key issues for
communicating such analyses in accessible ways, and for delivering such
techniques in forms which are compatible with design practice. We maintain that
theoretically grounded HCI design techniques such as these are powerful ways to
manage the implementational and cognitive complexity of emergent interface
technologies, particularly as they find their way into mainstream and large
scale software development. However, analytic techniques will only achieve
uptake if the end-user requirements of design practitioners are properly
understood, and the value of such techniques can be demonstrated. In the
exercise reported here, we have made progress in better understanding how to
make that match with a design culture which has traditionally shied away from
more formally based approaches.
Acknowledgments
This work is funded by ESPRIT BRA 7040. We are grateful to our AMODEUS
colleagues whose varied perspectives have contributed to the work presented
here. Our thanks also to the EuroCODE design team for their cooperation, to the
British Telecom, EuroPARC, PARC, SunSoft and U. Toronto media space designers
for their feedback, and to the CHI reviewers for their comments.
References
Note: AMODEUS documents are available by WWW & FTP:
