Abstract Interactor Model Specification
Working Draft 1. December 2011
- This version
- http://www.multi-access.de/mint/aim/2011/20111201/
- Latest version
- http://www.multi-access.de/mint/aim/
- Previous version
- none
- Editors
- Sebastian Feuerstack, UFSCar
- Jessica Colnago, UFSCar
This document is available under the Document License. See the Intellectual Rights Notice and Legal Disclaimers for additional information.
Overview
Using interactor models to describe interactive systems is an already well known and matured approach and has been extensively discussed, for instance, in [1]. They can be thought as an architectural abstraction similar to objects in object-oriented programming [1]. Several definitions of the interactor term have been proposed so far like for instance the PISA [2] or the YORK [3] interactor. Our approach relates to the latter one, which defines an interactor as:
“ a component in the description of an interactive system that encapsulates a state, the events that manipulate the state and the means by which the state is made perceivable to the user of the system.” [4]
Different to these early approaches that focused on the design of graphical interfaces, this specification proposes interactors to assemble multimodal user interfaces. An interactor can receive input from the user and send output from the system to the user. Each one is specified by a state machine that describes the interactor’s behaviour. A data structure is used to store and manipulate information that is received by each interactor or sent to others.
We use UML class diagrams to specify the data structure. Each interactor is associated with at least one class that needs to inherit from an abstract “Interactor” class. This class contains the basic data structure to support interactor persistence such as storing the actual states of an interactor and encapsulates the functionality to load and execute the state machines.
The following figure shows the interactor class and its relations of the three basic interactors: the Abstract Interactor Object (AIO), the graphical Concrete Interactor Object (CIO), and Interaction Resource interactors, which describe the interaction capabilities of physical devices.
The Concrete User Interface (CUI) model is used to design a user interface for a certain mode or media. The Abstract User Interface (AUI) model describes the part of the interface that contains the behaviour and data specification shared among all modes or media. Interactors share the same database (which is currently a tuple space) to store and manage their data. Each interactor (that e.g. represents a button on a concrete graphical interface) is initiated several times to reflect all occurrences of e.g. a button in an application. This interactor is further on initiated at the AUI model level, and additionally for each supported modality at the CUI level. Additionally, at system run-time, interaction resource (IR) interactors are instantiated once for each device that is connected to the system. IRs are, for instance, a gesture recognition that supports a certain set of gestures or postures or a WII remote control that supports a set of pre-defined gestures pointing to objects, and additionally has a joypad control
Abstract Interactor Model Structure
A multimodal setup combines at least one media with several modes. Thus, parts of the interface interactors are output-related (e.g. sound or a graphical interface for instance), whereas others serve to address the modes used by the user to control the interface (like a joystick, speech, or a mouse, for instance). At the abstract, mode and media independent model level this has two implications. First,: a) input needs to be separated from output; and b) a continuous input or output (such as moving the mouse or a displaying a graph) needs to be distinguished from a discrete one. The following two sections describe how these implications could be addressed in an abstract user interactor model (AIM). First by organizing all AIM interactors based on these findings in a static structure (a class diagram) and second, by specifying their behavior (by state charts).
The AIO is the parent interactor of all other interactors of the AIM and is derived from the core Interactor, which is the base interactor of all elements from all interface models.
The AIM distinguishes between an abstract interactor input (AIIN) and output (AIOUT) elements, which could be either of a continuous or a discrete nature. Continuous outputs are a graphical progress bar, a chart, a tone that changes its pitch, or a light, which can be dimmed, for instance. siscrete output could be implemented in different ways, such as by written text, by sound or by a spoken note.
Grouping of interactors is handled by abstract interactor containers (AIC). AICs can be specialized to realize single (AISingleChoice) or multiple choices (AIMultiChoice) and are derived from the discrete output interactor. A discrete output interactor can be explicitly associated with every AIO, which we call AIContext. The AIContext is used to add contextual information that helps the user to control the interface or to understand the interface interactor. This could be, for instance, a tooltip, a picture or a sound file.
User input could be performed in a continuous manner as well. Some examples are: moving a slider or showing a distance with two hands. Discrete user inputs (AIINDiscrete) could be performed by commands that are issued by voice, or by pressing a physical or virtual button, for instance. We consider choosing one or several things from a list as user input as well (AISingleChoose and AIMultiChoose respectively).
In order to support distributing input and output to different devices, we strictly separate user input from output in the user interface models. There are some implications of this strict separation: For instance user choices are required to be separeted to two different elements: a list of elements to choose from is considered as output (since it just signalizes the information that a user can choose together with a grouping of interactors), the current interactors to choose from are modeled separately as AIIN (by AISingleChoiceElement and AIMultiChoiceElement).
AIO - Abstract Interactor Object
An Abstract Interactor Object (AIO) is the abstract basic entity of all other elements of the AUI specification. Like all other elements of the AUI description it does not implement a representation in the user interfaces (which is done in the CUI description) but specifies common behavior that is required to be implemented for all multimodal setups (combinations of IR).
| SCXML | Since |
|---|---|
| AIO.scxml | 20111201 |
Lifecycle
After an AIO has been initialized it can be presenting, after it has been successfully organized. To calculate the AIO organisation, a basic navigation scheme is established for all AIOs that are presented together on a certain time in a user interface dialog. An AIO that is presenting can be removed from the user interface by suspending it to ''suspended'. Therafter it can be presenting again.
An AIO offers basic navigation capabilities to reach the previous (prev) and the next (next) AIO element of a user interface. By navigating forward and backward through the AIOs the focus that states the actual focus of attention of the user regarding the actual user interface representation is moved. A focus can be set by the focus event (results in isFocused=true) and removed by the defocus event. A general an arbitrary set of AIO can be focused (e.g. for multi-touch and collaborative, multi-user interfaces). By navigation the focus of one AIO can be changed at a time. Like it is specified by the behavior chart. If the user issues the next event, the next element gets focused whereas the previous one is defocused. The same mechanism is applied in the opposite direction (prev event).
AIIN - Abstract Interactor In
The Abstract Interactor In (AIIN) describes all elements of the AUI that can be used to input data to the computer. It derives the behavior and structure from the AIO class. It is used to describe mappings between input IRs and the appropriate abstract interactors (that needs to derive from this interactor).
| SCXML | Since |
|---|---|
| abstract | 20111201 |
AIINContinous - Abstract Interactor In Continous
The Abstract Interactor In Continous sums up all continous input elements, like graphical sliders and scroll bars, physical knobs or gestures that illustrates distances. The element is originally set to waiting and it's kept that way until it receives a decrease or increase event. When that happens it goes to state decreasing or increasing respectively. It returns to waiting whenever it's idle for more than tmax.
| SCXML | Since |
|---|---|
| 20111201 |
AIINDiscrete - Abstract Interactor In Discrete
The Abstract Interaction In Discrete sums up all discrete input elements.
| SCXML | Since |
|---|---|
| 20111201 |
AIReference - Abstract Interactor Reference
With the Abstract Interactor Reference that can be associated to every AIO the user can refer to this AIO. This might be a label or a hotkey on a graphical user interface, a voice menu command or a visual tag for instance. If there should be more than one way to address a certain AIO (in every modality), more than one AIReference can be associated to one AIO.
| SCXML | Since |
|---|---|
| 20111201 |
AICommand - Abstract Interactor Command
The Abstract Interactor Command details the AIIN/AIO behavior for the focused state. If an AICommand is focused it is initially deactivated. A command can be issued by the user setting it to activated. The command is activated until the user deactivates it again. An AICommand is derived from the AIIN class since it describes an input that can only be triggered by the user.
| SCXML | Since |
|---|---|
| 20111201 |
AIChoiceElement - Abstract Interactor Choice Element
The Abstract Interactor Choice Element extends the AIO presenting state to allow elements to be dragged and dropped. So if a choice element is focused and gets the drag event it is checked if it's unchosen. In case that's true it issues a choose event for itself and then goes to dragging. In case the check returns false it goes directly to dragging. When it gets the drop event it's defocused again.
| SCXML | Since |
|---|---|
| 20111201 |
AISingleChoiceElement - Abstract Interactor Single Choice Element
The Abstract Interactor Single Choice Element extends the AIChoiceElement presenting state in the sense that it adds selection to run in parallel with the presenting. A AISIngleChoiceElement is originally unchosen and if it gets the choose event (and it's focused) all other elements get unchosen. This ensures that only one element is chosen at a time.
| SCXML | Since |
|---|---|
| AISingleChoiceElement.scxml | 20111201 |
AIMultiChoiceElement - Abstract Interactor Multi Choice Element
Works like the AISingleChoiceElement except that it doesn't unchoose all other elements.
| SCXML | Since |
|---|---|
| 20111201 |
AIOUT - Abstract Interactor Out
The Abstract Interaction Out describes all abstract interactors that can be used to output information to the user. It derives structure and behavior from the AIO interactor description and is used to couple output interaction resources to abstract interators.
| SCXML | Since |
|---|---|
| abstract | 20111201 |
AIOUTDiscrete - Abstract Interactor Out Discrete
The Abstract Interactor Out Discrete describes all discrete outputs which includes lists of elements presented to the user.
| SCXML | Since |
|---|---|
| 20111201 |
AIOUTContinous - Abstract Interactor Out Continous
The Abstract Interactor Out Continous sums up all continous output elements, like graphical progress bars for graphical interaction or to express changes of the tone pitch for sound output for instance.The element is originally set to waiting and it's kept that way until it receives a progress or regress event. When that happens it goes to state progressing or regressing respectively. It returns to waiting when it receives the halt event.
| SCXML | Since |
|---|---|
| 20111201 |
AIContext - Abstract Interactor Context
The Abstract Interactor Context offers contextual information for the user to improve the understanding of an AIO's interaction capabilities. This might be for instance a helping text, a picture or video. An AIO context object can be associated to every AIO. Thus for a grouping AIC it might be used to explain the user what a certain user interface part is designed to be used for.
| SCXML | Since |
|---|---|
| 20111201 |
AIC - Abstract Interactor Container
The Abstract Interactor Container allows grouping of AIOs. Each AIC must have one parental AIC (with the exception of the root AIC) and stores references to all AIOs that it contains. The grouping of AIOs is considered an output mode. The AIC details the navigation options of the AIO's presenting state. Besides next,prev, focus, and defocus events the AIC supports navigation to parental AICs as well as setting a focus to the first child AIO by the child event.
| SCXML | Since |
|---|---|
| AIC.scxml | 20111201 |
AIChoice - Abstract Interactor Choice
The Abstract Interactor Choice (AIChoice) extends the AIC to implement a list of AIOs from that can be choosen by the user.
| SCXML | Since |
|---|---|
| 20111201 |
AIChoice - Abstract Interactor Choice
The Abstract Interactor Choice specifies a container from that the user can choose elements. The container is initially set as listing and gets dropped when receives the drop event (when this happens all elements in dragging get droped). After that it unfocus itself and focus the AIChoiceElement.
| SCXML | Since |
|---|---|
| AIChoice.scxml | 20111201 |
AISingleChoice - Abstract Interactor Single Choice
The Abstract Interactor Single Choice specifies a container from that the user can choose only one element at a time. The container is initially set as listing and gets dropped when receives the drop event (when this happens all elements in dragging get droped). After that it unfocus itself and focus the AIChoiceElement.
| SCXML | Since |
|---|---|
| 20111201 |
AIMultiChoice - Abstract Interactor Multi Choice
The Abstract Interactor Multi Choice specifies a container from that the user can choose an arbitrary amount of elements. It works similarly to AISingleChoiceElement except that it has the option to choose_all and unchoose_all childs.
| SCXML | Since |
|---|---|
| 20111201 |
Abstract Interactor Single Presence (AISinglePresence)
The Abstract Interactor Single Presence extends the AIC, but only presents one of it's children at a time to the user. When entering state presenting it must be checked whether or not there is an active child and if not the first page gets presented. The interactor is initially defocused, so that to be available for use it must be the currently focused object. When focused, it stays on state waiting until it receives the enter event. While on state waiting the interactor can receive a next, prev (previous) or parent event to enable navigation between interactors. The navigation within the interactor is possible by sending the next or prev events when on the entered state. This will suspend the active child and select the desired one.
| SCXML | Since |
|---|---|
| AISingePresence.scxml | 20111201 |
References
[1] Panos Markopoulos. A compositional model for the formal specification of user interface software. PhD thesis, Queen Mary and Westfield College, University of London., 1997.
[2] F. Paterno. A theory of user-interaction objects. Journal of Visual Languages & Computing, 5(3):227 – 249, 1994.
[3] D.Duke, G.Faconti, M.Harrison, and F.Paternó. Unifying views of interactors. In AVI ’94: Proceedings of the Workshop on Advanced Visual Interfaces, pages 143–152, New York, NY, USA, 1994. ACM, ISBN:0-89791-733-2.
[4] D.J. Duke and M.D. Harrison. Abstract interaction objects. Computer Graphics Forum, 12(3):25–36, 1993.