Redocumenting computer-mediated activity from its traces: a model-based approach for narrative construction
Our activities are becoming more and more computer-mediated. For documenting these activities, it is no longer sufficient to automatically record their traces. In this paper we introduce the redocumentation process of computer-mediated activity as a narrative construction that ties together the content of activity traces and the users’ knowledge in describing their activities in new easily exchangeable documents. We present a generic semi-automatic approach for this process, which is based on rhetorical structure theory. This approach uses formal models for process input and output, and handles the process through two main phases: an automatic phase to generate a fragmented document from traces as a first description of the activity and an interactive phase to allow the user to tailor this first description according to his particular needs and choices. We also present ActRedoc, a tool developed for text-based redocumentation, for which a first evaluation was conducted.
The concept of "document" seems to be intuitive for all of us since documents are ubiquitous in our daily life; however, when trying to define this concept, only special cases (like books, articles, files, photos, etc.) are usually cited. As a pivotal definition, Suzanne Briet considers the document as "any physical or symbolic sign, preserved or recorded, intended to represent, to reconstruct, or to demonstrate a physical or conceptual phenomenon" (Buckland, 1997, pp. 806), and therefore combines information and evidence as two fundamental functions. Nowadays, documents as electronic forms lose their stability as they are not material objects and major upheavals are imposed on their production and exchange. Roger T. Pédauque suggests that digital documents should be studied according to their three dimensions at the same time: the document as a form (digital structure), the document as a sign (content) and the document as a medium (communication tool) (Pédauque, 2006). These three dimensions are related to modalities that determine the degree of "maturity" of a document: anthropological (for legibility), intellectual (for assimilation) and social (for diffusion scope).
Documentation —as a process for creating and managing documents— focuses on gathering and recording information so as to provide evidence of something according to a particular purpose. For instance, personal documents may serve to reminisce or inform others. The rise of digital technologies entailed significant changes in documentation, both by making the process easier with the support of dedicated tools, and by allowing new possibilities of documentation. Accordingly, the emergent technology of Lifelogging offers the ability to capture life experiences of an individual through digital means1. Moreover, it opens new opportunities for retelling and sharing these experiences by creating digital narratives (Byrne and Jones, 2009).
We are especially interested in the documentation of computer-mediated activities, a process that should provide a persistent description of such activities, tailored to the choices and preferences of an individual and easily exchangeable. The produced document is interesting for the user since it provides him with the opportunity to review, reflect and reminisce upon his performed activity. Furthermore, it serves as a medium for the communication and exchange of personal computer-mediated experiences, as well as an activity guide or a behavioural analysis support for others.
Computer-mediated activities may be automatically recorded through the instrumentation of the computer system so as to record and collect events that are considered significant2. The products (log files, screenshots, video records…) are called traces and they supply a basic documentation for the traced activity. However, these traces usually have a low degree of "maturity" for ordinary users (for instance, the legibility, the assimilation and the diffusion scope of a log file make it a document for system designers, but hardly for the system users), which therefore calls for their transformation into more usable forms. Moreover, such traces are usually distributed across several files; hence, it is important to be able to integrate, correlate and organize such multimodal collection as a coherent unit that may be easily exchanged. This calls for the design of redocumentation processes that aim to tie together the basic traces of a computer-mediated activity into coherent and comprehensive forms that actually document the activity.
Several projects and tools (like Oracle Trace Analyser3, Cyclope Employee Monitoring Software4 and (Gray et al., 2004)) adopted automatic approaches for redocumentation so as to generate easy-to-read reports (usually hypertext). However, such approaches are too restrictive for people who may prefer applying their own choices and preferences for tailoring the content or the form of the activity redocumentation product. Other projects use semantic modelling of activity traces (Champin, Prié and Mille, 2003; Settouti, Prié, Marty and Mille, 2009; Clauzel, Sehaba and Prié, 2009) to propose model-based transformations and graphical visualizations that ensure a high level of abstraction in order to adequately represent the activity. Nevertheless, if semantic modelling of traces enhances their expressiveness and facilitates their use by both humans and machines, important information about the traced activity context may not be collected automatically or modelled explicitly (e.g., a user's opinion about the acivity). To overcome this shortcoming, we believe that the user must be able to intervene during the redocumentation process to actively decide which content from the traces should be preserved, what should be added and what form should be adopted for the produced document. However, given the possible complexity, volume and multimodal nature of traces, we cannot expect a user to manually construct a document describing his traced activity directly from basic traces. Hence, there is a need for tools that assist users in exploring their activity traces and minimize their authoring efforts during the redocumentation process. Moreover, as narratives were identified as a central part of how humans learn to make sense of the world around them (Nelson, 1989), a narrative construction approach seems to be the most familiar for users to descibe their experiences about the performed activities.
In this paper, we propose a semi-automatic model-based approach for redocumenting computer-mediated activity. Our general framework for implementing this process, its input traces and its various outcomes are based on models, namely trace and document ontologies. Two main phases are defined for this process. The first one aims to automatically generate a fragmented document from traces, as a first description of the activity. The second phase allows the user to enhance and tailor this first description according to his particular needs and choices by adopting a narrative construction based on Rhetorical Structure Theory (RST) (Mann and Thompson, 1988). We also describe ActRedoc, a generic authoring tool for assisting users in redocumentating their activity into a text document. This tool is developed based on a number of existing semantic web technologies.
The paper is organized as follows. In section 2 we refine our presentation of model-based redocumentation process for computer-mediated activities. Section 3 is dedicated to related work. In section 4 we describe our two-phase general framework for activity redocumentation from traces. In section 5 we introduce our application domain: navigation in a digital library of newspapers, and in section 6 we describe our generic authoring tool ActRedoc. Section 7 deals with the evaluation of our tool and its use. Finally, section 8 presents our conclusion and future work.
2. Toward a redocumentation process for computer-mediated activity
The process of redocumenting a computer-mediated activity is an interpretative process. It may be carried out by the user who has performed the activity (the traced user) or by any other person (for instance, a colleague or a friend of the traced user, an analyst or a psychologist, etc.). This process allows exploiting the automatically-generated traces of a computer-mediated activity in order to rewrite them within a new digital document that describes the traced activity according to one’s intention and choices. Thus, redocumenting implies reorganizing, filtering, rewriting or adding information to traces content (e.g., for reflecting a particular point of view, for expressing the production context5 of the activity trace, etc.). Moreover, it implies the choice of a particular form and presentation for the produced document (Yahiaoui, Prié and Boufaida, 2008). Figure 1 summarizes our global vision about the redocumentation process. This process input is a set of activity traces stemming from the recording of the interaction of the traced user with the computer system used. The process output is a digital document describing the traced activity in a personalized way. This document may be easily exchangeable with other persons; however, it is the person concerned by the redocumentation process who is able to specify whether to make it entirely personal (archived as private), reserved for one’s intimates (like friends or colleagues) by sending it through email or social networks, or publicly accessible (published on the Web).
Figure 1. A global vision about the redocumentation process of computer-mediated activity.
The function of the produced document may vary according to the objective underlying the redocumentation process. When the traced user performs this process on his own activity traces, the produced document may be kept as a personal support for reminiscence or reflection about his activity. It may also be exchanged with other persons to share an experience (e.g., as a guide for those involved in a similar activity) or to offer a support for analysis and reflection about the human activity or behaviour. This function may be the same when the person performing the redocumentation process is not the traced user; however, the degree of commitment would be lower and some privacy concerns have to be highlighted (O’Hara, Tuffield and Shadbolt, 2009).
For instance, a teacher using a learning system may redocument his own computer-mediated activity to generate a pedagogical guideline for his students, explaining how to solve a particular problem or what resources are the most interesting in such case. Similarly, each student may describe his own activity in a document where he provides details (what exactly he has done and why, the difficulties encountered, the errors made, etc.). This document may serve the learner to remember and reflect upon his experience so as to enhance his learning and avoid making similar mistakes. Moreover, it may be useful to other learners using the same system, as well as professionals concerned with analysing learners' activity or behaviour (tutors, system developers, analysts, etc.). In some cases, the redocumentation process becomes more necessary than optional. For instance, if a knowledge worker is asked to prepare a weekly report for his superior, it is obvious that he would not directly communicate his activity traces, but rather, he would construct a coherent unit in which he summarizes and justifies his work. Furthermore, the redocumentation process will serve to generate an organisational memory for this worker's daily activities.
The redocumentation process encourages personalized document generation for describing and sharing users' experiences. This personalization promotes autonomy, supports competence and creativity, and maintains social relations (Oulasvirta and Blom, 2008). It also ensures the widest possible dissemination of information about the user activity since the produced document is usually endowed with a legitimacy that exceeds a local or an immediate usage. However, to implement such process, some characteristics should be satisfied:
- Being easy to be applied by users.
- Being useful for users.
- Ensuring lightweight documentation and a good quality product.
- Preserving the user’s freedom in tailoring the content and the form of the produced document.
To meet these requirements, we propose the following principles:
- A narrative approach for the redocumentation process: narratives are the natural forms for describing and sharing human experiences and knowledge, in which expression of choice for human authors is obvious. Thus, narrative construction is likely to be the approach underlying the redocumentation process of computer-mediated activity so as to facilite the role of the user and to satisfy his needs during this process.
- An interactive process assisted by an authoring tool: the interactive aspect is needed for maintaining the user freedom in tailoring the content and the form of traces throughout the redocumentation process; whereas an authoring tool is usually needed to help the user manipulate the activity traces and guide towards respecting the rules that ensure the coherence of the produced document.
- A modelling approach of traces and documents: semantic modelling of activity traces offers the ability to reduce their content into manageable chunks, aggregated into higher-level related semantic units that are meaningful for human beings, as well as machine-readable. This facilitates the user intervention for traces selection and visualization, with the possibility to automatically transform traces (e.g., to generate an initial coherent unit on which the user may easily intervene). Additionally, when the document produced through the redocumentation process is semantically modelled, its construction may be assisted by an authoring tool since the document content is machine readable. Furthermore, different ways of presentation may be offered and links to the activity traces’ content may also be offered.
- The hypermedia modality for the produced document: hypermedia is an appreciated and rich presentation that allows various forms of traces to be represented through multimedia aspects; whereas relationships within traces may be expressed through hypertext links.
3. Related work
Even if we are interested in redocumenting computer-mediated activities, our context is encapsulated into that of human activity documentation. Research literature related to this subject dates back to 1945, when Bush envisioned the Memex (Bush, 1945), a device in which an individual may store and retrieve his documents and communications, as an enlarged supplement to his memory. Later, projects like MyLifeBits (Gemmell, Bell and Lueder, 2006), LifeLog (IPTO 2003) and SemanticLIFE (Admed et al., 2004) realized this idea by focusing on collecting as much data as possible on the life and activities of an individual. Nowadays, the emergent technology of lifelogging aims to capture personal life histories through digital technologies, and generates huge volumes of rich multimodal data (emails, documents, photos, videos, diaries/calendars, geo-location data, blog entries, navigation history, etc.).
From lifelogging to Storytelling. While a lifelog is a collection of data that provides a detailed description about a daily activity through both content and context, the reviewing and sharing of such data becomes questionable (Byrne and Jones, 2009). As a solution, SemanticLIFE has allowed users to setup an information repository that enriches lifelog data with semantic annotations to provide enhanced querying capabilities. However, Semantic logger (Tuffield et al. 2006b) has focused on the notion of assembling and integrating web accessible resources with no attempt to homogenize the data of the lifelog. The utility of this system was grounded in Photocopain (Tuffield et al. 2006), a photo annotation system that uses context and content based methods to generate metadata to enrich one’s personal photos collection. Certainly, activity (re)documentation does not amount to annotating or integrating the gathered data about one’s activity to facilate and expedite access to these data content. It rather implies intelligently organizing and visualizing this data for the end-users. A natural way to do so relies on the narrative paradigm, for which Fisher and MacIntyre put the emphasis on the idea that humans are storytellers (Fisher, 1989), especially about their experiences and daily lives.
Narration follows a particular form to convey messages by structuring the possible actions and events that take place within the narrative (the story). Therefore, storytelling (Frenzel, Müller and Sottong, 2004) may allow the end-user to transform a multitude of observations, insights and impressions, along with data stored within the lifelog, into a coherent whole that makes sense to him. Nevertheless, the lifelog nature poses a challenge for the creation of such stories. Some researchers have explored narrative construction from lifelog collections. Their propositions may be classified as manual, automatic or semi-automatic (interactive) storytelling approaches.
- Manual approach of storytelling from lifelogs: as an example of such approach, Harper et al. (2007) have explored the manual composition of stories from SenseCam captured images. Six participants were asked to use SenseCams to capture digital traces of their experiences, then to produce digital narratives for presentation from these traces. Even if the study was not focused on the process of creating personal narratives, it has shown that digital traces of the user's experience were not the analogue of that experience (e.g., through SensCam data, participants have noticed previously unremarked features about their experiences). Hence, an indepth analysis of relationships between the remembered experience and the experience as recollected by using its digital traces was provided.
- Automatic approach of storytelling from lifelogs: Brooks (1997) has provided an interesting
environment for story design and presentation, namely Agents Stories. His
objective was to use computational agents in order to easily and quickly
generate multiple narrations as cinematic stories. The model proposed for
the narrative uses three components of storytelling – the structure of
the narrative, the collection and organization of story pieces with some
representation of their meaning; and a navigational strategy through that
collection to get nonlinear and multiple point-of-view stories.
Furthermore, the agent story is capable of responding to the feedback of
the audience and to sometimes modify the presentation of the story. Thus,
Brooks has used autonomous agents with a story structure model for
proposing his narrative computational model. Nevertheless, when dealing
with lifelogs as initial data, the use of such computational model becomes
questionable. Indeed, Appan et al.
(2004) have demonstrated that the proposed model is not efficient for
personal storytelling by highlighting the fact that the generated
narratives are "disjoint and not very meaningful".
The problem of automating storytelling for generating narratives deals mainly with modelling these narratives. The existing narrative systems have addressed that problem according to three different approaches (Tuffield, Millard and Shadbolt, 2006): modelling the content, modelling the user or modelling the story. For modelling content, character-based approach uses agent-based computing to model human beings autonomy and their interactions so as to get emergent narratives; however, narratives produced by such agents are usually not pertinent. This is due to the fact that no structure model is used for the story to avoid possible logical contradictions within it; furthermore, modelling human characteristics within software agents is still unrealistic. User-modelling based approach models the end-user perception of the narrative through the user profile and uses this model for driving the narrative generation. Finally, for modelling the story, plot-based approach uses narrative structures and rule-based methods inspired from narrative theories in narrative generation. This includes Rhetorical Structure Theory (RST) that has been used in a variety of ways (Taboada and Mann, 2006), including text analysis, computer generation of text and teaching writing skills. As examples, it was used to generate video documentaries (Bocconi, Nack and Hardman, 2005), to aid technical writing (Silva and Henderson, 2005) and even to develop generic models for creating storytelling applications (Nakasone and Ishizuka, 2006).
- Semi-Automatic approach of storytelling from lifelogs:
MyLifeBits, as a pioneering project in lifelogging, has made an effort to
make authoring easy with Interactive Story By Query (ISBQ) (Gemmell, Aris and Lueder, 2005). ISBQ lets users
make queries and then drag and drop selections from the query result into a
story. The story form is either a slide show or a time sheet and may
associate images with location information. Appan
et al. (2004), who criticized the automatic approach of
storytelling, has asserted that users do not want to spend time editing or
authoring their stories. Thus, they have favored the use of an emergent
story framework in which the story evolves through feedback and interaction
from the user. Additionally, they have advocated the use of low-sample
capture of media within their narratives. Despite the fact that Danah et al. (2009) have agreed with this
proposition of interactive storytelling, they however thought that the the
lifelog complexity should be overcome by leveraging the available solutions
about data management rather than reducing lifelog content.
The implementation of such approaches mainly depends on designing authoring tools. Such tools help end-users stimulate their intellectual endeavors while constructing the story. An example of interesting authoring tools, though not concerned with lifelogs, is the RST-based pedagogical agent DPs (Rizzo, Shaw and Johnson, 2002) that helps children author structured multimedia presentations about their course explanations. Also, the reflective learning tool proposed by Mahmud (2004) may automatically reveal discourse relations structure to help students structure their essays by investigating and integrating various linguistic attributes; whereas the Artequakt system (Kim et al., 2002) helps users generate personalized biographies from web fragments by using templates and predefined styles for presentation.
Discussion. For computer-mediated activity redocumentation, we opt for a storytelling approach. On the one hand, we think that the manual composition of stories about the activity ensures the end-user freedom and the ease of expressing relevant contextual information about the performed activity; however, it remains usually time consuming, expensive and even discouraging for users especially when traces collection is voluminous and heterogeneous. On the other hand, we think that automating storytelling from lifelogs is too constraining for users freedom and cannot ensure high quality stories. Therefore, we choose a semi-automatic approach.
According to Bal’s layered view of narrative (1997), any complete model of narrative must address each of the layers he defines: the Fabula (to describe the objects/events and their relations), the Story (to describe the objects/events arrangement for a purpose) and the Narrative (to describe how this is realized in a particular form). Indeed, the best way for machines to produce narratives is to have an understanding of the narrative itself by using semantic technologies, particularly ontologies. For instance, the OntoMedia ontology (Lawrence et al., 2005) has successfully modelled the Fabula (Tuffield, Millard and Shadbolt, 2006). For modelling the story, ontologies built around existing narrative theories like RST are needed to assist the story construction and to ensure its coherence. Even if RST was originally designed for text, to offer a framework for the explanation and analysis of text coherence (Taboada and Mann, 2006b), it may also offer a flexible support for multimedia story authoring. Ontologies of the narrative layer depend on the story form.
In our case, the Fabula modelling will be ensured through the semantic modelling of the activity traces, whereas the story modelling will be ensured through the semantic modelling of the produced document by using an RST-based ontology. For the narrative modelling, we propose only predefined forms to be chosen by the end-user. We stress the need for a direct intervention of the user during the construction of what we may qualify as "the activity story", especially about its semantic content and structure. We argue that even if ontologies are crucial for knowledge modelling, it is unrealistic to hope to model everything implied in a narrative. Consequently, we opt for a semi-automatic approach of storytelling. Furthermore, in order to facilitate the task of the end-user and maintain the coherence of the produced document (the story), we insist on the use of an authoring tool. For implementing such tool, many ideas were inspired from the existing tools, especially the pedagogical agent DPs. In the next section, we describe the general framework that we propose for the redocumentation process.
4. A general framework for the redocumentation process
The general framework proposed for the redocumentation process of computer-mediated activity (Cf. Figure 2) implements a model-based semi-automatic narrative approach. Three important elements should be considered in that framework: the process input (a set of modelled activity traces), the process output (the final modelled document describing the activity), and the process two phases (Yahiaoui, Prié and Boufaida, 2009). The first phase of the process (P0) deals with the automatic transformation of the input activity trace(s) chosen by the end-user (the traced user or any other person performing the process) to an initial fragmented document (D0). The second phase (P1) is interactive and assisted by an authoring tool. It allows the end-user to iteratively transform the initial document by applying a set of operations. When the user is satisfied with the produced document (Dj), it may be exported as a final document (DF). Before detailing these phases, we describe our modelling of activity traces and produced documents.
Figure 2. The Redocumentation Process Framework Based on Models (for trace and document).
4.1 Activity Trace Modelling
Activity traces stem from the recording of the interaction of users with computer systems to carry out their activity. Modelling these traces depends on the traced activities and the modelling approach. Several works have proposed such modelling (Anjewierden and Efimova, 2006; Choquet and Iksal, 2007; Dragunov et al., 2005); however, the way in which they collect, model, transform and visualize activity traces differ. Our modelling of activity traces is based on the research led within the Silex team6, especially the meta-model proposed within the Trace Based System framework (TBS) (Laflaquière, Settout, Prié and Mille, 2006).
4.1.1 The TBS meta-model for activity traces
The Trace Based System (TBS) approach is both a theoretical and a practical framework for computer-activity traces management. This includes traces collecting, modelling, transforming and visualizing. A modelled trace of a computer-mediated activity is defined as a set of observed elements, time stamped, recorded on a digital media, resulting from the observation of using a computer system to perform a particular activity (Laflaquière, Settout, Prié and Mille, 2006). An observed element (obsel) describes a particular part of the traced activity; it may represent a sub-activity, an action, an object or a resource that is used, an actor, a state or an event. A trace model is defined as a formal model (possibly an ontology) that describes the structure and properties of the trace observed elements.
4.1.2 A generic activity-trace model
Our trace model (Cf. Figure 3), is based on the TBS activity trace meta-model, however, this meta-model was refined into a generic trace model by considering actions performed within the activity as the main observed elements in the activity trace. The temporal property of the observed element (action) may be a time interval or a moment characterizing its occurrence. We consider that the trace concerns a particular user and it is composed of a set of actions, whereas each action may manipulate a set of entities (objects or resources). These considerations are inspired from the unified activity model (Thomas, 2005) that represents an activity as an association of properties (objective and status) in relationship with other entities (actors, resources, products, actions, etc.). Furthermore, in order to be able to specialize the proposed trace generic model for real applications, we consider that actions and entities may be organized in hierarchies and that structural relations are possible between entities (for instance, one resource may be part of another like the relationship between a file and its folder). For generating modelled traces, it is important to use a process to collect the available basic traces of the activity according to the proposed trace model.
Figure 3. The activity trace generic model (that should be specialized for dedicated applications)
4.2 Document modelling
As any digital document, the product of the redocumentation process is an information structure that may include simple or composite objects with different content forms (text, graphics, sound, video...), or even groups of objects (Martin and Alpay, 1996). Thus, such document is fragmented into many information units called "information fragments" or "segments"7, that are inter-related.
4.2.1 Document segment definition
4.2.2 Document model
A digital document may be described through different levels —semantic, logical and physical (Christophides, 1998); and a particular structure may exist for each level. The semantic structure(s) defines the organization of the document’s meaning (for instance, the narrative structure), the logical structure defines the organization of the document syntactic structure (titles, chapters, paragraphs, etc.), whereas the physical or layout structure defines the document’s appearance (typography, functionalities, etc.). It is important to consider all these structures for constructing a document. Therefore, several studies (Djemal, Soulé-Dupuy and Vallés-Parlangeau, 2009; Barrutieta, Abaitua and Diaz, 2002; Garlatti and Iksal, 2003) tend to model these structures simultaneously within the same document and consider the segment as the basic unit on which these structures are defined. We are interested in the content’s meaning of the document produced through the redocumentation process and how the coherence of such document may be maintained when manipulated by the end-user. Thus, we focus on modelling the semantic structure of the document. However, we do not intend to neglect its logical and layout structures since we agree on the fact that the elaboration of any document implies the definition of an application between its semantic structure and the structure of the final document (Nanard and Nanard, 1989).
Figure 4 shows a generic model of document (upper part) that we specialized as our document model (lower part). For the generic model, the document is composed of several typed structures. Each structure is composed of segments and relationships; and each relation can have one or many starting or ending segments. For our specific model, the document has three main structures: the semantic, the logical and the layout structures. A semantic structure of the document relates a set of information segments with a set of semantic relations, whereas the type of each relation is deduced from the type of the semantic structure to which the relation belongs. Indeed, many theories were proposed to analyze the meaning of documents' content and to make it explicit. Among these theories, we chose Rhetorical structure theory (RST). In addition to its simplicity and ease of use, RST may be applied for document authoring through the possibility to maintain the document’s coherence (Taboada and Mann, 2006b).
Basically, RST divides a text into segments (Nucleus or Satellite) with relationships between them (rhetorical relations), whereas the text coherence is achieved by the overall effect created by these relationships. However, the use of this theory is easily extended to document of forms other than simple text (like hypermedia). Rhetorical relations are mainly used to develop an argumentation that binds together the elements of the content. They include cause and consequence, interpretation and justification, junction and contrast, problem and solution, etc. they are classified either as Satellite to Nucleus relations (RSrelation-SN) or Nucleus to Nucleus relations (RSrelation-NN) (Taboada and Mann, 2006b). In the first case, one information segment (the Nucleus) is more essential to the document's content than the other (the Satellite); for instance, when a segment describing an action in an activity trace is justified (Justify relation) by another segment added by the end-user, it is clear that the justified segment is more important than the justifying segment. In the second case, no segment is more essential than the other; for instance, when two segments describing consecutive actions within an activity trace are linked with the rhetorical relation Sequence.
Therefore, the rhetorical structure may convey the content’s meaning since it attributes a particular function to each information segment within the document’s content, and may facilitate the document authoring by providing rhetorical rules to be implemented and respected. Moreover, it can be easily implemented as a formal model (ontology). For the logical structure of the document, segments and relations defined within this structure represent those of the rhetorical structure transformed and organized according to a predefined template (an XML template for instance). In the same way, a predefined model (a stylesheet) may be used for the layout structure.
Figure 4. The document model as a specialization of a generic model, focusing on the rhetorical structure
4.3 A two phase redocumentation process
The redocumentation process of computer-mediated activity is handled through two transformation phases (see Figure 2). While the first one is automatic, the second is interactive and assisted by an authoring tool.
4.3.1 Automatic redocumentation phase (transformation T0)
This phase aims to automate a part of the redocumentation process. It allows the automatic transformation of the input activity trace(s) (chosen by the end-user) into a fragmented document (D0). Such possibility is due to the formal models of traces and documents and the availability of dedicated tool for this purpose. Each generated fragment (Fi) is considered as an information unit describing one or several observed elements within input activity trace(s). It is characterized by a content, a form and other properties (e.g., an obsel id to keep a link with the the observed element of the trace that it describes or a timecode to express the temporal property of that element). Relations (Ri) between the information fragments in the initial produced document (D0) are mainly deduced from relationships between elements of the input trace that the document describes. These relationships are expressed either as rhetorical or structural relations. For instance, temporal relations between the observed elements of an activity trace may be expressed as Sequence (rhetorical) relations between the corresponding information fragments (considered as nuclei); however other structural relations are possible to express structural relations within the trace. The automatic transformation T0 may be also parameterized in order to allow the end-user choose only a part of the activity trace to redocument (e.g., only actions that occurred during a given time interval).
4.3.2 Interactive transformation phase (transformation T1)
Since we are interested in generating a hypermedia document as a final product of the redocumentation process, the fragmented document (D0) may serve as an initial information space to build this document. During the interactive transformation phase, the user is assisted by an authoring tool and the document authoring is both interactive and iterative. We think that a personalized composition concerning the semantic, logical and layout structures of the document is important in this case. However, this personalized composition should not be confused with that of adaptative or personalized hypermedia construction (Garlatti and Iksal, 2003; Ranwez and Crampes, 1999) since the user is pro-active in our case.
For the semantic structure related to the meaning of the document's content, the user is assisted in performing operations on the initial document (D0), or on any intermediate one (Dj). These operations may involve a single information fragment (for instance, delete a fragment or insert a fragment with a particular relationship), several information fragments (for instance, merge or reorder fragments according to particular criteria like the temporal property of fragments), a relationship between fragments (for instance, delete or insert a relation between fragments), or the whole document (for instance, introduce or summarize the document). Since we focus on the rhetorical structure of the document, we can distinguish rhetorical operations from merely organizational operations. We define a set of rhetorical operations in which each operation is based on the use of a particular rhetorical relation. More precisely, each rhetorical operation implies the creation of an information fragment (a Satellite) for which the content is filled by the end-user; then, the new fragment is linked to the whole document or to a chosen fragment within this document by a rhetorical relation8. For instance, the user may justify, elaborate or evaluate the content of a particular information fragment describing an action within the trace. Furthermore, he may introduce, summarize or motivate the whole content of the document.
As a result, each information fragment in the document (Dj) may describe: (i) an observed element(s) of the input activity trace, (ii) a relationship between two information fragments in the document produced during the previous iteration (Dj-1), or (iii) an information fragment added or composed by the end-user. Relationships defined within the intermediate document (Dj) describe either existing relationships within the activity trace or relationships explicitly added by the end-user (either structural or rhetorical). Once the user is satisfied with the intermediate document, it may be exported as a final document. For the logical structure of the produced document, predefined templates may be used. Since we presume the final document to be a hypermedia, a web page may be created for a set of information segments of the semantic content, where these segments may be disposed spatially according to their Sequence rhetorical relations. The other relations between segments (rhetorical or structural) may be modelled as internal or external significant links, whereas the differences between links may reflect the differences between the corresponding relations. For the layout structure, a predefined model (stylesheet) may be proposed to the end-user so as to attribute visual effect to elements of the logical structure, with the possibility to choose some properties of the selected model by this user (like typography, colors or forms).
5. An application domain for the redocumentation process
As a computer-mediated activity to redocument, we choose the use of an application developed at the municipal library of Lyon (France). This application allows public access and navigation through the patrimonial funds of the regional press of the 19th century, stored as a digital corpus. Figure 5 shows the interface of that application9 called "Presse illustrée", where the newspaper entitled "Le progrès illustré" was the first to be stored within the application’s database. For that database model, each newspaper is composed of a set of fascicles (editions); the fascicle is composed of pages, and each page may contain articles and illustrations. All these elements are described by a set of metadata (number, date, etc.) to offer advanced ways of access, that go beyond traditional lamination of newspapers (e.g., direct access to elements by metadata, searching in full text of articles, etc.).
Figure 5. The interface of the application of navigation in the regional press (municipal library of Lyon).
The left panel shows operations for exploring the digital corpus (searching by year, theme, indexed name, etc.), whereas the central panel shows the result (e.g., a list of newspaper fascicles). Searching for articles and illustrations by content or metadata is also possible (upper right field).
5.1 Basic activity traces of the "Presse illustrée" application
A tracing module was integrated in the described application in order to capture all relevant information about the end-user activity. The product is a log file simply formatted as a set of lines and columns, as shown by Figure 6. In this file, the column "Time" gives the starting date and time of an executed action, the column "IP adr" gives the IP address of the machine used for accessing the application site (it is combined with session number to distinguish different users) and the column "URL/Action" gives either an identifier of the performed action (for instance <PDF> to display a page in a pdf format) or the url10 of the displayed resource (newspaper, fascicle, page, article, illustration, list of searched elements, etc.). In the second case, the column "Type" gives the type of the displayed resource, whereas the column "Properties" shows its properties (for instance, if the resource is a fascicle, this column displays this fascicle date, number and newspaper name).
Figure 6. An example of the log file produced from the application tracing (basic trace)
5.2 A specialized model for high-level activity traces
In order to enrich the content of the produced activity traces (log file), to facilitate their understanding and processing by both human beings and machines, and to enhance their presentation and visualization, we have developed a semantic model for activity traces. This model is an OWL-ontology that provides a specialization of our generic trace model (see Figure 3). It is deduced from the analysis of the use of the above application and describes the user’s activity according to an appropriate level of abstraction, which is relatively far from machine details. In this ontology, a trace is modelled as a set of observed elements (actions), organized in a hierarchy. Each action may manipulate one or more entities (an article, an illustration, a graphical element of the application interface, etc.) classified in a hierarchy. Figure 7 is a graphical representation of that activity trace ontology; however, only the most important concepts, properties and relations are shown. The transformation of basic activity traces (log file) to modelled traces (as instances of the trace ontology) is done through the collecting process. This process is automated through a set of transformation rules that use data issued from parsing each activity trace (log file) to instantiate the trace ontology (trace's actions and entities).
Figure 7. A graphical representation of the high-level trace ontology of the activity of using the "Presse Illustrée" application
6. A generic tool for text-based activity redocumentation
The task of developing a tool for supporting the semi-automatic redocumentation process of computer-mediated activity according to the proposed framework requires a rethinking of digital document authoring issues. The idea is to assist the end-user in generating a personalized document from activity traces to describe the activity. Firstly, the redocumentation tool should be able to present activity traces for the end-user in an intelligible way and to facilitate their manipulation. Secondly, it should help the end-user in defining the content and the form of the produced document and maintain its coherence. Thus, the redocumentation tool "ActRedoc" that we developed is an authoring tool that supports the redocumentation process. It is composed of two main modules: the automatic transformation module and the interactive transformation module. These modules are accessible through a graphical interface, as shown by Figure 8. The tool input is a set of semantically modelled traces (with the trace ontology), from which the user may choose one activity trace. Its output is a text document (in English) that describes the traced activity, which can be easily exported as a hypertext document enriched with metadata. For maintaining the coherence of the produced document, ActRedoc uses RST principles.
Figure 8. The architecture of the redocumentation tool ActRedoc.
6.1 Automatic redocumentation phase (T0) in ActRedoc
In order to benefit from the semantic modelling of activity traces for facilitating the role of the end-user in creating a document describing the traced activity, a natural language generation tool is used. NaturalOWL (Galanis and Androutsopoulos, 2007) is a tool that may produce text spans (in English) from one or many objects described by an OWL-ontology. However, the developer of this ontology would have to annotate it linguistically11 before its use by NaturalOwl. This linguistic annotation produces three RDF12 files that correspond to three different levels. The first level concerns the lexicon that deals with expressions of gender and number for the ontology classes and instances, the second level concerns micro-plans to order and express (as templates) properties of each class and relationships between the ontology classes and the third level concerns preferences about the maximum number of facts per sentence/page generated and the depth of the semantic inference. For ActRedoc, the OWL-ontology we annotated is the activity trace ontology (see figure 7) whose instances are our activity traces (instances of class Trace).
6.1.1 Activity trace selection and text segments generation
Through the interface of ActRedoc, the end-user may choose a particular activity trace. For supporting such possibility, the automatic transformation module exploits the trace ontology as a model for querying on the set of activity traces according to semantic criteria (for instance, by date or by user). Once a particular trace is selected, this module invokes NaturalOWL to generate a text description about the selected trace and NaturalOWL uses both the trace ontology and its linguistic annotation files to generate this description. Then, the automatic transformation module extracts all the observed elements (actions) of the selected trace (as instances), ordered according to their temporal property (start time). For each observed element, the automatic transformation module invokes again NaturalOWL to generate a text description about that element (action). Moreover, it enhances the generated text descriptions by applying automatically a set of linguistic rewriting rules (e.g., to correct the text punctuation or to avoid repeating the same date when describing consecutive actions). Later, the generated text descriptions (segments) about the activity trace are used in the composition of a fragmented text document (D0) according to our document model.
6.1.2 The specialized document model
The content of the document (D0) generated by the automatic redocumentation module is modelled semantically by an OWL-ontology (document ontology) that provides a specialization of our document generic model (see Figure 4). This ontology focuses on the content’s meaning and models the document as a set of text segments linked by rhetorical relations. Figure 9 shows the document ontology, inspired in part from ontoRest (Naja-Jazzar et al., 2009). In this ontology, each document is identified and linked to the activity trace that it describes (by the property uriTrace); it has a particular segment, its root, to which all the other segments of this document are (directly or indirectly) connected. Each segment belongs to one document and has a set of properties. The property Uri-source of the segment may refer to the element of the activity trace described by the segment (e.g. URI of an instance action), whereas the property typeSegment indicates whether the segment is original (stemming from the trace), composed, added or replaced by the end-user. Moreover, each segment has text content and a particular position in the document that it composes. Relationships between segments are rhethorical and of two types (RSrelation-SN and RSrelation-NN); they are described in Table 1. However, we used only a subset of rhetorical relations according to what we consider important for the redocumentation process of computer-mediated activity.
Figure 9. A graphical representation of the document ontology
6.1.3 The initial document (D0) composition
The automatic transformation module first creates a new instance of document (D0) in the document ontology. The content of the first segment (the root) in the generated document (D0) is the text description generated from the activity trace (as an instance), enriched with a text expression to introduce the rest of the document. The content of the second created segment is the text description generated from the first observed element of the trace (the first action of the traced activity); it is related to the root segment by the rhetorical relation Sequence. In the same way, the other segments are created from the remaining observed elements of the trace. They are ordered and linked according to the temporal order of the corresponding observed elements in the trace. Figure 10 shows a graphical representation of the document (D0). Finally, this document may be exported as a plain text or hypertext document, that can be easily previewed or exchanged. Furthermore, it will serve as an input for the interactive redocumentation phase.
Figure 10. A graphical representation of the initial document (D0) produced by the automatic redocumentation module
6.2 Interactive redocumentation phase (T1) in ActRedoc
Once the automatic transformation module has created an initial document from the activity trace, this document may be manipulated by the end-user through the interactive transformation module. This module allows a set of transformation operations to be applied iteratively on the document (D0) to get an intermediate document (Dj) after each iteration. These operations, classified as organizational or rhetorical, sum up and implement the operations that have been proposed within the general framework of the redocumentation process.
6.2.1 Rhetorical relations used within the redocumentation tool ActRedoc
Table 1 gives a detailed description about the rhetorical relations underlying the rhetorical operations allowed by the interactive redocumentation module of ActRedoc. Nevertheless, to ensure the possibility of previewing the transformed document by the end-user at any moment, the column Expression in Table 1 gives a way to express each rhetorical relation by text, either as a sentence to be added to this document content (i), or as an action to be performed on it (ii). For instance, the rhetorical relation Justify is expressed by adding a sentence like "the reason is that" or "that is because" between the justified text (segment) and the justifying text added by the end-user, whereas the rhetorical relation Sequence is expressed by simply ordering the related segments.
|Relation N-N Name||Nucleus||Nucleus||Expression|
|Sequence||A text segment||The next segment||<order segments> (ii)|
|Contrast||A text describing one alternative||The other alternative||Contrary to that (i)|
|Joint||A text||Another text||And (i)|
|Relation S-N Name||Nucleus||Satellite||Expression|
|Justify||A text describing an idea or an action (performed by the traced user for instance)||A text justifying the exposed idea or the performed action||The reason is that, that is because (i)|
|Elaboration||A basic information||An additional information||Moreover (i)|
|Background||A text whose understanding is being facilitated||A text for facilitating understanding||Notice that (i)|
|Enablement||A text describing an action||information intended to aid the reader in performing the action||To do that, (i)|
|Evaluation||A text describing a situation||an evaluative comment about the situation||So, Therefore (i)|
|Interpretation||A text describing a situation||an interpretation of the situation||This means that , more clearly, (i)|
|Introduction||A text||An introduction to the text||<place the saltellite before the nucleus and add "Introduction:" at the beginning> (ii)|
|Purpose||A text about an intended situation||the intent behind the situation||The purpose is that (i)|
|Summary||A text||a short summary of the text||<place the saltellite after the nucleus, preceded by the word "Summary:" > (ii)|
|Solutionhood||A text about a situation/method supporting satisfaction of the need||a question, request, problem, or other expressed need||This was a solution for the case (i)|
Table 1. Rhetorical relations used by the redocumentation tool ActRedoc
6.2.2 Interface of the redocumentation tool ActRedoc
The graphical interface of ActRedoc, shown by Figure 11, allows the end-user to choose a particular trace, to transform it automatically to a fragmented text document, then to apply different operations on this document until the user is satisfied with the product. The tool interface is composed of three panels. The right panel allows access to operations that the end-user may perform on the (fragmented) produced document, with a help area that explains the use of each operation (see Table 2). The central panel shows the text segments of the manipulated document, whereas the color of each segment’s boundary indicates its function (original, composed, replaced or added through a rhetorical relation). Moreover, when the mouse is placed over a segment’s area, properties of this segment are shown (id, position, detailed type and information about its relationship with other segments). The left panel shows the document preview as hypertext document (each hightlighted URL is a link to a ressource manipulated by the described action), whereas all the rhetorical relations are expressed by text (expressions). The red dashed arrows show examples of how text segments and their relationships are expressed within the document preview. For this document preview, a predefined model of presentation is used.
Figure 11. The graphical interface of the redocumentation tool ActRedoc
|add segment||Adds a text segment to the document by linking it to one of the existing segments of the document through a rhetorical relation of type N-N (see Table 1)|
|delete Segment||Deletes a text segment from the document|
|replace Segment||Replaces the content of an existing text segment of the document by a new content, to be provided by the end-user|
|explain Segment||Adds a text segment to the document by linking it to one of the existing segments of the document through a rhetorical relation of the type S-N (see Table 1)|
|reorder segments||Allows moving a text segment (s1) according to another one (s2) in the document (s1 before/after s2 or s2 before/after s1)|
|merge Segments||Merges the contents of two or many (consecutive) text segments of the document in one segment|
Table 2. The set of operations allowed by ActRedoc tool
6.2.3 Document export
When the end-user is satisfied with the produced document (Dj), this document may be exported through the export module as a hypertext document enriched with metadata. For this document content, all rhetorical relations existing within the instance document are represented according to their expressions. Additionally, metadata explicitly added by the end-user (document title, author, etc.) are incorporated in the header of the document. Moreover, rhetorical relations within this document are explicitly represented as a set of metadata by using RDFa13. (Adida and Birbeck, 2008; Pemberton, 2009). RDFa syntax expresses RDF structured data within a hypertext (HTML) document using dedicated attributes, so as to augment visual data in documents with machine-readable hints. In our case, the idea is to allow the mark up of human-readable data (the text produced document) with machine-readable indicators (referring elements of the document ontology or the trace ontology) for browsers and other programs to interpret. Figure 12 shows an overview of the final document content. For the document presentation, the export module uses a predefined template for its logical structure and a predefined stylesheet for its layout structure. Nevertheless, it may be parameterized to support the end-user's choices about the form and the presentation of that document.
Figure 12. An overview of the final exported document as content enriched with metadata (xhtml+rdfa)
6.3 Implementation Issues
The redocumentation tool ActRedoc is implemented as a JAVA application under Eclipse (v 3.3.1). The trace ontology and the document ontology were created by using the Protégé-OWL editor, whereas the API of this editor was used for programming the access and the manipulation of the ontologies elements (locally or remotely). A module of NaturalOWL tool was integrated as a plugin in the Protégé-OWL editor to allow a graphical linguistic annotation of the trace ontology. This annotation produced three RDF documents. However, for generating text descriptions from activity traces (as instances of the trace ontology), the NaturalOWL engine was imported as a package in our application. This module allowed the generation of natural language (in English) as text spans from the trace ontology and its linguistic annotation (RDF files).
7. A first evaluation of ActRedoc
A first version of the redocumentation tool ActRedoc was tested by a sample of users in the context of a preliminary study. This study aimed at providing an initial evaluation of the redocumentation tool by focusing on its usability14 and its utility15. It also hinted at making an in-depth study of the redocumentation process of computer-mediated activity and allowed to gain an understanding of how this process was achieved by using our tool.
Collecting usability and utility metrics usually requires an important number of users to overcome the substantial individual differences in user performance, and therefore can quicky become expensive; however, it is easy to get a quantitative study wrong and end up with misleading data. Jakob Nielsen, showed that for first evaluations about system design, it is enough to test with a handful of users (typically five users) and revise the design in the direction indicated by a qualitative analysis of their behaviour (Nielsen and Loranger, 2006); then through several similar evaluations, the design may be enhanced progressively and iteratively. We adopted this idea in our evaluation of the ActRedoc tool. Before describing participants and their experiment of redocumentation, we describe the activity traces they used as input for the redocumentation process.
7.1 Activity traces
The computer-mediated activity we traced is the use of the application "Presse illustrée" (see Section 5). To collect a set of traces, users were traced while performing significant scenarios of use. Basic traces were first extracted from the log files that were produced (by using IP addresses and starting/ending times) and then semantically modelled according to our trace ontology (collecting process).
In our experiment, in addition to the evaluation of ActRedoc tool, we intended to show through a minimal set of selected traces:
- the effect of traces length on the redocumentation process,
- the effect of the time span between the moment when the traced activity took place and the moment of redocumenting the same activity,
- the possible variation in the redocumentation process induced by the fact that one user is redocumenting his own activity or someone else's.
Therefore, amongst the available traces, we chose 6 traces with the following characteristics:
- Traces had different lengths (estimated by the number of actions in the activity trace). 4 of them had similar length to allow the study of the redocumentation process without the effect of traces length, one trace was the longest (trace-1) and another trace was the shortest (traces-5).
- Among the selected traces, two traces concerned the same traced user. The first trace (trace-1) was about an old activity, whereas the second one (trace-2) was about a recent activity.
- In most cases, the selected traces were redocumented once by the traced user himself, and once by another participant.
We asked 10 individuals in our research group (3 females, 7 males; with an age range of 26-36 years) to engage in the redocumentation process of the computer-mediated activity. Each participant had to use one or two traces with the redocumentation tool ActRedoc, in order to generate a text document(s) for describing the traced activity. Among these participants, 5 of them had not used the "Presse illustrée" application before (participants 4,5,6,7,9), whereas the others were traced users (participants 1,2,3,8,10). We did not attribute traces to participants in a purely random way; some considerations were taken:
- Each traced user was asked to redocument his own traced activity, whereas one participant (participant 1) had to redocument two of his traced activities, one old and another recent.
- One traced user (participant 10) was asked to redocument the activity of another traced user (participant 1).
- Each trace was used by at least two participants. Thus, each traced activity was redocumented by at least one participant different from the traced user.
- One participant (participant 8) was asked to redocument both the longest and the shortest traces.
Prior to the experiment, we asked participants who were not traced to use quickly (for 15 mins) the application "Presse illustrée" in order to get an idea about the content of activity traces. Moreover, we asked each participant to briefly review the basic trace (log file) of the activity that he would redocument, then to describe that activity through an oral retelling. We noticed that participants had difficulties when trying to interpret the content of the basic trace in a coherent way, especially when the participant was not the traced user. After that, we introduced participants to the use of ActRedoc tool (for nearly 20 mins) through an example of redocumenting a given traced activity. This activity dealt with researching for articles and illustrations about the French fashion of the 19th century. We described to participants the use of each operation supported by the redocumentation tool and we focused on rhetorical operations. Indeed, participants seemed not to be familiar with this type of operations despite the fact that rhetorical relations are usually implicitly used when writing or retelling about experiences
7.3 Experiment and results
For our experiment of activity redocumentation, we tested participants separately. For each participant, we chose first the activity trace that he had to redocument; then we invited the participant to use the redocumentation tool upon this trace. We stress the fact that we did not impose on participants any constraint about the objective underlying the redocumentation process. Hence, each participant was free to decide about his particular objective.
During the redocumentation process, we were observing what participants were doing and taking notes. We were especially interested in which operations they used more or less than others, how they combined them, how much time they spent on each type of operation, where they succeeded or had difficulties, how many errors occurred and how participants reacted. Moreover, during this time, we tried to let them solve whatever problem they encountered on their own and only intervened when it was necessary.
After the completion of the task, each participant was interviewed in order to evaluate the redocumentation tool. The interviews with subjects provided some insight into their ease of use of the tool and their satisfaction about the tool interface ergonomic, functionality and products (the initial document D0 and the final document). Participants were asked if they were pleased to use ActRedoc and what further needs they had. Finally, each participant was asked to clarify the objective that he tried to achieve through the redocumentation process since it was this objective that defined the content and the genre of the final document produced.
As a synthesis, we combined our observations with the answers of participants in Table 3. In this table, participants and traces that they exploited are both numbered, the traced user is mentioned for each trace (Traced participant) and the duration of the redocumentation process is recorded (Process time), whereas the remaining columns show details about the evaluation of the tool and its products:
- Produced document describes the genre, the length and the intended recipient of the final document. For the genre, we classified each produced document as synthesis report, analysis report or annotated report. This classification was done according to our analysis of these documents (about their contents/forms) and the participant's objectives. In a synthesis report, the participant checked out from the initial document D0 (the trace’s content) only what he qualified as important; then he appropriated and enriched this content according to a personal style. In an annotated report, the participant tended to keep much of the initial document content and focused on explaining this content parts (mainly descriptions of actions) through rhetorical relations; hence explanations anchored to parts of the initial document appeared as annotations. In an analysis report, the participant focused on the traced user reasoning and behaviour when performing the traced activity. He tried to tailor and enrich the content of the initial document D0 according to his analysis and personal style. For the final document length, we qualified it as either long (L) or short (S). The intended recipient of the produced document may be oneself (personal) to serve for remembrance, especially for the traced user; or it could be for helping other persons possibly concerned by the same activity or the traced user behaviour (guiding).
- Tool utility gives an evaluation of ActRedoc through participants’ evaluations of the initial produced document (D0) and their satisfaction about the tool’s functionality; in addition to findings about the use of the operations supported by the tool.
- Tool usability gives an evaluation of ActRedoc through learnability, efficiency and memorability. Learnability was estimated to be how easy it was for participants to accomplish basic tasks the first time they encountered the tool. Efficiency was estimated to be how quickly participants could perform tasks once they had learned the design of the tool. Memorability was estimated to be how easily participants could re-establish proficiency when they had returned to the tool after a period of not using it; thus, only participants engaged in the redocumentation of more than one trace were concerned.
|Participant||Trace||Traced participant||Process Time (mn)||Produced document||Tool utility||Tool usability|
|Evaluation of D0||Used operations||Frequent operations||User satisfaction||Learnability||Efficiency||Memorability|
|1||1||1||40||Personal annotated report (L)||G||All except solutionhood||Rhetorical||G||G|
|1||2||1||15||Personal analysis report (S)||G||organizational, justify||Merge, replace, justify||G||G||G||G|
|2||3||2||20||Guiding synthesis report (S)||A||organizational, introduction, justify||Replace, merge, delete, justify||A||A||G|
|3||4||3||30||Guiding annotated report (L)||G||rhetorical||evaluation, justify, purpose||G||G||G|
|4||2||1||25||Guiding synthesis report (S)||A||organizational, introduction, summary, justify||Merge, replace, justify||A||G||G|
|5||5||8||90||Guiding analysis report (L)||G||All except delete,reorder, rhetorical-NN, solutionhood||Background, evaluation, justify, replace||G||A||A|
|6||2||1||35||Guiding synthesis report (L)||G||merge,replace, introduction, summary, justify||Justify, replace||G||G||G|
|7||4||3||20||Guiding annotated report (S)||G||Rhetorical, merge||Evaluation, justify, merge||A||A||G|
|8||5||8||35||Guiding synthesis report (S)||G||All except rhetorical-NN, solutionhood||Merge, delete, replace, justify||G||G||A|
|8||1||1||50||Guiding annotated report (L)||G||All except delete,reorder, rhetorical-NN, solutionhood||Justify, background, elaboration||G||G||A||G|
|9||3||2||15||Guiding analysis report(S)||G||replace, delete, merge||Replace, delete||G||G||G||G|
|9||6||10||20||Guiding synthesis document(S)||G||Introduce, summary, merge, replace||Merge, replace||G||G||A|
|10||6||10||40||Guiding annotated repport(S)||G||All except rhetorical-NN, reorder||Merge, replace, Background, justify, purpose||G||G||A|
Table 3. Summary about the redocumentation experiment. Organizational operation (delete, replace, reorder or merge segments); Rhetorical operation (add segment with rhetorical-NN or explain segment with rhetorical-SN); Rhetorical-SN (justify, elaboration, background, enablement, evaluation, interpretation, introduction, summary, purpose, solutionhood); Rhetorical-NN (sequence, joint, contrast); evaluation (G: good, A: average, and L: low).
Tool evaluation. This experiment allowed us to notice the satisfaction of the majority of participants about the redocumentation tool utility, usability and products. The initial document (D0), produced automatically by ActRedoc from the activity trace, was clearly preferred by participants over the basic trace (log file). This initial document was considered as a good support for the interactive redocumentation phase since it described the performed activity in a detailed and coherent way. Nevertheless, two participants (participants 2 and 3) claimed that the content of this document may be reduced and enhanced. Participants were also satisfied about the ease of use of the tool. Seven out of ten participants had good learnability and good efficiency about using ActRedoc, whereas eight participants were relatively satisfied about the tool functionality and seven about the tool interface ergonomic.
Two participants (2 and 7) stated that it would be better if the middle panel of the tool interface showed relationships between segments in a graphical form, instead of letting the user infer this information from segments positions, colors and displayed metadata. Therefore, we plan to replace the middle panel of the interface with a graph of typed segments and typed relationships. Moreover, three participants required further needs, especially the possibility to enrich the content of the produced document with other media than text (illustrations, videos, etc.). This gives us indications of the challenges we will face in our future work, including how to allow and facilitate the intervention of the end-user on the form and presentation of the final document; in addition to providing a better assistance during the process. Finally, two participants had difficulties while trying to apply rhetorical operations; however the main reason was the inappropriate use of these operations. For instance, participant-9 tried to merge a text segment added through the rhetorical relation justify with an original segment (that described an action within the trace), whereas participant-2 tried to reorder the first segment of document (D0) and the segment added through a rhetorical relation with the fourth segment in this this document. We deduced that users needed more time and practice to learn the meaning and the use of rhetorical relations. We also plan to enhance the tool interface in order to as much as possible avoid this kind of situation, and we will probably reduce the set of rhetorical relations used within ActRedoc by omitting those that were rarely used (like solutionhood, joint, contrast and interpretation).
Time spent on the redocumentation process. The time spent on the redocumentation process is widely variable. It may be related to the trace length, as noticed for participant-8 who had redocumented both the longest and the shortest traces. Nevertheless, it was not the only reason since participant-5 also spent a long time redocumenting the shortest trace. The time spent may also be linked to the objective of each participant or to the variation in participants’ performance. For the first case, we give the example of participants 8 and 5: participant-8 aimed at describing his own traced activity for other persons through a synthesis report (how he had searched for information about Napolean Bonaparte and what he had found as interesting), whereas participant-5 aimed at providing a detailed analysis of both the traced activity and behaviour of participant-8 in a coherent report (for instance, he had attested in his report that participant-8 seemed to confuse Bonaparte with Napolean II or III) . Thus, participant-5 may have needed to spend a great deal of time appraising and inspecting the content of the activity trace. Indeed, he stated that it was time consuming, taking several minutes to complete an analysis of an activity trace that spans just a few minutes. For the second case, we give the example of participants 4 and 6. Although the two participants redocumented the same traced activity of another participant (trace-2 of participant-1) and had both the same objective, they spent different amounts of time on the redocumentation process.
Moreover, the traced user (participant 1) took less time in redocumenting his own activity (trace 2) than the other two participants working on the same trace. We guess that it was easier for him to redocument his own activity rather than that of another person. Contrary to that, we also noticed a reverse situation: participant-2 took more time redocumenting his own traced activity than participant-9 with the same activity, even if his aim was to analyze the activity and the traced user rather than just providing a synthesis about the performed actions. We think that it could be due to the fact that participant-2 seemed more engaged in the redocumentation process than participant-9. For traced users, there was another factor that affected the time spent on the redocumentation process since it really mattered for them whether the traced activity to redocument was old or recent. In the first case, they took more time trying to remember the context in which the activity was performed.
Operations use and the final document. Most of the participants did not review the initial produced document (D0) with any regularity nor did they use the operations supported by ActRedoc. We noticed that, usually, traced users seemed more engaged in the redocumentation process than other participants. For instance, participants 1 and 10 tended to review the initial document (D0) with excitement, using highly emotive language reflecting the ability of this document to provoke reminiscence over their past activities. During the redocumentation process, traced users applied various operations on D0 (both rhetorical and organizational operations) and previewed the produced document each time. Indeed, they spent more time than other participants on choosing which operation fitted more to their purpose at any time (particularly, participants 1, 2, and 8), even if they had a long trace to redocument (participant-1). However, participant-8 stated that the length of the activity trace of participant-1 discouraged him; this trace review through D0 took over 30 mins, by the end of which this participant had become visibly disengaged from the trace content and therefore from the use of operations within ActRedoc.
It is obvious that participants were more familiar with organizational operations than with rhetorical operations. Therefore, some participants (5, 3, and 7) consulted help about rhetorical relations many times, whereas other participants (2, 4 and 9) used only some rhetorical operations (mainly those based on Justify, Introduction and Summary) and the organizational operation replace to avoid the use of the remaining rhetorical operations. Moreover, participant-4 stated that it would be better if the set of rhetorical operations was reduced. Contrary to that, participant-1 stated that he was pleased to have such rich set of operations and stressed the importance of each of these operations in the redocumentation process of his activity. Indeed, we noticed that traced users were usually motivated by using different rhetorical operations since they had many ideas with which to explain their different actions (background, elaboration, justification, purpose, etc.).
For organizational operations, replace and merge were used by most of the participants, unlike delete and reorder. While traced users who redocumented their activities did not show hesitation about using such operations, participants who redocumented others’ activities did. We argue that it might be due to privacy concerns. For instance, participants 2 and 10 deleted parts from the initial document (D0) during the redocumentation of their activities and when interviewed, participant-10 justified this by saying that his intention was to hide all actions that he qualified as private, whereas participant-2 considered what he removed as irrelevant. Meanwhile, participants 7 and 8 stated that they had avoided such operations in order to keep as much as possible the activity trace content. The objectives of participants underlying the redocumentation process (why and for whom) also had their effect on the use of operations within ActRedoc. Indeed, when participants aimed at providing a synthesis about the activity, they used lot of operations like merge, replace and justify (like participants 4, 2 and 9), whereas they used other rhetorical operations like purpose, background and elaboration to go deeper in their analysis of the traced activity (like participant-5). However, when they tried to enrich the trace content only, they focused on using rhetorical operations and therefore produced an annotated report (like participants 6, 7, 10).
8. Conclusion and discussion
The documentation of computer-mediated activity is a process that requires collecting all the important data for the description of that activity. The main part of this data may result from automated recording of the use of computer systems (activity traces), whereas the remaining part usually requires human intervention for explaining the context of the traced activity, user intentions and rationale. Furthermore, collecting data about the performed activity is not sufficient, and an organizational and interpretative process is needed for expressing only what is important in a coherent and easily exchangeable way. In this paper, we presented a model-based semi-automatic approach for the redocumentation of computer-mediated activities based on the Rethorical Structure Theory (RST). This generic approach uses formal models for the input (activity trace) and output (produced document) of the redocumentation process. Moreover, it handles the process as the construction of a narrative (storytelling), a natural way for human beings to describe their experiences. The narrative construction is characterized by two main phases. The first phase is automatic and generates an initial coherent document from the activity trace; while the second is interactive and allows the user to tailor the content and form of the initial document according to his needs and choices, in order to produce an easily exchangeable document.
As a first implementation of our approach, we developed a redocumentation tool named ActRedoc, dedicated to the production of text documents. ActRedoc exploits system activity traces that have first been semantically modelled, in order to facilitate the automatic generation of a coherent initial text document. The initial document serves as an input for the interactive redocumentation phase, in which it is tailored according to the user's choices and preferences. By semantically modelling the initial document, and by specifying with rhetorical rules the operations that can be performed, we aim to preserve the coherence of the document throughout its manipulation by the end-user. The first evaluation of ActRedoc by a small sample of users has shown that they were satisfied about the usability and the utility of the tool. However, they raised some issues, especially about the form of the final document and facilities for using the rhetorical operations.
In this work, we used many technologies of the semantic web: ontologies for modelling traces and documents in order to allow their use by both humans and machines, natural language generation (NaturalOWL) to transform formal data into humanly understandable text, and RDFa for enriching the content of the final document with metadata describing its semantic structure (rhetorical structure) as well as the elements of the activity trace which it describes, to make it machine-readable. Our goal was to maintain, during the entire process, enough machine-usable information about the document to accurately support the redocumentation task, while leaving the user in charge of the process. The choice of RST, rather than other theories, is due to its simplicity, its possible use in maintaining document coherence and its wide use within many projects for document authoring. In our future work, we aim at enhancing the redocumentation tool that we have developed and to test it for other activities, especially by potentially committed users. Indeed, the purpose is to consider activities for which the redocumentation process will be more necessary than optional so as to really evaluate our tool utility. Furthermore, we aim at providing richer forms for the produced document (hypermedia) and more freedom for the end-user to tailor the document presentation; in addition to dealing with traces of collaborative activities.
- Adida, B. and Birbeck, M. (2008) "RDFa primer: Bridging the Human and Data Webs". W3C Working Group Note 14 October 2008 http://www.w3.org/TR/xhtml-rdfa-primer/ (Accessed January 3, 2011)
- Admed, M., Hoang, H. H., Karim, M. S., Khusro, S., Lanzenberger, M., Latif, K., Michlmayr, E., Mustofa, K., Hguyen, H. T., Rauber, A., Schatten, A., Tho, M. N. and Tjoa, A. M. (2004) "SemanticLIFE- A framework for managing information of a human lifetime". In Proceedings of the 6th International Conference on Information Integration and Web-based Applications and Services (Jakarta, Indonesia), pp. 1-10.
- Anjewierden, A., Efimova, L. (2006) "Understanding weblog communities through digital traces: a framework, a tool and an example". In Proceedings of OTM Confederated International Workshops, (Montpellier, France) pp. 279-289.
- Appan, P., Sundaram, H. and Birchfield, D. (2004) "Communicating Everyday Experiences". In Proceedings of the 1st ACM workshop on Story representation, mechanism and context (SRMC 04), pp. 17-24.
- Bal, M. (1997) Introduction to the theory of Narrative. University of Toronto Press, Toronto:Canada, 2nd edition. 1997.
- Barrutieta, G., Abaitua, J. and Diaz, J. (2002) "An XML/RST-based approach to multilingual document generation for the web". In Proceedings of the Natural Language, No. 29 (Italy), pp 247-253.
- Bocconi, S., Nack, F. and Hardman, L. (2005) "Using rhetorical annotations for generating video documentaries". In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME 05), pp. 1070-1073.
- Brooks, K. (1997) "Do Story Agents Use Rocking Chairs? The Theory and Implementation of One Model for Computational Narrative". In Proceedings Multimedia 1996 (ACM 1997), pp. 317-328.
- Buckland, M. K. (1997) "What Is a Document?". Journal of the American Society of Information Science, Vol. 48, No. 9, 804-809.
- Bush, V. (1945) "As We May Think". The Atlantic Monthly, Vol. 176, No. 1, 101-108.
- Byrne, D. and Jones, G. J. F. (2009) "Creating Stories for Reflection from Multimodal Lifelog Content: An Initial Investigation". In Designing for Reflection on Experience, Workshop at CHI 2009 (Boston, U.S.A.), pp. 09-14.
- Champin, P. A., Prié, Y. and Mille, A. (2003) "MUSETTE: Modelling USEs and Tasks for Tracing Experience". In WS5: From Structured Cases to Unstructured Problem Solving Episodes For Experience-Based Assistance at ICCBR'03 (Trondheim, Norway), pp. 279-294.
- Christophides, V. (1998) "Electronic Document Management Systems". http://www.ics.forth.gr/~christop/. (Accessed February 12, 2011)
- Clauzel, D., Sehaba, K. and Prié,Y. (2009) "Modelling and visualising traces for reflexivity in synchronous collaborative systems". In proceedings of the International Conference on Intelligent Networking and Collaborative Systems (INCoS 2009, Spain), edited by Hal Thwaites and Lon Addison (Los Alamitos, CA: IEEE Computer Society Press), pp. 16-23
- Choquet, C. and Iksal, S. (2007) "Modeling Tracks for the Model Driven Reengineering of a TEL System". Journal of Interactive Learning Research (JILR), Vol.18, No. 2, 161-184.
- Djemal, K., Soulé-Dupuy, C. and Vallés-Parlangeau, C. (2009) "Multistructured documents: from modelling to multidimensional analyses". SciWatch Journal, hexalog (Barcelona, Spain), Vol. 2, No. 4, 12-24.
- Dragunov, A.N., Diettrich, T.G., Johnsruhe, K., McLaughlin, M., Li, L., Herlocker, J.L. (2005) "TaskTracer: A Desktop Environment to Support Multi-tasking Knowledge Workers". In Proceedings of the 10th International Conference on Intelligent User Interfaces (San Diego, California), pp. 75-82.
- Fisher, W. R. (1989) Human Communication as Narration: Toward a Philosophy of Reason, Value, and Action. Columbia: University of South Carolina Press, 1989.
- Frenzel, K., Müller, M. and Sottong, H. (2004) Storytelling. Das Harun-al-Raschid-Prinzip, München: Hanser.
- Galanis, D. and Androutsopoulos, I. (2007) "Generating Multilingual Descriptions from Linguistically Annotated OWL Ontologies: the NaturalOWL System". In proceedings of the 11th European workshop on natural language generation (ENLG 2007, Germany), pp. 1-4.
- Garlatti,S. and Iksal, S. (2003) "Declarative Specifications for Adaptive Hypermedia Based on a Semantic Web Approach". In User Modeling 2003: 9th International Conference (UM03, USA), pp. 81-85.
- Gemmell, J., Aris, A. and Lueder, R. (2005) "Telling Stories With MyLifeBits". In Proceedings of IEEE International Conference on Multimedia & Expo, pp. 6-9.
- Gemmell, J., Bell, G., & Lueder, R. (2006) "MyLifeBits: a personal database for everything". In Communications of the ACM (CACM), Vol. 49, No. 1, 88-95.
- Gray, P., McLeod, I., Draper, S., Crease, M. and Thomas, R. (2004) "A Distributed Usage Monitoring System". In proceedings of Computer-Aided Design of User Interfaces IV (CADUI 2004, Portugal), pp. 121-132.
- Harper, R., Randall, D., Smyth, N., Evans, C., Heledd, L. and Moore, R. (2007) "Thanks for the Memory". In Proceedings BCS HCI 2007 (BCS, Swinton, UK), Vol. 2, pp. 39-42.
- Ioannides, D. (2007) A study of information fragment association in information management and retrieval applications. Phd thesis in Philosophy, University of Pittsburgh.
- IPTO (2003) Lifelog proposer information pamphlet. Defense Advanced Research Projects Agency Information Processing Technology Office document PIP 03-30, 2003.
- Kim, S., Alani, H., Hall, W., Lewis, P. H., Millard, D. E., Shadbolt, N, R. and Weal, M.J. (2002) "Generating Tailored Biographies from Automatically Annotated Fragments from the Web". In proceedings of the 15th European Conference on Artificial Intelligence (ECAI’02, France), pp. 1-6.
- Laflaquière, J., Settout, L. S., Prié, Y. and Mille, A. (2006) "A trace-based System Framework for Experience Management and Engineering". In proceedings of the second International Workshop on Experience Management and Engineering (EME 2006, UK) in conjunction with KES2006, pp. 1-8.
- Lawrence, K. F., Tuffield, M. M., Jewell, M. O., Prugel-Bennett, A., Millard, D. E., Nixon, M. S., schraefel, M. C. and Shadbolt, N. R. (2005) "OntoMedia - Creating an Ontology for Marking Up the Contents of Heterogeneous Media". In Ontology Patterns for the Semantic Web ISWC-05 Workshop (Galway, Ireland), pp. 1-8.
- Mahmud, R.(2004) "Possible Attributes to Identify Discourse Relations for a Reflective Learning Tool". In the 7th Annual CLUK Research Colloquium (Birmingham, UK), pp1-7.
- Mann. W, & Thompson. S. (1988). "Rhetorical structure theory:Toward a functional theory of text organization". Text, Vol. 8, No. 3, 243-281.
- Martin, P and Alpay, L. (1996) "Conceptual Structures and Structured Documents". In Proceedings of the 4th International Conference on Conceptual Structures (ICCS'96, Australia), pp. 145-159.
- Nakasone, A. and Ishizuka, M. (2006) "SRST: A Storytelling Model Using Rhetorical Relations". Lecture Notes in Computer Science, Vol 4326, Technologies for Interactive Digital Storytelling and Entertainment, pp. 127-138.
- Nanard, M. and Nanard, J. (1989) "MacWeb : un outil pour élaborer des documents". In WOODMAN’89, Workshop on Object Oriented Document Manipulation (Rennes).
- Naja-Jazzar, H., de Silva, N., Skaf-Molli, H., Rahhal,C. and Molli, P. (2009). "OntoReST: A RST-based Ontology for Enhancing Documents Content Quality in Collaborative Writing ". Journal of Computer Science (INFOCOMP), Vol. 8, No. 3, 03-13.
- Nelson, K. (1989) Narratives from the Crib. Harvard University Press, Cambridge:Mass. 1989.
- Nielsen, J. and Loranger, H. (2006) Prioritizing Web Usability. New Riders Press.
- O’Hara, K., Tuffield, M. and Shadbolt, N. (2009) "Lifelogging: Privacy and empowerment with memories for life". Identity in the Information Society journal, Vol. 1, No. 2.
- Oulasvirta, A. and Blom, J. (2008) "Motivations in personalization behavior". Interacting with Computers journal, Vol. 20, No. 1, 1–16.
- Pédauque, R. (2006) "Le document à la lumière du numérique (Forme, texte, médium : comprendre le rôle du document numérique dans l’émergence d’une nouvelle modernité". In C&F Eds. English version : Document : form, sign and medium, as Reformulated for electronic documents (http://archivesic.ccsd.cnrs.fr/docs/00/06/22/28/PDF/sic_00000594.pdf). (Accessed March 10, 2011).
- Pemberton, S. (2009) "RDFa for HTML Authors". http://www.w3.org/MarkUp/2009/rdfa-for-html-authors (Accessed January 11, 2011)
- Ranwez, S. and Crampes, M. (1999) "Conceptual Documents and Hypertext Documents are two Different Forms of Virtual Documents". In Proceedings of the Workshop on Virtual Documents, Hypertext Functionnality and the Web, 9th International World Wide Web Conference (Toronto, Canada).
- Rizzo, P., Shaw, E. and Johnson, W. N. (2002) "An Agent That Helps Children to Author Rhetorically-Structured Digital Puppet Presentations". In Proceedings of the 6th International Conference on Intelligent Tutoring Systems, pp. 903-912.
- Settouti, L. S., Prié, Y., Marty, J. C. and Mille, A. (2009) "A Trace-Based System for Technology-Enhanced Learning Systems Personalisation". In proceedings of the 9th IEEE International Conference on Advanced Learning Technologies (Riga, Latvia).
- Silva, N. D. and Henderson, P. (2005) "Narrative support for technical documents: Formalising rhetorical structure theory". In Proceedings of Enterprise Information Systems (In Press). ICEIS.
- Taboada, M & Mann, W. C. (2006) "Applications of Rhetorical Structure Theory". Journal of Discourse Studies, Vol. 8, No. 4, 567-588.
- Taboada, M. and Mann, W .C. (2006b) RST Website: Introduction to Rhetorical Structure Theory. http://www.sfu.ca/rst/05bibliographies/report.html. (Accessed March 3, 2011)
- Thomas, P. M. (2005) "Unified Activity Management: Explicitly Representing Activity in Work-Support System". In the European Conference on Computer Supported Cooperative Work Workshop on Activity: From a theoretical to a Computational Construct (ECSCW 2005, France).
- Tuffield, M. M., Harris, S., Dupplaw, D. P., Chakravarthy, A., Brewster, C., Gibbins, N., O’Hara, K, Ciravegna, F., Sleeman, D., Shadbolt, N. R. and Wilks, Y. (2006) "Image annotation with Photocopain". In Proceedings of the First International Workshop on Semantic Web Annotations for Multimedia (SWAMM 2006, Edinburgh).
- Tuffield, M. M., Millard, D. E. and Shadbolt, N. R. (2006) "Ontological Approaches to Modelling Narrative". In the 2nd AKT DTA Symposium (Aberdeen University, UK).
- Tuffield, M. M., Loizou, M., Dupplaw, D. P., Dasmahapatra, S., Lewis, P. H., Millard, D. E. and Shadbolt, N. R. (2006b) "Semantic Logger: Supporting service building from personal context ". In Proceedings of Capture, Archival and Retrieval of Personal Experiences Workshop (ACM MultiMedia), 1-9.
- Yahiaoui, L., Prié, Y. and Boufaïda, Z. (2008) "Redocumentation des traces d'activité médiée informatiquement dans le cadre des transactions communicationnelles". In 19ème Journées francophones d'Ingénierie des Connaissances (IC'2008, France), pp. 197-209.
- Yahiaoui, L., Prié, Y. and Boufaïda, Z. (2009) "The redocumentation process of computer mediated activity traces: a general framework". In the 20th ACM Conference on Hypertext and Hypermedia (HT09, Italie), pp. 363-364.
- Lifelogging includes desktop activity capture (activities durations, used documents content, etc.), mobile activity capture (used applications, calls, SMS, etc.), biometric data capture (by sensCams), context capture (by GPS, Bluetooth) and other passive data capture (photos and video).
- The intentionality of the designer of the tracing module allows considering what might be observed or unobserved in activity traces.
- Oracle trace analyzer: http//www.oracleutilities.com/Packages/TraceAnalyzer_IOUG.pdf
- Cyclope Employee Monitoring Software: http://www.cyclope-series.com/employee-surveillance/employee-monitoring-software.html
- For instance, information about this context may be supplied trough justifications of the traced user.
- Silex (a team within liris laboratory): http://liris.cnrs.fr/silex
- As some authors use the term "segment" instead of "fragment", we use both terms indifferently in this paper.
- For a detailed description about rhetorical relations see (http://www.sfu.ca/rst/01intro/intro.html)
- the "Presse illustrée" application site: http://collections.bm-lyon.fr/presseIllustree/
- Each url in the log file should be prefixed by : http://collections.bm-lyon.fr
- The linguistic annotation is done by a module of NaturalOWL, integrated as a plug-in in Protégé-owl editor.
- RDF: Resource Description Framework language
- RDFa : Resource Description Framework in attributes
- The usability of a tool is defined in terms of quality metrics about its ease of use such as learning time, efficiency of use, memorability, users’ errors and the subjective satisfaction of its users.
- The utility refers to the tool’s functionality and if it really does what users need.