EvaluationEvaluation
Last Updated on Thursday, 14 May 2009 07:03
The evaluation of AMOR will be achieved in three independent steps. First, the conflict detection enhancements will be quantitatively analyzed in the course of experiments. Second, the conflict resolution phase will be examined in an empirical study. And finally, the whole AMOR system will be adapted for the Enterprise Architect modeling tool which has been developed by our project partner.
Experimental Evaluation of Conflict Detection. For evaluating the supposed improvements concerning the conflict detection between model versions, a quantitative evaluation based on experiments is planed. The first task is the establishment of a test-set for model versioning systems. This model versioning test-set should cover various model versioning scenarios with respect to used DSLs, such as structural and behavioral languages, and kinds of conflicts, such as structural and semantic conflicts. For evaluating model versioning systems including AMOR, all true-positive conflicts for the versioning scenarios have to be properly defined, in order to elicit how much of them are actually found by the model versioning systems to determine recall and precision. In addition to the comparison of AMOR against other model versioning systems, various configuration possibilities of AMOR have to be evaluated. For example, an interesting scenario would be to compare AMOR’s semantic-based versioning facilities with AMOR’s operation-based versioning facilities.
Empirically Study-Based Evaluation of Conflict Resolution. In contrast to the evaluation of conflict detection, we plan to apply a qualitative empirical study. In particular, we plan to conduct experiments with students of our Model Engineering course (around 200 master students every year). The methodology for this empirical study is as follows. First, the students are divided into groups of three persons. Each group receives a pre-defined base UML model, which has to be extended by three change requests defined as textual definitions with a certain overlap and side effects between them. Each change request has to be incorporated into the base model by exactly one student. After adapting the models, the students have to check-in their models into a repository in a pre-defined order. For each group, three check-in processes are performed by permuting the check-in sequences of the students.
The evaluation focus lies on answering the following questions.
o How appropriate is the provided conflict resolution support? – Evaluated by questionnaires included in the Versioning Assistant.
o How much time is required for conflict resolution? – Evaluated by time measurements included in the Versioning Assistant.
o How much automatic adaptation can be performed? – Post-analysis of the experiment by determining how much and which additional supporting information can be mined during the experiment.
Case-Study-based Evaluation of Adaptation. In order to demonstrate the integration and adaptation of AMOR with respect to state-of-the-art modeling tools, a case study is planed in cooperation with SparxSystems in which language-specific conflict detection and resolution for UML as well as dedicated versioning front-ends for Enterprise Architect are developed.
The first part of the case study comprises adaptation of the AMOR back-end for UML. In this part of the case study, the goal is not to investigate how AMOR is integrated with stand-alone modeling tools, but rather how to define UML-specific conflict detections and conflict resolutions. In particular, semantic views should be provided for UML diagrams in order to enhance the detection of semantic conflicts which are frequently occurring between UML model versions. The focus on the UML language lies on class diagrams, sequence diagrams, activity diagrams and state machines due to the fact that these diagrams are the mostly used parts of UML. The results of the first part of the case study will be a UML specific adaptation of the versioning back-end and an adapter component between Enterprise Architect and AMOR in order to exchange models between these two systems.
The second part of the case study aims at further improving the versioning capabilities for UML models by using in addition, to semantically enhanced versioning capabilities, also operation-based versioning. This requires that an Editing Observer component for Enterprise Architect is built which will be achieved in close cooperation with SparxSystems. Finally, a dedicated Versioning Assistant is implemented as a plug-in for Enterprise Architect which allows exploring conflicts between models shown in their concrete syntax instead of working with the abstract syntax only. The plug-in development will also be conducted in close cooperation with Sparx Systems.