VMTL
VMQL
VMCL
Sidebar
vmtl:evaluation
This is an old revision of the document!
Table of Contents
Empirical Evaluation
The learnability of VMTL has been experimentally evaluated and compared to that of the Henshin and Epsilon model transformation languages. As of August 2015, a detailed discussion of this evaluation's results is under review for publication. This page will be updated to include it as soon as possible. In the interest of reviewers and of those looking to replicate our experiments or verify our analysis methods, we provide a complete replication package.
Replication Package
The replication package can be downloaded as a ZIP archive here.
It consists of the experimental material used for each experiment, the collected and encoded data, and statistical analysis scripts for analyzing the data. The experimental material and data are published under the Creative Commons Attribution 4.0 International License, and the statistical analysis scripts are published under the MIT License.
Experimental Material
We have conducted our evaluation in the form of two questionnaire-based experiments we refer to as Experiment 1 and Experiment 2. The replication package includes three PDF documents for each experiment, one for every applied treatment. The treatments present questions regarding the evaluated transformation languages in different orders, as a mitigation measure against learning effects. This measure is particularly relevant in the case of Experiment 1, where the same questions are asked for each transformation language. Participants were randomly assigned to one of the treatments.
The questionnaires consist of:
- a comprehension task requiring participants to select the correct descriptions of several transformation specifications from a list of options;
- a cognitive load assessment task requiring participants to rate the difficulty and effort associated to using each language;
- a demographics section collecting data about participant's background and self-assessed technical skills.
The replication package includes answer keys for the multiple-choice comprehension questions.
Data
The experimental data is presented as a comma-separated values (CSV) file consisting of the following columns:
| Column | Definition | Values |
|---|---|---|
| Experiment | experiment identifier | 1, 2 |
| Id | participant identifier, unique within an experiment | positive integers |
| UML | participant's self-assessed UML knowledge | 1, 2, 3, 4, 5 |
| OCL | participant's self-assessed OCL knowledge | 1, 2, 3, 4, 5 |
| MT | participant's self-assessed model transformation knowledge | 1, 2, 3, 4, 5 |
| Programming | participant's self-assessed programming knowledge | 1, 2, 3, 4, 5 |
| Order | the order in which a transformation language was presented to the participant | 1, 2, 3 |
| Language | a transformation language | Epsilon, Henshin, VMTL |
| Score | number of correct comprehension question answers | 0, 1, 2, 3 |
| Time | time in seconds required to complete the comprehension task | positive integers |
| Difficulty | participant's subjective evaluation of the difficulty of completing the comprehension task | 1 – 5 |
| Effort | participant's subjective evaluation of the effort required to complete the comprehension task | 1 – 5 |
Statistical Analysis
The replication package contains three statistical analysis scripts written in the R programming language for statistical computing. The individual scripts contain code for analyzing comprehension results, cognitive load ratings, and task completion times, respectively.
The scripts have a similar structure. They include descriptive statistics computations (mean, standard deviation), data visualizations (box plots, bar plots), and hypothesis tests (Fisher's exact test, ANOVA, the Wilcoxon signed rank test).
vmtl/evaluation.1440345038.txt.gz · Last modified: 2015/08/23 15:50 by rvac
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
