VMTL
VMQL
VMCL
Sidebar
vmtl:evaluation
This is an old revision of the document!
Table of Contents
Empirical Evaluation
The learnability of VMTL has been experimentally evaluated and compared to that of the Henshin and Epsilon model transformation languages. As of August 2015, a detailed discussion of this evaluation's results is under review for publication. This page will be updated to include it as soon as possible. In the interest of reviewers and of those looking to replicate our experiments or verify our analysis methods, we provide a complete replication package.
Replication Package
The replication package can be downloaded as a ZIP archive here.
It consists of the experimental material used for each experiment, the collected and encoded data, and statistical analysis scripts for analyzing the data. The experimental material and data are published under the Creative Commons Attribution 4.0 International License, and the statistical analysis scripts are published under the MIT License.
Experimental Material
We have conducted our evaluation in the form of two questionnaire-based experiments we refer to as Experiment 1 and Experiment 2. The replication package includes three PDF documents for each experiment, one for every applied treatment. The treatments present questions regarding the evaluated transformation languages in different orders, as a mitigation measure against learning effects. This measure is particularly relevant in the case of Experiment 1, where the same questions are asked for each transformation language. Participants were randomly assigned to one of the treatments.
The questionnaires consist of:
- a comprehension task requiring participants to select the correct descriptions of several transformation specifications from a list of options;
- a cognitive load assessment task requiring participants to rate the difficulty and effort associated to using each language;
- a demographics section collecting data about participant's background and self-assessed technical skills.
The replication package includes answer keys for the multiple-choice comprehension questions.
Data
The experimental data is presented as a comma-separated values (CSV) file consisting of the following columns:
Statistical Analysis
vmtl/evaluation.1440341281.txt.gz · Last modified: 2015/08/23 14:48 by rvac
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
