The learnability of VMTL has been experimentally evaluated and compared to that of the Henshin and Epsilon model transformation languages. As of February 2016, a detailed discussion of this evaluation's results is under review for publication. This page will be updated to include it as soon as possible. In the interest of reviewers and of those looking to replicate our experiments or verify our analysis methods, we provide a complete replication package.

The replication package can be downloaded as a ZIP archive here. It consists of the experimental material used for each experiment, the collected and encoded data, and statistical analysis scripts for analyzing the data. The experimental material and data are published under the Creative Commons Attribution 4.0 International License, and the statistical analysis scripts are published under the MIT License.

We have conducted our evaluation in the form of two questionnaire-based experiments we refer to as Experiment 1 and Experiment 2. The replication package includes three PDF documents for each experiment, one for every applied treatment. The treatments present questions regarding the evaluated transformation languages in different orders, as a mitigation measure against learning effects. This measure is particularly relevant in the case of Experiment 1, where the same questions are asked for each transformation language. Participants were randomly assigned to one of the treatments.

The questionnaires consist of:

• a comprehension task requiring participants to select the correct descriptions of several transformation specifications from a list of options;
• a cognitive load assessment task requiring participants to rate the difficulty and effort associated to using each language;
• a demographics section collecting data about participant's background and self-assessed technical skills.

The replication package includes answer keys for the multiple-choice comprehension questions.

The experimental data is presented as a comma-separated values (CSV) file consisting of the following columns:

The replication package contains three statistical analysis scripts written in the R programming language for statistical computing. The individual scripts contain code for analyzing comprehension results, cognitive load ratings, and task completion times, respectively.

The scripts have a similar structure. They include descriptive statistics computations (mean, standard deviation), data visualizations (box plots, bar plots), and hypothesis tests (Fisher's exact test, ANOVA, the Wilcoxon signed rank test).

The learnability of VMQL has also been experimentally evaluated and compared to that of the Object Constraint Language for the task of querying business process models expressed using the Business Process Model and Notation (BPMN). The results of this user experiment indicate that VMQL surpasses OCL in terms of the evaluated task metrics: query comprehension and query production. VMQL also imposes a lower cognitive load on its users. The full results of this experiment are described in the paper Querying business process models with VMQL by Harald Störrle and Vlad Acretoaie.

The replication package can be downloaded as a ZIP archive here. It consists of the experimental material and the collected and encoded data. The replication package is published under the Creative Commons Attribution 4.0 International License.

We have conducted our evaluation in the form of a questionnaire-based experiment. The replication package includes two PDF documents, one for every questionnaire version. The two questionnaire versions present questions regarding the evaluated query languages in different orders, as a mitigation measure against learning effects. Participants were randomly assigned to one of the questionnaire versions.

The questionnaires consist of:

• a comprehension task requiring participants to select the correct plain English descriptions of eight query specifications from a list of options;
• a production task requiring participants to express four queries described in plain English using VMQL and OCL;
• a cognitive load assessment task requiring participants to rate the difficulty and effort associated to using VMQL and OCL;
• a demographics section collecting data about participant's background and self-assessed technical skills.

The replication package includes answer keys for the multiple-choice comprehension questions.

The experimental data is presented as a comma-separated values (CSV) file consisting of the following columns: