This repository contains the replication package for the study titled Utility of Domain Model Completion as described in our paper. The package includes relevant materials, some data that serves as example to understand the format , scripts, and documentation necessary to replicate and analyze the findings.
This repository contains the protocol, task descriptions, anonymized participant data, and analysis scripts used to address the research questions posed in our study. The study focuses on evaluating the impact of modeling assistance on domain model completion through five key research questions that explore productivity, contribution, creativity, participant preferences, and overall user experience.
The study addresses the following research questions:
- Productivity (RQ1): What is the impact of modeling assistance on the time required to complete domain models?
- Contributivity (RQ2): What is the level of contribution of the suggestions to the resulting models?
- Creativity (RQ3): Does the assistance reduce the solution diversity among the modelers?
- Assistance Preference (RQ4): What kind of assistance do participants prefer when completing their tasks?
- Modeling Experience (RQ5): How does the assistance affect the modeling experience, as perceived by the modelers?
The experiments were conducted at the University of Montreal in Canada and the University of Malaga in Spain, involving 30 participants in total (15 per location). The participants completed four domain modeling tasks under different recommendation modes to evaluate the utility of the assistance provided. We used a combination of quantitative and qualitative data collection methods, including questionnaires, log files, and model analysis.
- Locations: University of Montreal, Canada; University of Malaga, Spain.
- Participants: 30 participants, recruited from a diverse pool of undergraduate and graduate students, postdoctoral researchers, professors, and industry experts.
- Language: English.
- Tasks: Four tasks were designed, including three domain modeling tasks with predetermined recommendation modes (Automatic, On-Request, At-End) and one free modeling task where participants could switch between modes.
- Data Collection: Time, acceptance and contribution rates, task completion, and participant feedback were logged for analysis.
The replication package includes the following files and directories:
- Protocol and Questionnaires: All materials related to the study setup, including task descriptions, participant instructions, and questionnaires used during the experiments.
- Solutions: xml files of different domain models designed by participants for different tasks.
- Anonymized Data: The anonymized log files and data collected from some participants, including time-to-completion records and questionnaire responses. PS: We don't provide all the collected data.
- Scripts: Python scripts and Jupyter notebooks used for data processing, analysis, and plotting results.
- Figures: Demographic distributions and experimental results generated during the study.
For questions, comments, or issues, please contact Meriem Ben Chaaben.