This is the official repository for the research paper "Cognitive abilities and visual complexity impact first impressions in five-second testing". The work addresses the validity of different time intervals for their use in the five-second testing user research method. It further evaluates the impact of user’s cognitive abilities and visual complexity of the web page on received feedback. This repository provides researchers with the dataset collected during the experiment, the experiment stimuli and data analysis scripts. Authors' contact information is also provided.
Published in the journal Behaviour & Information Technology - Open Access
@article{doi:10.1080/0144929X.2023.2272747,
author = {Eduard Kuric, Peter Demcak, Matus Krajcovic and Giang Nguyen},
title = {Cognitive abilities and visual complexity impact first impressions in five-second testing},
journal = {Behaviour \& Information Technology},
volume = {43},
number = {13},
pages = {3209–3236},
year = {2023},
publisher = {Taylor & Francis},
doi = {10.1080/0144929X.2023.2272747},
URL = {https://doi.org/10.1080/0144929X.2023.2272747},
eprint = {https://doi.org/10.1080/0144929X.2023.2272747}
}In our case study, we have collected first impression testing data from participants - responses from initial and final study questionnaires, responses to follow up questions and results from cognitive ability tests. Raw data from the UXtweak Five Second Test tool was exported and further processed (cleaning of irrelevant data, calculation of metrics and aggregations).
Temporary files were created during our multi-stage analysis. The final processed data is contained in two files, one composed of the participant's details (initial and final questionnaire, cognitive tests and related metrics), the other of first impression answers (including metrics and aggregations). The final data files were used in later stages of the analysis to validate the case study's hypotheses.
3 variants of UXtweak Five Second Study were created to collect data during the experiment. Each variant is the same contentwise, only using different exposure times - 2, 5 and 10 seconds. The studies can be accessed below:
- Two-second test - Results, Study preview as participant
- Five-second test - Results, Study preview as participant
- Ten-second test - Results, Study preview as participant
6 screenshots of website visual designs were used in our case study, with one additional screenshot serving as the warmup task.
 TSHIRT |
 GLASSES |
 SPORT |
 WEATHER |
 PET |
 NEWS |
All of the scripts are written using Python (version 3.7.9) and other external libraries installed using pip (version 22.3.1). Scripts were executed using jupyter notebooks. A requirements file is provided for installing dependencies. After installing Python, the below commands can be used in the analysis directory to install the environment and launch the notebooks:
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
jupyter notebook
This will create your virtual environment, install the dependencies and launch the python notebooks with the analysis files. After the installation, to launch the notebooks again, use these commands:
.venv\Scripts\activate
jupyter notebook
1 - Data preparation
Loads the raw dataset exported from UXtweak and prepares its data for further use by other scripts (appropriate naming, formats, data transformation, etc.).
2 - Result labeling
Automatic categorical labeling of some free text follow up questions for later use and preparation for manual UX expert labeling (number of words, use of words relevant to understanding).
3 - Data calculation
Calculation of derived metrics from the raw data, aggregations, lemmatization, assigning participants into groups. Outputs the final participant details and first impression answers datasets used in further analysis. After this step, the collected and calculated data include:
Participant details:
- data from short-term memory test (passed levels)
- data from perceptual speed test (passed levels, mistakes)
- demographic information (age, gender, education, etc.)
- notes from participants
- attention check question answers
- device characteristics (device, os, browser etc.)
- other misc data (date, time taken, etc.)
First impression answers:
- task's respondent
- answers to follow-up questions and their aggregates
- task's information (complexity, design)
4 - Data analysis of respondents and results
Statistical analysis of calculated metrics and respondent's data and related visualizations.
5 - Impact of exposure time, visual complexity and cognitive abilities
Testing the hypothesis on impact of time interval, visual complexity and cognitive abilities on participant's responses.
6 - Relationships of exposure time intervals with visual complexity and cognitive abilities
Analyzing the relationships of time with visual complexity and cognitive abilities. Testing the possibility of prolonging or shortening the exposure time for different stimuli complexities or participants with different cognitive abilities to achieve similar answers.
Email: 5stest.research([AT])gmail.com
Eduard Kuric
He received his PhD degree in computer science from the Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava. He is a researcher and assistant professor at the same university. His research interests include human-computer interaction, user modeling, personalized web-based systems, and machine learning. Eduard is also the head of the UX Research Department and the founder of UXtweak.
- Google Scholar
- Email: eduard.kuric([AT])stuba.sk
Peter Demcak
He received his master’s degree in computer science from the Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava. He is a researcher with background in software engineering, whose current topics of interest involve user behavior, human-computer interaction, UX research methods & design practices, and machine learning. Currently occupies the position of a scientific and user experience researcher at UXtweak, with focus on research that supports work of UX professionals.
- Email: peter.demcak([AT])uxtweak.com
Matus Krajcovic
User experience researcher at UXtweak and computer science student at Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava. Currently focuses on data analysis and research in machine learning use in the field of human-computer interaction.
- Email: matus.krajcovic([AT])uxtweak.com
Giang Nguyen
She is a senior researcher and associate professor at Faculty of Informatics and Information Technologies, Slovak University of Technology in Bratislava. She focuses on machine learning, deep learning, soft computing, and security and reliability. She is also a reviewer and editor for Web of Science journals and member of program committee, editor, organizator for international conferences.
- Google Scholar
- Email: giang.nguyen([AT])stuba.sk
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
