README.md

LLMTestSubjects

Series of experiments to evaluate whether LLMs can be used to replace participants in linguistic experiments.

Pronomina

Based on the two papers (Patterson and Schumacher 2021 and Patterson et al. 2022), we analyse whether LLMs

• could make similar judgements about the acceptability of pronoun references as human subjects.
• prefer the same R-expressions for pronouns as human subjects.
• can be used reliably in anaphora resolution.

Step 1: Collect data from experiments, generate lists for participants/llms

The data of the experiments from the two papers is available online, references can be found on the linked pages. We received the combined data from the authors in a file that we pre-processed with the script 01_CreateParticipantsList.R

The code generates two data frames (ExpADate, ExpBDate) that we use to generate our prompts for the large language models (folder ExperimentParticipantsLists) and exports them to RDS files. These files allow us (and you) to feed the LLMs and use our code (02/03 A/B) to analyse our/your results without having to rely on the original data.

Step 2: Use LLMs as simulated participants

We used the generated participants lists to generate the prompts for the various LLMs (that we used as simulated participants).

The notation for LLaMA-based models is as follows:

ModelAbbrevation_Parameters_Temperature_Experiment

The OpenAI GPT notation is similar, but without the Parameters.

The following models were used:

Abbreviation	Parameters	Link to model
EGLM	7B	EM German Leo Mistral
EMG	70B	EM German 70b v01
SKLM	7B	SauerkrautLM Her0
GPT4	NA	OpenAI GPT4 Turbo

Step 3: Collect answers from LLMs

See 02A_CollectCompletionAnswers.R and 02B_CollectRatingAnswers.R

Step 4: Analyse behaviour of LLMs compared to behaviour of participants of the original experiments

See 02A_AnalyseCompletionAnswers.R and 03B_AnalyseRatingAnswers.R