Human validation of automated media-bias detection in news paragraphs

We would like to invite you to take part, as a volunteer, in a research study on the automatic detection of media bias in news text, conducted jointly by São Paulo State University (UNESP) and Emergent Methods.

We have developed a language model (BiasExpert) that detects specific kinds of bias in news coverage, for example loaded word choice, sensational language, or an opinion presented as established fact. The model covers 18 kinds of bias in total; this study focuses on eight of them, the ones that can be judged from a short paragraph on its own. The data used to build the model was labeled by a consensus of several language models rather than by hand. This study collects human judgments on short news paragraphs, so that we can understand how the model’s labels compare with the way people read the same text.

You will read short news paragraphs, shown on their own, and answer focused questions about the kinds of bias they may contain. The source and the headline of each article are hidden on purpose, so that your judgment is based only on the text in front of you.

To understand how often human readers agree with the model’s bias labels, for each type of bias, and to identify the types where human perception and the model diverge.

From a theoretical standpoint, the study contributes to the evaluation of automated media-bias detection, an area where human reference data is scarce. From an applied standpoint, the collected judgments form a dataset that can support the development of more transparent and better-calibrated tools for analysing news content.

Before starting, you will be asked for some basic profile information: your name, email address, country or region, your professional relationship to journalism (for example professional journalist, journalism student, researcher, or regular news reader), how often you read the news, and how comfortable you are reading in English. This lets us compare responses between groups of readers.

The study itself has two parts:

1. Practice. One news paragraph in the same format as the study (two questions, yes, no, or not sure), so that you know what to expect. Practice answers are not scored and are not part of your results.
2. Bias judgments. Six news paragraphs. Before each one, a short reminder introduces the two kinds of bias you will be asked about, with examples. For each paragraph you answer two questions of the form “does this paragraph show X?”, with the options yes, no, and not sure. Answers to this part are not revealed during the study.

At the end you are shown a single summary figure: how often the model’s labels matched your answers. You may then optionally review each paragraph together with the model’s reasoning and say whether you agree with it, answer one further set of six paragraphs, and leave a free-text comment about your experience. The whole session takes approximately 10 to 15 minutes.

Alongside your answers, the platform records how long you spend on each screen, including periods of inactivity and whether you switch away from the browser tab. This is used only to assess the quality and attentiveness of responses.

5.1 Expected risks

The risks are minimal and comparable to those of everyday activities such as reading on a computer. You may experience mild fatigue or eye strain from reading on screen; the session is deliberately short to limit this. The paragraphs are excerpts from real published news articles and may therefore refer to subjects such as crime, politics, or conflict, which some people may find unpleasant. You are free to end your participation at any moment.

5.2 Benefits for participants

You will not pay, and will not be paid, to take part. Your voluntary participation will nevertheless contribute to research on media bias and on the evaluation of language models applied to journalism.

6.1 Inclusion

Any person aged eighteen years or older, with internet access, who is able to read news text in English, may take part.

6.2 Exclusion

People under the age of eighteen may not take part.

We clarify that your participation is entirely voluntary: you may decline to take part, or withdraw at any moment, without giving a reason and without any penalty or disadvantage to you.

The data you provide is stored in a private research database and is used solely for academic and scientific purposes. No personal participant data will be published. Any results we report, whether in scientific articles, presentations, or public communication, are aggregated and anonymous, and do not allow individual participants to be identified. The researchers will treat your identity according to professional standards of confidentiality.

Any questions regarding this research may be addressed to the researchers:

• Wagner Costa Santos (UNESP / Emergent Methods) · wagner@emergentmethods.ai
• Arnaldo Candido Junior (UNESP) · arnaldo.candido@unesp.br

Having read this document and had the opportunity to clarify my questions, I believe I am sufficiently informed. It is clear to me that my participation is voluntary and that I may withdraw this consent at any moment, without penalty or loss of any benefit. I am also aware of the objectives of the research, of the procedures I will be subjected to, of the possible risks arising from them, and of the guarantee of confidentiality.

Given the above, I express my agreement, of my own free will, to take part in this study. This free and informed consent form is accepted by ticking the consent box before the study begins.