|
| 1 | +--- |
| 2 | +layout: page |
| 3 | +title: MM-ArgFallacy2025 |
| 4 | +permalink: /mm-argfallacy/2025/ |
| 5 | +description: |
| 6 | +nav: false |
| 7 | +#nav_order: 8 |
| 8 | +related_publications: false |
| 9 | +--- |
| 10 | +<!-- pages/parkinsons-speech-xai.md --> |
| 11 | + |
| 12 | + |
| 13 | +Multimodal Argumentative Fallacy Detection and Classification on Political Debates Shared Task. |
| 14 | + |
| 15 | +Co-located with The [12th Workshop on Argument Mining](https://argmining-org.github.io/2025/) in Vienna, Austria. |
| 16 | + |
| 17 | + |
| 18 | +# Overview |
| 19 | +This shared task focuses on detecting and classifying fallacies in **political debates** by integrating text and audio data. Participants will tackle two sub-tasks: |
| 20 | +- **Argumentative Fallacy Detection** |
| 21 | +- **Argumentative Fallacy Classification** |
| 22 | + |
| 23 | +We offer three input settings: |
| 24 | +- **Text-only:** Analyze textual arguments. |
| 25 | +- **Audio-only:** Explore paralinguistic features. |
| 26 | +- **Text + Audio:** Combine both for a multimodal perspective. |
| 27 | + |
| 28 | +Join us to advance multimodal argument mining and uncover new insights into human reasoning! 💬 |
| 29 | + |
| 30 | + |
| 31 | +# Tasks |
| 32 | + |
| 33 | +**Task A** |
| 34 | + |
| 35 | +- **Input**: a sentence, in the form of text or audio or both, extracted from a political debate. |
| 36 | +- **Task**: to determine whether the input contains an argumentative fallacy. |
| 37 | + |
| 38 | +**Task B** |
| 39 | +- **Input**: a sentence, in the form of text or audio or both, extracted from a political debate, containing a fallacy. |
| 40 | +- **Task**: to determine the type of fallacy contained in the input, according to the classification introduced by [Goffredo et al. (2022)](https://www.ijcai.org/proceedings/2022/575). We only refer to macro categories. |
| 41 | + |
| 42 | +----------------------------------- |
| 43 | + |
| 44 | +For each sub-task, participants can leverage the debate context of a given input: all its previous sentences and corresponding aligned audio samples. For instance, consider the **text-only** input mode. Given a sentence from a political debate at index *i*, participants can use sentences with indexes from *0* to *i - 1*, where *0* denotes the first sentence in the debate. |
| 45 | + |
| 46 | +------------------------------------ |
| 47 | + |
| 48 | + |
| 49 | +# Data |
| 50 | + |
| 51 | + |
| 52 | +We use **MM-USED-fallacy** and release a version of the dataset specifically designed for argumentative fallacy detection. This dataset includes 1,891 sentences from [Haddadan et al.'s (2019)](https://aclanthology.org/P19-1463.pdf) dataset on US presidential elections. Each sentence is labeled with one of six argumentative fallacy categories, as introduced by [Goffredo et al. (2022)](https://www.ijcai.org/proceedings/2022/575). |
| 53 | + |
| 54 | +Inspired by observations from [Goffredo et al. (2022)](https://www.ijcai.org/proceedings/2022/575) on the benefits of leveraging multiple argument mining tasks for fallacy detection and classification, we also provide additional datasets to encourage multi-task learning. A summary is provided in the table below: |
| 55 | + |
| 56 | +For argumentative fallacy detection, we will compute the binary F1-score on predicted sentence-level labels. |
| 57 | +For argumentative fallacy classification, we will compute the macro F1-score on predicted sentence-level labels. |
| 58 | +Metrics will be computed on the hidden test set to determine the best system for each sub-task and input mode. |
| 59 | + |
| 60 | +--- |
| 61 | + |
| 62 | +| **Dataset** | **Description** | **Size** | |
| 63 | +|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------| |
| 64 | +| **UKDebates** | 386 sentences and audio samples from the 2015 UK Prime Ministerial elections. Sentences are labeled for argumentative sentence detection: containing or not containing a claim. | 386 sentences | |
| 65 | +| **M-Arg** | A multimodal dataset for argumentative relation classification from the 2020 US Presidential elections. Sentences are labeled as attacking, supporting, or unrelated to another sentence. | 4,104 pairs | |
| 66 | +| **MM-USED** | A multimodal extension of the USElecDeb60to16 dataset, covering US presidential debates (1960–2016). Includes labels for argumentative sentence detection and component classification. | 26,781 sentences | |
| 67 | + |
| 68 | +--- |
| 69 | + |
| 70 | +All datasets will be available through [MAMKit](https://nlp-unibo.github.io/mamkit/). |
| 71 | + |
| 72 | +Since many multimodal datasets cannot release audio samples due to copyright restrictions, MAMKit provides an interface to dynamically build datasets and promote reproducible research. |
| 73 | + |
| 74 | +Datasets are formatted as `torch.Dataset` objects, containing input values (text, audio, or both) and corresponding task-specific labels. More details about data formats and dataset building are available in MAMKit's documentation. |
| 75 | + |
| 76 | +# Evaluation |
| 77 | +For argumentative fallacy detection, we will compute the binary F1-score on predicted sentence-level labels. |
| 78 | +For argumentative fallacy classification, we will compute the macro F1-score on predicted sentence-level labels. |
| 79 | +Metrics will be computed on the hidden test set to determine the best system for each sub-task and input mode. |
| 80 | + |
| 81 | +# Key Dates (Anywhere on Earth) |
| 82 | +Will be updated soon. |
| 83 | + |
| 84 | +# Submission |
| 85 | +We be updated soon. |
| 86 | + |
| 87 | + |
| 88 | + |
| 89 | +# Task Organizers |
| 90 | + |
| 91 | +<div class="row row-cols-2 projects pt-3 pb-3"> |
| 92 | + {% include people_horizontal.html name="Eleonora Mancini" affiliation="Language Technologies Lab, University of Bologna, Italy" url="https://helemanc.github.io/" img="/assets/img/people/eleonora_mancini.jpeg" %} |
| 93 | + {% include people_horizontal.html name="Federico Ruggeri" affiliation="Language Technologies Lab, University of Bologna, Italy" url="https://www.unibo.it/sitoweb/federico.ruggeri6" img="/assets/img/people/fede.png" %} |
| 94 | + {% include people_horizontal.html name="Paolo Torroni" affiliation="Language Technologies Lab, University of Bologna, Italy" url="https://www.unibo.it/sitoweb/p.torroni/en" img="/assets/img/people/paolo.png" %} |
| 95 | + {% include people_horizontal.html name="Serena Villata" affiliation="Inria-I3S WIMMICS Laboratoire I3S, CNRS, Sophia Antipolis, France" url="https://webusers.i3s.unice.fr/~villata/Home.html" img="assets/img/people/serena_villata.jpg" %} |
| 96 | +</div> |
| 97 | + |
| 98 | + |
| 99 | + |
| 100 | +# Contacts |
| 101 | +**[Join the MM-ArgFallacy2025 Slack Channel!](https://join.slack.com/t/mm-argfallacy2025/shared_invite/zt-2yjct5udc-vbuGSsSelR5FMiopSne~wQ)** |
0 commit comments