Added paper draft

Author: x-tabdeveloping

paper.bib

@@ -0,0 +1,210 @@
+@misc{s3,
+      title={$S^3$ -- Semantic Signal Separation}, 
+      author={Márton Kardos and Jan Kostkan and Arnault-Quentin Vermillet and Kristoffer Nielbo and Kenneth Enevoldsen and Roberta Rocca},
+      year={2024},
+      eprint={2406.09556},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2406.09556}, 
+}
+
+@misc{keynmf,
+      title={Context is Key(NMF): Modelling Topical Information Dynamics in Chinese Diaspora Media}, 
+      author={Ross Deans Kristensen-McLachlan and Rebecca M. M. Hicke and Márton Kardos and Mette Thunø},
+      year={2024},
+      eprint={2410.12791},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2410.12791}, 
+}
+
+@misc{bertopic_paper,
+      title={BERTopic: Neural topic modeling with a class-based TF-IDF procedure}, 
+      author={Maarten Grootendorst},
+      year={2022},
+      eprint={2203.05794},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+
+@inproceedings{topmost,
+    title = "Towards the {T}op{M}ost: A Topic Modeling System Toolkit",
+    author = "Wu, Xiaobao  and
+      Pan, Fengjun  and
+      Luu, Anh Tuan",
+    editor = "Cao, Yixin  and
+      Feng, Yang  and
+      Xiong, Deyi",
+    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
+    month = aug,
+    year = "2024",
+    address = "Bangkok, Thailand",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2024.acl-demos.4/",
+    doi = "10.18653/v1/2024.acl-demos.4",
+    pages = "31--41",
+    abstract = "Topic models have a rich history with various applications and have recently been reinvigorated by neural topic modeling. However, these numerous topic models adopt totally distinct datasets, implementations, and evaluations. This impedes quick utilization and fair comparisons, and thereby hinders their research progress and applications. To tackle this challenge, we in this paper propose a Topic Modeling System Toolkit (TopMost). Compared to existing toolkits, TopMost stands out by supporting more extensive features. It covers a broader spectrum of topic modeling scenarios with their complete lifecycles, including datasets, preprocessing, models, training, and evaluations. Thanks to its highly cohesive and decoupled modular design, TopMost enables rapid utilization, fair comparisons, and flexible extensions of diverse cutting-edge topic models. Our code, tutorials, and documentation are available at https://github.com/bobxwu/topmost."
+}
+
+@inproceedings{stream,
+    title = "{STREAM}: Simplified Topic Retrieval, Exploration, and Analysis Module",
+    author = {Thielmann, Anton  and
+      Reuter, Arik  and
+      Weisser, Christoph  and
+      Kant, Gillian  and
+      Kumar, Manish  and
+      S{\"a}fken, Benjamin},
+    editor = "Ku, Lun-Wei  and
+      Martins, Andre  and
+      Srikumar, Vivek",
+    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
+    month = aug,
+    year = "2024",
+    address = "Bangkok, Thailand",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2024.acl-short.41/",
+    doi = "10.18653/v1/2024.acl-short.41",
+    pages = "435--444",
+    abstract = "Topic modeling is a widely used technique to analyze large document corpora. With the ever-growing emergence of scientific contributions in the field, non-technical users may often use the simplest available software module, independent of whether there are potentially better models available. We present a Simplified Topic Retrieval, Exploration, and Analysis Module (STREAM) for user-friendly topic modelling and especially subsequent interactive topic visualization and analysis. For better topic analysis, we implement multiple intruder-word based topic evaluation metrics. Additionally, we publicize multiple new datasets that can extend the so far very limited number of publicly available benchmark datasets in topic modeling. We integrate downstream interpretable analysis modules to enable users to easily analyse the created topics in downstream tasks together with additional tabular information.The code is available at the following link: https://github.com/AnFreTh/STREAM"
+}
+
+@inproceedings{ctm,
+    title = "Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence",
+    author = "Bianchi, Federico  and
+      Terragni, Silvia  and
+      Hovy, Dirk",
+    editor = "Zong, Chengqing  and
+      Xia, Fei  and
+      Li, Wenjie  and
+      Navigli, Roberto",
+    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)",
+    month = aug,
+    year = "2021",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.acl-short.96",
+    doi = "10.18653/v1/2021.acl-short.96",
+    pages = "759--766",
+    abstract = "Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the data. However, the resulting word groups are often not coherent, making them harder to interpret. Recently, neural topic models have shown improvements in overall coherence. Concurrently, contextual embeddings have advanced the state of the art of neural models in general. In this paper, we combine contextualized representations with neural topic models. We find that our approach produces more meaningful and coherent topics than traditional bag-of-words topic models and recent neural models. Our results indicate that future improvements in language models will translate into better topic models.",
+}
+
+@inproceedings{zeroshot_tm,
+    title = "Cross-lingual Contextualized Topic Models with Zero-shot Learning",
+    author = "Bianchi, Federico  and
+      Terragni, Silvia  and
+      Hovy, Dirk  and
+      Nozza, Debora  and
+      Fersini, Elisabetta",
+    editor = "Merlo, Paola  and
+      Tiedemann, Jorg  and
+      Tsarfaty, Reut",
+    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume",
+    month = apr,
+    year = "2021",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2021.eacl-main.143",
+    doi = "10.18653/v1/2021.eacl-main.143",
+    pages = "1676--1683",
+    abstract = "Many data sets (e.g., reviews, forums, news, etc.) exist parallelly in multiple languages. They all cover the same content, but the linguistic differences make it impossible to use traditional, bag-of-word-based topic models. Models have to be either single-language or suffer from a huge, but extremely sparse vocabulary. Both issues can be addressed by transfer learning. In this paper, we introduce a zero-shot cross-lingual topic model. Our model learns topics on one language (here, English), and predicts them for unseen documents in different languages (here, Italian, French, German, and Portuguese). We evaluate the quality of the topic predictions for the same document in different languages. Our results show that the transferred topics are coherent and stable across languages, which suggests exciting future research directions.",
+}
+
+@article{blei_prob_topic_models, title={Probabilistic topic models}, volume={55}, url={https://doi.org/10.1145/2133806.2133826}, DOI={10.1145/2133806.2133826}, number={4}, journal={Communications of the ACM}, author={Blei, David M.}, year={2012}, month=apr, pages={77–84} }
+
+@misc{top2vec,
+      title={Top2Vec: Distributed Representations of Topics}, 
+      author={Dimo Angelov},
+      year={2020},
+      eprint={2008.09470},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+
+@inproceedings{prodlda,
+  title={Autoencoding Variational Inference For Topic Models},
+  author={Akash Srivastava and Charles Sutton},
+  booktitle={International Conference on Learning Representations},
+  year={2017},
+  url={https://api.semanticscholar.org/CorpusID:29842525}
+}
+
+@article{scikit-learn,
+ title={Scikit-learn: Machine Learning in {P}ython},
+ author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+ journal={Journal of Machine Learning Research},
+ volume={12},
+ pages={2825--2830},
+ year={2011}
+}
+
+@inproceedings{blei_dynamic,
+author = {Blei, David M. and Lafferty, John D.},
+title = {Dynamic Topic Models},
+year = {2006},
+isbn = {1595933832},
+publisher = {Association for Computing Machinery},
+address = {New York, NY, USA},
+url = {https://doi.org/10.1145/1143844.1143859},
+doi = {10.1145/1143844.1143859},
+abstract = {A family of probabilistic time series models is developed to analyze the time evolution of topics in large document collections. The approach is to use state space models on the natural parameters of the multinomial distributions that represent the topics. Variational approximations based on Kalman filters and nonparametric wavelet regression are developed to carry out approximate posterior inference over the latent topics. In addition to giving quantitative, predictive models of a sequential corpus, dynamic topic models provide a qualitative window into the contents of a large document collection. The models are demonstrated by analyzing the OCR'ed archives of the journal Science from 1880 through 2000.},
+booktitle = {Proceedings of the 23rd International Conference on Machine Learning},
+pages = {113–120},
+numpages = {8},
+location = {Pittsburgh, Pennsylvania, USA},
+series = {ICML '06}
+}
+
+@inproceedings{blei_hierarchical,
+author = {Blei, David M. and Jordan, Michael I. and Griffiths, Thomas L. and Tenenbaum, Joshua B.},
+title = {Hierarchical Topic Models and the Nested Chinese Restaurant Process},
+year = {2003},
+publisher = {MIT Press},
+address = {Cambridge, MA, USA},
+abstract = {We address the problem of learning topic hierarchies from data. The model selection problem in this domain is daunting—which of the large collection of possible trees to use? We take a Bayesian approach, generating an appropriate prior via a distribution on partitions that we refer to as the nested Chinese restaurant process. This nonparametric prior allows arbitrarily large branching factors and readily accommodates growing data collections. We build a hierarchical topic model by combining this prior with a likelihood that is based on a hierarchical variant of latent Dirichlet allocation. We illustrate our approach on simulated data and with an application to the modeling of NIPS abstracts.},
+booktitle = {Proceedings of the 16th International Conference on Neural Information Processing Systems},
+pages = {17–24},
+numpages = {8},
+location = {Whistler, British Columbia, Canada},
+series = {NIPS'03}
+}
+
+@misc{ctm_docs,
+author={Bianchi, Federico and Terragni, Silvia and Hovy, Dirk},
+title={Contextualized Topic Models — Contextualized Topic Models 2.5.0 documentation}, url={https://contextualized-topic-models.readthedocs.io/en/latest/introduction.html}, year={2020} }
+
+@misc{fastopic,
+      title={FASTopic: A Fast, Adaptive, Stable, and Transferable Topic Modeling Paradigm}, 
+      author={Xiaobao Wu and Thong Nguyen and Delvin Ce Zhang and William Yang Wang and Anh Tuan Luu},
+      year={2024},
+      eprint={2405.17978},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2405.17978}, 
+}
+
+@article{sentence_transformers,
+  author       = {Nils Reimers and
+                  Iryna Gurevych},
+  title        = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
+  journal      = {CoRR},
+  volume       = {abs/1908.10084},
+  year         = {2019},
+  url          = {http://arxiv.org/abs/1908.10084},
+  eprinttype    = {arXiv},
+  eprint       = {1908.10084},
+  timestamp    = {Thu, 26 Nov 2020 12:13:54 +0100},
+  biburl       = {https://dblp.org/rec/journals/corr/abs-1908-10084.bib},
+  bibsource    = {dblp computer science bibliography, https://dblp.org}
+}
+
+@software{topicwizard,
+  author = {Kardos, Márton},
+  month = nov,
+  title = {{topicwizard: Pretty and opinionated topic model visualization in Python}},
+  url = {https://github.com/x-tabdeveloping/topic-wizard},
+  version = {0.5.0},
+  year = {2023}
+}

paper.md

@@ -0,0 +1,77 @@
+---
+title: 'Turftopic: Topic Modelling with Contextual Embeddings'
+tags:
+  - Python
+  - topic modelling
+  - sentence-transformers
+  - embeddings
+authors:
+  - name: Márton Kardos
+    orcid: 0000-0001-9652-4498
+    affiliation: 1
+  - name: Kenneth C. Enevoldsen
+    orcid: 0000-0001-8733-0966
+    affiliation: 1
+  - name: Jan Kostkan
+    orcid: 0000-0002-9707-7121
+    affiliation: 1
+  - name: Ross Deans Kristensen-McLachlan
+    orcid: 0000-0001-8714-1911
+    affiliation: "1, 3"
+  - name: Roberta Rocca
+    orcid: 0000-0002-0680-7097
+    affiliation: 2
+affiliations:
+ - name: Center for Humanities Computing, Aarhus University, Denmark
+   index: 1
+ - name: Interactive Minds Center, Aarhus University, Denmark
+   index: 2
+ - name: Department of Linguistics, Cognitive Science, and Semiotics, Aarhus University, Denmark
+   index: 3
+date: 17 March 2025
+bibliography: paper.bib
+---
+
+# Summary
+
+Turftopic is a topic modelling library including a number of recent topic models that go beyond bag-of-words and can understand text in context, utilizing representations from transformers.
+The library focuses on ease-of-use, providing a unified, interface for a number of different modern topic models, and boasting both model-specific and model-agnostic interpretation and visualization utilities.
+The user is afforded great flexibility in model choice and customization, but the library comes with reasonable defaults, not to overwhelm first-time users with a plethora of choices.
+In addition, our library is capable of modeling topics over time, modeling topics in streams of texts, finding hierarchical themes, and multilingual usage.
+For ease of interpretation, users can utilize large language models to assign human-readable names to topics, and can model key phrases, lemmas or stems right out of the box.
+
+![An Overview of Turftopic's Functionality](assets/paper_banner.png)
+
+# Statement of need
+
+While a number of software packages have been developed for contextual topic modelling in recent years, including BERTopic [@bertopic_paper], Top2Vec [@top2vec], CTM [@ctm], these packages include implementations of one or two topic models, and most of the utilities they provide are model-specific. This has resulted in the unfortunate situation that practitioners need to switch between different topic modelling libraries if they intend to try different models for their use case, and adapt to their particularities in both interface and functionality.
+Some attempts have been made at creating unified packages for modern topic models, including STREAM [@stream] and TopMost [@topmost]. These packages, however have a focus on neural models and topic model evaluation, have abstract and highly specialized interfaces, and do not include all popular topic models.
+Additionally, we considered the interpretation utilities included in these libraries fairly limited.
+
+Turftopic unifies state-of-the-art contextual topic models under a superset of the scikit-learn [@scikit-learn] API, which users are likely already familiar with, and can be readily included in scikit-learn workflows and pipelines.
+We focused on making Turftopic first and foremost an easy-to-use library, that does not necessitate expert knowledge or excessive amounts of code to get started with, but gives great flexibility to power users.
+The library also includes three topic models, which to our knowledge only have implementations in Turftopic, these are: KeyNMF [@keynmf], S^3^ [@s3] and GMM.
+
+# Functionality
+
+Turftopic includes a wide array of contextual topic models from the literature, these include:
+FASTopic [@fastopic], Clustering models, such as BERTopic [@bertopic_paper] and Top2Vec [@top2vec], auto-encoding topic models, like CombinedTM [@ctm] and ZeroShotTM [@zeroshot_tm], KeyNMF [@keynmf], Semantic Signal Separation [@s3] and GMM.
+We believe these models to be representative of the state of the art in contextual topic modelling and intend to expand on them in the future.
+
+![Components of a Topic Modelling Pipeline in Turftopic](https://x-tabdeveloping.github.io/turftopic/images/topic_modeling_pipeline.png){width="800px"}
+
+Each model in Turftopic has an *encoder* component, which is used for producing continuous document-representations, and a *vectorizer* component, which extracts term counts in each documents, thereby dictating which terms will be considered in topics.
+The user has full control over what components should be used at different stages of the topic modelling process, thereby having fine-grained influence on the nature and quality of topics.
+
+The library comes loaded with a lot of utilities to help users interpret their results, including *pretty printing* utilities for exploring topics, *interactive visualizations* partially powered by the `topicwizard` [@topicwizard] Python package, and *automated topic naming* with LLMs.
+
+To accommodate a variety of use-cases, Turftopic can be used for dynamic topic modelling, where we expect topics to change over time, can be used for uncovering hierarchical structure in topics.
+Some models can also be fitted in an *online* fashion, where documents are accounted for as they come in by batches.
+Turftopic also includes *seeded* topic modelling, where a seed phrase can be used to retrieve topics relevant to the specific research question.
+
+# Target Audience
+
+Topic models can be utilized in a number of research settings, including exploratory data analysis, discourse analysis of many domains, such as newspapers, social media or policy documents.
+Turftopic has already been utilized by @keynmf for analyzing information dynamics in Chinese Diaspora Media, and is currently being used in multiple ongoing research projects, including one analyzing discourse on the HPV vaccine in Denmark.
+In addition, the library has already seen extensive use in student exam papers at Aarhus University.
+We expect that computational researchers, especially in digital humanities and in social sciences will benefit from an easy-to-use, unified contextual topic modelling package.