Publications
Overview
You can also find me on Google Scholar.
In prep
-
Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, Dieuwke Hupkes. Lost in inference: rediscovering the role of natural language inference for large language models.
[preprint] -
Aaditya K Singh*, Muhammed Yusuf Kocyigit*, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, Dieuwke Hupkes. Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?
[preprint] -
Kartik Choudhary*, Aman Singh Thakur*, Venkat Srinik Ramayapally*, Sankaran Vaidyanathanand Dieuwke Hupkes. Judging the judges: evaluating alignment and vulnerabilities in LLMs-as-Judges.
[preprint] -
Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang and Dieuwke Hupkes. Quantifying Variance in Evaluation Benchmarks .
[preprint]
2024
-
Llama team. The Llama 3 Herd of Models.
Contribution: pretraining evaluations lead
[paper] -
Xenia Ohmer, Elia Bruni, Dieuwke Hupkes. From form(s) to meaning: probing the semantic depths of language models using multisense consistency. Computational Linguistics
[paper] -
Lucas Weber, Jaap Jumelet, Elia Bruni, Dieuwke Hupkes. Interpretability of Language Models via Task Spaces. ACL 2024
[preprint] -
Lucas Weber, Jaap Jumelet, Paul Michel, Elia Bruni, Dieuwke Hupkes. Curriculum learning with Adam: The devil is in the wrong details.
[preprint]
2023
-
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe et al. State-of-the-art generalisation research in NLP: a taxonomy and review. Nature Machine Intelligence
[paper] [(longer) preprint] [website] -
Kaiser Sun, Adina Williams, Dieuwke Hupkes. A replication study of compositional generalization works on semantic parsing. MLRC 2022. Best paper award
[paper] -
Lucas Weber, Elia Bruni, Dieuwke Hupkes. Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning. CoNLL 2023. Honourable mention
[paper] -
Kaiser Sun, Adina Williams, Dieuwke Hupkes. The validity of evaluation results: assessing concurrence across compositionality benchmarks. CoNLL 2023. Honourable mention
[paper] [code] -
Verna Dankers, Ivan Titov, Dieuwke Hupkes. Memorisation cartography: mapping out the memorisation generalisation continuum in neural machine translation. Accepted at EMNLP 2023
[paper] [demo] -
Xenia Ohmer, Elia Bruni, Dieuwke Hupkes. Evaluating task understanding through multilingual self-consistency: A ChatGPT case study. GEM
[paper] [code] -
Lucas Weber, Elia Bruni, Dieuwke Hupkes. The ICL Consistency test. GenBench 2023 CBT submission
[paper] [code] -
Valentin Taillandiers, Dieuwke Hupkes, Benoit Sagot, Emmanuel Dupoux and Paul Michel. Neural Agents Struggle to Take Turns in Bidirectional Emergent Communication. ICLR2023
[paper] -
Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes and Pascal Vincent. WorldSense: A synthetic benchmark for grounded reasoning in large language models.
[preprint]
2022
-
Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes and Adina Williams. The curious case of absolute position embeddings. EMNLP
[paper] -
Maartje ter Hoeve, Evgeny Kharitonov, Dieuwke Hupkes, Emmanuel Dupoux. Towards Interactive Language Modeling. SPA-NLP workshop @ACL
[paper] -
Daniel Simig, Tianlu Wang, Verna Dankers, Peter Henderson, Khuyagbaatar Batsuren, Dieuwke Hupkes, Mona Diab. Text Characterization Toolkit (TCT). AACL – system demonstrations.
[paper] -
Nicola De Cao, Leon Schmid, Dieuwke Hupkes, Ivan Titov. Sparse Interventions in Language Models with Differentiable Masking. BlackBoxNLP
[paper] -
Yair Lakretz, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene. Causal transformers perform below chance on recursive nested constructions, unlike humans, COLING
[paper]. -
Verna Dankers, Elia Bruni and Dieuwke Hupkes. The paradox of the compositionality of natural language: a neural machine translation case study, ACL.
[paper] -
Eugene Kharitonov, Marco Baroni, Dieuwke Hupkes. How BPE Affects Memorization in Transformers.
[preprint]
2021
-
Verna Dankers, Anna Langedijk, Kate McCurdy, Adina Williams, Dieuwke Hupkes. Generalising to German plural noun classes, from the perspective of a recurrent neural network. CoNLL 2021, Best paper award.
[paper] -
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. EMNLP 2021.
[paper] -
Jaap Jumelet, Milica Denić, Jakub Szymanik, Dieuwke Hupkes, Shane Steinert-Threlkeld. Language models use monotonicity to assess NPI Licensing. ACL findings
[paper] -
Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni and Stanislas Dehaene. Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans. Cognition.
[paper] -
Lucas Weber, Jaap Jumelet, Elia Bruni and Dieuwke Hupkes. Language modelling as a multi-task problem. EACL 2021. [paper]
-
Gautier Dagan, Dieuwke Hupkes and Elia Bruni. Co-evolution of language and agents in referential games. EACL 2021.
[paper] -
Tom Kersten, Hugh Mee Wong, Jumelet Jumelet and Dieuwke Hupkes. Attention vs Non-attention for a Shapley-based Explanation Method. ADeeLIO @ NAACL 2021
[paper]
2020
-
Diana Rodriguez Luna, Edoardo Maria Ponti, Dieuwke Hupkes, and Elia Bruni. Internal and External Pressures on Language Emergence: Least Effort, Object Constancy and Frequency. EMNLP-findings.
[paper] -
Oskar van der Wal, Silvan de Boer, Elia Bruni and Dieuwke Hupkes. The grammar of emergent languages. EMNLP 2020.
[paper] [source code] -
Dieuwke Hupkes, Verna Dankers, Mathijs Mul and Elia Bruni. Compositionality decomposed: how do neural networks generalise? JAIR.
[paper] [source code] [extended abstract (IJCAI)] [ 15 min presentation (IJCAI)] -
Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Location Attention for Extrapolation to Longer Sequences. ACL2020.
[paper]
2019
-
Jaap Jumelet, Willem Zuidema and Dieuwke Hupkes. Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment. CONLL 2019. Honourable mention.
[paper] [source code] -
Joris Baan, Jana Leible, Mitja Nikolaus, David Rau, Dennis Ulmer, Tim Baumgärtner, Dieuwke Hupkes and Elia Bruni. On the Realization of Compositionality in Neural Networks. BlackboxNLP, ACL 2019.
[paper] -
Dennis Ulmer, Dieuwke Hupkes and Elia Bruni Assessing incrementality in sequence-to-sequence models. Repl4NLP, ACL 2019.
[paper] -
Kris Korrel, Dieuwke Hupkes, Verna Dankers and Elia Bruni. Transcoding compositionally: using attention to find more generalizable solutions. BlackboxNLP, ACL 2019.
[paper] -
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene and Marco Baroni. The emergence of number and syntax units in LSTM language models. NAACL 2019.
[paper] [bibtex]. -
Dieuwke Hupkes, Anand Kumar Singh, Kris Korrel, German Kruszewski, and Elia Bruni Learning compositionally through attentive guidance. CICLing 2019.
[paper] [source code] -
Rezka Leonandya, Dieuwke Hupkes, Elia Bruni, Germán Kruszewski. The Fast and the Flexible: training neural networks to learn to follow instructions from small data. IWCS 2019.
[paper]
2018
-
Hupkes D., Veldhoen S., and Zuidema W. (2018). Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. JAIR
[paper] [bibtex] [demo] [extended abstract (IJCAI)] -
Giulianelli, M., Harding, J., Mohnert, F., Hupkes, D. and Zuidema, W. (2018). Under the hood: using diagnostic classifiers to investigate and improve how language models track agreement information.
BlackboxNLP 2018, ACL. Best paper award.
[paper] [bibtex]. -
Zuidema W., Hupkes D., Wiggins G., Scharf C. and Rohrmeier M. (2018). Formal models of Structure Building in Music, Language and Animal Song. In The Origins of Musicality
[chapter] -
Jumelet, J. and Hupkes, D. (2018). Do language models understand anything? On the ability of LSTMs to understand negative polarity items. BlackboxNLP 2018, ACL.
[paper] [bibtex]. -
Hupkes, D., Bouwmeester, S. and Fernández, R. (2018). Analysing the potential of seq2seq models for incremental interpretation in task-oriented dialogue. BlackboxNLP 2018, ACL.
[paper] [bibtex].
2017
- Hupkes D. and Zuidema W. Diagnostic classification and symbolic guidance to understand and improve recurrent neural networks.
Interpreting, Explaining and Visualizing Deep Learning, NIPS2017.
[paper] [poster] [source code]
2016
-
Veldhoen S., Hupkes D. and Zuidema W. (2016). Diagnostic classifiers: revealing how neural networks process hierarchical structure. CoCo, NIPS 2016.
[paper] [bibtex] [poster] [source code] [demo] -
Hupkes D. and Bod R. POS-tagging of Historical Dutch. LREC 2016.
[bibtex] [paper] [source code]
Abstracts
-
Ponti E., Hupkes D. and Bruni E. (2019) The typology of emergent languages. Interaction and the Evolution of Linguistic Complexity
[abstract] -
Ulmer D., Hupkes D. and Bruni E. (2019) An incremental encoder for sequence-to-sequence modelling. CLIN29.
[abstract] -
Lakretz, Y., Kruszewski, G., Hupkes, D., Desbordes, T., Marti, S., Dehaene, S.and Baroni, M., (2018). The representation of syntactic structures in Long-Short Term Memory networks and humans. L2HM.
[poster] -
Zuidema, W., Hupkes, D., Abnar, S. (2018). Interpretable Machine Learning for Predicting Brain Activation in Language Processing. L2HM.
[poster] -
Leonandya, R., Kruszewski, G., Hupkes, D., Bruni, E. (2018). The Fast and the Flexible: pretraining neural networks to learn from small data. L2HM. [poster]
-
Nielsen A., Hupkes D., Kirby S. and Smith K. (2016). The Arbitrariness Of The Sign Revisited: The Role Of Phonological Similarity. EVOLANG11.
[abstract]