Dieuwke Hupkes bio photo

Dieuwke Hupkes

Research scientist at FAIR, ELLIS Amsterdam enthusiast, interested in structures in language

Email Twitter Instagram Github

Curriculum Vitae


Publications

Overview

You can also find me on Google Scholar.

In prep

  • Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes and Pascal Vincent. WorldSense: A synthetic benchmark for grounded reasoning in large language models.
    [preprint]

  • Lucas Weber, Jaap Jumelet, Paul Michel, Elia Bruni, Dieuwke Hupkes. Curriculum learning with Adam: The devil is in the wrong details.
    [preprint]

2023

  • Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe et al. State-of-the-art generalisation research in NLP: a taxonomy and review. Nature Machine Intelligence
    [paper] [(longer) preprint] [website]

  • Kaiser Sun, Adina Williams, Dieuwke Hupkes. A replication study of compositional generalization works on semantic parsing. MLRC 2022. Best paper award
    [paper]

  • Lucas Weber, Elia Bruni, Dieuwke Hupkes. Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning. CoNLL 2023. Honourable mention
    [paper]

  • Kaiser Sun, Adina Williams, Dieuwke Hupkes. The validity of evaluation results: assessing concurrence across compositionality benchmarks. CoNLL 2023. Honourable mention
    [paper] [code]

  • Verna Dankers, Ivan Titov, Dieuwke Hupkes. Memorisation cartography: mapping out the memorisation generalisation continuum in neural machine translation. Accepted at EMNLP 2023
    [paper] [demo]

  • Xenia Ohmer, Elia Bruni, Dieuwke Hupkes. Evaluating task understanding through multilingual self-consistency: A ChatGPT case study. GEM
    [preprint]

  • Lucas Weber, Elia Bruni, Dieuwke Hupkes. The ICL Consistency test. GenBench 2023 CBT submission
    [paper] [code]

  • Valentin Taillandiers, Dieuwke Hupkes, Benoit Sagot, Emmanuel Dupoux and Paul Michel. Neural Agents Struggle to Take Turns in Bidirectional Emergent Communication. ICLR2023
    [paper]

2022

  • Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes and Adina Williams. The curious case of absolute position embeddings. EMNLP
    [paper]

  • Maartje ter Hoeve, Evgeny Kharitonov, Dieuwke Hupkes, Emmanuel Dupoux. Towards Interactive Language Modeling. SPA-NLP workshop @ACL
    [paper]

  • Daniel Simig, Tianlu Wang, Verna Dankers, Peter Henderson, Khuyagbaatar Batsuren, Dieuwke Hupkes, Mona Diab. Text Characterization Toolkit (TCT). AACL – system demonstrations.
    [paper]

  • Nicola De Cao, Leon Schmid, Dieuwke Hupkes, Ivan Titov. Sparse Interventions in Language Models with Differentiable Masking. BlackBoxNLP
    [paper]

  • Yair Lakretz, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene. Causal transformers perform below chance on recursive nested constructions, unlike humans, COLING
    [paper].

  • Verna Dankers, Elia Bruni and Dieuwke Hupkes. The paradox of the compositionality of natural language: a neural machine translation case study, ACL.
    [paper]

  • Eugene Kharitonov, Marco Baroni, Dieuwke Hupkes. How BPE Affects Memorization in Transformers.
    [preprint]

2021

  • Verna Dankers, Anna Langedijk, Kate McCurdy, Adina Williams, Dieuwke Hupkes. Generalising to German plural noun classes, from the perspective of a recurrent neural network. CoNLL 2021, Best paper award.
    [paper]

  • Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. EMNLP 2021.
    [paper]

  • Jaap Jumelet, Milica Denić, Jakub Szymanik, Dieuwke Hupkes, Shane Steinert-Threlkeld. Language models use monotonicity to assess NPI Licensing. ACL findings
    [paper]

  • Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni and Stanislas Dehaene. Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans. Cognition.
    [paper]

  • Lucas Weber, Jaap Jumelet, Elia Bruni and Dieuwke Hupkes. Language modelling as a multi-task problem. EACL 2021. [paper]

  • Gautier Dagan, Dieuwke Hupkes and Elia Bruni. Co-evolution of language and agents in referential games. EACL 2021.
    [paper]

  • Tom Kersten, Hugh Mee Wong, Jumelet Jumelet and Dieuwke Hupkes. Attention vs Non-attention for a Shapley-based Explanation Method. ADeeLIO @ NAACL 2021
    [paper]

2020

  • Diana Rodriguez Luna, Edoardo Maria Ponti, Dieuwke Hupkes, and Elia Bruni. Internal and External Pressures on Language Emergence: Least Effort, Object Constancy and Frequency. EMNLP-findings.
    [paper]

  • Oskar van der Wal, Silvan de Boer, Elia Bruni and Dieuwke Hupkes. The grammar of emergent languages. EMNLP 2020.
    [paper] [source code]

  • Dieuwke Hupkes, Verna Dankers, Mathijs Mul and Elia Bruni. Compositionality decomposed: how do neural networks generalise? JAIR.
    [paper] [source code] [extended abstract (IJCAI)] [ 15 min presentation (IJCAI)]

  • Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Location Attention for Extrapolation to Longer Sequences. ACL2020.
    [paper]

2019

  • Jaap Jumelet, Willem Zuidema and Dieuwke Hupkes. Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment. CONLL 2019. Honourable mention.
    [paper] [source code]

  • Joris Baan, Jana Leible, Mitja Nikolaus, David Rau, Dennis Ulmer, Tim Baumgärtner, Dieuwke Hupkes and Elia Bruni. On the Realization of Compositionality in Neural Networks. BlackboxNLP, ACL 2019.
    [paper]

  • Dennis Ulmer, Dieuwke Hupkes and Elia Bruni Assessing incrementality in sequence-to-sequence models. Repl4NLP, ACL 2019.
    [paper]

  • Kris Korrel, Dieuwke Hupkes, Verna Dankers and Elia Bruni. Transcoding compositionally: using attention to find more generalizable solutions. BlackboxNLP, ACL 2019.
    [paper]

  • Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene and Marco Baroni. The emergence of number and syntax units in LSTM language models. NAACL 2019.
    [paper] [bibtex].

  • Dieuwke Hupkes, Anand Kumar Singh, Kris Korrel, German Kruszewski, and Elia Bruni Learning compositionally through attentive guidance. CICLing 2019.
    [paper] [source code]

  • Rezka Leonandya, Dieuwke Hupkes, Elia Bruni, Germán Kruszewski. The Fast and the Flexible: training neural networks to learn to follow instructions from small data. IWCS 2019.
    [paper]

2018

  • Hupkes D., Veldhoen S., and Zuidema W. (2018). Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure. JAIR
    [paper] [bibtex] [demo] [extended abstract (IJCAI)]

  • Giulianelli, M., Harding, J., Mohnert, F., Hupkes, D. and Zuidema, W. (2018). Under the hood: using diagnostic classifiers to investigate and improve how language models track agreement information.
    BlackboxNLP 2018, ACL. Best paper award.
    [paper] [bibtex].

  • Zuidema W., Hupkes D., Wiggins G., Scharf C. and Rohrmeier M. (2018). Formal models of Structure Building in Music, Language and Animal Song. In The Origins of Musicality
    [chapter]

  • Jumelet, J. and Hupkes, D. (2018). Do language models understand anything? On the ability of LSTMs to understand negative polarity items. BlackboxNLP 2018, ACL.
    [paper] [bibtex].

  • Hupkes, D., Bouwmeester, S. and Fernández, R. (2018). Analysing the potential of seq2seq models for incremental interpretation in task-oriented dialogue. BlackboxNLP 2018, ACL.
    [paper] [bibtex].

2017

  • Hupkes D. and Zuidema W. Diagnostic classification and symbolic guidance to understand and improve recurrent neural networks. Interpreting, Explaining and Visualizing Deep Learning, NIPS2017.
    [paper] [poster] [source code]

2016

  • Veldhoen S., Hupkes D. and Zuidema W. (2016). Diagnostic classifiers: revealing how neural networks process hierarchical structure. CoCo, NIPS 2016.
    [paper] [bibtex] [poster] [source code] [demo]

  • Hupkes D. and Bod R. POS-tagging of Historical Dutch. LREC 2016.
    [bibtex] [paper] [source code]

Abstracts

  • Ponti E., Hupkes D. and Bruni E. (2019) The typology of emergent languages. Interaction and the Evolution of Linguistic Complexity
    [abstract]

  • Ulmer D., Hupkes D. and Bruni E. (2019) An incremental encoder for sequence-to-sequence modelling. CLIN29.
    [abstract]

  • Lakretz, Y., Kruszewski, G., Hupkes, D., Desbordes, T., Marti, S., Dehaene, S.and Baroni, M., (2018). The representation of syntactic structures in Long-Short Term Memory networks and humans. L2HM.
    [poster]

  • Zuidema, W., Hupkes, D., Abnar, S. (2018). Interpretable Machine Learning for Predicting Brain Activation in Language Processing. L2HM.
    [poster]

  • Leonandya, R., Kruszewski, G., Hupkes, D., Bruni, E. (2018). The Fast and the Flexible: pretraining neural networks to learn from small data. L2HM. [poster]

  • Nielsen A., Hupkes D., Kirby S. and Smith K. (2016). The Arbitrariness Of The Sign Revisited: The Role Of Phonological Similarity. EVOLANG11.
    [abstract]