About Us
TakeLab is an academic research group at the Faculty of Electrical Engineering and Computing in Zagreb, Croatia, focused on advancing artificial intelligence, machine learning, and natural language processing (NLP). Our work centers on large language models (LLMs), with a commitment to refining methods for language comprehension and analyzing complex, unstructured data.
- Advancing LLM research, with a focus on enhancing their generalization, robustness, and interpretability.
- Creating representation learning techniques to improve semantic and contextual understanding in computational systems.
- Exploring computational social science, using data-driven methods to study social interactions and societal trends.
Our research focuses on multiple aspects of representation learning, seeking a deeper understanding of the internal workings of LLMs. We also engage in interdisciplinary work within computational social science, utilizing NLP tools to analyze large datasets that reveal insights into human behavior, communication patterns, and evolving societal trends.
Latest Research
Explore our recent research studies. Select a publication to read more about it.
Disentangling Latent Shifts of In-Context Learning Through Self-Training
arXiv preprint
In-context learning (ICL) has become essential in natural language processing, particularly with autoregressive large language models capable of learning from demonstrations provided within the prompt. However, ICL faces challenges with stability and long contexts, especially as the number of demonstrations grows, leading to poor generalization and inefficient inference. To address these issues, we introduce STICL (Self-Training ICL), an approach that disentangles the latent shifts of demonstrations from the latent shift of the query through self-training. STICL employs a teacher model to generate pseudo-labels and trains a student model using these labels, encoded in an adapter module. The student model exhibits weak-to-strong generalization, progressively refining its predictions over time. Our empirical results show that STICL improves generalization and stability, consistently outperforming traditional ICL methods and other disentangling strategies across both in-domain and out-of-domain data.
Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings
In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5061–5074, Singapore. Association for Computational Linguistics.
Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.
Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysis
In Proceedings of the 2023 CLASP Conference on Learning with Small Data (LSD), pages 11–24, Gothenburg, Sweden. Association for Computational Linguistics.
Developed to alleviate prohibitive labeling costs, active learning (AL) methods aim to reduce label complexity in supervised learning. While recent work has demonstrated the benefit of using AL in combination with large pre-trained language models (PLMs), it has often overlooked the practical challenges that hinder the effectiveness of AL. We address these challenges by leveraging representation smoothness analysis to ensure AL is feasible, that is, both effective and practicable. Firstly, we propose an early stopping technique that does not require a validation set – often unavailable in realistic AL conditions – and observe significant improvements over random sampling across multiple datasets and AL methods. Further, we find that task adaptation improves AL, whereas standard short fine-tuning in AL does not provide improvements over random sampling. Our work demonstrates the usefulness of representation smoothness analysis for AL and introduces an AL stopping criterion that reduces label complexity.
Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods
In Findings of the Association for Computational Linguistics: ACL 2023, pages 9147–9162, Toronto, Canada. Association for Computational Linguistics.
A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. A common practice for evaluating whether an interpretability method is faithful has been to use evaluation-by-agreement – if multiple methods agree on an explanation, its credibility increases. However, recent work has found that saliency methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods. In our work, we demonstrate that rank correlation is not a good fit for evaluating agreement and argue that Pearson-r is a better-suited alternative. We further show that regularization techniques that increase faithfulness of attention explanations also increase agreement between saliency methods. By connecting our findings to instance categories based on training dynamics, we show that the agreement of saliency method explanations is very low for easy-to-learn instances. Finally, we connect the improvement in agreement across instance categories to local representation space statistics of instances, paving the way for work on analyzing which intrinsic model properties improve their predisposition to interpretability methods.
ALANNO: An Active Learning Annotation System for Mortals
In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 228–235, Dubrovnik, Croatia. Association for Computational Linguistics.
Supervised machine learning has become the cornerstone of today’s data-driven society, increasing the need for labeled data. However, the process of acquiring labels is often expensive and tedious. One possible remedy is to use active learning (AL) – a special family of machine learning algorithms designed to reduce labeling costs. Although AL has been successful in practice, a number of practical challenges hinder its effectiveness and are often overlooked in existing AL annotation tools. To address these challenges, we developed ALANNO, an open-source annotation system for NLP tasks equipped with features to make AL effective in real-world annotation projects. ALANNO facilitates annotation management in a multi-annotator setup and supports a variety of AL methods and underlying models, which are easily configurable and extensible.
Projects
Explore our projects.
Retriever
TakeLab Retriever is a platform that scans articles and their metadata from Croatian news outlets and does text mining in real-time.
Alanno
We created a powerful annotation platform powered by active learning and designed to support a wide range of machine learning and deep learning models.
PsyTxt
With this project, we aim to set the ground for a truly interdisciplinary perspective on computational personality research by developing datasets and models for personality prediction and analysis based on online textual interactions.
Teaching
We take great pride and care in teaching the things we're good at and that inspire us. We design our courses mainly around the key topics in artificial intelligence, machine learning, NLP and IR, the topics that we deem relevant for our students' career success and professional development. Here's a list of courses we currently offer at the Faculty of Electrical Engineering and Computing, University of Zagreb.
Intro to AI
An introductory course covering fundamental concepts and techniques in artificial intelligence.
Machine Learning 1
A foundational course in machine learning, focused on key algorithms and exploring their underlying mechanisms.
Text Analysis and Retrieval
Examines modern approaches to text analysis and retrieval, grounded in fundamental principles.
Selected Topics in Natural Language Processing
Advanced topics in natural language processing, covering current research and applications.
News
Stay up to date with the latest news and updates from TakeLab.
Team
Get to know the people behind the work.
Ana Barić
CaporegimeExplicabo voluptatem mollitia et repellat qui dolorum quasi
David Dukić
CaporegimeExplicabo voluptatem mollitia et repellat qui dolorum quasi
Iva Vukojević
CaporegimeExplicabo voluptatem mollitia et repellat qui dolorum quasi
Jan Šnajder
DonExplicabo voluptatem mollitia et repellat qui dolorum quasi
Josip Jukić
CaporegimeIt’s no coincidence a 90° angle is called the right one.
Laura Majer
CaporegimeExplicabo voluptatem mollitia et repellat qui dolorum quasi
Martin Tutek
SottocapoExplicabo voluptatem mollitia et repellat qui dolorum quasi
Matej Gjurković
ConsigliereExplicabo voluptatem mollitia et repellat qui dolorum quasi