Recent advances in large language models have demonstrated promising capabilities in following simple instructions through instruction tuning. However, real-world tasks often involve complex, multi-step instructions that remain challenging for current NLP systems. Despite growing interest in this area, there lacks a comprehensive survey that systematically analyzes the landscape of complex instruction understanding and processing. Through a systematic review of the literature, we analyze available resources, representation schemes, and downstream tasks related to instructional text. Our study examines 177 papers, identifying trends, challenges, and opportunities in this emerging field. We provide AI/NLP researchers with essential background knowledge and a unified view of various approaches to complex instruction understanding, bridging gaps between different research directions and highlighting future research opportunities.
@misc{safa2024systematicsurveyinstructionaltext,title={A Systematic Survey on Instructional Text: From Representation Formats to Downstream NLP Tasks},author={Safa, Abdulfattah and Kapanadze, Tamta and Uzunoğlu, Arda and Şahin, Gözde Gül},year={2024},eprint={2410.18529},archiveprefix={arXiv},primaryclass={cs.CL},}
Sophisticated grammatical error detection/correction tools are available for a small set of languages such as English and Chinese. However, it is not straightforward – if not impossible – to adapt them to morphologically rich languages with complex writing rules like Turkish which has more than 80 million speakers. Even though several tools exist for Turkish, they primarily focus on spelling errors rather than grammatical errors and lack features such as web interfaces, error explanations and feedback mechanisms. To fill this gap, we introduce GECTurk WEB, a light, open-source, and flexible web-based system that can detect and correct the most common forms of Turkish writing errors, such as the misuse of diacritics, compound and foreign words, pronouns, light verbs along with spelling mistakes. Our system provides native speakers and second language learners an easily accessible tool to detect/correct such mistakes and also to learn from their mistakes by showing the explanation for the violated rule(s). The proposed system achieves 88,3 system usability score, and is shown to help learn/remember a grammatical rule (confirmed by 80% of the participants).
@misc{gebeşçe2024gecturkwebexplainableonline,title={GECTurk WEB: An Explainable Online Platform for Turkish Grammatical Error Detection and Correction},author={Gebeşçe, Ali and Şahin, Gözde Gül},year={2024},eprint={2410.12350},archiveprefix={arXiv},primaryclass={cs.CL},}
Multilingual language models often perform unevenly across different languages due to limited generalization capabilities for some languages. This issue is significant because of the growing interest in making universal language models that work well for all languages. Instruction tuning with multilingual instruction-response pairs has been used to improve model performance across various languages. However, this approach is challenged by high computational costs, a lack of quality tuning data for all languages, and the "curse of multilinguality" – the performance drop per language after adding many languages. Recent studies have found that working with datasets with few languages and a smaller number of instances can be beneficial. Yet, there exists no systematic investigation into how choosing different languages affects multilingual instruction tuning. Our study proposes a method to select languages for instruction tuning in a linguistically informed way, aiming to boost model performance across languages and tasks. We use a simple algorithm to choose diverse languages and test their effectiveness on various benchmarks and open-ended questions. Our results show that this careful selection generally leads to better outcomes than choosing languages at random. We suggest a new and simple way of enhancing multilingual models by selecting diverse languages based on linguistic features that could help develop better multilingual systems and guide dataset creation efforts.
@misc{soykan2024linguisticallyinformedmultilingualinstructiontuning,title={Linguistically-Informed Multilingual Instruction Tuning: Is There an Optimal Set of Languages to Tune?},author={Soykan, Gürkan and Şahin, Gözde Gül},year={2024},eprint={2410.07809},archiveprefix={arXiv},primaryclass={cs.CL},}
Dialogue State Tracking (DST) is crucial for understanding user needs and executing appropriate system actions in task-oriented dialogues. Majority of existing DST methods are designed to work within predefined ontologies and assume the availability of gold domain labels, struggling with adapting to new slots values. While Large Language Models (LLMs)-based systems show promising zero-shot DST performance, they either require extensive computational resources or they underperform existing fully-trained systems, limiting their practicality. To address these limitations, we propose a zero-shot, open-vocabulary system that integrates domain classification and DST in a single pipeline. Our approach includes reformulating DST as a question-answering task for less capable models and employing self-refining prompts for more adaptable ones. Our system does not rely on fixed slot values defined in the ontology allowing the system to adapt dynamically. We compare our approach with existing SOTA, and show that it provides up to 20% better Joint Goal Accuracy (JGA) over previous methods on datasets like Multi-WOZ 2.1, with up to 90% fewer requests to the LLM API.
@misc{safa2024zeroshotopenvocabularypipelinedialogue,title={A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding},author={Safa, Abdulfattah and Şahin, Gözde Gül},year={2024},eprint={2409.15861},archiveprefix={arXiv},primaryclass={cs.CL},}
Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with this https URL.
@misc{uzunoglu2024paradise,title={PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset},author={Uzunoglu, Arda and Safa, Abdulfattah and Şahin, Gözde Gül},month=mar,year={2024},eprint={2403.03167},archiveprefix={arXiv},primaryclass={cs.CL}}
Predicting the collaboration likelihood and measuring cognitive trust to AI systems is more important than ever. To do that, previous research mostly focus solely on the model features (e.g., accuracy, confidence) and ignore the human factor. To address that, we propose several decision-making similarity measures based on divergence metrics (e.g., KL, JSD) calculated over the labels acquired from humans and a wide range of models. We conduct a user study on a textual entailment task, where the users are provided with soft labels from various models and asked to pick the closest option to them. The users are then shown the similarities/differences to their most similar model and are surveyed for their likelihood of collaboration and cognitive trust to the selected system. Finally, we qualitatively and quantitatively analyze the relation between the proposed decision-making similarity measures and the survey results. We find that people tend to collaborate with their most similar models – measured via JSD – yet this collaboration does not necessarily imply a similar level of cognitive trust. We release all resources related to the user study (e.g., design, outputs), models, and metrics at our repo.
@misc{kural2024quantifying,title={Quantifying Divergence for Human-AI Collaboration and Cognitive Trust},author={Kural, Müge and Gebeşçe, Ali and Chubakov, Tilek and Şahin, Gözde Gül},month=dec,year={2023},eprint={2312.08722},archiveprefix={arXiv},primaryclass={cs.AI}}
Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish
Arda Uzunoglu, and Gözde Şahin
In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, Nov 2023
Understanding procedural natural language (e.g., step-by-step instructions) is a crucial step to execution and planning. However, while there are ample corpora and downstream tasks available in English, the field lacks such resources for most languages. To address this gap, we conduct a case study on Turkish procedural texts. We first expand the number of tutorials in Turkish wikiHow from 2,000 to 52,000 using automated translation tools, where the translation quality and loyalty to the original meaning are validated by a team of experts on a random set. Then, we generate several downstream tasks on the corpus, such as linking actions, goal inference, and summarization. To tackle these tasks, we implement strong baseline models via fine-tuning large language-specific models such as TR-BART and BERTurk, as well as multilingual models such as mBART, mT5, and XLM. We find that language-specific models consistently outperform their multilingual models by a significant margin across most procedural language understanding (PLU) tasks.
@inproceedings{uzunoglu-ahin:2023:ijcnlp,author={Uzunoglu, Arda and Şahin, Gözde},title={Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish},booktitle={Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics},month=nov,year={2023},address={Nusa Dua, Bali},publisher={Association for Computational Linguistics},pages={804--819},url={https://aclanthology.org/2023.ijcnlp-long.52}}
GECTurk: Grammatical Error Correction and Detection Dataset for Turkish
Atakan Kara, Farrin Marouf Sofian, Andrew Bond, and
1 more author
In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, Nov 2023
Grammatical Error Detection and Correction (GEC) tools have proven useful for native speakers and second language learners. Developing such tools requires a large amount of parallel, annotated data, which is unavailable for most languages. Synthetic data generation is a common practice to overcome the scarcity of such data. However, it is not straightforward for morphologically rich languages like Turkish due to complex writing rules that require phonological, morphological, and syntactic information. In this work, we present a flexible and extensible synthetic data generation pipeline for Turkish covering more than 20 expert-curated grammar and spelling rules (a.k.a., writing rules) implemented through complex transformation functions. Using the pipeline, we derive 130,000 high-quality parallel sentences from professionally edited articles. Additionally, we create a more realistic test set by manually annotating a set of movie reviews. We implement three baselines formulating the task as i) neural machine translation, ii) sequence tagging, and iii) few-shot learning with prefix tuning, achieving strong results. Then we perform a zero-shot evaluation of our pretrained models on the coarse-grained “BOUN -de/-da” and fine-grained expert annotated dataset. Our results suggest that our corpus, GECTurk, is high-quality and allows knowledge transfer for the out-of-domain setting. To encourage further research on Turkish GEC, we release our dataset, baseline models, and synthetic data generation pipeline with https://anonymous.4open.science/r/tr-gec-17D6/.
@inproceedings{kara-EtAl:2023:findings,author={Kara, Atakan and Marouf Sofian, Farrin and Bond, Andrew and Şahin, Gözde},title={GECTurk: Grammatical Error Correction and Detection Dataset for Turkish},booktitle={Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics},month=nov,year={2023},address={Nusa Dua, Bali},publisher={Association for Computational Linguistics},pages={278--290},url={https://aclanthology.org/2023.findings-ijcnlp.26}}
In-context learning (ICL) for large language models has proven to be a powerful approach for many natural language processing tasks. However, determining the best method to select examples for ICL is nontrivial as the results can vary greatly depending on the quality, quantity, and order of examples used. In this paper, we conduct a case study on text simplification (TS) to investigate how to select the best and most robust examples for ICL. We propose Metric-Based in-context Learning (MBL) method that utilizes commonly used TS metrics such as SARI, compression ratio, and BERT-Precision for selection. Through an extensive set of experiments with various-sized GPT models on standard TS benchmarks such as TurkCorpus and ASSET, we show that examples selected by the top SARI scores perform the best on larger models such as GPT-175B, while the compression ratio generally performs better on smaller models such as GPT-13B and GPT-6.7B. Furthermore, we demonstrate that MBL is generally robust to example orderings and out-of-domain test sets, and outperforms strong baselines and state-of-the-art finetuned language models. Finally, we show that the behaviour of large GPT models can be implicitly controlled by the chosen metric. Our research provides a new framework for selecting examples in ICL, and demonstrates its effectiveness in text simplification tasks, breaking new ground for more accurate and efficient NLG systems.
@inproceedings{mbl-subha23,author={Vadlamannati, Subha and Şahin, Gözde Gül},title={Metric-Based In-context Learning: {A} Case Study in Text Simplification},booktitle={Proceedings of the 16th International Natural Language Generation Conference},month=sep,year={2023},address={Prague, Czech Republic},publisher={Association for Computational Linguistics},}
Many Natural Language Processing (NLP) systems use annotated corpora for training and evaluation. However, labeled data is often costly to obtain and scaling annotation projects is difficult, which is why annotation tasks are often outsourced to paid crowdworkers. Citizen Science is an alternative to crowdsourcing that is relatively unexplored in the context of NLP. To investigate whether and how well Citizen Science can be applied in this setting, we conduct an exploratory study into engaging different groups of volunteers in Citizen Science for NLP by re-annotating parts of a pre-existing crowdsourced dataset. Our results show that this can yield high-quality annotations and at- tract motivated volunteers, but also requires considering factors such as scalability, participation over time, and legal and ethical issues. We summarize lessons learned in the form of guidelines and provide our code and data to aid future work on Citizen Science.
@inproceedings{klie-etal-2023-lessons,title={Lessons Learned from a Citizen Science Project for Natural Language Processing},author={Klie, Jan-Christoph and Lee, Ji-Ung and Stowe, Kevin and Şahin, Gözde Gül and Moosavi, Nafise Sadat and Bates, Luke and Petrak, Dominic and Eckart De Castilho, Richard and Gurevych, Iryna},booktitle={Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},month=may,year={2023},address={Dubrovnik, Croatia},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2023.eacl-main.261},pages={3594--3608}}
MetaQA: Combining Expert Agents for Multi-Skill Question Answering
Haritz Puerto, Gözde Gül Sahin, and Iryna Gurevych
In Proceedings of the 17th Conference of the European Chapter of the
Association for Computational Linguistics, EACL 2023, Dubrovnik,
Croatia, May 2-6, 2023, May 2023
The recent explosion of question answering (QA) datasets and models has increased the interest in the generalization of models across multiple domains and formats by either training on multiple datasets or by combining multiple models. Despite the promising results of multi-dataset models, some domains or QA formats may require specific architectures, and thus the adaptability of these models might be limited. In addition, current approaches for combining models disregard cues such as question-answer compatibility. In this work, we propose to combine expert agents with a novel, flexible, and training-efficient architecture that considers questions, answer predictions, and answer-prediction confidence scores to select the best answer among a list of answer candidates. Through quantitative and qualitative experiments we show that our model i) creates a collaboration between agents that outperforms previous multi-agent and multi-dataset approaches in both in-domain and out-of-domain scenarios, ii) is highly data-efficient to train, and iii) can be adapted to any QA format. We release our code and a dataset of answer predictions from expert agents for 16 QA datasets to foster future developments of multi-agent systems on https://github.com/UKPLab/MetaQA.
@inproceedings{DBLP:conf/eacl/PuertoSG23,author={Puerto, Haritz and Sahin, G{\"{o}}zde G{\"{u}}l and Gurevych, Iryna},editor={Vlachos, Andreas and Augenstein, Isabelle},title={MetaQA: Combining Expert Agents for Multi-Skill Question Answering},booktitle={Proceedings of the 17th Conference of the European Chapter of the
Association for Computational Linguistics, {EACL} 2023, Dubrovnik,
Croatia, May 2-6, 2023},pages={3548--3562},publisher={Association for Computational Linguistics},year={2023},month=may,url={https://aclanthology.org/2023.eacl-main.259},timestamp={Thu, 11 May 2023 17:08:21 +0200},biburl={https://dblp.org/rec/conf/eacl/PuertoSG23.bib},bibsource={dblp computer science bibliography, https://dblp.org}}