nlu-slot-constraints

In goal-oriented dialogue systems, users provide information through slot values to achieve specific goals. Practically, some combinations of slot values can be invalid according to external knowledge, i.e., slot constraints. For example, a combination of "cheese pizza" (a menu item) and "oreo cookies" (a topping) from an input utterance "Can I order a cheese pizza with oreo cookies on top?" exemplifies such invalid combinations according to the menu of a restaurant business.

In this project, we present a new task of slot constraint violation detection accompanied with benchmarking data (adapted from the MultiDoGO dataset).

Slot constraint violation detection: given a bot schema with constraints, a current utterance, and a conversation history, predict whether the current state of conversation violates any constraints or not and which constraints are violated.

Also, we propose three approaches to solve this problem (based on a pipeline approach and an end-to-end approach). This github repository contains both the benchmarking data and the code of the approaches used in our experiments.

For more information, please refer to our NAACL 2021 paper:

@inproceedings{lertvittayakumjorn-etal-2021-knowledge, title = "Knowledge-driven slot constraints for goal-oriented dialogue systems", author = "Lertvittayakumjorn, Piyawat and Bonadiman, Daniele and Mansour, Saab", booktitle = "Proceedings of the 2021 Conference of the North {A}merican Chapter of the Association for Computational Linguistics", month = jun, year = "2021", }

Dataset description

Our benchmarking data has two domains -- insurance (sentence-level annotation) and fast food (turn-level annotation) of the MultiDoGO dataset. Originally, MultiDoGO was constructed as an English multi-domain goal-oriented IC/SL dataset. Then we modified the two domains to support slot constraint violation detection by (1) creating a list of possible entities for each custom slot type and providing ground-truths for entity linking (2) manually writing slot constraints for each intent and (3) constructing dialogue states and adding constraint violation detection results to the dataset.

Installation

Requirements

• git
• python >= 3.6
• Package requirements:
  • pytorch 1.6.0
  • JointBERT dependencies
  • editdistance 0.5.3
  • regex-2020.7.14
  • hydra-core 1.0.3

Install

After installing all requirements run:

bash ./INSTALL.sh

This script will clone the JointBERT repository and apply the patches required to run this project.

Run

Run Pipeline

• To run the pipeline approaches, change the permission of the shell script runpipeline.sh to allow execution and run: ```bash ./runpipeline.sh fastfood ``` or

bash ./run_pipeline.sh insurance where fastfood and insurance are the two datasets.

Run End To End

To run the end-to-end approach, change the permission of the shell script runendtoend.sh to allow execution and run: ```bash ./runendtoend.sh fastfood or bash ./run_endtoend.sh insurance ```

File Organization

• entity_linking.py: This file contains methods for entity linking used in the pipeline approach
• generate_result_table.py: This file is used to generate result table files in csv format
• main.py: This is the main file for running the pipeline approach (deterministic and probabilistic)
• run_endtoend.sh: Shell script to run the end-to-end approach
• run_pipeline.sh: Shell script to run deterministic and probabilistic pipeline approaches
• violation_detection.py: This file contains methods to check if the constraints are violated or not given a combination of entities (used in the pipeline approach)

Dataset Organization

The dataset folder contains data in the format that is suitable for JointBERT. Each domain (fastfood and insurance) has a folder containing a bot definition, an intent label file, a slot label file, and three folders for training, development, and test sets. The structure of each folder, for example for fast food, is as follows. - bot_definition_*.json: This file provides a list of all intents, a list of domain-specific intents (i.e., main intents), a list of slots, a list of all possible entities for each custom slot type, and a list of constraints. - intent_label.txt: a list of all intents (together with combinations of intents) which appear in the training / development / test splits (used in the intent classification task) - slot_label.txt: a list of all slot labels which appear in the training / development / test splits (used in the slot labelling task)
- train: containing the training data - train.tsv: the tsv data file (same format as the original MultiDoGO dataset) - train.json: the training dataset augmented with entity linking and constraint violation gold labels. This is used to train the MultilabelBERT model (end-to-end approach) - seq.in, seq.out, label, ids.txt: training data in the format that is applicable for the JointBERT model - dev: containing the development data in the same formats as train - test: containing the test data in the same formats as train

Besides, the vocab_process.py in the outer folder was used to generate intent_label.txt and slot_label.txt. It is not needed when training the model though.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

Data License

The data is licensed under the Community Data License Agreement - Permissive – Version 1.0