Zhangyijingaskfollowupquestionsforhealthmonitoringusinggen_models

Master's Degree in Linguistics: Text Mining, Vrije Universiteit Amsterdam 2023/2024

This repository provides the codes and datasets used in the thesis: Ask Follow-up Questions for Health Monitoring Using Generative Models

Project Structure

Data

• predefined_data/: Contains predefined datasets for prompt generation.

  • custom_valid_words.json: A customized list of words identified as invalid but potentially valid.
  • examples.json: Examples used for few-shot prompting.
  • icf_cate_comb.json: ICF sub-categories with definitions, sub-activities, and examples for generating prompts.
  • icf_def.json: ICF categories and their definitions for generating prompts.

• response_data/: Contains conversation data generated by GPT-3.5 and processed follow-up questions.

  • raw_conversations/: Raw conversations generated by GPT-3.5.
  • Communication.json
  • mobility.json
  • self-care.json
  • split_conversations/: Cleaned and split conversation data into training, validation, and test sets.
  • communication/
    • test.json
    • train_1.json
    • val_1.json
  • (Same structure for mobility and self-care)
  • raw_fq/: Raw follow-up questions generated by GPT-3.5.
  • fewshot/: Questions generated using few-shot prompting.
  • zeroshot/: Questions generated using zero-shot prompting.
  • clean_fq/: Cleaned follow-up questions.
  • test_data/: Contains test inputs and reference outputs.
  • in_conver/: Input conversation data.
  • out_fq/: Output follow-up questions.
    • out_clean/
    • out_raw/
  • references/: Reference data used for evaluation.
  • results/: Evaluation scores of the models.

Scripts and Notebooks

This includes code: data-related and colab:model-related

code

• s1_valid_words.py: Script to create predefined data.
• data_process.py: Script for initial data processing.
• s2_data_process_new.py: Updated data processing script.

• s3_evaluation.py: Script to evaluate the final results.

  • Note: Run the scripts in the order listed here (until Evaluation.ipynb).

• Data_Prompt.ipynb: Jupyter notebook for zero-shot and few-shot prompting using GPT-3.5 to generate conversations and follow-up questions (FQs).

• save_doc.ipynb: Notebook to push test data to Google Sheets for collaborative reference creation.

• export_reference.ipynb: Export selected and revised test conversations used for generating reference outputs.

• Llama-3_train_fewshot.ipynb: Use Llama-3 with few-shot prompting to generate reference outputs.

• Llama-3_train_zeroshot.ipynb: Use Llama-3 with zero-shot prompting to generate reference outputs.

• base_llama.ipynb: Use Llama-3 with basic instructions to generate reference outputs.

• ft_Llama-3_train_onfewshot.ipynb: Fine-tune Llama-3 with training data generated by few-shot prompting with GPT-3.5, and infer the results from the fine-tuned model.

• ft_Llama-3_train_onzeroshot.ipynb: Fine-tune Llama-3 with training data generated by zero-shot prompting with GPT-3.5, and infer the results from the fine-tuned model.

• Evaluation.ipynb: Evaluate all resulting sentences against the reference data.

colab

• Llama-3_train_fewshot.ipynb: Use Llama-3 with few-shot prompting to generate reference outputs.
• Llama-3_train_zeroshot.ipynb: Use Llama-3 with zero-shot prompting to generate reference outputs.
• base_llama.ipynb: Use Llama-3 with basic instructions to generate reference outputs.
• ft_Llama-3_train_onfewshot.ipynb: Fine-tune Llama-3 with training data generated by few-shot prompting with GPT-3.5, and infer the results from the fine-tuned model.
• ft_Llama-3_train_onzeroshot.ipynb: Fine-tune Llama-3 with training data generated by zero-shot prompting with GPT-3.5, and infer the results from the fine-tuned model.

Requirements

• requirements.txt: Contains all the packages used in this project. Ensure you have all dependencies installed before running the scripts.

Steps to Run the Project

1. Install Dependencies: Run pip install -r requirements.txt to install all necessary packages.
2. Set Up Paths: Ensure that your working directory is set to the Data folder before running any scripts.
3. Configure Google Drive: If using Google Sheets for collaborative work, update the API key or authentication credentials as required.
4. Execute Notebooks: Follow the order mentioned in code until Evaluation.ipynb, then run five notebooks in colab and finally run Evaluation.ipynb in code. This will ensure the correct sequence of data processing, model training, and evaluation.

Thesis Report

TheUseandComparisonofPromptEngineeringApproachesUsedforFollowupQuestion_Generation.pdf

References

The code for prompting GPT-3.5 is inspired by ICF-activities-classifier, specifically 1.gptgeneratedconversations.ipynb. The code for fine-tuning llama3 is adopted from unslothai