question-pair-nlp
Science Score: 44.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (13.7%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: fahaddeshmukh
- License: mit
- Language: Jupyter Notebook
- Default Branch: master
- Size: 28.4 MB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
Readme.md
Semantic Duplicate Detection: Quora Question Pairs Analysis using Machine Learning
This repository contains files for a machine learning (ML)-based solution to the Quora Question Pairs problem, hosted on Kaggle. This challenge falls within the scope of natural language processing (NLP) and involves the task of identifying whether pairs of questions from the Quora platform are duplicates.
Directories
The project is structured as follows:
bin: This directory contains the source code files required to run the analysis.
quora.ipynb: This Jupyter Notebook file provides an alternative interface for running the analysis interactively. It contains the same code assrc.pybut provides easy accessablity along with a narrative.
data: This directory contains the dataset used for the analysis.
train.csv: The CSV file that contains the original dataset, that contains questions. This file is required for running the analysis. Please ensure that it is located in this directory.
docs: This directory contains any documentation related to the project.
plots: This directory contains the EDA plots used for the analysis.
Installation
To use this script, you need to have Python installed on your system (3.8 or higher). You also need to install the required dependencies by running the following command:
pip install -r requirements.txt
Requirements
The script requires the following Python packages:
- pandas
- numpy
- matplotlib
- seaborn
- scikit-learn
- lightgbm
- spacy
- FuzzyWuzzy
You can install these packages by running the command mentioned in the "Installation" section.
License
This project is licensed under the MIT License. See the LICENSE file for details.
Citation Information
Citation information for this project can be found in the Citation files in the directory. See the CITATION file for details.
Contact Information
For any questions, suggestions, or issues, please feel free to contact:
- Name: Fahad Deshmukh
- Email: deshmukh@uni-potsdam.de
Owner
- Name: Fahad Deshmukh
- Login: fahaddeshmukh
- Kind: user
- Location: Dortmund, Germany.
- Repositories: 1
- Profile: https://github.com/fahaddeshmukh
I'm a Data Science student at Uni Potsdam and I'm passionate about ML and DL techniques. This Github showcases some of my academic and personal projects.
Citation (CITATION.cff)
# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!
cff-version: 1.2.0
title: >-
Semantic Duplicate Detection: Quora Question Pairs
Analysis using Machine Learning
message: >-
If you use this software, please cite it using the
metadata from this file.
type: software
authors:
- given-names: Fahad
family-names: Deshmukh
email: deshmukh@uni-potsdam.de
affiliation: University of Potsdam
identifiers:
- type: url
value: 'https://github.com/fahaddeshmukh/Question-pair-nlp'
description: github repo
abstract: >-
This project presents a machine learning (ML)-based
solution for the Quora Question Pairs challenge hosted on
Kaggle. With a focus on natural language processing (NLP),
the project addresses the task of identifying whether
pairs of questions from the Quora platform are duplicates.
The primary objective is to construct binary
classification models capable of determining whether two
provided questions are semantically similar or represent
duplicates.
The project involves a dataset comprising over 400,000
observations across six columns: 'ID', 'QID1', 'QID2',
'Question1', 'Question2', and 'Is_duplicate'. Through
systematic processing, exploratory data analysis, and
feature engineering, the project seeks to derive insights
for model development. The 'Is_duplicate' column,
containing boolean values indicating question duplicity,
serves as the dependent variable.
The project encompasses loading and preprocessing the
dataset, conducting exploratory data analysis, and
performing feature engineering using NLP techniques. The
engineered features are then subjected to additional
exploratory analysis. Predictive modeling forms the
culminating phase of the project, where various binary
classification models are trained and evaluated to
determine the most effective solution.
By addressing the Quora Question Pairs challenge, this
project contributes to the field of NLP and
classification, demonstrating the application of machine
learning techniques in detecting semantic similarity
within textual data. The outcomes shed light on the
efficacy of different models and feature engineering
strategies in tackling real-world NLP tasks.
keywords:
- NLP
- Machine Learning
- Python
- jupyter notebbok
- Semantic duplicate
license: MIT
date-released: '2023-08-10'
GitHub Events
Total
Last Year
Dependencies
- fuzzywuzzy ==0.18.0
- matplotlib ==3.4.3
- nltk ==3.6.3
- numpy ==1.21.2
- pandas ==1.3.3
- scikit-learn ==0.24.2
- spacy ==3.1.3