Data

Tools

Indexes

Applications

Experiments

Open Source Science

semantic-regex

https://github.com/dataunitylab/semantic-regex

Science Score: 52.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ✓
    Institutional organization owner
    Organization dataunitylab has institutional domain (cs.rit.edu)
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.0%) to scientific vocabulary
Last synced: 6 months ago · JSON representation ·

Repository

Basic Info
  • Host: GitHub
  • Owner: dataunitylab
  • License: mit
  • Language: Python
  • Default Branch: main
  • Size: 432 KB
Statistics
  • Stars: 2
  • Watchers: 1
  • Forks: 0
  • Open Issues: 1
  • Releases: 0
Created almost 4 years ago · Last pushed 6 months ago
Metadata Files
Readme License Citation

README.md

Learning from Uncurated Regular Expressions

CI pre-commit.ci status

Dependencies of all Python code are managed with Pipenv and can be installed with pipenv install. Note that the dataset from the Sherlock project should be available in a copy of the repository in alongside the directory for this project. jq is also required for some JSON processing.

Model training

  1. Download all regular expressions from regex101

./download_patterns.sh

This will create a directory regex101 which has the individual regular expressions and patterns.json which contains only the expressions strings.

  1. Compile a database of all the downloaded regular expressions

pipenv run python compile_db.py < patterns.json > patterns_final.json

patterns_final.json is a subset of the expressions in patterns.json which are supported by Hyperscan. This step will also create hs.db which are the compiled regular expressions that can be used during preprocessing.

  1. Preprocess the data to generate feature vectors

pipenv run python preprocess.py train

This will generate preprocessed_train.txt which contains all the feature vectors extracted using the regular expression extracted using the regular expressions.

  1. Train the model on the extracted features

pipenv run python train.py

The model architecture will be stored in nn_model_sherlock.json with the weights in nn_model_sherlock.weights.keras.

Evaluation

First, the test data must be preprocessed.

pipenv run python preprocess.py test

Then, the model can be evaluated.

pipenv run python test.py

Model explanation

Explains for predictions for an individual class can be generated using SHAP. First, follow the steps for training the model above. The file patterns_final.json will be used to match the patterns back to the original regular expressions.

pipenv run python find_patterns.py > pattern_ids.txt

This file of pattern IDs will then be used to label the SHAP plot with the ID of the regular expression. To generate the SHAP plot in shap.png, run the command below where <class_name> is one of the semantic types defined by Sherlock.

pipenv run python explain.py <class_name>

The IDs displayed in the SHAP plot can be used to reference the regular expressions by ID in the regex101/patterns directory or viewing it directly on regex101 at the URL https://regex101.com/library/<ID>.

Owner

  • Name: Data Unity Lab
  • Login: dataunitylab
  • Kind: organization
  • Email: mmior@mail.rit.edu
  • Location: United States of America

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: Mior
  given-names: Michael
  orcid: "https://orcid.org/0000-0002-4057-8726"
  email: mmior@mail.rit.edu
  affiliation: Rochester Institute of Technology
title: "Learning from Uncurated Regular Expressions"
url: "https://github.com/dataunitylab/semantic-regex"
license: MIT
preferred-citation:
  type: conference-paper
  authors:
    - family-names: Mior
      given-names: Michael
      orcid: "https://orcid.org/0000-0002-4057-8726"
      email: mmior@mail.rit.edu
      affiliation: Rochester Institute of Technology
  doi: 10.1145/3596225.3596226
  conference:
    name: "1st Workshop on Simplicity in Management of Data"
    city: Bellevue
    region: WA
    country: US
    date-start: 2023-06-23
    date-end: 2023-06-23
  start: 1 # First page number
  end: 5 # Last page number
  title: "Learning from Uncurated Regular Expressions"
  year: 2023
  month: 6

GitHub Events

Total
  • Delete event: 5
  • Issue comment event: 4
  • Push event: 52
  • Pull request event: 18
  • Create event: 4
Last Year
  • Delete event: 5
  • Issue comment event: 4
  • Push event: 52
  • Pull request event: 18
  • Create event: 4