Splice

The Role of Information Extraction Tasks in Automatic Literary Character Network Construction

Reproducing Results

First, you should:

• install dependencies. Either use poetry install if you have poetry, or pip install -r requirements.txt otherwise.
• get the litbank dataset

The main experiment can be run with xp.py:

sh python xp.py with\ min_graph_nodes=10\ co_occurrences_dist=32\ litbank.root="/path/to/litbank"

Degradation Experiments

The following script will run all of the degradation experiments:

```sh MAINXPRUN="/path/to/main/xp/run"

python xpmetricsoverdegradation.py with inputdir="${MAINXPRUN}" taskname=NER degradationname=addwrongentity degradationsteps=1000 degradationreportfrequency=0.05 python xpmetricsoverdegradation.py with inputdir="${MAINXPRUN}" taskname=NER degradationname=removecorrectentity degradationsteps=200 degradationreportfrequency=0.5 python xpmetricsoverdegradation.py with inputdir="${MAINXPRUN}" taskname=coref degradationname=addwrongmention degradationsteps=200 degradationreportfrequency=0.05 python xpmetricsoverdegradation.py with inputdir="${MAINXPRUN}" taskname=coref degradationname=removecorrectmention degradationsteps=1000 degradationreportfrequency=0.05 python xpmetricsoverdegradation.py with inputdir="${MAINXPRUN}" taskname=coref degradationname=addwronglink degradationsteps=500 degradationreportfrequency=0.05 python xpmetricsoverdegradation.py with inputdir="${MAINXPRUN}" taskname=coref degradationname=removecorrectlink degradationsteps=1000 degradationreportfrequency=0.05 python xpmetricsoverdegradation.py with inputdir="${MAINXPRUN}" taskname=coref degradationname=corefall degradationsteps=1000 degradationreportfrequency=0.05 ```

End-to-end LLM-based Pipelines

The E2E-Coref experiment can be reproduced with the xp_e2e_llm_coref.py script:

```sh MAINXPRUN="/path/to/main/xp/run" LITBANK_PATH="/path/to/litbank"

python xpe2ellmcoref.py with\ inputdir="${MAINXPRUN}"\ model="gpt3.5"\ openAIAPIkey="insert your openAI key"\ litbank.root="${LITBANK_PATH}"

python xpe2ellmcoref.py with\ inputdir="${MAINXPRUN}"\ model="gpt40"\ openAIAPIkey="insert your openAI key"\ litbank.root="${LITBANK_PATH}"

python xpe2ellmcoref.py with\ inputdir="${MAINXPRUN}"\ model="llama3-8b-instruct"\ hgaccesstoken="insert your Huggingface access token"\ device="cuda"\ litbank.root="${LITBANK_PATH}" ```

Similarly, the *E2E-Graphml experiment can be reproduced with the xp_e2e_llm_graphml.py script:

```sh MAINXPRUN="/path/to/main/xp/run"

python xpe2ellmgraphml.py with\ inputdir="${MAINXPRUN}"\ model="gpt3.5"\ openAIAPIkey="insert your openAI key"\ litbank.root="${LITBANK_PATH}"

python xpe2ellmgraphml.py with\ inputdir="${MAINXPRUN}"\ model="gpt40"\ openAIAPIkey="insert your openAI key"\ litbank.root="${LITBANK_PATH}"

python xpe2ellmgraphml.py with\ inputdir="${MAINXPRUN}"\ model="llama3-8b-instruct"\ hgaccesstoken="insert your Huggingface access token"\ device="cuda"\ litbank.root="${LITBANK_PATH}" ```

Printing / Plotting Results

| Figure | Corresponding Script | |----------|-------------------------------------| | Table 1 | print_main_task_results.py | | Table 2 | print_main_graph_results.py | | Table 3 | | | Figure 1 | plot_degradation_metrics.py | | Figure 2 | plot_ner_degradation_metrics.py | | Figure 3 | plot_coref_degradation_metrics.py | | Table 4 | print_e2e_graph_results.py |