https://github.com/boostcampaitech5/level2_klue-nlp-09
level2_klue-nlp-09 created by GitHub Classroom
Science Score: 10.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (3.4%) to scientific vocabulary
Last synced: 10 months ago
·
JSON representation
Repository
level2_klue-nlp-09 created by GitHub Classroom
Basic Info
- Host: GitHub
- Owner: boostcampaitech5
- Language: Jupyter Notebook
- Default Branch: main
- Size: 2.93 MB
Statistics
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
- Releases: 0
Created about 3 years ago
· Last pushed about 3 years ago
https://github.com/boostcampaitech5/level2_klue-nlp-09/blob/main/
#
---
# 1.
### Task: ** **
(Relation Extraction) (Entity) . , , , , . triple , .
, , . . , .
###
- ****: AI Stage (NVIDIA V100 32GB)
- **IDE**: VSCode, Jupyter Lab
- ****: Git(GitHub), Notion, Slack
- ****: WandB
---
# 2.
### \_T5038
- EDA preprocessing(easy data aug.)
- Hyperparameter tuning: kogpt2, twhin-bert-large, xlm-roberta-large
- Model evaluation( )
### \_T5157
- pytorch lightning base code
- EDA
- Hyperparameter tuning: klue/bert-base, klue/roberta-large
### \_T5227
- EDA ( , [UNK] )
-
- Hyperparameter tuning : mluke, kobart
### \_T5139
- EDA
- Hyperparameter tuning: google/rembert, klue/bert-base
- preprocessing(Entity Representation)
### \_T5194
- Preprocessing
- Clean Foreign Language
- snunlp/kr-electra-discriminator Modeling
- EDA
---
# 3.
## 3.0. Base Code ()
- **pytorch lightning base code **
- (logging, sweep )
- Dataloader ( , )
## 3.1. EDA
### 3.1.1. Data Distribution ()
- `id`, `sentence`, `subject_entity`, `object_entity`, `label`, `source`
- `train` 32,470, `test` 7,765
- `no_relation`, `per`(person), `org`(organization) main-label, `per` 17, `org` 12 sub-label 30 class
**Sub-label data distribution [ 3.1]**
3.1. Sub-label data distribution
- 30 sub-label
- `no_relation` 32,470 9,534( 29.36%) , `org:top_member/employee`(4,284, 13.19%), `per:place_of_death`(40, 0.12%) , imbalance
**Main-label data distribution [ 3.2]**
3.2. Main-label data distribution
- Sub-label distribution main-label sub-label , main-label sub-label 3 main-label
- `no_relation`(NR) `per` 9,534(29.36%), 9,081(27.97%) , org 13,855(42.67%)
- Main-label sub-label [ 3.3, 3.4]
3.3. `org` label data distribution
3.4. `per` label data distribution
- , sub-label sub-label no_relation , org per imbalance handling
**Source data distribution**
- `wikipedia`, `wikitree`, `policy-briefing` source , source
- train [ 3.5] `wikipedia` 60% , `policy-briefing` 1%
3.5. Train source
- train source `policy-briefing` `wikipedia`, `wikitree` sub-label
3.6. Wikipedia source sub-label
3.7. Wikitree source sub-label
- , `wikipedia` 9,534 `no_relation` 7,382 , `wikitree` `org:top_member/employee`
- , source
- test [ 3.8] `wikitree` , test sub-label train
3.8. Test source
**Token sequence length distribution [ 3.9, 3.10]**
3.9. train token sequence length(BERT tokenizer)
3.10. test token sequence length(BERT tokenizer)
- `klue/bert-base` BERT tokenizer token sequence length , train test
### 3.1.2. `[UNK]` Tokens ()
**unk **
```python
{"'": 337, '': 225, '': 60, '': 54, '': 48, '': 43, '.': 42, '': 38, '': 31, '': 31,
'': 28, '': 28, '': 25, '': 24, '': 23, ')': 22, '': 22, '': 22, '': 22, '': 21,
'': 21, '': 21, '!': 21, '': 20, '': 19, '': 19, '': 18, '': 18, '': 18, '': 18,
'': 18, '': 18, '': 16, '': 16, '': 16, '': 16, '': 15, ',': 15, '': 15,
'': 15, '': 15, '': 14, '': 14, '': 14, '': 14, '': 14, '': 13, '': 13,
'': 13, ',': 13, '': 13, '031': 13, '': 13, '': 13, '': 13, '': 12, '': 12,
'': 12, ',': 12, '': 12, '': 12, '': 11, '': 11, ',': 11, '': 11, ',': 10,
'': 10, '': 10, '': 10, '': 10, '': 10, '': 10, '': 10, '': 10, ',': 10,
'': 10, '': 10, ',': 10, '': 10, ',': 10, '': 10, '': 10, '': 10, '': 10,
'': 10, '': 9, '': 9, '': 9, '': 9, '': 9, '': 9, '': 9, '': 9, '': 9,
'': 9, ',': 9, '': 9, '': 9, '': 9, '': 9, '': 9, '': 9, ... }
```
- unk Data Preprocessing
### 3.1.3. Word Frequency ()
- data augmentation
- Special token `: `
```
{'': 2676, '': 2348, '': 1902, '': 1872, 'FC': 1811, '': 1735, '': 1710,
'': 1639, '': 1571, '': 1556, '': 1496, '': 1435, '': 1434, '': 1417,
'': 1403, '': 1348, '': 1346, '': 1325, '': 1245, '': 1184, '': 1175,
'': 1139, '': 1130, '': 1120, '': 1117, '': 1093, '': 1071, '': 1060, ...}
```
## 3.2. Preprocessing
### 3.2.1. Chinese-Characters Cleaning ()
```python
r'([-=+#/\?:^$.@*\"~&%!\\|\(\)\[\]\<\>`\'---\s])'
```
-
- ** **
```python
Before: (, 1937 4 29( 3 19)( 3 19) ~ 2009 11 4) , KBO .
After: (1937 4 29( 3 19)( 3 19) ~ 2009 11 4) , KBO .
```
### 3.2.2. Data Augmentation ()
**Easy data augmentation**
- Wei and Zhou (2019) 4 easy data augmentation , augmentation entity (entity augmentation )
- Synonym replacement (SR): [ 3.?]
3.11. SR augmentation (`` ``, `` ``)
- Random deletion (RD): [ 3.?]
3.12. RD augmentaion (`` )
- Random swap (RS): [ 3.?]
3.?. RS augmentation (`` - `` )
- Random insertion (RI): [ 3.?]
3.14. RI augmentation (`` : ` UEFA` ` UEFA`)
- SR, RS, RI, RD ( ), augmentation
- (0, 100), [100, 200), [200, 450), [450, 700) 4, 3, 2, 1
**Entity replacement (ER)**
- Entity type , entity type entity
3.15. ER augmentation (`subject_entity` )
- Threshold(1,000 or 2,000) , threshold threshold augmentation. augmentation 2 , ER 1
** augmentation **
- ,
- 6 () ,
- No aug.
/ easy data aug. only
/ ER(thres: 1,000) only
/ easy data aug. + ER 1,000
/ easy data aug. + ER 2,000
/ easy data aug. + ER 2,000 + no_relation cut
## 3.2.3. Entity Representation ()
- ****
- Train Test dataset Entity type
3.16. dataset entity
- base type
-
- ****
- Typed Entity Marker([Zhong and Chen, 2021](https://www.notion.so/KLUE-Wrap-Up-Report-7e063543d6154e02ad26f350bcabe04b?pvs=21)) Typed Entity Marker (punct)([Zhou and Chen, 2021](https://www.notion.so/KLUE-Wrap-Up-Report-7e063543d6154e02ad26f350bcabe04b?pvs=21)), Sentence Swap
- marker special token
- ****
3.17
- ****
- klue/bert-base Typed Entity Marker
- klue/roberta-large Typed Entity Marker (punct)
-
## 3.3. Model Selection ()
| | | F1 / AUPRC (dev) | F1 / AUPRC (public) |
| ---------------------------------------- | ----------- | ---------------- | ------------------- |
| klue/bert-base | 125M | 83.302 / 77.652 | |
| klue/roberta-large | 355M | 85.04 / 77.537 | |
| xlm-roberta-large | 355M | 84.344 / 78.023 | |
| wooy0ng/korquad1-klue-roberta-large | 355M | 86.03 / 79.491 | 69,475 / 73.0159 |
| kykim/albert-kor-base | 11M | 79.315 / 65.227 | |
| kykim/electra-kor-base | 85M | 77.358 / 49.062 | |
| beomi/KcELECTRA-base | 85M | 73.47 / 43.678 | |
| snunlp/KR-ELECTRA-discriminator | 85M | 79.729 / 63.963 | |
| monologg/koelectra-base-v3-discriminator | 85M | 78.667 / 55.246 | |
| skt/kogpt2-base-v2 | 125M | 78.174 / 67.998 | |
| google/rembert | 469M | 84.84 / 78.663 | 67.5282 / 68.6837 |
| setu4993/LaBSE | 470M | 81.447 / 73.052 | |
| timpal01/mdeberta-v3-base-squad2 | 86M | 79.992 / 61.719 | |
| studio-ousia/mluke-large-lite | 561M | 85.623 / 79.949 | 69.1844 / 71.59 |
| hfl/cino-large-v2 | 442M | 84.624 / 78.901 | |
- dev F1 score public score
- ,
- fully connected layer T5, bart model
** **
| | Description |
| ---- | ----------- |
| RoBERTa
-based | Dynamic masking BERT |
| ELECTRA
-based | BERT |
| | RemBERT (Chung et al., 2020)
-
MLUKE (Yamada et al.. 2020)
-
-
CINO
- chinese 6 minority languages xlm RoBERTa model |
## 3.4. Hyperparameter Tuning ()
### WandB - Sweep Hyperparameter Tuning
3.18 klue/roberta-large sweep
### Tuning Configuration
- **Learning Rate**
- **Max Epoch**
- **Batch Size**
- **Weight Decay**
- **LR Scheduler**
- **Warmup Steps**
- **Typed-Entity Marker**
- **Augmentation**
## 3.5. Ensemble ()
### 3.5.1. Soft-Voting
```python
dfs = [pd.read_csv(path) for path in model_paths]
probs = []
for row in zip(*[df['probs'].tolist() for df in dfs])
temp = []
for col in zip(*[eval(p) for p in row]):
temp.append(sum(col) / len(col))
probs.append(temp)
pred_label = [n2l[i.index(max(i))] ofri in probs]
```
- test data csv
- `probs`: class
- `pred_label`: class
---
# 4.
## 4.1. Single Models ()
4.1
## 4.2. Ensemble Models ()
4.2
---
# 5.
## 5.1. Whats Good
- Level 2 . .
- , merge .
## 5.2. Whats Bad
- commit branch . . .
- . . .
## 5.3. Whats Learned
- Hugging Face . .
- merge push . Git . .
---
# Reference
1. [Chung, H. W., Fevry, T., Tsai, H., Johnson, M., & Ruder, S. (2020). Rethinking embedding coupling in pre-trained language models.*arXiv preprint arXiv:2010.12821*.](https://arxiv.org/pdf/2010.12821.pdf)
2. [Wei, J., & Zou, K. (2019). Eda: Easy data augmentation techniques for boosting performance on text classification tasks.*arXiv preprint arXiv:1901.11196*.](https://arxiv.org/pdf/1901.11196)
3. [Yamada, I., Asai, A., Shindo, H., Takeda, H., & Matsumoto, Y. (2020). LUKE: Deep contextualized entity representations with entity-aware self-attention.*arXiv preprint arXiv:2010.01057*.](https://arxiv.org/pdf/2010.01057)
4. [Yang, Z., Xu, Z., Cui, Y., Wang, B., Lin, M., Wu, D., & Chen, Z. (2022). CINO: A Chinese Minority Pre-trained Language Model.*arXiv preprint arXiv:2202.13558*.](https://arxiv.org/pdf/2202.13558)
5. [Zhou, W., & Chen, M. (2021). An improved baseline for sentence-level relation extraction.*arXiv preprint arXiv:2102.01373*](https://arxiv.org/pdf/2102.01373.pdf)
6. [Zhong, Z., & Chen, D. (2020). A frustratingly easy approach for entity and relation extraction.*arXiv preprint arXiv:2010.12812*.](https://aclanthology.org/2021.naacl-main.5.pdf)
---
#
```
level2_klue-nlp-09
|-- README.md
|-- best_model
|-- config
| `-- config.yaml
|-- preprocessing
|-- eda
| |-- JYS.ipynb
| |-- KSH.ipynb
| |-- LDH.ipynb
| `-- LJS.ipynb
|-- inference.py
|-- load_data.py
|-- prediction
|-- requirements.txt
|-- requirements_pl.txt
|-- results
|-- train.py
|-- pl_train.py
|-- pl_sweep.py
|-- pl_inference.py
|-- .gitgnore
`-- utils.py
```
Owner
- Name: 부스트캠프 AI Tech 5기
- Login: boostcampaitech5
- Kind: organization
- Email: boostcamp_ai@connect.or.kr
- Location: Korea, South
- Website: https://boostcamp.connect.or.kr/program_ai.html
- Repositories: 1
- Profile: https://github.com/boostcampaitech5
AI 엔지니어의 지속 가능한 성장을 위한 학습 커뮤니티, 부스트캠프 AI Tech입니다.