Next, generate a database for files in data.

bash python scripts/cyclomatic-complexity.py --input ./data --db ./scripts/data/cyclomatic-complexity-github.db

IMPORTANT For the above and complexity calculation below, duplicates are not removed. We store the sha256 and sha1 so you can do this!

5. LC Jobspec Database

This database is kind of messy - not sure I like it as much as the one I generated. Someone else can deal with it :)

• Total unique jobspec jsons: 210351
• Total with BatchScript: 116117

bash cd ./lc python scripts/cyclomatic-complexity.py --input ./raw/jobdata_json --db ./data/cyclomatic-complexity-lc.db

IMPORTANT Since this is a combination of json and .tar files (for which we extract members) the database has an extra column for the jobid, and the original filename path corresponds to the file here. The file that we actually read is parsed from the BatchScript directive of the json file, which is only the batch portion of the data to match what we use in GitHub.

Reading Sqlite Databases

Examples to read in the two databases:

```python import sqlite3 conn = sqlite3.connect("scripts/data/cyclomatic-complexity-github.db") cursor = conn.cursor()

This gets the field names and metadata

cursor.execute('PRAGMA table_info(jobspecs);').fetchall() console [(0, 'id', 'INTEGER', 0, None, 1), (1, 'name', 'TEXT', 0, None, 0), (2, 'sha256', 'TEXT', 0, None, 0), (3, 'sha1', 'TEXT', 0, None, 0), (4, 'ccn', 'NUMBER', 0, None, 0)] ```

And this gets the jobspecs (one for example)

```python query = cursor.execute("SELECT * from jobspecs;") query.fetchone()

rows = query.fetchall()

console (1, './data/ZIYU-DEEP/reprover-test/2gpu.sh', '48ef130f0700b606c3b5d4b2a784cd78f97439b935bd7d7df4673d9683d420e1', 'c88fdc41ae091734565c63ed67c51a45de66a07f', 1) ```

Don't forget to close.

python conn.close()

And don't forget LC will have an extra field, for the name of the file plus the jobid since some are members in a tar.

python conn = sqlite3.connect("lc/data/cyclomatic-complexity-lc.db") cursor = conn.cursor() cursor.execute('PRAGMA table_info(jobspecs);').fetchall() console [(0, 'id', 'INTEGER', 0, None, 1), (1, 'name', 'TEXT', 0, None, 0), (2, 'jobid', 'TEXT', 0, None, 0), (3, 'sha256', 'TEXT', 0, None, 0), (4, 'sha1', 'TEXT', 0, None, 0), (5, 'ccn', 'NUMBER', 0, None, 0)] python conn.close()

6. Summarizing

We want to look at, for each databases, what we have for:

• applications
• job managers
• length of jobs

The first two will be tags based on presence of directives. The last will be a calculation.

bash python scripts/summarize-jobspecs.py --input ./data --db ./scripts/data/jobspec-summary-github.db console {'slurm': 23958, 'pbs': 6381, 'lsf': 2614, 'oar': 145, 'flux': 74, 'cobalt': 630}

This saves to scripts/data/jobspec-summary

Reading Sqlite Database

Examples to read in the two database:

```python import sqlite3 conn = sqlite3.connect("./scripts/data/jobspec-summary-github.db") cursor = conn.cursor()

This gets the field names and metadata

cursor.execute('PRAGMA table_info(jobspecs);').fetchall() console [(0, 'id', 'INTEGER', 0, None, 1), (1, 'name', 'TEXT', 0, None, 0),

job manager tags

(2, 'manager_tags', 'TEXT', 0, None, 0),

these are tags from gemini without a template

(3, 'software_tags', 'TEXT', 0, None, 0),

These were tags with a template (nicer / cleaner set)

(4, 'softwaretagswith_template', 'TEXT', 0, None, 0), (5, 'length', 'NUMBER', 0, None, 0)] ```

Each of managertags and softwaretags are json dumped lists.

```python query = cursor.execute("SELECT * from jobspecs;") result = query.fetchone()

rows = query.fetchall()

console (1, './data/ZIYU-DEEP/reprover-test/2gpu.sh',

job manager tags

'["slurm"]',

these are tags from gemini without a template

'["rs", "head", "scontrol", "yt", "distributed", "node", "les", "main", "an", "run", "generator", "ame", "at", "srun", "nam", "python", "bash", "gp", "gt", "aria", "aml", "scrip", "ct", "re", "tact", "ed", "su", "vi", "tac", "ic", "li", "hostname", "ip", "go", "train", "env", "bin", "os", "export", "cc", "docker", "random", "training", "ml", "tr", "sh", "yaml", "hon", "gene", "od", "gpu", "host"]',

These were tags with a template (nicer / cleaner set)

'["python", "bash", "srun", "docker", "r"]', 733) ```

To read in the tags:

console import json print(json.loads(result[2])) print(json.loads(result[3])) print(json.loads(result[4])) ['slurm'] ['rs', 'head', 'scontrol', 'yt', 'distributed', 'node', 'les', 'main', 'an', 'run', 'generator', 'ame', 'at', 'srun', 'nam', 'python', 'bash', 'gp', 'gt', 'aria', 'aml', 'scrip', 'ct', 're', 'tact', 'ed', 'su', 'vi', 'tac', 'ic', 'li', 'hostname', 'ip', 'go', 'train', 'env', 'bin', 'os', 'export', 'cc', 'docker', 'random', 'training', 'ml', 'tr', 'sh', 'yaml', 'hon', 'gene', 'od', 'gpu', 'host'] ['python', 'bash', 'srun', 'docker', 'r']

And that's it! Go hither and jobspec, away!!!!

License

HPCIC DevTools is distributed under the terms of the MIT license. All new contributions must be made under this license.

See LICENSE, COPYRIGHT, and NOTICE for details.

SPDX-License-Identifier: (MIT)

LLNL-CODE- 842614