Dataverse Metadata Crawler

📜Description

A Python CLI tool for extracting and exporting metadata from Dataverse repositories. It supports bulk extraction of dataverses, datasets, and data file metadata from any chosen level of dataverse collection (an entire Dataverse repository/sub-Dataverse), with flexible export options to JSON and CSV formats.

✨Features

1. Bulk metadata extraction from Dataverse repositories at any chosen level of collection (top level or selected collection)
2. JSON & CSV file export options

☁️ Installation (Cloud - Slower)

Click to launch the crawler directly in your web browser—no Git or Python installation required!

⚙️Installation (Locally - Better performance)

📦Prerequisites

1. Git

2. Python 3.10+

1. Clone the repository sh git clone https://github.com/scholarsportal/dataverse-metadata-crawler.git

2. Change to the project directory sh cd ./dataverse-metadata-crawler

3. Create an environment file (.env) ```sh touch .env # For Unix/MacOS nano .env # or vim .env, or your preferred editor

   OR

   New-Item .env -Type File # For Windows (Powershell) notepad .env ```

4. Configure the environment (.env) file using the text editor of your choice. ```sh

   .env file

   BASEURL = "TARGETREPOURL" # Base URL of the repository; e.g., "https://demo.borealisdata.ca/" APIKEY = "YOURAPIKEY" # Found in your Dataverse account settings. Can also be specified in the CLI interface using the -a flag. Your `.env` file should look like this: sh BASEURL = "https://demo.borealisdata.ca/" APIKEY = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" ```

5. Set up virtual environment (recommended) ```sh python3 -m venv .venv source .venv/bin/activate # For Unix/MacOS

   OR

   .venv\Scripts\activate # For Windows ```

6. Install dependencies sh pip install -r requirements.txt

🛠️Usage

Basic Command

sh python3 dvmeta/main.py [-a AUTH] [-l] [-d] [-p] [-f] [-e] [-s] -c COLLECTION_ALIAS -v VERSION Required arguments:

| Option | Short | Type | Description | Default | |--------------------|-----------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------| | --collection_alias | -c | TEXT | The alias of the collection to crawl.
See the guide here to learn how to look for a the collection alias.
[required] | None | | --version | -v | TEXT | The Dataset version to crawl. Options include:
• draft - The draft version, if any
• latest - Either a draft (if exists) or the latest published version
• latest-published - The latest published version
• x.y - A specific version
[required] | None (required) |

Optional arguments:

| Option | Short | Type | Description | Default | |-------------------|-----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------| | --auth | -a | TEXT | Authentication token to access the Dataverse repository. | None | | --log --no-log | -l | | Output a log file. Use --no-log to disable logging. | log (unless --no-log) | | --dvdfdsmetadata | -d | | Output a JSON file containing metadata of Dataverses, Datasets, and Data Files. | | | --permission | -p | | Output a JSON file that stores permission metadata for all Datasets in the repository. | | | --emptydv | -e | | Output a JSON file that stores all Dataverses which do NOT contain Datasets (though they might have child Dataverses which have Datasets). | | | --failed | -f | | Output a JSON file of Dataverses/Datasets that failed to be crawled. | | | --spreadsheet | -s | | Output a CSV file of the metadata of Datasets.
See the spreadsheet column explanation notes. | | | --debug-log | -debug | | Enable debug logging. This will create a debug log file in the logfiles directory. | | | --help | | | Show the help message. | |

Examples

```sh

Export the metadata of latest version of datasets under collection 'demo' to JSON

python3 dvmeta/main.py -c demo -v latest -d

Export the metadata of version 1.0 of all datasets under collection 'demo' to JSON and CSV

python3 dvmeta/main.py -c demo -v 1.0 -d -s

Export the metadata and permission metadata of version 1.0 of all datasets under collection 'demo' to JSON and CSV, with the API token specified in the CLI interface

python3 dvmeta/main.py -c demo -v 1.0 -d -s -p -a xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx ```

📂Output Structure

| File | Description | |-------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------| | dsmetadatayyyymmdd-HHMMSS.json | Datasets representation & data files metadata in JSON format. | | emptydvyyyymmdd-HHMMSS.json | The id of empty dataverse(s) in list format. | | failedmetadataurisyyyymmdd-HHMMSS.json | The URIs (URL) of datasets failed to be downloaded. | | permissiondictyyyymmdd-HHMMSS.json | The permission metadata of datasets with their dataset id. | | piddictyyyymmdd-HHMMSS.json | Datasets' basic info with hierarchical information dictionary.Only exported if -p (permission) flag is used without -d (metadata) flag. | | piddictddyyyymmdd-HHMMSS.json | The Hierarchical information of deaccessioned/draft datasets. | | dsmetadatayyyymmdd-HHMMSS.csv | Datasets and their data files' metadata in CSV format. | | log_yyyymmdd-HHMMSS.txt | Summary of the crawling work. |

sh exported_files/ ├── json_files/ │ └── ds_metadata_yyyymmdd-HHMMSS.json # With -d flag enabled │ └── empty_dv_yyyymmdd-HHMMSS.json # With -e flag enabled │ └── failed_metadata_uris_yyyymmdd-HHMMSS.json # With -f flag enabled │ └── permission_dict_yyyymmdd-HHMMSS.json # With only -p flag enabled │ └── pid_dict_yyyymmdd-HHMMSS.json # With only -p flag enabled │ └── pid_dict_dd_yyyymmdd-HHMMSS.json # Hierarchical information of deaccessioned/draft datasets. ├── csv_files/ │ └── ds_metadata_yyyymmdd-HHMMSS.csv # with -s flag enabled └── logs_files/ └── log_yyyymmdd-HHMMSS.txt # Exported by default, without specifying --no-log └── debug.log # Export by using -debug flag

⚠️Disclaimer

[!WARNING] To retrieve data about unpublished datasets or information that is not available publicly (e.g. collaborators/permissions), you will need to have necessary access rights. Please note that any publication or use of non-publicly available data may require review by a Research Ethics Board.

✅Tests

No tests have been written yet. Contributions welcome!

💻Development

1. Dependencies management: uv - Use uv to manage dependencies and reflect changes in the pyproject.toml file.
2. Linter: ruff - Follow the linting rules outlined in the pyproject.toml file.

🙌Contributing

1. Fork the repository
2. Create a feature branch
3. Submit a pull request

📄License

MIT

🆘Support

• Create an issue in the GitHub repository

📚Citation

If you use this software in your work, please cite it using the following metadata.

APA: Lui, L. H. (2025). Dataverse Metadata Crawler (Version 0.1.6) [Computer software]. https://github.com/scholarsportal/dataverse-metadata-crawler

BibTeX: @software{Lui_Dataverse_Metadata_Crawler_2025, author = {Lui, Lok Hei}, month = {June}, title = {Dataverse Metadata Crawler}, url = {https://github.com/scholarsportal/dataverse-metadata-crawler}, version = {0.1.6}, year = {2025} }

✍️Authors

Ken Lui - Data Curation Specialist, Map and Data Library, University of Toronto - kenlh.lui@utoronto.ca