Data

Tools

Indexes

Applications

Experiments

Open Source Science

v-d4rl

Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations

https://github.com/conglu1997/v-d4rl

Science Score: 64.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ✓
    Committers with academic emails
    1 of 3 committers (33.3%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.5%) to scientific vocabulary
Last synced: 10 months ago · JSON representation ·

Repository

Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations

Basic Info
  • Host: GitHub
  • Owner: conglu1997
  • License: mit
  • Language: Python
  • Default Branch: main
  • Size: 388 KB
Statistics
  • Stars: 100
  • Watchers: 3
  • Forks: 9
  • Open Issues: 2
  • Releases: 1
Created about 4 years ago · Last pushed about 2 years ago
Metadata Files
Readme License Citation

README.md

V-D4RL

Twitter arXiv

V-D4RL provides pixel-based analogues of the popular D4RL benchmarking tasks, derived from the dm_control suite, along with natural extensions of two state-of-the-art online pixel-based continuous control algorithms, DrQ-v2 and DreamerV2, to the offline setting. For further details, please see the paper:

Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations; Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, Yee Whye Teh. Published at TMLR, 2023.

View on arXiv

Benchmarks

The V-D4RL datasets can be found on Google Drive[^1]. These must be downloaded before running the code. Assuming the data is stored under vd4rl_data, the file structure is:

vd4rl_data └───main │ └───walker_walk │ │ └───random │ │ │ └───64px │ │ │ └───84px │ │ └───medium_replay │ │ │ ... │ └───cheetah_run │ │ ... │ └───humanoid_walk │ │ ... └───distracting │ ... └───multitask │ ...

A complete listing of the datasets in main is given in Table 5, Appendix A.

(Update, Dec 2023): Our datasets are now compatible with torch/rl!

Baselines

Environment Setup

Requirements are presented in conda environment files named conda_env.yml within each folder. The command to create the environment is: conda env create -f conda_env.yml

Alternatively, dockerfiles are located under dockerfiles, replace <<USER_ID>> in the files with your own user ID from the command id -u.

V-D4RL Main Evaluation

Example run commands are given below, given an environment type and dataset identifier:

ENVNAME=walker_walk # choice in ['walker_walk', 'cheetah_run', 'humanoid_walk'] TYPE=random # choice in ['random', 'medium_replay', 'medium', 'medium_expert', 'expert']

Offline DV2

python offlinedv2/train_offline.py --configs dmc_vision --task dmc_${ENVNAME} --offline_dir vd4rl_data/main/${ENV_NAME}/${TYPE}/64px --offline_penalty_type meandis --offline_lmbd 10 --seed 0

DrQ+BC

python drqbc/train.py task_name=offline_${ENVNAME}_${TYPE} offline_dir=vd4rl_data/main/${ENV_NAME}/${TYPE}/84px nstep=3 seed=0

DrQ+CQL

python drqbc/train.py task_name=offline_${ENVNAME}_${TYPE} offline_dir=vd4rl_data/main/${ENV_NAME}/${TYPE}/84px algo=cql cql_importance_sample=false min_q_weight=10 seed=0

BC

python drqbc/train.py task_name=offline_${ENVNAME}_${TYPE} offline_dir=vd4rl_data/main/${ENV_NAME}/${TYPE}/84px algo=bc seed=0

Distracted and Multitask Experiments

To run the distracted and multitask experiments, it suffices to change the offline directory passed to the commands above.

Note on data collection and format

We follow the image sizes and dataset format of each algorithm's native codebase. The means that Offline DV2 uses *.npz files with 64px images to store the offline data, whereas DrQ+BC uses *.hdf5 with 84px images.

The data collection procedure is detailed in Appendix B of our paper, and we provide conversion scripts in conversion_scripts. For the original SAC policies to generate the data see here. See here for distracted/multitask variants. We used seed=0 for all data generation.

Acknowledgements

V-D4RL builds upon many works and open-source codebases in both offline reinforcement learning and online pixel-based continuous control. We would like to particularly thank the authors of: - D4RL - DMControl - DreamerV2 - DrQ-v2 - LOMPO

Contact

Please contact Cong Lu or Philip Ball for any queries. We welcome any suggestions or contributions!

[^1]: The files on Google Drive may be separated into separate zip files, use this to combine.

Owner

  • Name: Cong Lu
  • Login: conglu1997
  • Kind: user
  • Location: Oxford

Reinforcement Learning PhD Student in MLRG and OxCSML @ Oxford

Citation (CITATION.cff) 

@article{
lu2023challenges,
title={Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations},
author={Cong Lu and Philip J. Ball and Tim G. J. Rudner and Jack Parker-Holder and Michael A Osborne and Yee Whye Teh},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2023},
url={https://openreview.net/forum?id=1QqIfGZOWu},
note={}
}

GitHub Events

Total
  • Issues event: 1
  • Watch event: 17
Last Year
  • Issues event: 1
  • Watch event: 17

Committers

Last synced: over 1 year ago

All Time
  • Total Commits: 23
  • Total Committers: 3
  • Avg Commits per committer: 7.667
  • Development Distribution Score (DDS): 0.13
Past Year
  • Commits: 6
  • Committers: 1
  • Avg Commits per committer: 6.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Cong Lu c****u@b****k 20
Cong Lu t****u@m****m 2
Cong Lu b****4@h****l 1
Committer Domains (Top 20 + Academic)
microsoft.com: 1 balliol.ox.ac.uk: 1

Issues and Pull Requests

Last synced: about 1 year ago

All Time
  • Total issues: 17
  • Total pull requests: 0
  • Average time to close issues: 16 days
  • Average time to close pull requests: N/A
  • Total issue authors: 12
  • Total pull request authors: 0
  • Average comments per issue: 2.35
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 1
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 1
  • Pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • gunnxx (3)
  • return-sleep (2)
  • ghost (2)
  • dwsmart32 (2)
  • yihaosun1124 (1)
  • GGOSinon (1)
  • linkct (1)
  • BrightMoonStar (1)
  • 930r91 (1)
  • Henry0132 (1)
  • BlankSHC (1)
  • wukui-muc (1)
  • levilovearch (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads: unknown
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 1
proxy.golang.org: github.com/conglu1997/v-d4rl
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 5.5%
Average: 5.6%
Dependent repos count: 5.8%
Last synced: 11 months ago