cropharvest
Open source remote sensing dataset with benchmarks
Science Score: 33.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: zenodo.org -
✓Committers with academic emails
1 of 5 committers (20.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (15.6%) to scientific vocabulary
Repository
Open source remote sensing dataset with benchmarks
Basic Info
Statistics
- Stars: 202
- Watchers: 7
- Forks: 48
- Open Issues: 35
- Releases: 9
Metadata Files
README.md
CropHarvest
CropHarvest is an open source remote sensing dataset for agriculture with benchmarks. It collects data from a variety of agricultural land use datasets and remote sensing products.
The dataset consists of 95,186 datapoints, of which 33,205 (35%) have multiclass labels. All other datapoints only have binary crop / non-crop labels.
70,213 (74%) of these labels are paired with remote sensing and climatology data, specifically Sentinel-2, Sentinel-1, the SRTM Digital Elevation Model and ERA 5 climatology data.
21 datasets are aggregated into CropHarvest - these are documented here.
More details about CropHarvest and the benchmarks are available in this paper.
Pipeline
The code in this repository
1) combines the constituent datasets into a single geoJSON file,
2) exports the associated satellite data from Earth Engine,
3) combines both datasets to create (X,y) training tuples and
4) exposes those tuples via a Dataset object.
The pipeline through which this happens is shown below:
All blue boxes are associated with code in this repository. Anything green is data accessible via Zenodo. By default, the Zenodo data will get downloaded to the data folder - the data folder's Readme has more information about the exact structure of the data.
There are unique cases where you may need to use the EarthEngineExporter directly, these use cases are demonstrated in the demo_exporting_data.ipynb notebook.
Installation
Linux and MacOS users can install the latest version of CropHarvest with the following command:
bash
pip install cropharvest
Windows users must install the CropHarvest within a conda environment to ensure all dependencies are installed correctly:
bash
conda install 'fiona>=1.5' 'rasterio>=1.2.6'
pip install cropharvest
In addition, it may be necessary to install h5py using conda as well (conda install 'h5py>3.7.0') prior to pip-installing cropharvest.
Getting started 
See the demo.ipynb notebook for an example on how to download the data from Zenodo and train a random forest against this data.
For more examples of models trained against this dataset, see the benchmarks.
Contributing
If you would like to contribute a dataset, please see the contributing readme.
~~FAQ~~ Questions asked at least once
How do I use CropHarvest for a specific geography?
All the data is accessible through the `cropharvest.datasets.CropHarvest` object. The main parameters which you might be interested in manipulating are controllable through a `cropharvest.datasets.Task`, which takes as input the following parameters: - A bounding box, which defines the spatial boundaries of the labels retrieves - A target label, which defines the class of the positive labels (if this is left to `None`, then the positive class will be crops and the negative class will be non-crops) - A boolean defining whether or not to balance the crops and non-crops in the negative class - A test_identifier string, which tells the dataset whether or not to retrieve a file from the `test_features` folder and return it as the test data. So if I wanted to use this to train a model to identify crop vs. non crop in France, I might do it like this: ```python from sklearn.ensemble import RandomForestClassifier from cropharvest.datasets import Task, CropHarvest from cropharvest.countries import get_country_bbox # folder into which the data will be downloaded / already exists data_dir = "data" # get_country_bbox returns a list of bounding boxes. # the one representing Metropolitan France is the 2nd box metropolitan_france_bbox = get_country_bbox("France")[1] task = Task(bounding_box=metropolitan_france_bbox, normalize=True) my_dataset = CropHarvest(data_dir, task) X, y = my_dataset.as_array(flatten_x=True) model = RandomForestClassifier(random_state=0) model.fit(X, y) ```How do I load a specific pixel timeseries?
The specific use case here is to retrieve NDVI values for a specific row in the `labels.geojson`. Here is how you might go about doing that: Firstly, I will load the geosjon. I'll do it through a `CropHarvestLabels` object, which is just a wrapper around the geojson but provides some nice utility functions. ```python >>> from cropharvest.datasets import CropHarvestLabels >>> >>> labels = CropHarvestLabels("cropharvest/data") >>> labels_geojson = labels.as_geojson() >>> labels_geojson.head() harvest_date planting_date ... is_test geometry 0 None None ... False POLYGON ((37.08252 10.71274, 37.08348 10.71291... 1 None None ... False POLYGON ((37.08721 10.72398, 37.08714 10.72429... 2 None None ... False POLYGON ((37.08498 10.71371, 37.08481 10.71393... 3 None None ... False POLYGON ((37.09021 10.71320, 37.09014 10.71341... 4 None None ... False POLYGON ((37.08307 10.72160, 37.08281 10.72197... [5 rows x 13 columns] ``` Now, I can use the `labels` object to retrieve the filepath of the processed satellite data for each row in the labels geojson: ```python >>> path_to_file = labels._path_from_row(labels_geojson.iloc[0]) ``` This processed satellite data is stored as `h5py` files, so I can load it up as follows: ```python >>> import h5py >>> h5py_file = h5py.File(path_to_file, "r") >>> x = h5py_file.get("array")[:] >>> x.shape (12, 18) ``` The shape of `x` represents 12 timesteps and 18 bands. To retrieve the band I am interested in: ```python >>> from cropharvest.bands import BANDS >>> x[:, BANDS.index("NDVI")] array([0.28992072, 0.28838343, 0.26833579, 0.22577633, 0.27138986, 0.06584114, 0.498998 , 0.50147203, 0.50437743, 0.44326343, 0.33735849, 0.28375967]) ``` These are 12 NDVI values, corresponding to the 12 months captured in this timeseries. To find out exactly which month each timestep represents, I can do ```python >>> labels_geojson.iloc[0].export_end_date '2021-02-01T00:00:00' ``` Wich tells me that the last timestep represents January 2021. I can work backwards from there.What is the data format?
The structure of the different data files is now described in depth in the data folder's [Readme](https://github.com/nasaharvest/cropharvest/blob/main/data/README.md)License
CropHarvest has a Creative Commons Attribution-ShareAlike 4.0 International license.
Citation
If you use CropHarvest in your research, please use the following citation:
@inproceedings{
tseng2021cropharvest,
title={CropHarvest: A global dataset for crop-type classification},
author={Gabriel Tseng and Ivan Zvonkov and Catherine Lilian Nakalembe and Hannah Kerner},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=JtjzUXPEaCu}
}
Owner
- Name: nasaharvest
- Login: nasaharvest
- Kind: organization
- Repositories: 12
- Profile: https://github.com/nasaharvest
GitHub Events
Total
- Issues event: 5
- Watch event: 35
- Issue comment event: 3
- Fork event: 9
Last Year
- Issues event: 5
- Watch event: 35
- Issue comment event: 3
- Fork event: 9
Committers
Last synced: 10 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Gabriel Tseng | g****g@m****a | 275 |
| ivanzvonkov | i****v@g****m | 136 |
| Adebowale | d****7@g****m | 19 |
| alexhernandezgarcia | a****a@g****m | 1 |
| hannah-rae | h****r@g****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 10 months ago
All Time
- Total issues: 54
- Total pull requests: 83
- Average time to close issues: 3 months
- Average time to close pull requests: 9 days
- Total issue authors: 22
- Total pull request authors: 7
- Average comments per issue: 0.85
- Average comments per pull request: 0.94
- Merged pull requests: 72
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 3
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 3
- Pull request authors: 0
- Average comments per issue: 0.33
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- gabrieltseng (15)
- ivanzvonkov (9)
- cnakalembe (6)
- Joaquin-Gajardo (3)
- hannah-rae (3)
- yichiac (1)
- pavanthanay (1)
- ck-amrahd (1)
- LL0912 (1)
- jlpearso (1)
- EhrmannS (1)
- wllih (1)
- ofcosar (1)
- fmenat (1)
- dks4-hw (1)
Pull Request Authors
- gabrieltseng (53)
- ivanzvonkov (23)
- adebowaledaniel (4)
- rishisinhanj (1)
- hannah-rae (1)
- alexhernandezgarcia (1)
- TrellixVulnTeam (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 684 last-month
- Total dependent packages: 1
- Total dependent repositories: 2
- Total versions: 10
- Total maintainers: 1
pypi.org: cropharvest
Open source remote sensing dataset with benchmarks
- Homepage: https://github.com/nasaharvest/cropharvest
- Documentation: https://cropharvest.readthedocs.io/
- License: Other/Proprietary License
-
Latest release: 0.7.0
published about 2 years ago
Rankings
Maintainers (1)
Dependencies
- dill *
- learn2learn ==0.1.3
- pytorch-lightning ==0.7.1
- torch *
- black *
- earthengine-api >=0.1.271
- flake8 ==3.9.2
- jupyter *
- mypy *
- pytest-flake8 *
- pytest-mock *
- geopandas ==0.9.0
- h5py >=3.1.0,
- h5py *
- https *
- openpyxl >=2.5.9
- rasterio >=1.2.6
- scikit-learn >=0.22.2
- tqdm >=4.61.1
- xarray >=0.16.2
- actions/cache v2 composite
- actions/checkout v2 composite
- actions/checkout master composite
- actions/setup-python v2 composite
- actions/setup-python v1 composite
- conda-incubator/setup-miniconda v2 composite
- pypa/gh-action-pypi-publish master composite
- actions/checkout master composite
- actions/setup-python v1 composite
- pypa/gh-action-pypi-publish master composite