BlackMarbleR

<!-- badges: end -->

BlackMarbleR is a R package that provides a simple way to use nighttime lights data from NASA's Black Marble. Black Marble is a NASA Earth Science Data Systems (ESDS) project that provides a product suite of daily, monthly and yearly global nighttime lights. This package automates the process of downloading all relevant tiles from the NASA LAADS DAAC to cover a region of interest, converting and mosaicing the raw files (in HDF5 format) to georeferenced rasters.

• Installation
• Bearer token
• Usage
  • Setup
  • Make raster
  • Make raster across multiple time periods
  • Make map
  • Make figure of trends in nighttime lights
  • Workflow to update data
• Functions and arguments
  • Functions
  • Required Arguments
  • Optional Arguments
  • Argument only for bm_extract
• Black Marble Resources

Installation

The package can be installed via CRAN.

r install.packages("blackmarbler")

To install the development version from Github:

```r

install.packages("devtools")

devtools::install_github("worldbank/blackmarbler") ```

Bearer Token

1. Create a NASA Earth Data account account. On the top right of the webpage, click "Login" then "Earthdata Login". Then click "register" (blue button).
2. Enter the information in the registration page. You must include the following information; this information is not required to create an account, but the bearer token will not work without this information:

- Study Area
- User Type
- Organization

1. Click "Register for EarthData Login" (green button at bottom). Check your email, and click the link in the email to activate the account.
2. Go to the Earth Data Login page and login.
3. On the panel near the top, click "EULAs" then "Accept New EULAs". Accept:

- MERIS EULA
- Sentinel EULA

1. On the "Profile Home" page, you should see something like below. Information should be filled in for each category, and "Agreed To Meris EULA" and "Agreed To Sentinel-3 EULA" should be True.

1. Go to the NASA LAADS Archive and login (login button on top right). You will see a page to authorize use of Sentinel3 and Meris. Click the green "Authorize" button.

2. To obtain the bearer token, go to the Earth Data Login page and login. On the top panel, click "Generate token". On this page, click "Show Token" to see the bearer token.

1. If the bearer token ever stops working, you make need to go to the "Generate token" page (see step 8), delete any existing tokens, and generate a new token.

Programmatically retrieve token

Usage

Setup

Make raster of nighttime lights

Make raster of nighttime lights across multiple time periods

Map of nighttime lights

Trends over time

We can use the bm_extract function to observe changes in nighttime lights over time. The bm_extract function leverages the exactextractr package to aggregate nighttime lights data to polygons. Below we show trends in annual nighttime lights data across Ghana's first administrative divisions.

```r

Extract annual data

ntldf <- bmextract(roisf = roisf, product_id = "VNP46A4", date = 2012:2022, bearer = bearer)

Trends over time

ntldf |> ggplot() + geomcol(aes(x = date, y = ntlmean), fill = "darkorange") + facetwrap(~NAME1) + labs(x = NULL, y = "NTL Luminosity", title = "Ghana Admin Level 1: Annual Average Nighttime Lights") + scalexcontinuous(labels = seq(2012, 2022, 4), breaks = seq(2012, 2022, 4)) + thememinimal() + theme(strip.text = element_text(face = "bold")) ```

Workflow to update data

Functions and arguments

Functions

Required arguments

• roi_sf: Region of interest; sf polygon. Must be in the WGS 84 (epsg:4326) coordinate reference system. For bm_extract, aggregates nighttime lights within each polygon of roi_sf.

• product_id: One of the following:

  • "VNP46A1": Daily (raw)
  • "VNP46A2": Daily (corrected)
  • "VNP46A3": Monthly
  • "VNP46A4": Annual

• date: Date of raster data. Entering one date will produce a SpatRaster object. Entering multiple dates will produce a SpatRaster object with multiple bands; one band per date.

  • For product_ids "VNP46A1" and "VNP46A2", a date (eg, "2021-10-03").
  • For product_id "VNP46A3", a date or year-month (e.g., "2021-10-01", where the day will be ignored, or "2021-10").
  • For product_id "VNP46A4", year or date (e.g., "2021-10-01", where the month and day will be ignored, or 2021).

• bearer: NASA bearer token. For instructions on how to create a token, see here.

Optional arguments

• variable: Variable to used to create raster (default: NULL). To see all variable choices, set variable = "" (this will create an error message that lists all valid variables). For additional information on all variable choices, see here; for VNP46A1, see Table 3; for VNP46A2 see Table 6; for VNP46A3 and VNP46A4, see Table 9. If NULL, uses the following default variables:

  • For product_id "VNP46A1", uses DNB_At_Sensor_Radiance_500m.
  • For product_id "VNP46A2", uses Gap_Filled_DNB_BRDF-Corrected_NTL.
  • For product_ids "VNP46A3" and "VNP46A4", uses NearNadir_Composite_Snow_Free.

• qualityflagrm: Quality flag values to use to set values to NA. Each pixel has a quality flag value, where low quality values can be removed. Values are set to NA for each value in ther quality_flag_rm vector. (Default: NULL).

  • For VNP46A1 and VNP46A2 (daily data):
  • 0: High-quality, Persistent nighttime lights
  • 1: High-quality, Ephemeral nighttime Lights
  • 2: Poor-quality, Outlier, potential cloud contamination, or other issues
  • For VNP46A3 and VNP46A4 (monthly and annual data):
  • 0: Good-quality, The number of observations used for the composite is larger than 3
  • 1: Poor-quality, The number of observations used for the composite is less than or equal to 3
  • 2: Gap filled NTL based on historical data

• checkalltiles_exist: Check whether all Black Marble nighttime light tiles exist for the region of interest. Sometimes not all tiles are available, so the full region of interest may not be covered. If TRUE, skips cases where not all tiles are available. (Default: TRUE).

• interpol_na: When data for more than one date is downloaded, whether to interpolate NA values in rasters using the terra::approximate function. Additional arguments for the terra::approximate function can also be passed into bm_raster/bm_extract (eg, method, rule, f, ties, z, NA_rule). (Default: FALSE).

• h5_dir: Black Marble data are originally downloaded as h5 files. If h5_dir = NULL, the function downloads to a temporary directory then deletes the directory. If h5_dir is set to a path, h5 files are saved to that directory and not deleted. The function will then check if the needed h5 file already exists in the directory; if it exists, the function will not re-download the h5 file.

• download_method: Method to download data (h5 files) from NASA LAADS Archive: "httr" or "wget". If httr, uses the httr2 R package to download data. If wget, uses the wget command line tool. httr is fully integrated in R, while wget requires the wget system command. wget can be more efficient and can help avoid network issues. (Default: "httr").

• outputlocationtype: Where output should be stored (default: r_memory). Either:

  • r_memory where the function will return an output in R
  • file where the function will export the data as a file. For bm_raster, a .tif file will be saved; for bm_extract, a .Rds file will be saved. A file is saved for each date. Consequently, if date = c(2018, 2019, 2020), three datasets will be saved: one for each year. Saving a dataset for each date can facilitate re-running the function later and only downloading data for dates where data have not been downloaded.

If output_location_type = "file", the following arguments can be used:

• file_dir: The directory where data should be exported (default: NULL, so the working directory will be used)
• file_prefix: Prefix to add to the file to be saved. The file will be saved as the following: [file_prefix][product_id]_t[date].[tif/Rds]
• fileskipif_exists: Whether the function should first check wither the file already exists, and to skip downloading or extracting data if the data for that date if the file already exists (default: TRUE). If the function is first run with date = c(2018, 2019, 2020), then is later run with date = c(2018, 2019, 2020, 2021), the function will only download/extract data for 2021. Skipping existing files can facilitate re-running the function at a later date to download only more recent data.

• filereturnnull: Whether to return NULL instead of a output to R (SpatRaster or dataframe). When output_location_type = 'file', the function will export data to the file_dir directory. When file_return_null = FALSE, the function will also return the queried data---so the data is available in R memory. Setting file_return_null = TRUE, data will be saved to file_dir but no data will be returned by the function to R memory (default: FALSE).

• ...: Additional arguments for terra::approximate, if interpol_na = TRUE

Argument for bm_extract only

Black Marble Resources

• Academic paper
• Substack Post
• Webinar