LandGriffon <!-- omit from toc -->

Homepage: github.com/Vizzuality/landgriffon

• 1. Project description
• 2. Architecture
• 3. Folder Structure
• 4. Environment variables
  • 4.1. How to use environment variables in local development
  • 4.2. Environment variables & secrects in Github Actions
  • 4.3. Environment variables & secrets in Terraform
• 5. Services
  • 5.1. API Service
  • 5.2. Client Service
  • 5.3. Marketing site
  • 5.4. Tiler server (TiTiler)
  • 5.5. Database server (Postgresql)
  • 5.6. Cache and Message Broker server (Redis)
• 6. Utilities
  • 6.1. Data package
  • 6.2. Infrastructure package
• 7. Start Up
  • 7.1. System requirements
  • 7.2. Data initialization
  • 7.3. Configure environment variables
  • 7.4. Starting all the services
• 8. Local development
  • 8.1. Using docker with debugging and hot reload
    • 8.1.1. Only frontend
    • 8.1.2. Only backend
  • 8.2. Debugging with hot reload manually
    • 8.2.1. Only frontend
    • 8.2.2. Only backend
• 9. Testing
• 10. CI/CD and Deployment
  • 10.1. GitHub Actions
    • 10.1.1. Testing workflows
    • 10.1.2. Deployment Workflows
    • 10.1.4. Manual triggering
  • 10.2. Pull request template
  • 10.3. Cloud infrastructure (Terraform)
• 11. Additional documentation?
• 12. TODO: Documentation/application improvements

1. Project description

Deforestation and water stress have a negative impact on agricultural supply chains, preventing agribusiness and food companies from becoming more sustainable. Advanced technology such as the Copernicus programme provides precise, timely and easily accessible data that improve environmental management and mitigate climate change effects.

The EU-funded LAND GRIFFON project will develop digital decision-making instruments based on Copernicus data to observe, prognoses, analyse and follow environmental impacts on the entire agricultural supply chain. These innovative instruments will support agribusiness and food enterprises in becoming more sustainable and transparent.

Related Information:

• Executive summary with an overview of how LandGriffon works.
• Full methodology with an in-depth description of every feature.

TODO: User Guide would also be useful as the Full methodology document is not exactly a user guide, only explains how the indicators work and other related information, which does not exactly fit as a user guide.

2. Architecture

This repository is a monorepo containing all the microservices of the LandGriffon platform, each organized in a top-level folder.

All services are packaged as Docker images, ensuring consistency and ease of deployment. For local development, the microservices run seamlessly using Docker Compose.

In CI, testing, staging, and production environments, the microservices are orchestrated via Kubernetes.

TODO: Add a diagram showing how all components relate to each other.

3. Folder Structure

• . (root): Root folder with configuration, documentation, startup (Docker Compose) and environment variable files.
  • .editorconfig: defines consistent formatting rules for different file types.
  • .gitignore: excludes files from version control.
  • .pre-commit-config.yaml: configures pre-commit hooks to automate code quality checks and formatting before commits.
  • .python-version: specifies the Python version to use for the LandGriffon project.
  • CHANGELOG.md: (TODO: outdated) track important changes made to LandGriffon over time.
  • docker-compose.yml: (TODO: improve) Docker Compose file to start all services in local environment.
  • ENV_VARS.md: Document explaining the environment variables needed by service in LandGriffon.
  • env.default: Template with placeholders and default values for the environment variables needed for the several services of LandGriffon.
  • LICENSE: License file.
  • Makefile: Starting point for several processes. (deprecated?)
  • README.md: this document.
• .github/: stores GitHub-specific configuration files for testing, publishing, building and deploying workflows. Also contains a pull request template.
• api/: contains the API service code.
• client/: contains the client (frontend) service code.
• cookie-traceability/: (TODO: deprecated? remove?)
• data/: contains the data and scripts needed to initialize the database.
• database/: contains a Dockerfile to build a custom Postgresql image.
• infrastructure/: Terraform files for automatic cloud provisioning.
• landing/: (TODO: deprecated? remove?)
• marketing/: current static site for LandGriffon
• redis/: contains a Dockerfile to build a custom Redis image.
• tiler/: necessary files to start up a Tiler service with TiTiler including a Dockerfile.

4. Environment variables

TODO: Not all services are included in the ENV_VARS.md, only the API.

You can find all the environment variables needed for every service in the ENV_VARS.md file.

4.1. How to use environment variables in local development

TODO: It would be nice to have this method in place.

• Create a .env file at the root of the repository out of the env.default template.
• Default values are already included.
• User-provided values need to go in the environment variables that do not have value. You can download these values from the shared secret storage. Choose the ones for local development.

4.2. Environment variables & secrects in Github Actions

TODO: Explain how are they used

4.3. Environment variables & secrets in Terraform

TODO: Explain how are they used

5. Services

The services that make up LandGriffon are:

• API service (Nest.js)
• Client service (Next.js)
• Marketing site (Next.js)
• Tiler server (TiTiler)
• Database server (Postgresql)
• Cache and queue server (Redis)

5.1. API Service

The API provides endpoints that the frontend utilizes to retrieve all necessary information and execute the calculations essential for Landgriffon's proper functionality. It ensures seamless communication between the frontend and backend, handling data processing, computations, and any required transformations to support the application's features effectively.

Its contents are in the api/ folder.

More information its README.md file.

5.2. Client Service

Web frontend client for the API.

Its contents, including a Dockerfile, are in the client/ folder.

More information its README.md file.

5.3. Marketing site

The marketing service serves as the current static site for LandGriffon. The live marketing site can be accessed at landgriffon.com.

Its contents are in the marketing/ folder.

In the Methodology section of the marketing site, you'll find two key documents: one providing an overview of how LandGriffon works and another offering a detailed explanation of its features.

• Executive summary with an overview of how LandGriffon works.
• Full methodology with an in-depth description of every feature.

More information in its README.md file.

5.4. Tiler server (TiTiler)

Tiler service that uses TiTiler, "a modern dynamic tile server built on top of FastAPI and Rasterio/GDAL".

Its contents, including a Dockerfile, are in the tiler/ folder.

More information in its README.md file.

5.5. Database server (Postgresql)

Database service containing the relational data needed for LandGriffon. It uses PostgreSQL v14 with two additional plugins:

• PostGIS v3.5.2
• PostgreSQL bindings for H3 v3.7.2

The Dockerfile and the entrypoint.sh script are in the database/ folder.

More information in this link.

5.6. Cache and Message Broker server (Redis)

Redis serves as both a message broker and a caching mechanism for certain precomputed results. While caching can be disabled through configuration environment variables, Redis is essential for message broker functionality and cannot be turned off.

More information about the use cases where Redis is needed in LandGriffon are explained in its README.md file.

6. Utilities

6.1. Data package

This directory contains seed data and scripts required to import essential data into the database, ensuring that the API has access to the necessary information for proper operation. These scripts automate the data population process, facilitating a smooth setup and maintenance of the database while supporting the APIs functionality.

Its contents are in the data/ folder.

More information about the data process used in LandGriffon are explained in the following README.md file.

6.2. Infrastructure package

The code related to setting up the cloud infrastructure is located in the infrastructure/ folder.

This setup adheres to the principle of Infrastructure as Code (IaC), which allows for the management and provisioning of computing resources through machine-readable configuration files. By using IaC, we ensure that our infrastructure is consistent, repeatable, and version-controlled.

We utilize Terraform, a powerful IaC tool, to define and provision our infrastructure on AWS. Terraform enables us to create, update, and manage AWS resources efficiently and reliably, ensuring that our infrastructure is scalable and maintainable.

For more details on how to deploy and manage the infrastructure, please refer to this README.md file.

7. Start Up

To start the application for the first time you have to import some data into the database, starting just the database and running the data seed process.

TODO: review if this is the real way of working.

Once you already have the data imported, you just can start the whole system by running:

sh docker compose up

7.1. System requirements

TODO: Complete with more dependencies?

• Python
• Node.js
• Docker

7.2. Data initialization

In order to run the application, it first needs to start a database and ingest the needed data. This is a time consuming process (a one time process for local development) that needs to be run separately, before starting the application.

How to setup the database is explained in this README.md file.

Also, for detailed information on how to import the data, read this README.md file.

7.3. Configure environment variables

TODO: Explain:

• [ ] which environment variables
• [ ] templates with default values suitable for local development (env.default)
• [ ] where to get values for the different environment variables with "real" credentials for external services (Geocoding, email, ...)
• [ ] explain where the real credentials are stored per environment (staging, production, ...)
• [ ] explain how the application uses the environment variables (maybe at service level in its corresponding README.md files)

7.4. Starting all the services

TODO: This should be ideal, not sure if it works with a proper configuration of the environment variables.

After all the previous steps you just start the whole application:

sh docker compose up

8. Local development

TODO: Explain the different possibilities to start specific services for development maybe with some environment variables we can tweak the ports exposed by the containers to be usable for debugging and hot reloading inside the container.

8.1. Using docker with debugging and hot reload

8.1.1. Only frontend

8.1.2. Only backend

8.2. Debugging with hot reload manually

8.2.1. Only frontend

8.2.2. Only backend

9. Testing

TODO: Explain the different ways of testing.

SUGGESTION: In a monorepo setup and using some monorepo utilities (Turborepo, Nx, ...) we can test all the services at the same time (frontend, backend, data?) with a single command from the root directory, and take advantage of the caching features all this tools come with.

In LandGriffon we have the following types of tests:

• Unit tests: verifies the correctness of individual code units in isolation.
• Integration tests: individual components or modules are combined and tested as a group to ensure that they work together correctly.
• API tests: validate the functionality of the API to ensure they correctly handle requests, deliver expected responses, and integrate seamlessly with other components.
• E2E tests: simulate real user scenarios to validate that all components of an application work together as intended from start to finish.

For mor information on how to trigger testing in specific modules, refer to its corresponding documentation:

• Testing the backend
• Testing the frontend
• Testing the data processing

10. CI/CD and Deployment

10.1. GitHub Actions

This document provides an overview of the GitHub Actions workflows used in the LandGriffon project.

10.1.1. Testing workflows

• testing-api.yml: Runs tests for the API service, ensuring code quality and functionality.
• testing-client.yml: Executes integration tests for the client application.
• testing-client-unit-tests.yml: Runs unit tests specifically for the client components.
• testing-data-import.yml: Validates data import functionality and integrity.
• testing-tiler.yml: Tests the tiler service for geospatial data handling.

10.1.2. Deployment Workflows

• deploy-to-kubernetes.yml: Deploys the application to a Kubernetes cluster, typically triggered on merge to main/specific branches.
• publish-docker-images.yml: Builds and pushes Docker images to a container registry.
• publish-marketing-site.yml: Builds and deploys the marketing website.
• build-database-docker.yml: Builds a Docker image for the PostgreSQL database with required extensions.

10.1.4. Manual triggering

To manually trigger a workflow:

1. Navigate to the Actions tab in the repository
2. Select the desired workflow
3. Click Run workflow
4. Select the branch and provide any required inputs

10.2. Pull request template

This GitHub pull request template provides a standardized format for contributors, requiring them to document changes with descriptions and testing instructions while enforcing quality standards through a pre-merge checklist. It ensures PRs include Jira ticket references, proper testing, CI validation, and code reviews, ultimately streamlining the review process and maintaining project quality standards.

10.3. Cloud infrastructure (Terraform)

TODO: Explain the main options taken for the deployment in the cloud:

• [ ] AWS provider options.
• [ ] GCP provider options.
• [ ] GitHub Actions secrets.

11. Additional documentation?

• Executive summary with an overview of how LandGriffon works.
• Full methodology with an in-depth description of every feature.

TODO: User guide.

12. TODO: Documentation/application improvements

• Environment variables needed for every service are, incomplete, outdated and dispersed over several files.
• The process to setup the environment variables for local development needs some clarification and a way to "do-it-without-asking-anyone" would be a nice thing to have. A template with default values, a guide on how to create a .env file and a reference to the shared secret storage for the sensible values (passwords, users, secrets...) should be enough.
• Initial data setup takes too much time so the possiblity to have a "working relevant subset" of the data for just the minimum local development would also be nice to have.