Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/clustercockpit/cc-backend

Web frontend and API backend server for ClusterCockpit Monitoring Framework

https://github.com/clustercockpit/cc-backend

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    3 of 14 committers (21.4%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.6%) to scientific vocabulary

Keywords

backend golang graphql hpc monitoring rest-api sveltejs

Keywords from Contributors

profiling projection benchmarking archival threading interactive generic sequences performance-engineering performance-analysis
Last synced: 5 months ago · JSON representation

Repository

Web frontend and API backend server for ClusterCockpit Monitoring Framework

Basic Info
Statistics
  • Stars: 20
  • Watchers: 7
  • Forks: 17
  • Open Issues: 29
  • Releases: 12
Topics
backend golang graphql hpc monitoring rest-api sveltejs
Created almost 7 years ago · Last pushed 5 months ago
Metadata Files
Readme License

README.md

NOTE

Please have a look at the Release Notes for breaking changes!

ClusterCockpit REST and GraphQL API backend

Build

This is a Golang backend implementation for a REST and GraphQL API according to the ClusterCockpit specifications. It also includes a web interface for ClusterCockpit. This implementation replaces the previous PHP Symfony based ClusterCockpit web interface. The reasons for switching from PHP Symfony to a Golang based solution are explained here.

Overview

This is a Golang web backend for the ClusterCockpit job-specific performance monitoring framework. It provides a REST API for integrating ClusterCockpit with an HPC cluster batch system and external analysis scripts. Data exchange between the web front-end and the back-end is based on a GraphQL API. The web frontend is also served by the backend using Svelte components. Layout and styling are based on Bootstrap 5 using Bootstrap Icons.

The backend uses SQLite 3 as a relational SQL database by default. Optionally it can use a MySQL/MariaDB database server. While there are metric data backends for the InfluxDB and Prometheus time series databases, the only tested and supported setup is to use cc-metric-store as the metric data backend. Documentation on how to integrate ClusterCockpit with other time series databases will be added in the future.

Completed batch jobs are stored in a file-based job archive according to this specification. The backend supports authentication via local accounts, an external LDAP directory, and JWT tokens. Authorization for APIs is implemented with JWT tokens created with public/private key encryption.

You find a detailed documentation on the ClusterCockpit Webpage.

Build requirements

ClusterCockpit requires a current version of the golang toolchain and node.js. You can check go.mod to see what is the current minimal golang version needed. Homebrew and Archlinux usually have current golang versions. For other Linux distros this often means that you have to install the golang compiler yourself. Fortunately, this is easy with golang. Since much of the functionality is based on the Go standard library, it is crucial for security and performance to use a current version of golang. In addition, an old golang toolchain may limit the supported versions of third-party packages.

How to try ClusterCockpit with a demo setup

We provide a shell script that downloads demo data and automatically starts the cc-backend. You will need wget, go, node, npm in your path to start the demo. The demo downloads 32MB of data (223MB on disk).

sh git clone https://github.com/ClusterCockpit/cc-backend.git cd ./cc-backend ./startDemo.sh

You can also try the demo using the latest release binary. Create a folder and put the release binary cc-backend into this folder. Execute the following steps:

shell ./cc-backend -init vim config.json (Add a second cluster entry and name the clusters alex and fritz) wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-demo.tar tar xf job-archive-demo.tar ./cc-backend -init-db -add-user demo:admin:demo -loglevel info ./cc-backend -server -dev -loglevel info

You can access the web interface at http://localhost:8080. Credentials for login are demo:demo. Please note that some views do not work without a metric backend (e.g., the Analysis, Systems and Status views).

How to build and run

There is a Makefile to automate the build of cc-backend. The Makefile supports the following targets:

  • make: Initialize var directory and build svelte frontend and backend binary. Note that there is no proper prerequisite handling. Any change of frontend source files will result in a complete rebuild.
  • make clean: Clean go build cache and remove binary.
  • make test: Run the tests that are also run in the GitHub workflow setup.

A common workflow for setting up cc-backend from scratch is:

```sh git clone https://github.com/ClusterCockpit/cc-backend.git

Build binary

cd ./cc-backend/ make

EDIT THE .env FILE BEFORE YOU DEPLOY (Change the secrets)!

If authentication is disabled, it can be empty.

cp configs/env-template.txt .env vim .env

cp configs/config.json . vim config.json

Optional: Link an existing job archive:

ln -s ./var/job-archive

This will first initialize the job.db database by traversing all

meta.json files in the job-archive and add a new user.

./cc-backend -init-db -add-user :admin:

Start a HTTP server (HTTPS can be enabled in the configuration, the default port is 8080).

The --dev flag enables GraphQL Playground (http://localhost:8080/playground) and Swagger UI (http://localhost:8080/swagger).

./cc-backend -server -dev

Show other options:

./cc-backend -help ```

Project file structure

  • api/ contains the API schema files for the REST and GraphQL APIs. The REST API is documented in the OpenAPI 3.0 format in ./api/openapi.yaml.
  • cmd/cc-backend contains main.go for the main application.
  • configs/ contains documentation about configuration and command line options and required environment variables. A sample configuration file is provided.
  • docs/ contains more in-depth documentation.
  • init/ contains an example of setting up systemd for production use.
  • internal/ contains library source code that is not intended for use by others.
  • pkg/ contains Go packages that can be used by other projects.
  • tools/ Additional command line helper tools.
    • archive-manager Commands for getting infos about and existing job archive.
    • convert-pem-pubkey Tool to convert external pubkey for use in cc-backend.
    • gen-keypair contains a small application to generate a compatible JWT keypair. You find documentation on how to use it here.
  • web/ Server-side templates and frontend-related files:
    • frontend Svelte components and static assets for the frontend UI
    • templates Server-side Go templates
  • gqlgen.yml Configures the behaviour and generation of gqlgen.
  • startDemo.sh is a shell script that sets up demo data, and builds and starts cc-backend.

Owner

  • Name: ClusterCockpit
  • Login: ClusterCockpit
  • Kind: organization

GitHub Events

Total
  • Create event: 26
  • Release event: 6
  • Issues event: 81
  • Watch event: 6
  • Delete event: 16
  • Issue comment event: 66
  • Push event: 280
  • Pull request review event: 2
  • Pull request event: 127
  • Fork event: 3
Last Year
  • Create event: 26
  • Release event: 6
  • Issues event: 81
  • Watch event: 6
  • Delete event: 16
  • Issue comment event: 66
  • Push event: 280
  • Pull request review event: 2
  • Pull request event: 127
  • Fork event: 3

Committers

Last synced: 8 months ago

All Time
  • Total Commits: 1,463
  • Total Committers: 14
  • Avg Commits per committer: 104.5
  • Development Distribution Score (DDS): 0.592
Past Year
  • Commits: 405
  • Committers: 7
  • Avg Commits per committer: 57.857
  • Development Distribution Score (DDS): 0.36
Top Committers
Name Email Commits
Christoph Kluge c****e@f****e 597
Jan Eitzinger j****n@m****g 377
Jan Eitzinger j****g@g****m 229
Lou Knauer l****r@g****e 195
Michael Panzlaff m****f@f****e 16
Pay Giesselmann g****n@d****e 15
dependabot[bot] 4****] 10
Aditya Ujeniya a****j@g****m 6
Michael Schwarz s****z@u****e 6
exterr2f R****k@r****e 6
Joachim Meyer j****r@c****e 3
Alex a****s@m****e 1
Thomas Roehl T****l@g****m 1
brinkcoder R****k@r****e 1
Committer Domains (Top 20 + Academic)
fau.de: 2 ruhr-uni-bochum.de: 1 mail.upb.de: 1 cs.uni-saarland.de: 1 rub.de: 1 uni-paderborn.de: 1 dkrz.de: 1 gmx.de: 1 moebiusband.org: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 172
  • Total pull requests: 216
  • Average time to close issues: 3 months
  • Average time to close pull requests: 5 days
  • Total issue authors: 18
  • Total pull request authors: 10
  • Average comments per issue: 1.27
  • Average comments per pull request: 0.2
  • Merged pull requests: 193
  • Bot issues: 0
  • Bot pull requests: 15
Past Year
  • Issues: 44
  • Pull requests: 98
  • Average time to close issues: 23 days
  • Average time to close pull requests: 2 days
  • Issue authors: 10
  • Pull request authors: 6
  • Average comments per issue: 1.02
  • Average comments per pull request: 0.22
  • Merged pull requests: 84
  • Bot issues: 0
  • Bot pull requests: 9
View more stats
Top Authors
Issue Authors
  • moebiusband73 (82)
  • giesselmann (20)
  • spacehamster87 (14)
  • oscarminus (13)
  • aw32 (8)
  • TomTheBear (5)
  • brinkcoder (5)
  • fodinabor (5)
  • behnle (4)
  • Autumn-Roy (4)
  • weijianwen (3)
  • Armagetron (2)
  • ipatix (2)
  • KParas (1)
  • Behtsis (1)
Pull Request Authors
  • moebiusband73 (117)
  • spacehamster87 (86)
  • dependabot[bot] (18)
  • giesselmann (12)
  • brinkcoder (12)
  • ipatix (5)
  • oscarminus (3)
  • fodinabor (3)
  • sanjay7178 (3)
  • TomTheBear (1)
  • adityauj (1)
Top Labels
Issue Labels
enhancement (63) backend (58) frontend (51) bug (23) completed (8) v1.4 (2) duplicate (1)
Pull Request Labels
dependencies (18) go (13) javascript (2) enhancement (1) backend (1)

Packages

  • Total packages: 1
  • Total downloads: unknown
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 16
proxy.golang.org: github.com/ClusterCockpit/cc-backend
  • Versions: 16
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 7.0%
Forks count: 7.0%
Average: 8.1%
Stargazers count: 9.0%
Dependent repos count: 9.3%
Last synced: 5 months ago