cc-metric-collector
A node agent for measuring, processing and forwarding node level metrics
Science Score: 49.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 3 DOI reference(s) in README -
✓Academic publication links
Links to: zenodo.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.3%) to scientific vocabulary
Repository
A node agent for measuring, processing and forwarding node level metrics
Basic Info
- Host: GitHub
- Owner: ClusterCockpit
- License: mit
- Language: Go
- Default Branch: main
- Size: 2.47 MB
Statistics
- Stars: 4
- Watchers: 2
- Forks: 7
- Open Issues: 11
- Releases: 17
Metadata Files
README.md
cc-metric-collector
A node agent for measuring, processing and forwarding node level metrics. It is part of the ClusterCockpit ecosystem.
The metric collector sends (and receives) metric in the InfluxDB line protocol as it provides flexibility while providing a separation between tags (like index columns in relational databases) and fields (like data columns).
There is a single timer loop that triggers all collectors serially, collects the collectors' data and sends the metrics to the sink. This is done as all data is submitted with a single time stamp. The sinks currently use mostly blocking APIs.
The receiver runs as a go routine side-by-side with the timer loop and asynchronously forwards received metrics to the sink.
Configuration
Configuration is implemented using a single json document that is distributed over network and may be persisted as file. Supported metrics are documented here.
There is a main configuration file with basic settings that point to the other configuration files for the different components.
json
{
"sinks": "sinks.json",
"collectors" : "collectors.json",
"receivers" : "receivers.json",
"router" : "router.json",
"interval": "10s",
"duration": "1s"
}
The interval defines how often the metrics should be read and send to the sink. The duration tells collectors how long one measurement has to take. This is important for some collectors, like the likwid collector. For more information, see here.
See the component READMEs for their configuration:
Installation
$ git clone git@github.com:ClusterCockpit/cc-metric-collector.git
$ make (downloads LIKWID, builds it as static library with 'direct' accessmode and copies all required files for the collector)
$ go get (requires at least golang 1.16)
$ make
For more information, see here.
Running
$ ./cc-metric-collector --help
Usage of metric-collector:
-config string
Path to configuration file (default "./config.json")
-log string
Path for logfile (default "stderr")
-once
Run all collectors only once
Scenarios
The metric collector was designed with flexibility in mind, so it can be used in many scenarios. Here are a few:
mermaid
flowchart TD
subgraph a ["Cluster A"]
nodeA[NodeA with CC collector]
nodeB[NodeB with CC collector]
nodeC[NodeC with CC collector]
end
a --> db[(Database)]
db <--> ccweb("Webfrontend")
mermaid
flowchart TD
subgraph a [ClusterA]
direction LR
nodeA[NodeA with CC collector]
nodeB[NodeB with CC collector]
nodeC[NodeC with CC collector]
end
subgraph b [ClusterB]
direction LR
nodeD[NodeD with CC collector]
nodeE[NodeE with CC collector]
nodeF[NodeF with CC collector]
end
a --> ccrecv{"CC collector as receiver"}
b --> ccrecv
ccrecv --> db[("Database1")]
ccrecv -.-> db2[("Database2")]
db <-.-> ccweb("Webfrontend")
Contributing
The ClusterCockpit ecosystem is designed to be used by different HPC computing centers. Since configurations and setups differ between the centers, the centers likely have to put some work into the cc-metric-collector to gather all desired metrics.
You are free to open an issue to request a collector but we would also be happy about PRs.
Contact
Owner
- Name: ClusterCockpit
- Login: ClusterCockpit
- Kind: organization
- Repositories: 8
- Profile: https://github.com/ClusterCockpit
GitHub Events
Total
- Create event: 26
- Release event: 3
- Issues event: 9
- Delete event: 14
- Issue comment event: 9
- Push event: 106
- Pull request event: 66
- Fork event: 2
Last Year
- Create event: 26
- Release event: 3
- Issues event: 9
- Delete event: 14
- Issue comment event: 9
- Push event: 106
- Pull request event: 66
- Fork event: 2
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 19
- Total pull requests: 151
- Average time to close issues: about 1 month
- Average time to close pull requests: about 1 month
- Total issue authors: 7
- Total pull request authors: 11
- Average comments per issue: 1.0
- Average comments per pull request: 0.07
- Merged pull requests: 121
- Bot issues: 0
- Bot pull requests: 13
Past Year
- Issues: 7
- Pull requests: 48
- Average time to close issues: 28 days
- Average time to close pull requests: 4 days
- Issue authors: 4
- Pull request authors: 4
- Average comments per issue: 0.86
- Average comments per pull request: 0.08
- Merged pull requests: 43
- Bot issues: 0
- Bot pull requests: 13
Top Authors
Issue Authors
- TomTheBear (8)
- behnle (4)
- agemuend (2)
- Autumn-Roy (2)
- moebiusband73 (1)
- brinkcoder (1)
- ipatix (1)
Pull Request Authors
- TomTheBear (120)
- brinkcoder (16)
- dependabot[bot] (12)
- ho-ob (6)
- oscarminus (3)
- iamlouk (3)
- sanjay7178 (2)
- Obihoernchen (2)
- mehsoy (1)
- fodinabor (1)
- moebiusband73 (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 2
- Total downloads: unknown
-
Total dependent packages: 1
(may contain duplicates) -
Total dependent repositories: 0
(may contain duplicates) - Total versions: 24
proxy.golang.org: github.com/clustercockpit/cc-metric-collector
- Documentation: https://pkg.go.dev/github.com/clustercockpit/cc-metric-collector#section-documentation
- License: mit
-
Latest release: v0.7.2
published 8 months ago
Rankings
proxy.golang.org: github.com/ClusterCockpit/cc-metric-collector
- Homepage: https://github.com/ClusterCockpit/cc-metric-collector
- Documentation: https://pkg.go.dev/github.com/ClusterCockpit/cc-metric-collector#section-documentation
- License: MIT
-
Latest release: v0.7.2
published 8 months ago
Rankings
Dependencies
- actions/checkout v2 composite
- actions/download-artifact v2 composite
- actions/upload-artifact v2 composite
- softprops/action-gh-release v1 composite
- actions/checkout v2 composite
- actions/setup-go v3 composite
- github.com/ClusterCockpit/cc-units v0.3.0
- github.com/ClusterCockpit/go-rocm-smi v0.3.0
- github.com/NVIDIA/go-nvml v0.11.6-0
- github.com/PaesslerAG/gval v1.2.1
- github.com/apapsch/go-jsonmerge/v2 v2.0.0
- github.com/beorn7/perks v1.0.1
- github.com/cespare/xxhash/v2 v2.2.0
- github.com/deepmap/oapi-codegen v1.12.4
- github.com/golang/protobuf v1.5.2
- github.com/google/uuid v1.3.0
- github.com/gorilla/mux v1.8.0
- github.com/influxdata/influxdb-client-go/v2 v2.12.1
- github.com/influxdata/line-protocol v0.0.0-20210922203350-b1ad95c89adf
- github.com/matttproud/golang_protobuf_extensions v1.0.4
- github.com/nats-io/nats-server/v2 v2.8.4
- github.com/nats-io/nats.go v1.21.0
- github.com/nats-io/nkeys v0.3.0
- github.com/nats-io/nuid v1.0.1
- github.com/pkg/errors v0.9.1
- github.com/prometheus/client_golang v1.14.0
- github.com/prometheus/client_model v0.3.0
- github.com/prometheus/common v0.37.0
- github.com/prometheus/procfs v0.8.0
- github.com/shopspring/decimal v1.3.1
- github.com/stmcginnis/gofish v0.13.0
- github.com/tklauser/go-sysconf v0.3.11
- github.com/tklauser/numcpus v0.6.0
- golang.org/x/crypto v0.3.0
- golang.org/x/net v0.3.0
- golang.org/x/sys v0.3.0
- google.golang.org/protobuf v1.28.1
- 581 dependencies