Log Parser for Karpenter (LogParserForKarpenter)

Log Parser for Karpenter is a Golang based command line tool which can parse the output of Karpenter Controller logs, tested against logs from Karpenter versions: * v0.37.7 * * v1.0.x * * v1.1.x * v1.2.x * v1.3.x * v1.4.x

* Note: "messageKind":"spot_interrupted" is first supported with Karpenter version v1.1.x, so LogParserForKarpenter (lp4k) does not provide interruptiontime and interruptionkind in earlier versions

It allows using either STDIN (for example for piping live Karpenter controller logs) or multiple Karpenter log files as input and will print CSV style formatted output of nodeclaim data ordered by createdtime to STDOUT, so one can easily redirect it into a file and analyse with tools like Amazon QuickSight or Microsoft Excel.

If neither STDIN nor log files are used as input, lp4k will attach to a running K8s/EKS cluster and parses Karpenter logs (streamed logs, similar to kubectl logs -f using LP4KKARPENTERNAMESPACE and LP4KKARPENTERLABEL) and creates a ConfigMap lp4k-cm-<date> in same namespace, which gets updated every LP4KCMUPDATE_FREQ.

K8s handling can be configured using the following OS environment variables:

| Environment variable | Default value | Description | ------------- | ------------- | ------------- | | LP4KKARPENTERNAMESPACE | "karpenter" | K8s namespace where Karpenter controller is running | LP4KKARPENTERLABEL | "app.kubernetes.io/name=karpenter" | Karpenter controller K8s pod labels | LP4KCMUPDATEFREQ | "30s" | update frequency of ConfigMap and STDOUT if enabled (default), must be valid Go time.Duration string like "30s" or 2m30s" | LP4KCMPREFIX | "lp4k-cm" | nodeclaim ConfigMap prefix, if KARPENTERLP4KCMOVERRIDE=false or ConfigMap name, if KARPENTERLP4KCMOVERRIDE=true | LP4KCMOVERRIDE | "false" | determines, if ConfigMap will just use prefix and will be overriden upon every start of lp4k | LP4KNODECLAIMPRINT | "true" | print nodeclaim information every KARPENTERCMUPDATEFREQ to STDOUT

* Note: In mode LP4K_CM_OVERRIDE=true lp4k will read existing nodeclaim data from ConfigMap specified by LP4KCMPREFIX

Use: bash LP4K_CM_UPDATE_FREQ=10s ./bin/lp4k or permanently ```bash export LP4KCMUPDATE_FREQ=10s ./bin/lp4k

```

To start using LogParserForKarpenter

Just run: bash make A binary lp4k for your OS and platform is build in directory bin.

Then run it like: bash ./bin/lp4k sample-input.txt or bash ./bin/lp4k <Karpenter log output file 1> [... <Karpenter log output file n>] or bash kubectl logs -n kube-system deployments/karpenter --all-pods=true [-f] | ./lp4k or for attaching to K8s/EKS cluster in current KUBECONFIG context bash ./bin/lp4k The sample output file sample-multi-file-klp-output.csv shows all exposed nodeclaim information and can be used as a sample starter to build analysis on top of it. * Note: lp4k will recognise new nodeclaims and populate its internal structures first when Karpenter controller logs show a logline containing "message":"created nodeclaim". That means after a Karpenter controller restart and a subsequent and required restart lp4k will not recognise already existing nodeclaims and shows No results - empty "nodeclaim" map

lp4kcm

lp4kcm is a helper tool to display lp4k ConfigMap data in same CSV format.

Just run: bash make tools A binary lp4kcm for your OS and platform is build in directory bin.

Then run it like: bash ./bin/lp4k <lp4k ConfigMap name 1> [... <lp4k ConfigMap name n>]

Analyse LogParserForKarpenter output

The simplest way for analysis is to use the output and parse it using standard Linux utilities like awk, cut and grep. ```console

indexed header

$ head -1 sample-multi-file-klp-output.csv nodeclaim[1],createdtime[2],nodepool[3],instancetypes[4],launchedtime[5],providerid[6],instancetype[7],zone[8],capacitytype[9],registeredtime[10],k8snodename[11],initializedtime[12],nodereadytime[13],nodereadytimesec[14],disruptiontime[15],disruptionreason[16],disruptiondecision[17],disruptednodecount[18],replacementnodecount[19],disruptedpodcount[20],annotationtime[21],annotation[22],tainttime[23],taint[24],interruptiontime[25],interruptionkind[26],deletedtime[27],nodeterminationtime[28],nodeterminationtimesec[29],nodelifecycletime[30],nodelifecycletimesec[31],initialized[32],deleted[33]

print nodeclaim[index/column=1], nodereadytime[13],nodereadytimesec[14]

$ cat sample-multi-file-klp-output.csv | awk -F ',' '{print $1,$13,$14 }' | more nodeclaim[1] nodereadytime[13] nodereadytimesec[14] spot-844xp 1m18.591s 78.6 default-brbk4 0s 0.0 default-lpc62 50.935s 50.9 default-j4lj7 43.617s 43.6 default-mpz2w 46.277s 46.3 default-8sxj9 36.714s 36.7 default-zgb22 35.008s 35.0 local-storage-raid-al2023-kdsvk 41.839s 41.8 local-storage-raid-al2023-tq7v5 43.922s 43.9 local-storage-raid-al2023-9kx8z 48.781s 48.8 ...

search for specific value sof a specific nodeclaim

$ cat sample-multi-file-klp-output.csv | awk -F ',' '/local-storage-raid-al2023-nccxt/ { print $1,$13,$14 }' local-storage-raid-al2023-nccxt 1m8.447s 68.4

combine directly with lp4k and raw Karpenter controller logs

$ ./bin/lp4k karpenter-log-0.37.7.txt | awk -F ',' '{ print $1,$13,$14 }' nodeclaim(1) nodereadytime(13) nodereadytimesec(14) default-brbk4 0s 0.0 default-lpc62 50.935s 50.9 default-j4lj7 43.617s 43.6 default-mpz2w 46.277s 46.3 ``` Amazon QuickSight or Microsoft Excel are possible choices to use the CSV output for advanced analysis to create graphs and/or pivot tables.

Contributing

We welcome contributions to lp4k ! Please see CONTRIBUTING.md for more information on how to report bugs or submit pull requests.

Code of conduct

This project has adopted the Amazon Open Source Code of Conduct. See CODEOFCONDUCT.md for more details.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache 2.0 License.