gen3-infra-bootstrap

Bootstrap shared Gen3 infrastructure (S3/SQS/etc.) and seed AWS Secrets Manager entries consumed by External Secrets Operator (ESO) --- with a strict create-if-missing contract (no overwrites, no deletes).

• Open-source friendly: no account/region or secrets stored in the repo.

• Env-driven: PROJECT, ENV_NAME, HOSTNAME drive naming and outputs.

• Infra-aware: secrets can embed the actual bucket names and SQS URLs you create here.

• Safe by default: passwords are random alphanumeric (script-friendly). OIDC creds can be placeholders (||) until you supply real values.

Table of contents

• What this repo creates

• Strict create-if-missing

• Requirements

• Quick start

• Configuration (environment variables)

• Resources & naming

• Secrets produced (schema)

• g3auto bundles

• External Secrets Operator (ESO) wiring

• Network & WAF (kept in a separate foundation repo)

• GitHub Actions (optional)

• Security notes

• Rotation & deletion

• Repo layout

• License

What this repo creates

Infrastructure (examples; adjust as needed):

• An S3 bucket for manifests (name derived from HOSTNAME, sanitized).

• One or more SQS queues (e.g., audit queue, data-upload queue).

Secrets in AWS Secrets Manager:

• Per-service DB credential secrets for:\ index, requestor, fence, peregrine, wts, audit, manifestservice, metadata, arborist, sheepdog\ using the name pattern:

  <project>-<env>-<service>

• Optional g3auto secrets (toggle per environment):

  • <project>-<env>-metadata-g3auto
  • <project>-<env>-wts-g3auto
  • <project>-<env>-manifestservice-g3auto
  • <project>-<env>-pelicanservice-g3auto (prefer IRSA; access keys optional)
  • <project>-<env>-audit-gen3auto (YAML stored under config.yaml)
  • <project>-<env>-ssjdispatcher-creds

DB host/port are read from the existing master DB secret:\ <project>-master-<env>-rds (must contain JSON keys host, port).

Strict create-if-missing

• On each cdk deploy, a Lambda custom resource runs.

• For every secret:

  • If it exists → left as-is (no overwrite).
  • If it's missing → created with generated or supplied content.

• On cdk destroy, secrets are not deleted.

To rotate a secret, manually delete it in Secrets Manager, then re-deploy.

Requirements

• Node.js 18+ (or 20+)

• AWS CDK v2

• AWS credentials (locally or via CI OIDC) with permissions to:

  • create S3, SQS as needed
  • read the master DB secret (<project>-master-<env>-rds)
  • create new Secrets Manager secrets

Quick start

`` # 1) Configure AWS credentials (locally) or use CI OIDC export AWS_REGION=ap-southeast-2

2) Provide minimal env inputs (no account/region in code)

export PROJECT=omix3 export ENV_NAME=test export HOSTNAME=omix3.test.biocommons.org.au

Optional: feature toggles (CSV); optional OIDC creds (placeholders written if absent)

export FEATURES=metadataG3auto,wtsG3auto,manifestserviceG3auto,auditGen3auto,ssjdispatcherCreds

export WTSOIDCCLIENT_ID=...

export WTSOIDCCLIENT_SECRET=...

npm ci npm run synth npm run deploy ``

Ensure the master DB secret exists and has {"host": "...", "port": 5432} under the name <project>-master-<env>-rds.

Configuration (environment variables)

| Var | Required | Example | Notes | | --- | --- | --- | --- | | PROJECT | ✅ | omix3 | Used in names: <project>-<env>-... | | ENV_NAME | ✅ | test | Environment suffix in names | | HOSTNAME | ✅ | omix3.test.biocommons.org.au | Used in g3auto and bucket naming | | FEATURES | ➖ | *metadata*G3auto...,ssjdispatcherCreds | Enables optional g3auto bundles | | WTS_OIDC_CLIENT_ID | ➖ | abc123 | If absent, wts-g3auto writes placeholder | |WTSOIDCCLIENTSECRET| ➖ |supersecret| If absent,wts-g3autowrites placeholder | | `CDKDEFAULTACCOUNT/ CDKDEFAULT_REGION` | ➖ | set by AWS creds | CDK picks these up automatically |

Resources & naming

• Secrets: <project>-<env>-<service> (e.g., omix3-test-metadata)

• g3auto: <project>-<env>-<name> (e.g., omix3-test-manifestservice-g3auto)

• S3 bucket: manifest-<hostname-sanitized>\ (e.g., manifest-omix3-test-biocommons-org-au)

  • Hostnames are sanitized (dots → hyphens) to avoid TLS/VH quirkiness.

• SQS queues: example names audit-service-<project>-<env>, data-upload-<project>-<env>

Secrets produced (schema)

Per-service DB credentials

Name: <project>-<env>-<service>, payload:

{ "username": "<service>", "password": "<random-alnum>", "host": "<from master rds secret>", "port": "<from master rds secret>", "database": "<service>" }

Services covered (loop):

index, requestor, fence, peregrine, wts, audit, manifestservice, metadata, arborist, sheepdog

Passwords are alphanumeric for bash/URL compatibility.

g3auto bundles

All are create-if-missing. Enable via FEATURES CSV.

• metadata-g3auto

  • Creates:

    • dbcreds.json (metadata DB creds, generated password)
    • metadata.env (includes ADMIN_LOGINS=gateway:<generated>)
    • base64Authz.txt (base64 of gateway:<generated>)

• wts-g3auto

  • appcreds.json with:

    • wts_base_url → default https://<HOSTNAME>/wts/
    • fence_base_url → default https://<HOSTNAME>/user/
    • encryption_key & secret_key → random base64
    • oidc_client_id / oidc_client_secret → placeholders || if not provided

• manifestservice-g3auto

  • { "manifest_bucket_name": "<bucket>", "hostname": "<HOSTNAME>", "prefix": "" }

• pelicanservice-g3auto

  • Same shape as manifestservice; optional access keys only if explicitly supplied (prefer IRSA).

• audit-gen3auto

  • Stores YAML under "config.yaml" with your SQS URL/region.

• ssjdispatcher-creds

  • Includes AWS.region, SQS.url, jobs array with Indexd/Metadata creds (generated if not supplied), and pattern (S3 prefix).