Deployment and operations

The browser edge is packaged in the Heimdall Helm chart and is disabled by default. The spark-dev1 overlay explicitly enables the edge and its dedicated owned-TLS Ingress, while the generated values file supplies the reviewed origin map. Dev1 has an immutable image tag and the required heimdall-edge-secrets Secret. The spark-prod-mexico1 edge and Ingress remain disabled. Dev has a verified Quasar player-validation contract; the production player-api contract remains unverified.

Prepared environment contract

The prepared dependency contracts are:

Environment	Player-validation URL	Contract status	Existing shared Redis Service	Redis database
spark-dev1	http://spark-dev1-quasar-svc.core-spark-test1.svc.cluster.local/getPlayer	Quasar source and live response verified; edge configured on	newton-stack-rediscluster-haproxy.newton-system.svc.cluster.local:6379	database 7 assigned to Heimdall
spark-prod-mexico1	http://micros-prod-na-01-player-api-svc.micros-prod-na-01.svc.cluster.local:8666/casinoapi/getPlayer	retained but unverified; edge disabled	spark-cache.spark-resources.svc.cluster.local:6379	database 6 proposed, not reserved

Heimdall reuses these Redis Services; do not deploy a new Redis instance for the edge. Dev1 uses database 7 because live inspection showed database 6 was already active. Keep database 7 assigned exclusively to Heimdall. The production database remains pending until separately reserved.

The exact prepared egress allowlists are:

Environment	Dependency	Namespace selector	Pod selector	Port
spark-dev1	Redis	kubernetes.io/metadata.name: newton-system	app: rediscluster-haproxy, release: newton-stack	6379
spark-dev1	Quasar	kubernetes.io/metadata.name: core-spark-test1	app.kubernetes.io/name: quasar, app.kubernetes.io/instance: spark-dev1-quasar-svc	80
spark-prod-mexico1	Redis	kubernetes.io/metadata.name: spark-resources	app.kubernetes.io/name: spark-cache, app.kubernetes.io/part-of: spark-resources, role: standalone	6379
spark-prod-mexico1	Player API	kubernetes.io/metadata.name: micros-prod-na-01	app.kubernetes.io/name: player-api-svc, app.kubernetes.io/instance: micros-prod-na-01-player-api-svc	8666

Dev1 also allows UDP and TCP port 53 to the exact CoreDNS Service CIDR 10.43.0.10/32, alongside the CoreDNS pod selector. RKE2 Canal evaluates this egress path against the Service ClusterIP before translation, so the pod selector alone does not permit DNS resolution.

Read-only cluster inspection confirmed that the corresponding CNIs enforce Kubernetes NetworkPolicy: dev uses rke2-canal with Calico policy.type=k8s; prod uses Cilium with enable-k8s-networkpolicy=true and enable-policy=default. The prepared overlays also select the observed HAProxy ingress controller and Prometheus pods exactly. The dev Quasar Service source confirms port 80 and the selectors above. These are infrastructure and source facts, not proof that a Heimdall edge pod has run in either cluster.

Read-only live contract proof confirmed that the public BFF and direct Quasar calls both returned HTTP 200 with the same positive client_player_id and global_player_id pair. The bearer and identifiers were redacted. The dev public certificate was expired, so the BFF check required local curl --insecure; internal Quasar HTTP was unaffected. This proves the dev player-response contract only. It does not prove that the configured edge is deployable or deployed, and it does not verify the production player-api target.

Kubernetes path

The dev1 browser must use exactly https://heimdall-edge.spark-test.newtoncelestial.com/api/heimdall/events. A browser cannot call a Kubernetes ClusterIP Service. The dedicated Ingress owns one hostname, terminates HTTPS before the player bearer reaches the edge, and routes only the exact analytics path. Frontend origins are CORS and trusted tenant/site mappings, not Ingress hosts. The Ingress must replace any incoming client-IP value with one dedicated X-Heimdall-Client-IP header. The edge trusts that header only from configured proxy CIDRs and uses Kubernetes Services for Redis, player validation, and internal ingest because those calls originate inside the cluster.

Generate the origin values

Raw Domainator and config-provider exports must first be normalized and reconciled. The normalized file must already contain matching host sets and only unique valid route and tenant/site entries. tools/edgevalues validates that whole file and converts it into deterministic Helm origin values; it does not filter raw records. Its default file mode emits a chart-managed ConfigMap. The public Ingress hostname remains the single environment-owned hostname above and is not generated from the frontend origins:

go run ./tools/edgevalues \
  --input /path/to/spark-dev1.normalized.json \
  --config-map-name heimdall-edge-origins \
  --output /tmp/spark-dev1.heimdall-edge-values.yaml

The input is one explicit normalized JSON file, not either raw export. Standard input is not accepted. Its contract is:

{
  "schema_version": 1,
  "environment": "spark-dev1",
  "player_url": "http://spark-dev1-quasar-svc.core-spark-test1.svc.cluster.local/getPlayer",
  "domainator_hosts": [
    {
      "host": "casino.example",
      "managed_by": "domainator",
      "tls_enabled": true,
      "ingress_class": "haproxy",
      "path": "/",
      "service": "micros-spark-test1-wl-front-svc"
    }
  ],
  "config_provider_mappings": [
    { "host": "casino.example", "tenant_id": "tenant-1", "site_id": "18" }
  ]
}

The compiler fails closed on unknown fields, unsupported environments, a non-verified player-validation URL, non-canonical hosts, duplicate hosts, non-Domainator ownership, missing TLS, a non-HAProxy class, a non-root route, or a route to the wrong frontend Service. It requires the normalized Domainator and config-provider host sets to match in both directions. Any unresolved, conflicting, or non-routed entry rejects the whole normalized input, and no fragment is written. Reconcile the raw sources and regenerate the normalized file rather than inferring or copying identity mappings.

File mode minifies and deterministically sorts the origin JSON, sets edge.originsFile.enabled=true, and creates the named ConfigMap with key origins.json. The compiler and chart conservatively reject content larger than 900 KiB, below the runtime's hard 1 MiB file limit. Chart-owned content is hashed automatically into the edge pod template so a content change causes a rollout. If an environment instead references an existing ConfigMap with create: false, it must provide a non-empty edge.originsFile.rolloutChecksum and change that opaque revision whenever the external content changes. The only inline alternative is explicit local use:

go run ./tools/edgevalues \
  --mode local-inline \
  --input /path/to/local.normalized.json \
  --output /tmp/local.heimdall-edge-values.yaml

Inline mode is rejected when the resulting HEIMDALL_EDGE_ORIGINS_JSON environment entry exceeds the safe 128 KiB single-entry bound. Environment overlays must use the default file mode.

Secrets and Redis reservation

The prepared overlays reference a Secret named heimdall-edge-secrets in the Heimdall release namespace. It must be provisioned through the approved environment secret workflow with exactly these keys:

Key	Content contract
redis-url	Full connection URL for the environment's assigned logical database; dev1 uses database 7
cache-hmac-key	Independent random HMAC material of at least 32 bytes

Do not put either value in a Helm values file, generated origin fragment, shell history, documentation, or source control. Verify the Secret and key names without printing their values. Do not reuse the ingest bearer, admin bearer, NATS credentials, Kafka credentials, or a key from another service.

Environment mode also requires a non-empty edge.secretEnv.rolloutChecksum. This is an opaque Secret version or revision, not secret material. Change it whenever either referenced Secret value rotates so Kubernetes creates a new pod template and the edge reloads the values. The enabled dev1 overlay sets an opaque checksum, while the disabled production overlay leaves it empty until provisioning is complete. The referenced Secret must still exist; a checksum does not create or validate it.

Dedicated cross-origin TLS

Dev1 uses edge.ingress.tlsMode: owned with exactly one host, heimdall-edge.spark-test.newtoncelestial.com, and a TLS entry covering that same host. The Ingress carries the cert-manager.io/cluster-issuer: letsencrypt-prod annotation so cert-manager owns certificate issuance and renewal. Rendering fails when owned mode has no TLS entry or when a host is not covered. DNS, the Ingress host, certificate SAN, frontend analyticsApiUrl, and any connect-src policy must all use the exact same hostname.

The dedicated topology has one exact route:

edge:
  ingress:
    tlsMode: owned
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    hosts:
      - host: heimdall-edge.spark-test.newtoncelestial.com
    tls:
      - secretName: heimdall-edge-spark-test1-tls-secret
        hosts:
          - heimdall-edge.spark-test.newtoncelestial.com

Do not rely on an HTTP redirect to protect the bearer: the frontend is configured with HTTPS and rejects redirects. Verify DNS ownership, certificate validity and renewal, exact SNI, direct HTTPS routing, and HSTS before activation. A dedicated cross-origin endpoint must also be added as one exact CSP connect-src source wherever the frontend response policy is owned; do not use a wildcard or parent-domain suffix.

Generic frontend chart values remain analytics_enabled: "0" and analytics_api_url: "". Local development is on with the exact relative path, while the spark-dev1 override is owned by automation-micros-argo and uses "1" plus the exact dedicated HTTPS URL. The Heimdall dev1 environment overlay is also explicitly enabled, but its current latest image tag prevents a deployable render and the referenced Secret is still required for pod startup and readiness. The frontend allowlist rejects a poisoned runtime URL before it queues the token, and requests use credentials: "omit", redirect: "error", and referrerPolicy: "no-referrer". Production and unrelated environments remain off. Enabled local configuration does not authorize a remote deployment.

Network isolation

When the edge is enabled, the chart always renders NetworkPolicies that:

• allow edge HTTP only from the selected Ingress controller
• allow edge metrics only from the selected monitoring pods
• allow edge egress only to DNS, Redis, the environment's player-validation backend, and Heimdall ingest
• allow private ingest port 8082 only from matching edge pods
• preserve the public ingest and metrics listeners for their existing callers

The exact prepared ingress allowlists are:

Environment	Traffic	Namespace selector	Pod selector
spark-dev1	Edge HTTP	kubernetes.io/metadata.name: ingress-system	app.kubernetes.io/instance: haproxy-ingress, app.kubernetes.io/name: kubernetes-ingress
spark-dev1	Edge metrics	kubernetes.io/metadata.name: monitoring-system	app.kubernetes.io/name: prometheus, operator.prometheus.io/name: kube-prometheus-stack-spar-prometheus
spark-prod-mexico1	Edge HTTP	kubernetes.io/metadata.name: haproxy-ingress-system	app.kubernetes.io/instance: haproxy-ingress, app.kubernetes.io/name: kubernetes-ingress
spark-prod-mexico1	Edge metrics	kubernetes.io/metadata.name: monitoring-system	app.kubernetes.io/name: prometheus, operator.prometheus.io/name: monitoring-kube-prometheus-prometheus

DNS egress is limited to pods labeled k8s-app: kube-dns in the namespace labeled kubernetes.io/metadata.name: kube-system, on TCP and UDP port 53. Reconfirm every selector against the target cluster immediately before any enablement because labels and controller releases have their own lifecycle.

Trusted client IP

Environment deployments must configure HEIMDALL_EDGE_TRUSTED_PROXY_CIDRS with the exact ingress proxy source ranges. The chart accepts canonical IPv4 network prefixes only and rejects addresses with host bits set. The Heimdall runtime also accepts canonical IPv6 prefixes when run outside this environment chart. The Ingress must remove any browser-supplied value and write one X-Heimdall-Client-IP header from trusted connection metadata. Heimdall rejects requests from outside those CIDRs with 403; a missing, duplicate, or malformed header fails with 503. The browser cannot request this header through CORS.

On a validation miss, Heimdall sends Quasar a bearer-authenticated GET /getPlayer with JSON body { "ip": "<canonical-client-ip>", "site_id": <trusted-positive-decimal-site-id> }. The trusted IP is not logged, stored as a raw Redis key or value, or copied into the analytics envelope. It contributes only to the HMAC cache identity so a cached validation cannot cross client IPs. Local mode may leave the CIDR list empty and use the direct socket IP.

The private ingest listener uses plain HTTP and has no application token. Do not expose the internal Service through Ingress, a public load balancer, or an unrestricted NetworkPolicy. This boundary depends on the confirmed NetworkPolicy-capable CNIs; local Docker uses its private Compose network instead.

Only private ingest port 8082 is edge-only. Public ingest and metrics retain the chart's explicit access rules for their existing callers.

The default edge egress policy selects the configured player-validation pods. An arbitrary external URL will not be reachable unless an environment owner deliberately extends the egress policy and reviews the security impact.

Important edge settings

Environment variable	Default	Purpose
HEIMDALL_EDGE_HTTP_PORT	8083	Browser edge listener
HEIMDALL_EDGE_MAX_INFLIGHT	256	Immediate load-shed budget per edge pod
HEIMDALL_EDGE_ORIGINS_FILE	chart-managed in file mode	Mounted exact origin to tenant, site, and player-validation mapping
HEIMDALL_EDGE_ORIGINS_JSON	local inline only	Explicit size-limited local alternative to the mounted file
HEIMDALL_EDGE_TRUSTED_PROXY_CIDRS	required in environment mode	Comma-separated ingress proxy source CIDRs allowed to set the dedicated client-IP header
HEIMDALL_EDGE_REDIS_URL	required secret	Redis connection used by cache and limits
HEIMDALL_EDGE_CACHE_HMAC_KEY	required secret	Token-key HMAC material, at least 32 bytes
HEIMDALL_EDGE_REDIS_TIMEOUT_MS	200	Redis operation timeout
HEIMDALL_EDGE_PLAYER_TIMEOUT_MS	2000	Player validation timeout
HEIMDALL_EDGE_PLAYER_CACHE_TTL_SECONDS	300	Valid-player cache lifetime
HEIMDALL_EDGE_PLAYER_NEGATIVE_TTL_SECONDS	5	Definitive invalid-player cache lifetime
HEIMDALL_EDGE_PLAYER_INVALID_STATUSES	401,403,422	Upstream 4xx statuses treated as definitive invalid-player results
HEIMDALL_EDGE_INGEST_URL	internal Service URL	Exact /internal/v1/events target
HEIMDALL_EDGE_INGEST_TIMEOUT_MS	1000	Internal ingest request and readiness timeout

The chart also exposes request and validation limits for token, tenant, and global scopes. Defaults are documented in helm_chart/values.yaml; adjust them from observed traffic and alerts rather than removing the bounds.

Scaling defaults

The edge HPA defaults are:

• minimum 2 replicas
• maximum 20 replicas
• target CPU utilization 60 percent
• 256 maximum in-flight requests per pod
• 250m CPU and 256 MiB memory requests
• 1 CPU and 512 MiB memory limits

The ingest and worker HPAs also keep at least two replicas. Their maxUnavailable: 1 PodDisruptionBudgets therefore preserve a serving replica during a voluntary disruption. The admin API keeps a minimum of one replica and its PodDisruptionBudget is disabled; a maxUnavailable: 1 budget on a single replica would not provide high availability. Edge, ingest, and worker availability still depends on replicas being schedulable across the cluster.

Local validation covered 1,000 distinct logged players for one tenant and 10,000 players across ten tenants. All requests returned 202 and the buffered events drained to Kafka. This is development evidence, not a production capacity guarantee. Production sizing must include representative payloads, player-validation latency, Redis latency, Kafka throughput, failure tests, and sustained rather than burst traffic.

Health, readiness, and metrics

Process	Liveness	Readiness	Metrics
edge	/health on 8083	/ready on 8083, checks Redis and internal ingest	/metrics on 9090
ingest	/health on 8080	/ready on 8080, checks JetStream state	/metrics on 9090
worker	process and metrics listener	no HTTP readiness endpoint	/metrics on 9090
admin API	/health on 8081	/ready on 8081	/metrics on 9090

Edge-specific counters are:

• heimdall_edge_requests_total{outcome}
• heimdall_edge_player_validation_total{outcome}
• heimdall_edge_player_cache_total{outcome}

Pipeline counters include accepted and rejected events, JetStream publish outcomes, Kafka production and retries, redelivery, and DLQ outcomes.

Enablement and canary gate

The following is the environment-owner checklist for a traffic canary. Secret provisioning and the dev1 Redis assignment are complete; the remaining checks still apply.

Before the enabled dev1 configuration becomes deployable:

1. Keep database 7 assigned to Heimdall on the existing dev1 Redis Service. Do not create a new Redis deployment or reuse database 7 for another service.
2. Verify that heimdall-edge-secrets contains redis-url and cache-hmac-key without printing their values, and change edge.secretEnv.rolloutChecksum whenever either value rotates.
3. Export current Domainator routes and config-provider mappings, then normalize and reconcile them into matching host sets containing only unique valid route and tenant/site entries. Run tools/edgevalues on that normalized file; any validation error rejects the whole input. Do not hand-edit or infer generated origin content. Chart-owned origin content gets an automatic checksum; an existing external ConfigMap requires a non-empty edge.originsFile.rolloutChecksum that changes with its content revision.
4. Confirm DNS ownership for heimdall-edge.spark-test.newtoncelestial.com, one owned-mode Ingress host, the exact /api/heimdall/events path, the cert-manager issuer annotation, and one TLS entry covering that host. Before sending a bearer, verify the issued certificate SAN and validity, direct HTTPS with no redirect, HSTS, and the absence of any broader route.
5. Reconfirm the ingress, monitoring, Redis, player-validation, and DNS NetworkPolicy selectors against the target cluster. For dev, verify the Quasar namespace, labels, and port 80; do not enable the unverified production player-api target.
6. Confirm the CI overlay still contains an immutable, non-latest global image tag. Environment rendering rejects an empty or mutable global tag and rejects an effective edge tag of latest.
7. Inspect the live ingress controller Service and confirm spec.externalTrafficPolicy: Local; edge.ingress.controllerExternalTrafficPolicy is a render-time attestation because this chart cannot inspect that external Service. Configure exact trusted proxy CIDRs, require the Ingress to replace X-Heimdall-Client-IP, and confirm HEIMDALL_EDGE_PLAYER_INVALID_STATUSES includes the environment's definitive invalid responses. Dev requires 422 for Quasar.
8. Keep generic frontend defaults off. In the automation-micros-argo spark-dev1 override only, set analyticsEnabled to "1" and analyticsApiUrl to exactly https://heimdall-edge.spark-test.newtoncelestial.com/api/heimdall/events. Confirm the frontend response CSP allows that exact origin in connect-src and no broader analytics destination.
9. Merge the enabled dev1 base, CI image values, and generated origin values locally, then render the full chart. Inspect the one-host Ingress, exact path, owned TLS, cert-manager annotation, header rule, ConfigMap, Secret references, mounts, rollout checksum, NetworkPolicies, Services, HPAs, and PodDisruptionBudgets.
10. Load-test cold and cached CORS preflight as well as POST. OPTIONS carries no bearer but is answered before POST in-flight and player-validation controls, so it needs separate ingress capacity and abuse monitoring. In a browser network trace, confirm that the bearer reaches only the exact host and path, cookies and referrer are absent, redirects fail, allowed origins receive the exact CORS headers, and an excluded origin receives 403 without an allow-origin header.
11. Obtain a separate deployment decision. Local rendering and tests do not authorize a cluster change.

For a separately authorized canary:

1. Start with a small approved subset of origins and logged-player traffic only; the public Heimdall hostname remains the single dedicated host.
2. Verify the cert-manager-owned certificate with SNI, the absence of a root route, exact OPTIONS and POST /api/heimdall/events routing, and rejection of an unapproved origin.
3. Confirm edge /health and /ready, Redis and player-validation outcomes, internal ingest readiness, GET /admin/v1/queues, worker retries and DLQ counters, and Kafka watermark movement.
4. Confirm that no raw bearer or trusted client IP appears in Redis keys or values, events, logs, or metrics.
5. Exercise dependency failure and recovery without exceeding the frontend's three-attempt budget.
6. Expand approved frontend origins gradually while monitoring preflight volume, the dedicated Ingress, and certificate. Roll back by switching the frontend override off, then disabling the edge Ingress and edge workload if readiness, routing, certificate, queue, or Kafka checks regress.

Operational checks

For a deployment or incident, combine the following signals:

1. edge, ingest, and admin readiness
2. edge request, cache, and validation outcomes
3. GET /admin/v1/queues for JetStream pending and acknowledgement state
4. Kafka partition high watermarks
5. worker retry and DLQ counters

A 202 at the browser boundary proves only that JetStream acknowledged the event. Queue drain and Kafka watermark movement prove downstream delivery.

Failure guide

Symptom	Likely area	Check
Browser reports a CORS failure	Origin map, dedicated endpoint, CSP, or preflight	Confirm the exact frontend origin and analytics URL, inspect CSP, and inspect the OPTIONS response
Browser never sends the POST	Failed preflight or CSP connect-src	Verify the exact endpoint is allowed by CSP and compare the preflight response headers with the requested method/header names
401 from edge	Missing, malformed, or invalid player bearer	Check frontend auth type and configured player-validation response
403 or 503 before validation	Untrusted proxy or invalid dedicated client-IP header	Check the direct peer against trusted CIDRs and confirm the Ingress replaces exactly one X-Heimdall-Client-IP header
429 from edge	Request or validation limit	Inspect Retry-After and edge outcome metrics
Edge readiness is 503	Redis or internal ingest unavailable	Check Redis connectivity and internal ingest /ready
Edge returns 503 under load	Per-pod in-flight budget or dependency failure	Inspect HPA state, edge metrics, and downstream latency
JetStream pending count grows	Worker or Kafka problem	Check worker logs, retries, Kafka connectivity, and DLQ counters
HTTP succeeds but Kafka does not advance	Downstream drain problem	Inspect queues and Kafka high watermarks instead of replaying blindly

Local deployment and verification

The commands in this section target only the local Docker and Helm-rendering workflow. Do not turn them into a real cluster deployment without a new, explicit environment-owner decision.

make local-up
make local-status
make local-edge-inject
curl --fail http://localhost:18084/ready
curl --fail http://localhost:18081/admin/v1/queues

Static verification:

go test ./...
go test -race ./...
go vet ./...
make test-helm
docker compose -f docker-compose.local.yml config --quiet

All testing and deployment performed for this integration is local. Real dependencies were inspected read-only, including the redacted dev player-contract proof above; no Heimdall edge traffic, Secret write, overlay enablement, Helm release, or canary was run there. Enabling a real environment requires completed prerequisites, explicit environment-owned values, and a separate deployment decision.