Beacon operations
Ownership and trust boundary
Beacon accepts only envelope version 1, body schema version 1, event type
player.casino-presence.v1, and source online-services-front. Tenant, site,
and player IDs are trusted because Heimdall replaces browser values before
publishing. metadata.received_at controls ordering and expiry;
metadata.occurred_at is diagnostic only. Records more than five minutes in the
future are rejected so a bad timestamp cannot keep a player online indefinitely.
The source field is forgeable and is not authentication. The repository's
current Kafka configuration uses plaintext broker listeners and supplies no
Kafka credentials. The required production boundary is:
- Heimdall can produce only to
heimdall.player-casino-presence.v1. - Beacon can read that topic with group
beacon.player-state.v1. - Beacon can produce only to
beacon.player-casino-presence.dlq.v1.
Enforce that boundary with an authenticated Kafka listener and ACLs. Until that platform work exists, any workload with write access to the broker listener can forge the trusted envelope fields.
Kafka topics and bootstrap
input topic: heimdall.player-casino-presence.v1
consumer group: beacon.player-state.v1
DLQ topic: beacon.player-casino-presence.dlq.v1
The input topic is a Heimdall-owned prerequisite. Beacon does not create it; if
it is absent, the consumer cannot establish a healthy group session and
/ready stays unavailable.
Beacon currently creates a registered DLQ when it is missing. DLQ partition and replication settings apply only at creation and do not alter an existing topic. Move DLQ provisioning, retention, and ACL ownership to Kafka infrastructure together; then remove Beacon's runtime cluster-admin requirement.
For a topic with no committed offset, Beacon starts at Kafka's latest offset. Active sessions materialize on their next heartbeat instead of replaying stale retained presence. A future projector that needs history must define an explicit backfill or separate consumer lineage.
Event contract
| Field | Rule |
|---|---|
metadata.event_id | UUID; repeated last event for a site is a no-op even at a new Kafka offset |
metadata.received_at | Trusted ordering and expiry time; RFC3339 and at most five minutes ahead of Beacon |
metadata.occurred_at | Diagnostic browser time; never extends expiry |
body.tenant_id, body.site_id, body.user.user_id | Positive decimal strings supplied by Heimdall |
event_payload.presence_session_id | UUID identifying one browser presence lifecycle |
event_payload.action | opened, heartbeat, or closed |
event_payload.close_reason | Optional last_tab or logout, allowed only for closed; validated but not materialized |
Heartbeat may create a missing site. Closed changes state only when its session
matches the current site session. The persisted BSON/JSON field remains state
with values online or offline; Go uses the typed field name Status.
Mongo projection
Database and collection:
beacon.presence_state
The unique identity is {tenant_id, player_id}. Each site_id has independent
session, status, expiry, and last-event transport metadata in the sites array.
Top-level state is online when any site is online.
Startup creates and requires:
tenant_player_unique {tenant_id: 1, player_id: 1} unique
online_site_expiry {sites.state: 1, sites.expires_at: 1}
Updates use optimistic comparison on version. If two replicas read the same
document, only one whole-document replacement can match; the other reloads and
reapplies its event. The same rule prevents expiry from overwriting a newer
heartbeat.
Opened and heartbeat set expires_at from trusted receive time plus the
configured expiry, 120 seconds by default. A sweep runs every 30 seconds by
default, selects at most 10,000 due documents, and uses 32 bounded workers.
Every replica may sweep safely. The Mongo operation timeout, five seconds by
default, caps each pass.
Expiry changes status but does not delete a player document. Define a business
retention period before adding a top-level purge_at TTL for old, fully-offline
states. Current documents retain player, session, event, and timing data.
Idempotency, commits, and recovery
Each site stores the last event ID and Kafka topic, partition, and offset. Duplicate guards apply in this order:
- The last event ID catches a record reproduced at another offset.
- Offset comparison catches redelivery within the same topic-partition.
- Trusted receive time rejects older envelopes when offsets are not comparable.
- Session comparison prevents an old close from ending a newer session.
Automatic commits are disabled. A claim processes records in order and marks a record only after Mongo projection or acknowledged DLQ publication. Marked offsets are synchronously committed every 100 records, every second, and when a claim closes. A crash can replay the uncommitted suffix; guards make current state replay a successful no-op instead of losing data.
Mongo failures block and retry the affected partition with capped exponential
backoff and jitter. They do not enter the DLQ. While a claim is blocked,
/ready returns 503 and lag grows without advancing that partition.
DLQ contract
Permanent validation failures are synchronously written to the input topic's DLQ. Beacon marks the input only after the broker acknowledges the DLQ write. The DLQ value is the exact original record bytes, avoiding base64 size amplification. Metadata is stored in Kafka headers:
| Header | Meaning |
|---|---|
dlq_id | Stable original_topic:partition:offset identity |
original_topic | Source Kafka topic |
original_partition | Source partition |
original_offset | Source offset |
event_id | Valid UUID when extractable, otherwise empty |
error_code | Bounded validation classification |
error_detail | Diagnostic detail capped at 256 bytes |
failed_at | UTC failure timestamp |
The Kafka record key is also dlq_id. DLQ publication and input offset commit
are separate operations, so a crash between them may create a duplicate DLQ
record. Consumers and replay tools must deduplicate by dlq_id. Replay only
after correcting the producer or record. A failed DLQ publish remains
uncommitted and retries; there is no retry topic.
Each input topic owns one DLQ. Event types sharing an input share its DLQ; different inputs cannot share a DLQ. Malformed envelopes still reach the right DLQ because the destination is chosen from the Kafka input topic before decode. DLQ retention and access restrictions are Kafka-infrastructure responsibilities and are not currently defined by Beacon.
Operational HTTP surface
Beacon has no business, admin, or player-state read API. It needs no OpenAPI document or API explorer. Its unauthenticated HTTP listener exposes only:
| Method and path | Success | Failure or notes |
|---|---|---|
GET /health | 200 ok | Process liveness; available while Mongo startup retries |
GET /ready | 200 ready | 503 not ready without an active unblocked Kafka group or Mongo primary ping |
GET /metrics | 200 Prometheus exposition | Operational metrics only |
Other methods return 405. Kubernetes exposes this listener through a ClusterIP Service with no Ingress. The NetworkPolicy permits the configured Prometheus pod selector; node-originated Kubernetes probes still reach the pod. Do not expose the operations listener as an application API.
Metrics
| Metric | Purpose and labels |
|---|---|
beacon_records_total | Registered event_type, bounded outcome |
beacon_projection_duration_seconds | Mongo projection latency by registered event type |
beacon_consumer_lag | Approximate lag by configured topic and partition |
beacon_duplicate_records_total | Duplicate event ID or transport position no-ops |
beacon_stale_closes_total / beacon_stale_times_total | Session and trusted-time no-ops |
beacon_invalid_records_total | Permanent failures by bounded validation code |
beacon_dlq_publishes_total / beacon_dlq_failures_total | DLQ acknowledgement and retry failures |
beacon_mongo_errors_total | Mongo projection, startup, sweep, and readiness errors |
beacon_retry_delay_seconds | Dependency retry delay |
beacon_expiry_candidates_total / beacon_expired_sites_total | Expiry sweep work and results |
beacon_sweep_duration_seconds | Expiry sweep duration |
beacon_ready | Last readiness result, 1 or 0 |
Never add tenant, site, player, event ID, session, raw error, or offset values as metric labels.
Configuration
YAML values have environment equivalents. Defaults below come from
internal/config; environment overlays may override them.
| Environment variable | Default | Purpose |
|---|---|---|
BEACON_LOG_LEVEL | info | debug, info, warn, or error |
BEACON_HTTP_ADDR | :8080 | Operations listener |
BEACON_KAFKA_BOOTSTRAP_SERVERS | required | Comma-separated Kafka brokers |
BEACON_KAFKA_DLQ_PARTITIONS | 6 | New DLQ partition count only |
BEACON_KAFKA_DLQ_REPLICATION_FACTOR | 3 | New DLQ replication only |
MONGODB_URI | empty | Direct Mongo URI; takes precedence over secret lookup |
MONGODB_URI_SECRET_NAME | empty | Existing Key Vault secret containing the Mongo URI |
KEY_VAULT_URL | required with secret name | Azure Key Vault endpoint |
BEACON_MONGO_DATABASE | beacon | Fixed logical database; other values are rejected |
BEACON_MONGO_TIMEOUT_MS | 5000 | Mongo operation timeout |
BEACON_PRESENCE_EXPIRY_SECONDS | 120 | Presence lease duration |
BEACON_PRESENCE_SWEEP_INTERVAL_SECONDS | 30 | Expiry sweep cadence |
BEACON_PRESENCE_SWEEP_BATCH_SIZE | 10000 | Maximum candidate documents per pass |
BEACON_RETRY_INITIAL_MS | 250 | Initial dependency retry delay |
BEACON_RETRY_MAX_MS | 30000 | Retry delay cap |
The inject command publishes a trusted envelope directly to Kafka and bypasses
Heimdall. It is for local or explicitly approved development smoke/load work,
not a producer API or production application path.
Kubernetes deployment
| Environment | Release pipeline | Kafka | Mongo secret |
|---|---|---|---|
| Dev1 | release_beacon_spark_dev1 | kafka-main-kafka-bootstrap.kafka-clusters.svc.cluster.local:9092 | spark-dev-mongodb |
| Mexico | release_beacon_spark_prod_mexico1 | kafka-main-kafka-bootstrap.spark-resources.svc.cluster.local:9092 | spark-mexico1-prod-mongodb |
Beacon resolves the existing Key Vault secret through
MONGODB_URI_SECRET_NAME, matching other Go services. The shared connection
creates the logical beacon database, presence_state collection, and required
indexes. No Beacon-specific Mongo credential is required today.
This is logical isolation, not credential-level least privilege. The shared credential has access beyond Beacon. Move to a database-scoped Mongo user when the platform supports it, and verify TLS from the external Key Vault URI because the repository cannot inspect that secret value.
The NetworkPolicy limits operations ingress to the environment's Prometheus
selector and egress to DNS, Kafka, Mongo, plus HTTPS for Key Vault. Standard
Kubernetes NetworkPolicy cannot restrict the HTTPS rule by hostname, so it
currently permits 0.0.0.0/0:443; use the platform egress proxy or FQDN policy
when available. Local Compose binds Mongo, Kafka, and operations ports to
127.0.0.1.
After rollout, verify both replicas, /health, /ready, PodMonitor scraping,
consumer group assignments, the input and DLQ topics, and both Mongo indexes.
Deployment is an operator action and is not part of local repository checks.
Future projectors
Add each player-state projector as an explicit Go route containing its trusted
input topic, event type, owned DLQ, decoder, and projection function. A permanent
record failure must return event.Invalid; other errors are treated as transient
and retry the partition indefinitely.
Keep the code-owned consumer group unless an offset migration is explicitly planned. Kafka does not order across topics: independent state may use separate topics, while events mutating one shared invariant need one keyed topic or clear event-time conflict rules.
Keep the beacon database. Use a separate collection when identity, queries, or
retention differ. Split a projector into another consumer group or service only
for independent replay, scaling, credentials, release cadence, or failure
isolation. Do not use operational Mongo as raw analytics history.