bedrock
tenant layer

Design decisions

Most parts of bedrock replaced something else that was tried or considered first. This page lists the main decisions and the reasons for them.

Platform

ChosenInstead ofReason
RKE2 with the CIS profilek3sAudit logging, Pod Security, secrets encryption and CIS compliance out of the box
One cluster per tenantNamespaces on a shared clusterIsolation as a property of the design, not of configuration
Hand-written, pinned manifestsHelmWhat is in the repository is exactly what runs
Bootstrap once, then reviewed kubectl diff and applyArgo CD or another GitOps controllerWrite access to the repository would equal cluster admin, and a git host becomes attack surface

Storage

ChosenInstead ofReason
GarageMinIOMinIO’s enterprise edition is paid and licence gated, and the community edition is no longer maintained
GarageCeph RGWCeph is heavy in memory and operations for a job that only needs S3
Three Garage pods, each on a local volumeOne Garage pod on a replicated volumeNo single point of failure, and no data replicated twice
LonghornCeph RBD and CephFSLonghorn already provides block and shared volumes and runs on any node

Network

ChosenInstead ofReason
CiliumNetworkPolicyKubernetes NetworkPolicyCluster-wide deny, naming the API server as an entity, FQDN rules and layer 7 rules
Cilium kube-proxy replacementkube-proxy alongside CiliumWith both handling services, traffic from the node was rewritten and lost its identity, so host rules never matched
ClusterIP and port-forwardDashboards on NodePortsA datacenter has no trusted LAN
Cluster DNS with no forwardingForwarding to an upstream resolverOpen forwarding allows data to leave through DNS
Per-component policy in applicationAllow-all inside the namespaceOne compromised pod could otherwise silence alerts and delete logs

Policy and security

ChosenInstead ofReason
Kyverno on application namespaces onlyKyverno on every namespaceStorage and database recovery must never depend on a policy engine
Priority classes set in each manifestA Kyverno mutationPriority is scheduling, not security, so it should not sit behind a webhook
A small privileged node-agents namespaceHost agents inside applicationTwo agents needing the host made every monitoring pod eligible to run as root
OpenBao, one per tenantSealed secrets, SOPS or a shared vaultAn audit trail and dynamic credentials, and no shared component holding every tenant’s keys
Falco to stdout, collected by AlloyfalcosidekickOne component fewer

Operations

ChosenInstead ofReason
A single Alertmanager per tenantA highly available AlertmanagerIf it dies, the Watchdog stops reaching v2, which raises the alarm. High availability arrives with v2 delivery.
Local etcd snapshotsSnapshots to external S3The bootstrap must work with no outside services, and etcd can be rebuilt from the repository
Single replicas for Tempo, Grafana and OpenCostHigh availability everywhereThey lose nothing on restart