Kubernetes at the Edge: Latency Lessons

When we decided to deploy K3s clusters across 50+ geographic locations for a real-time IoT analytics platform, we thought the hardest part would be Kubernetes itself. We were wrong. The real challenge was physics.

The Architecture

Our setup was straightforward on paper: lightweight K3s clusters at each edge location, a central control plane on AWS EKS, and ArgoCD syncing configurations through GitOps. Each edge cluster ran 3-5 nodes handling local data ingestion, transformation, and real-time alerting.

# K3s edge cluster provisioning
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server \
  --cluster-init \
  --tls-san edge-node-01.region-ap.internal \
  --disable traefik \
  --disable servicelb \
  --write-kubeconfig-mode 644" sh -

Lesson 1: Latency Budgets Are Non-Negotiable

Our first mistake was treating latency as an afterthought. When your edge nodes in Mumbai need to sync state with a control plane in us-east-1, you're looking at 180-220ms round-trip. That may sound small, but when your reconciliation loop runs every 10 seconds and each sync involves 3-4 API calls, you've already consumed 800ms of your budget on network alone.

We solved this by implementing a tiered sync strategy:

▸Critical configs (security policies, RBAC): Synced immediately via webhook push
▸Application deployments: Synced every 30s with exponential backoff on failure
▸Observability data: Batched and shipped every 60s to reduce API pressure

Lesson 2: etcd is Your Bottleneck

K3s uses SQLite by default for its datastore, which works great for single-node setups. But at the edge, when you need HA across 3 nodes with unreliable networking between them, SQLite falls apart. We switched to embedded etcd with aggressive compaction settings:

# etcd optimization for edge environments
--etcd-arg quota-backend-bytes=2147483648
--etcd-arg auto-compaction-mode=periodic
--etcd-arg auto-compaction-retention=1h
--etcd-arg snapshot-count=5000

This reduced our etcd database size by 60% and cut leader election time from 12s to under 3s during network partitions.

Lesson 3: GitOps at Scale Needs Sharding

Running a single ArgoCD instance managing 50+ clusters is a recipe for OOMKilled pods. We sharded ArgoCD by region — one instance per geographic zone, each watching its own cluster set. This reduced memory usage by 75% and sync times by 80%.

# ArgoCD ApplicationSet for regional sharding
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: edge-apps
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            region: ap-south
  template:
    spec:
      destination:
        server: '{{server}}'
      source:
        repoURL: git@github.com:org/edge-configs.git
        path: 'clusters/{{name}}'

Key Takeaways

✓Speed of light is the only hard limit — design your sync strategy around it, not against it
✓Edge clusters must be autonomous — they should survive 100% control plane disconnection
✓Shard your GitOps tooling early — a single ArgoCD instance won't scale past 20 clusters
✓Monitor etcd obsessively at the edge — it's the first thing to break on unreliable networks

End of Transmission