What is the problem your project solves?

Cluster observability makes you read graphs instead of seeing your system

When a Kubernetes service starts failing, the standard tooling hands you a wall of time-series dashboards. You are asked to reconstruct a live, moving system from a dozen static line charts — correlating CPU panels against request-rate panels against a service map that was drawn from a config file rather than from observed traffic.

Three specific things go wrong:

• The topology is fiction. Most service maps are generated from declared config, so they show edges that never carry traffic and miss the ones that do. You cannot tell what is actually talking to what.
• Overload is buried in a number. "CPU: 1699m" means nothing without the request as a denominator. Whether that is fine or on fire depends on context the dashboard does not show you.
• Answering a question takes minutes. "Did product-catalog overload in the last hour, and for how long?" requires picking the right panel, the right time range, and eyeballing a threshold crossing.

Impact: during an incident, the time spent translating dashboards into a mental model is time the outage continues. The information is all present — it is just not in a form a human can absorb at a glance or ask a question of.

Watchman's bet is that a cluster should be seen and interrogated, not decoded.

How you are solving it?

A live pixel canvas over three real data sources, with Claude as the query layer

Services render as pixel blocks. Real request traffic is drawn as moving dots along observed edges. A service turns red when it crosses a measured overload threshold. Every number on screen comes from a live source — there is no simulated or mocked data anywhere in the wired path.

Architecture

Kubernetes API (watch)  ──┐
metrics-server API      ──┼──▶  Go aggregator (in-memory)  ──▶  WebSocket ──▶  pixel canvas
Hubble Relay (GetFlows) ──┘                │
                                           └──▶  Claude (Haiku 4.5)  ──▶  chat panel

A Go aggregator merges a client-go shared informer (pod state), metrics-server polling (CPU), and a Cilium/Hubble GetFlows gRPC subscription (eBPF-traced network flows) into one in-memory model, then pushes snapshot-on-connect + diffs over a WebSocket at 4Hz.

Engineering decisions that were measured, not guessed

The overload signal is CPU as a percentage of the pod's CPU request, threshold 200%. I measured both candidate signals against the real app under real load:

Idle tops out at 30% of request; loaded services land at 428–3398%. The 200% threshold sits in a wide empty gap, so it never flaps.

I rejected request-rate as the primary signal after measuring it. Flow-count sampling was non-monotonic — product-catalog measured 1289 flows/5s idle but blank under load, because the Hubble CLI drops events at high volume. Flow data is excellent for drawing traffic and useless for gating a blink.

Edges are earned. A line is drawn between two services only after Hubble has actually observed a flow for that pair. payment and email stay visibly edgeless because they are only reachable via checkout, which is disabled here — that is real, not a bug.

React owns the shell only. The canvas is driven by a plain requestAnimationFrame loop reading a mutable store the WebSocket mutates directly. Live pod state never enters React state — that would re-render every 500ms tick and visibly stutter.

Colour is not the only cue. Overload adds a heavier border, corner bolts, a ! badge, a shake, and steam particles. Palette validated for colour-blind separation (worst all-pairs CVD ΔE 11.5).

Load is real. k6 runs as an in-cluster pod generating real HTTP traffic against the real app — it never injects events into the WebSocket. Driving load through kubectl port-forward dropped 18% of requests at 60 VUs, so run-load.sh runs k6 in-cluster instead.

Disclosure

The visualization and aggregator were built before this hackathon. The work completed during this hackathon is the Claude integration — aggregator/claude.go, the POST /api/chat endpoint, and the frontend wiring (see the Claude Haiku 4.5 as the chat reasoning layer commit).

How Did You Use Claude?

Claude is the query layer over measured telemetry — and is constrained so it cannot hallucinate

Claude Haiku 4.5 turns the dashboard into something you can ask questions of. Instead of picking a panel and a time range, you ask "did product-catalog overload?" and get the peak value, wall-clock time, and incident duration.

How it is wired

The Go aggregator maintains an overload event log — every threshold crossing it actually measured, with service, metric, peak_value, start_time, end_time, duration, active. POST /api/chat hands Claude that log as JSON alongside a system prompt that forbids invention:

// aggregator/claude.go
resp, err := client.Messages.New(ctx, anthropic.MessageNewParams{
    Model:     "claude-haiku-4-5",
    MaxTokens: 256,
    System:    []anthropic.TextBlockParam{{Text: claudeSystemPrompt}},
    Messages:  []anthropic.MessageParam{
        anthropic.NewUserMessage(anthropic.NewTextBlock(userContent)),
    },
})

The prompt states: "You may ONLY use the JSON event log provided... Never invent a service, a number, or an incident that isn't in the log. If the log doesn't answer the question, say so plainly instead of guessing."

Three deliberate engineering choices

1. Grounding is verified, not assumed. I tested the failure case explicitly. With an empty event log:

Q: did product-catalog overload?
A: "The event log is empty, so I cannot determine whether product-catalog overloaded. There are no incidents recorded."

It refuses rather than producing a plausible fabrication. For an observability tool this is the property that matters most — a monitoring system that invents incidents is worse than no monitoring system.

2. Haiku 4.5, chosen deliberately. The task is grounded fact-lookup over a small JSON payload, not open-ended reasoning. Haiku ($1/$5 per MTok) is the right tier; spending Opus-tier tokens on templated fact retrieval would be waste, and the latency matters in an incident.

3. The key never reaches the browser. Claude is called server-side from Go via the official anthropic-sdk-go, with ANTHROPIC_API_KEY read from the aggregator's environment. The frontend only ever talks to POST /api/chat on localhost. A rule-based matcher over the same event log remains as a fallback, so the demo still answers if the key is absent — but Claude is the primary path and the only one that handles paraphrased questions.

Claude Code as the build tool

Claude Code did the integration work in this repo, including catching that the pre-existing chat module's docstring still claimed it was "deliberately NOT an LLM" and that a stale GROQ_API_KEY reference needed replacing — leftovers that would have made the submission's own documentation wrong.

What is the deployed URL for this project?

https://github.com/Syedowais312/watchman