Lamplighter

To start your Phoenix server:

• Run mise install to get the pinned Erlang/Elixir versions (see mise.toml)
• Run mix setup to install and setup dependencies
• Start Phoenix endpoint with mix phx.server or inside IEx with iex -S mix phx.server

Now you can visit localhost:4000 from your browser.

Running it for real? See Running in production.

Authentication

Users can log in with email (magic link or password) and, optionally, single sign-on through any OpenID Connect provider (Authentik, Authelia, Keycloak, Pocket ID, Dex, ...). SSO is enabled when these environment variables are set. For local development, copy .env.example to .env (gitignored); mise loads it automatically.

Variable	Description
OIDC_ISSUER	Issuer URL, e.g. https://auth.example.com/application/o/lamplighter/
OIDC_CLIENT_ID	Client ID registered with the provider
OIDC_CLIENT_SECRET	Client secret
OIDC_DISPLAY_NAME	Button label on the login page (default SSO)
OIDC_SCOPES	Space-separated scopes (default openid email profile)
OIDC_ALLOW_INSECURE_HTTP	Dev only: allow a plain-HTTP issuer for local testing
SSO_ONLY	true to allow only SSO (see below)

Register <app URL>/auth/oidc/callback as the redirect URI with your provider.

On first SSO login, an account is created for the user's email. If an account with that email already exists, it is linked only when the provider reports the email as verified (email_verified). After that, users are matched by the provider's stable subject ID, so changing the email at the provider is safe.

With SSO_ONLY=true, registration, password login, and magic links are disabled, and account email and password are managed by the provider. Since accounts are created on first SSO login, restrict who can use the Lamplighter app in your provider (e.g. an Authentik policy or Pocket ID user group). Lamplighter refuses to start with SSO_ONLY set if OIDC isn't configured.

Running in production

The Dockerfile builds a self-contained image of the release. Tagged releases are published for amd64 and arm64 as git.knope.dev/jfro/lamplighter (see Releasing):

docker run -p 4000:4000 \
  -e DATABASE_URL=ecto://user:pass@db/lamplighter \
  -e SECRET_KEY_BASE=... -e PHX_HOST=lamplighter.example.com \
  -e OIDC_ISSUER=https://id.example.com -e OIDC_CLIENT_ID=... -e OIDC_CLIENT_SECRET=... \
  -e SSO_ONLY=true \
  git.knope.dev/jfro/lamplighter

Migrations run on start. Everything Lamplighter keeps is in PostgreSQL, so the container needs no volume. It speaks plain HTTP on port 4000; put TLS in a reverse proxy.

Lamplighter can't send email in production yet, so magic links (used to sign in after registering) go nowhere: configure SSO (see Authentication) and set SSO_ONLY=true.

To watch the Docker host it runs on, mount its socket and add the socket's group, since the app runs as nobody (see Docker):

-v /var/run/docker.sock:/var/run/docker.sock:ro --group-add "$(stat -c %g /var/run/docker.sock)"

Environment

Variable	Required
DATABASE_URL	yes	ecto://USER:PASS@HOST/DATABASE
SECRET_KEY_BASE	yes	Signs cookies, and encrypts the source credentials stored in the database. Generate with mix phx.gen.secret. Changing it means re-entering those credentials.
PHX_HOST	yes	The hostname the app is reached at, over HTTPS. LiveView refuses connections from other hosts.
OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET	for SSO	See Authentication, along with OIDC_DISPLAY_NAME, OIDC_SCOPES, and SSO_ONLY.
PORT		HTTP port, default 4000.
RUN_MIGRATIONS		Set to false to skip migrations on start and run bin/migrate yourself.
POOL_SIZE		Database connections, default 10.
ECTO_IPV6		true to connect to the database over IPv6.
DNS_CLUSTER_QUERY		For clustering several nodes via DNS.

Releasing

Commit messages follow conventional commits (feat:, fix:, ...), which git-cliff turns into CHANGELOG.md (configured in cliff.toml); older commits starting with "Add" or "Fix" are grouped as features and fixes. To release:

mise run release -n        # preview the next version and its notes
mise run release           # bump mix.exs, update the changelog, commit, tag and push
mise run release minor     # or force major/minor/patch, or an exact 1.2.3

Pushing a v* tag builds the image for both architectures and publishes it tagged with the version, major.minor, major, and latest. CI runs the checks and tests on every push to main and on pull requests.

Sources and reports

Lamplighter polls sources for the current state of what they watch, and triggers turn sustained problems into reports on the dashboard:

1. A source (Lamplighter.Sources.SourceType) returns one Signal per watched thing on each poll: its status, the what (title, summary), the why if known, and raw facts for diagnosis.
2. Lamplighter.Triggers tracks how long each signal has been failing or healthy and opens or resolves reports once the source's open_after / resolve_after thresholds are met.
3. If a source can't be polled, that becomes a report of its own.

Each source is labeled with what it's for:

• Problems: it finds problems, which become reports. Most sources do this.
• Context: it answers queries for metrics, logs, and state while a report is diagnosed, such as PromQL against Prometheus. It's polled only to check it can be reached, so its form has no thresholds for problems.

A source can have both labels, like Kubernetes.

Sources are added and configured in the app under Settings → Sources, including connection details, credentials, and timing thresholds. Credentials are stored encrypted with a key derived from SECRET_KEY_BASE; if that changes, re-enter them. The settings form can test a connection before saving.

Uptime Kuma

Lamplighter reads Kuma's /metrics endpoint, so a read-only API key (Kuma: Settings → API Keys) is enough. Point it at Kuma's base address, not through a login proxy. It reports:

• Down monitors, one report per host when several of its monitors fail
• Brief outages between checks, caught from drops in Kuma's uptime ratio
• Slow responses, when the last day's average is several times the 30-day one
• TLS certificates that are invalid or close to expiry

Tune individual monitors with Kuma tags (Kuma exports a new tag only after it restarts, and keeps only letters, digits, and underscores in values):

Tag	Value	Effect
lamplighter	ignore	Skip the monitor entirely
lamplighter	critical, serious, warning	Severity when down (default critical)
lamplighter_group	any name, e.g. storage	Report these monitors together

Proxmox VE

Create an API token (Datacenter → Permissions → API Tokens) and give its user the PVEAuditor role on /, plus the token itself if it has privilege separation. It reports offline nodes, nodes that rebooted (seen from their uptime dropping), lost quorum, guests set to start at boot that aren't running, storage that's unavailable or filling up, ZFS pools that aren't ONLINE, disks failing SMART, failed backup runs, and guests not covered by any backup job. Tag guests lamplighter-ignore, or lamplighter-warning / -serious / -critical to set their severity.

Pending updates are opt-in: reading them needs Sys.Modify on the nodes, which also allows changing node settings.

A reboot says whether the node shut down cleanly first, from the stopall task Proxmox runs to stop guests in every clean shutdown (from the web UI, the shell, or a UPS). One without it, after a power loss, crash, or reset, is serious rather than a warning.

A stopped guest that should be running says why, from its latest task: the error of a failed start, who shut it down, or that it last started fine and nothing in Proxmox stopped it (it shut down from inside or crashed). A failed backup run's cause adds the error lines from its log, such as which guest failed and why, where Proxmox only says "job errors".

It also adds context to diagnoses: the cluster's nodes, guests, and storage with their usage; a guest in depth (live status, hardware, disks, any lock, and recent tasks); tasks and their logs, such as why a backup or a guest start failed; and Proxmox's own metric history for nodes, guests, and storage. A guest's notes, cloud-init password, SSH keys, and raw QEMU arguments are never read out. Diagnoses can also read nodes' system logs (kernel, corosync, and service messages) if you turn that on and grant Sys.Syslog with the setup guide's optional LamplighterSyslog role.

A role granted on /nodes replaces PVEAuditor there rather than adding to it, so the guide's optional roles include Sys.Audit. If you created LamplighterUpdates from an older guide, with Sys.Modify alone, failed backups have gone unreported; fix it with pveum role modify LamplighterUpdates --privs Sys.Audit,Sys.Modify. Test connection says when a role hides the nodes like this.

Proxmox Backup Server

Create an API token and give both the user and the token the Audit role on /; PBS limits a token to what its user may do. It reports datastore errors and capacity (including projected fill-up), backup groups with no recent backup in any namespace, latest backups that failed verification, failed jobs (verify, sync, prune, garbage collection), and pending updates. A failed job's cause adds the error lines from its task log, since PBS's own status often only says to check it.

It also adds context to diagnoses: datastores with their usage and last garbage collection, backup groups and snapshots with their verification state, tasks and their logs (e.g. which chunks failed verification), metric history, and the system log. The Audit role already allows all of it; a setting turns the system log off.

Kubernetes (k3s and others)

Create a read-only service account (the settings page has a copyable kubectl apply manifest; it can't read Secrets or change anything) and paste its token, the cluster's CA certificate, and the API server URL. It reports nodes that aren't ready or are under pressure, Deployments, StatefulSets, and DaemonSets missing pods (with the reason: unschedulable, crash-looping, image pull failures, and so on), containers restarting while otherwise healthy, failed CronJob runs (as backups when named or annotated like one), and cert-manager certificates that aren't ready.

A node reboot is one report, seen from the node's boot ID changing even when the reboot was too quick for it to go NotReady. Containers on the node that restart in the 15 minutes after are counted in that report instead of reported one by one, including ones that take a few tries while their dependencies come back.

Annotate namespaces or objects with lamplighter.dev/ignore: "true", or lamplighter.dev/severity: warning|serious|critical.

It also adds context to diagnoses: pods with why containers are waiting or how they last ended (e.g. OOMKilled), one pod in depth with its recent events, events, nodes, and container logs, including a crashed container's previous run. Environment variables, command arguments, and volumes are never read out. Logs can hold secrets, so a setting turns that off. If you set up the service account before events and logs were added, apply the manifest again; Test connection says when it can't read them yet.

Alertmanager

Point it at Alertmanager (not Prometheus), with basic-auth credentials or a bearer token if it's protected. Firing alerts become reports, one per alertname by default (configurable, e.g. alertname,instance), using the alert's summary/description, its severity label, and its real start time. info/none alerts such as Watchdog are skipped unless included. Silenced alerts keep their report open; alerts that stop firing resolve on the next poll. Set a lamplighter_kind label (security, backup, performance, updates) to choose a report's kind; otherwise it's guessed from the alert name.

QNAP NAS

Enable SNMPv3 in QTS (Control Panel → Network & File Services → SNMP) with authentication (HMAC-SHA), and privacy with AES if offered. DES isn't supported; if the QNAP only offers DES, turn privacy off on both sides: requests stay authenticated but the data isn't encrypted on the network. Lamplighter reads QNAP's storage tables: disk health and SMART status, disk and system temperatures, fans, power supplies, RAID groups, storage pools, and volume status and fullness, for the NAS and any expansion units. Firmware updates are optional: they need the web interface URL and an administrator login (without 2-step verification) and are checked at most every 12 hours.

Docker and Docker Swarm

Give Lamplighter read-only access to the Docker API, in one of three ways (the settings page has commands): mount /var/run/docker.sock read-only and keep the default unix:// address; run a socket proxy that only allows CONTAINERS, INFO, NODES, SERVICES, and TASKS, and use http://<host>:2375; or use Docker's TLS port (https://<host>:2376) with a client certificate and key and Docker's CA. For a swarm, connect to a manager. Inspecting containers reveals their environment variables, so keep a proxy off networks others can reach.

On the host, it reports containers that keep restarting, are unhealthy (with the health check's output), ran out of memory, couldn't start (such as a port already taken after a reboot), or crashed, plus restarts between checks. Containers stopped with docker stop are fine, and stopped containers without a restart policy are only watched once Lamplighter has seen them running, so old leftovers aren't reported. On a swarm, it reports nodes that are down, unreachable managers, services with fewer running tasks than desired, jobs and run-to-completion services (like swarm-cronjob backups) whose latest run failed, and rolling updates that were rolled back, paused, or stuck. Causes come from Docker: scheduling errors such as "no suitable node", exit codes, and the last lines the failed container or task printed (turn this off if logs may hold secrets).

Label containers, services, or nodes lamplighter.ignore=true, lamplighter.severity=warning|serious|critical, or lamplighter.kind=backup.

Prometheus

A context source: it reports no problems itself (point the Alertmanager source at your alert rules for that), but answers queries while a report is diagnosed: PromQL at an instant or over a range, the series and label values that exist (e.g. every metric recorded for a host), and scrape targets with their last error. Results are capped so they stay readable, and say when they were cut short. Point it at Prometheus itself (with kube-prometheus-stack, usually http://prometheus-operated.monitoring:9090), with basic-auth credentials or a bearer token if it's protected. Anything serving Prometheus's query API works, such as Thanos, VictoriaMetrics, or Grafana Mimir (include its /prometheus prefix). Test connection shows its version and which scrape targets are down.

Queries run through Lamplighter.Sources.query_context/3, for example from iex:

source = Lamplighter.Sources.get_source!(id)
Lamplighter.Sources.query_context(source, "query_range", %{
  query: ~s|node_memory_MemAvailable_bytes{instance=~"pve1.*"}|,
  start: "2026-10-02T13:00:00Z"
})

Loki

A context source, like Prometheus: it answers LogQL queries while a report is diagnosed, such as what a crashing pod logged before it died, and lists the labels, label values, and streams that exist so the right logs can be found. Log queries return lines grouped by stream (newest first, or oldest first to read from when a problem began), at most 500 and each cut at 1,000 characters; metric queries such as count_over_time return series. Point it at Loki's gateway or query frontend (or the single binary, port 3100), not Grafana. If Loki runs multi-tenant (auth_enabled: true), enter the tenant ID, sent as X-Scope-OrgID; for a proxy or Grafana Cloud, enter credentials too. Test connection shows its version and the labels its recent logs carry.

Self-signed certificates

Sources that talk HTTPS can trust a server's self-signed certificate by its SHA-256 fingerprint (it may then be reached by IP), or trust a private CA by pasting its PEM certificate. When verification fails, Test connection shows the fingerprint the server presented, to compare against the one the server itself displays (e.g. the PBS dashboard's "Show Fingerprint").

Adding a source type

Implement the Lamplighter.Sources.SourceType behaviour and register the module in Lamplighter.Sources.types/0. Describe its settings with Lamplighter.Sources.Field specs in settings_fields/0, and the settings form, validation, encrypted secrets, and connection test come for free. Polling, thresholds, persistence, and reports are handled generically.

A source that adds context returns :context from roles/0 (alongside :problems if it also finds them), describes its queries as Lamplighter.Sources.ContextQuery structs in context_queries/0, and runs them in query/3. Params are checked and cast before query/3 sees them.

AI diagnosis

Lamplighter can diagnose reports with Claude: it reads a report and its facts, queries your context sources (Prometheus, Loki) as tools, and writes the most likely cause, with its evidence, and what to check next. The report shows the diagnosis, the model, and the queries it ran. Set it up under Settings → AI with one of:

• Claude Code: uses a Claude subscription. Run claude setup-token on any machine with Claude Code and paste the token. Lamplighter runs the Claude Code CLI headless (the image includes it) with its own config directory, so nothing of a local Claude Code setup is used. The CLI reaches the tools through Lamplighter's own /mcp endpoint, over loopback, with a token that only works while that diagnosis runs.
• Claude API: uses an Anthropic API key, billed per use. The tool loop runs inside Lamplighter.

Check connection asks the model for a short reply before you save. New reports are diagnosed automatically unless you turn that off; any report can be diagnosed, or diagnosed again, from the dashboard. Diagnoses run two at a time and pause when a usage limit is hit, resuming when it resets. Each diagnosis uses your subscription's allowance or API credit.

Other AI providers can be added by implementing Lamplighter.AI.Provider and registering the module in Lamplighter.AI.providers/0; diagnosis only talks to that behaviour.

Learn more

• Official website: https://www.phoenixframework.org/
• Guides: https://hexdocs.pm/phoenix/overview.html
• Docs: https://hexdocs.pm/phoenix
• Forum: https://elixirforum.com/c/phoenix-forum
• Source: https://github.com/phoenixframework/phoenix