CrowdSec on a 4 GB VPS in an afternoon — Divyam Azad

The server behind this site has exactly one thing listening on the internet: Caddy, on ports 80 and 443. Everything else is reachable only on the compose network. That is a small surface, but it is a surface, and the access log filled with vulnerability scanners within an hour of the first certificate being issued. This is how I put CrowdSec in front of it, in a way that fits the rest of the repository: declared in git, bootstrapped by the deploy script, visible on a public page.

The shape

Caddy ──JSON access log──▶ shared volume ──▶ crowdsec agent ──decisions──▶ Caddy (bouncer module)
                                                   │
                                                   └─▶ community API (signals out, blocklist in)

Three pieces, all in docker-compose.yml:

Caddy writes a JSON access log to a named volume. A small snippet is imported by every site block:

(access_log) {
    log {
        output file /var/log/caddy/access.log { roll_size 10mb  roll_keep 3  mode 0644 }
        format json
    }
}

The crowdsec container tails that file (acquis.yaml with type: caddy), runs the collections it needs — crowdsecurity/caddy, base-http-scenarios, http-cve, whitelist-good-actors, and linux for the base parsers — and exchanges signals with CrowdSec’s central API: what this box sees goes out, the community blocklist comes in. It exposes its Local API on the compose network only.
Caddy is the bouncer. Instead of a separate process, the Caddy image is built on the box from a tiny Dockerfile that adds hslatman/caddy-crowdsec-bouncer with xcaddy. A global block points it at the Local API, and crowdsec is the first directive of every site, so a banned address gets 403 before routing, file serving or any proxy runs:
```
{
    order crowdsec first
    crowdsec {
        api_url http://crowdsec:8080
        api_key {$CROWDSEC_CADDY_BOUNCER_KEY:bootstrap-pending}
        ticker_interval 15s
    }
}
```

The whole thing costs about 100 MB of RAM at rest and a one-time, roughly five-minute Caddy build on two vCPUs.

Bootstrapping keys without a human

Bouncers authenticate to the Local API with a key that only cscli inside the agent can mint. That is a chicken-and-egg problem for a pull-based deploy: Caddy must start before the key exists, and the key must exist before Caddy can enforce anything.

The Caddyfile above solves the first half — note the default value in {$CROWDSEC_CADDY_BOUNCER_KEY:bootstrap-pending}. With a placeholder key Caddy starts and serves normally; the bouncer just logs that it cannot authenticate. The site never goes down because of the security layer being half-configured.

The deploy script solves the second half. After docker compose up, it checks whether the secrets file has both keys; if not, it waits for the Local API to answer, runs cscli bouncers add caddy (and one more read-only bouncer for the security page), writes the file with mode 600, and recreates the two containers that consume it. Every run after that is a no-op. If the file is ever lost, the same function re-registers the bouncers with fresh keys.

A page for it

Because CrowdSec’s own console is a hosted service and its old local dashboard is heavy, I added a tiny Go service that reads the Local API with a read-only bouncer key and renders what is being blocked right now: active decisions, how many were detected by this box versus pulled from the community, the top scenarios, and a sample of current bans with the addresses masked. Browsers get HTML; curl gets JSON — the same content negotiation the status page uses. It cannot ban or unban anything; that is deliberate for something anyone can open.

The first real ban

About ten minutes after the first deploy, cscli decisions list showed a ban I had not created: a scanner that had been probing for a Jira vulnerability path (crowdsecurity/jira_cve-2021-26086), then tripped http-probing as it walked through the rest of its list. The access log showed the requests; the decision showed the outcome; the security page showed both. Nothing to do — that is the point.

Then the end-to-end test, with my own address:

docker compose exec crowdsec cscli decisions add --ip <my-ip> --duration 2m --reason test
curl -sI https://divyam.top/ | head -1        # HTTP/2 403  (within 15 s)
docker compose exec crowdsec cscli decisions delete --ip <my-ip>
curl -sI https://divyam.top/ | head -1        # HTTP/2 200

The ops cheatsheet

docker compose exec crowdsec cscli metrics           # are caddy lines being parsed?
docker compose exec crowdsec cscli bouncers list     # caddy + secstats, "validated"
docker compose exec crowdsec cscli decisions list    # who is banned right now
docker compose exec crowdsec cscli alerts list       # what tripped, and when
docker compose logs -f crowdsec                      # parsing, CAPI sync, new decisions

The one trap

A custom Caddy build means the stock caddy validate cannot parse your Caddyfile any more — it does not know the crowdsec directive. If the merged configuration has a typo, the new container fails to start and the edge is down. So the deploy that introduces the module is the one to pre-flight: build the image once on the box, run caddy validate with that image against the merged Caddyfile, and only then let the real deploy recreate the container. After that, the build is cached and validation is a normal part of the loop.

Everything else was uneventful, which is the nicest thing one can say about a security change.