Self-Hosted Delivery Pipeline

Problem

This site already shipped through a clean GitHub Actions job: build, OIDC into AWS, sync to S3, invalidate CloudFront. That’s the right answer for a static site. But “push triggers a hosted runner” hides every moving part of a real delivery pipeline: where artifacts live, how build hosts are configured, how you know any of it is healthy. I wanted a version that makes those parts explicit and operable, the way a delivery pipeline looks behind a product team.

What I built

A self-hosted pipeline that produces a versioned, reproducible artifact and is fully observable, while keeping the cheap static serving path (S3 + CloudFront) unchanged. A push to main now flows:

GitHub push
    │
    ▼
Jenkins ─ scan source (Semgrep + tfsec) ─ build in Docker ─ scan image (Trivy) ─ smoke-test
              │
              ├─▶ image :<gitSHA> + dist tarball ─▶ Nexus
              ├─▶ dist/ ─▶ S3 ─▶ CloudFront                 (main only)
              └─▶ flag changed posts ─▶ Substack (manual)    (main only)

Everything observing it reports into Prometheus.

Key decisions and trade-offs

• Terraform provisions a self-contained VPC, three EC2 hosts, security groups locked to my IP, and the Jenkins IAM instance profile, reusing the same least-privilege S3-sync + CloudFront-invalidate policy the old GitHub role used, so the build host carries no long-lived AWS keys.
• Ansible owns configuration: Docker engine, Jenkins via Configuration as Code (plugins, credentials, global env, and the seed job all declared, zero click-through setup), Nexus with its repos provisioned over the REST API, and the monitoring stack.
• Docker builds the site in a multi-stage image (node build → nginx serve). That image is the release artifact, smoke-tested before anything publishes.
• Jenkins orchestrates the whole run: scan source → build image → scan image → smoke test → push image + tarball to Nexus → s3 sync + CloudFront invalidation → cross-post, with everything past the artifact gated to main. I own the build host, so every stage, credential, and failure mode is something I configured myself and can go read the logs for.
• Security gates run before anything publishes: Semgrep (SAST, including a secrets ruleset) over the app source, tfsec over the Terraform, and Trivy over the built image for fixable HIGH/CRITICAL CVEs. Any finding fails the build. The Semgrep pass is mirrored as a GitHub Actions job, so pull requests get the same security feedback before they ever reach the pipeline.
• Nexus keeps a hosted Docker registry and a raw repo of versioned dist tarballs and every release is retrievable by git SHA. Leaving artifacts in S3 alone would have shipped the site fine, but it treats the deployed files as the release; a registry plus a raw repo makes the artifact first-class, so any past build can be pulled and run by SHA rather than rebuilt and hoped for.
• Prometheus scrapes node_exporter and cAdvisor across the fleet, Jenkins’ own metrics, and a blackbox probe of the live site, with alert rules for host/CPU/disk pressure, a failing site probe, and TLS expiry. CloudWatch stops at the instance boundary and has nothing to say about a Jenkins stage or a container; Prometheus holds host, container, build, and public-URL health in one model.
• Substack cross-posting is where I stopped automating, on purpose. The script works and drafts a post with full text plus a canonical link home, but Substack is behind Cloudflare, which answers datacenter IPs with a bot challenge before reading the session cookie. Identical cookie and code: 403 from a CI runner, clean draft from my laptop. Beating that would mean forging a cf_clearance cookie bound to the runner’s IP and user agent, so the “fix” would be a fragile impersonation I’d have to rebuild every time Cloudflare tuned a rule. Instead the pipeline reports which posts are due and I create the drafts by hand, with an /rss.xml feed for Substack’s official importer as the bulk path. Knowing which integrations shouldn’t be automated is part of owning one.

Outcome

• The artifact is now first-class: any build is reproducible and pullable by SHA, not just “whatever the runner happened to produce.”
• The deploy credential is an instance profile, scoped to exactly the same two actions as the old GitHub role: sync the bucket, invalidate the cache.
• The whole path is observable end to end, from build-host CPU to whether the public URL actually returns 200.
• Nothing ships unscanned: SAST, IaC, and image scans gate every build, and the SAST mirrors into the pull-request check so problems surface before merge.