Deploying Sourcegraph executors on Kubernetes (docker-in-docker)

Docker-in-Docker Kubernetes executors are in beta. For production workloads where you want the strongest reliability and support guarantees, you can also deploy via Terraform or the Linux binary.

Kubernetes manifests are provided to deploy Sourcegraph Executors on a running Kubernetes cluster. If you are deploying Sourcegraph with helm, charts are available here.

Deployment

Executors on Kubernetes require privileged access to a container runtime daemon in order to operate correctly. To ensure maximum capability across Kubernetes versions and container runtimes, a Docker in Docker sidecar is deployed with each executor pod so that the executor does not have to access the host container runtime directly.

Step-by-step Guide

Ensure you have the following tools installed:

  • kubectl
  • Helm if you're installing Sourcegraph with helm.

Deployment via Helm

Please refer to the Sourcegraph Helm docs for the latest instructions.

To specifically deploy Executors,

  1. Create an overrides file, override.yaml, with any other customizations you may require.

    1. See details on configurations
    2. See here for a full list of executor environment variables

  2. Run the following command:

    BASH
    helm upgrade --install --values ./override.yaml --version <your Sourcegraph Version> sg-executor sourcegraph/sourcegraph-executor-dind

  3. Confirm executors are working by checking the Executors page under Site admin > Executors > Instances .

Security considerations

Docker-in-docker executors require the Docker sidecar to run as a privileged container. privileged: true understandably raises concerns, so it's worth understanding the threat model before evaluating it — and seeing why, with the right deployment practices, it is acceptable for the large majority of self-hosted Sourcegraph instances.

Executors run untrusted code by design

Every executor deployment method exists to run arbitrary, untrusted code. Auto-indexing invokes language indexers and package-manager hooks to resolve dependencies, and batch changes run user-defined tooling against the contents of a repository. See Executors and Firecracker for the full sandboxing model.

This means the risk of a malicious job attempting to break out of its sandbox, consume excessive compute, or exfiltrate code and credentialsis is inherent to running executors. It is not unique to Docker-in-Docker:

  • Firecracker provides the strongest per-job isolation (a MicroVM boundary), but it is not a complete security control on its own. Even with Firecracker, Sourcegraph additionally configures iptables to stop jobs from reaching private IP ranges and the cloud metadata service.
  • Native Kubernetes and Docker-in-Docker run jobs as containers without a MicroVM boundary, so they rely more heavily on the surrounding isolation and network controls.

The runtime you choose changes the isolation boundary, but they do not negate the need for additional controls to furthur limit the blast radius.

Security hardening and risk management

A privileged container shares the host kernel and can access host devices, so in principle a successful container breakout could affect the node. The mitigations below contain that blast radius to a pool of disposable nodes that run only untrusted executor jobs and that cannot reach your internal network — which is the same posture Sourcegraph applies to Firecracker-isolated executors.

We strongly recommend applying the following controls when running docker-in-docker executors:

  1. Run executors on a dedicated, isolated node pool. Schedule executor pods onto their own node pool using taints/tolerations and node selectors or affinity, kept separate from the rest of your Sourcegraph workloads and any other sensitive services. A breakout is then contained to nodes that run only untrusted executor jobs, and those nodes can be recycled aggressively.

  2. Sandbox the executor nodes where supported. Run the executor node pool on a sandboxed runtime such as gVisor (for example, GKE Sandbox). We offer built-in support for gVisor in the docker-in-docker helm chart. Learn more from the documentation.

  3. Restrict network access. Apply Kubernetes NetworkPolicies and/or cloud firewall rules to the executor node pool to:

    • Block access to internal and private resources and services.
    • Block access to the cloud instance metadata service at 169.254.169.254 (AWS / GCP).