/bazel

Hermeticity, Sandboxing, Caching, and Remote Execution

Use this reference when a build differs across machines, passes locally but fails in CI, has incorrect/stale outputs, misses caches, needs remote caching/execution, or accesses runtime data/tools.

Contents

• Hermetic action contract
• Declared inputs, tools, environment, and outputs
• Sandboxing
• Runfiles
• Generation patterns
• Cache layers
• Remote cache and remote execution
• Cache diagnostics
• Non-determinism checklist
• Review checklist

Hermetic Action Contract

Given the same declared inputs, tools, command, environment, and platform configuration, an action should produce the same declared outputs. This is the foundation for correct incremental builds and safe cache sharing.

Conceptually, an action's identity includes:

• content/identity of declared input artifacts;
• executable/tool artifacts;
• argument vector and parameter-file content;
• declared action environment;
• expected output paths;
• relevant execution/target platform and toolchain properties;
• rule/action semantics represented in analysis.

If the action also reads undeclared host state, the cache key no longer describes the real computation. That can cause either:

• incorrect cache hits: different real inputs share one key;
• unexpected misses: an unnecessary declared/global difference changes the key;
• remote-only failures: a local tool/file/service is absent remotely;
• non-reproducible outputs: identical keys generate different bytes.

Declared Inputs

Every file an action reads should reach it through an attribute/provider or be generated by a declared dependency.

Common Hidden Inputs

• files found by walking the source tree at execution time;
• config read from a parent directory or repository root but not declared;
• .env, user config, home directory, or global package-manager state;
• compiler headers/SDK files found outside a declared toolchain;
• Git metadata read by a build action;
• locale/timezone databases;
• files written by another action but not connected by an artifact edge;
• runtime fixtures omitted from data;
• network responses.

Do not “fix” a missing input by granting a broader filesystem view. Add the narrow dependency edge or redesign the rule/tool.

Directory Inputs

Avoid directory inputs whose contents Bazel cannot enumerate. Depend on individual files or tree artifacts produced/consumed through rule APIs. In BUILD files, use explicit files, filegroup, or a controlled glob() rather than a raw directory label.

If a custom rule uses a tree artifact, the producing action owns the entire directory and consumers declare that artifact. Do not let unrelated actions write into it.

Declared Tools

A build tool is an input that executes under the execution configuration. It should be:

• an executable target on an appropriate rule attribute;
• supplied by a selected toolchain;
• accompanied by its runfiles/support files;
• compatible with the execution platform;
• passed to ctx.actions.run as an executable/tool, not rediscovered by name.

Avoid:

/usr/bin/protoc
which node
python from PATH
xcrun-discovered SDK without declared toolchain setup

Bazel cannot safely track arbitrary tools outside the workspace/output tree. Two machines can then share an action key while executing different binaries.

Host vs Target Tools

In cross-compilation, a generator/compiler runs on the execution platform while its output may target another platform. A tool dependency must use the exec transition/toolchain. Treating it as an ordinary target dependency can build an ARM tool that Bazel then tries to execute on x86, or vice versa.

Use cquery to see configured tool dependencies and aquery to see the concrete executable.

Environment

An action should receive a deliberate, minimal environment. Do not inherit the developer's entire shell environment into cacheable actions.

Potentially output-affecting values include:

• locale and timezone;
• compiler/runtime flags;
• SDK discovery variables;
• source-date or stamping inputs;
• feature toggles;
• language package-manager paths;
• home/temp/cache directories;
• network/proxy behavior.

Rules should declare environment inputs through supported APIs, toolchains, or build settings. Repository .bazelrc may set --action_env for unavoidable values, but this broadens action identity and can reduce cross-machine cache reuse. A pass-through environment variable is not automatically tracked correctly by every rule.

Prefer deterministic tool arguments such as a fixed locale/timezone/source date when semantics allow. Do not erase a meaningful input solely to gain a cache hit.

Outputs

Actions must write only declared outputs and must create each declared output in the expected form.

Avoid:

• writing back into source directories;
• side files next to a declared output;
• shared mutable cache directories as action outputs;
• appending to another action's file;
• output filenames based on clock/random/PID;
• undeclared package-manager installation trees.

Bazel isolates actions and publishes declared outputs. An undeclared output may disappear after the action or never reach a remote consumer.

Deterministic Bytes

Even when output filenames are declared, file contents may be non-deterministic because of:

• embedded timestamps or hostnames;
• unstable directory/map iteration;
• absolute execroot/temp paths;
• random identifiers;
• archive entry ordering/metadata;
• locale-dependent formatting;
• compiler/linker build IDs or debug paths;
• concurrent writers;
• unpinned downloads.

Normalize these at the tool/rule level and add a reproducibility test that runs equivalent builds in distinct output roots or machines.

Sandboxing

Bazel's filesystem sandbox gives an action a working directory containing declared inputs and captures declared outputs. It is a strong detector of missing file dependencies and approximates remote execution.

Sandboxing does not make every action completely hermetic:

• host environment can still leak unless controlled;
• platform facilities and some absolute paths may remain visible;
• network access is a separate concern;
• toolchains/repository rules can still discover host state;
• OS-specific sandbox implementations differ.

Treat sandbox success as necessary but not sufficient evidence.

Debug a Sandboxed Failure

bazel build //app:target \
  --verbose_failures \
  --sandbox_debug

This shows the concrete command and preserves sandbox directories for inspection. Compare with:

bazel aquery 'deps(//app:target)' --include_param_files

Questions:

• Is the missing file in the action's declared inputs?
• Is the executable declared as a tool and present for the execution platform?
• Does the command contain an absolute local path?
• Does the action write outside declared outputs?
• Is a runtime file incorrectly treated as a build-time file or vice versa?

Disable --sandbox_debug after diagnosis; preserved sandboxes consume disk.

Local Strategy Is a Diagnostic

If --spawn_strategy=local makes a failure disappear, that is evidence of undeclared host state. Do not submit the local strategy as the general fix unless the action is intentionally non-hermetic and explicitly excluded from remote/cache guarantees with a documented reason.

Runfiles

Runfiles are runtime dependencies for binaries and tests. Declare them through data or ruleset-specific runtime dependency providers.

some_test(
    name = "parser_test",
    srcs = ["parser_test.ext"],
    data = ["//testdata/parser:cases"],
    deps = [":parser"],
)

At runtime:

• use the language-specific Bazel runfiles library;
• resolve an rlocation path based on apparent repository name and package path;
• do not assume the current working directory;
• do not hardcode bazel-bin, .runfiles, execroot, or a canonical @@repo name;
• do not use a source-tree relative path that bypasses runfiles.

Runfiles behavior differs across operating systems (directory trees vs manifests). A direct filesystem path that works on one host may fail on Windows or remote execution.

Tests

Bazel's test runner provides isolated runtime directories and variables. Tests should:

• write temporary files under the test temporary directory;
• use runfiles for fixtures and helper binaries;
• avoid shared mutable ports/state where possible;
• not depend on user HOME, credentials, or current shell cwd;
• declare sharding support before setting shard_count;
• emit undeclared diagnostic artifacts only through the supported test output directory.

Generation Patterns

Prefer Purpose-Built Rules

Use an existing ruleset rule when available. It knows provider, platform, toolchain, runfiles, and output semantics that a shell command may miss.

genrule Boundaries

genrule is useful for simple transformations, but shell portability and tool declaration become fragile quickly.

genrule(
    name = "compile_schema",
    srcs = ["schema.idl"],
    outs = ["schema.generated"],
    cmd = "$(location //tools/schema:compiler) $(location schema.idl) > $@",
    tools = ["//tools/schema:compiler"],
)

Even this example depends on the tool's stdout contract and shell behavior. For complex generation, multiple outputs, platform-specific arguments, or provider propagation, write/use a custom rule with ctx.actions.run.

Avoid:

• invoking an undeclared tool by name;
• cd into source directories and broad tree scans;
• writing more files than outs;
• depending on shell utilities that are not declared/portable;
• downloading inside cmd;
• using $(location) for a multi-file target when $(locations) or a dedicated attribute/API is required.

Custom Actions

For ctx.actions.run:

• declare inputs, outputs, executable, tools, and arguments precisely;
• use ctx.actions.args() for scalable/portable argument construction;
• pass transitive inputs as a depset, not a flattened list;
• use toolchains for compiler/runtime selection;
• choose an informative stable mnemonic and progress message;
• avoid use_default_shell_env = True unless the rule's contract truly needs it;
• model runfiles and providers for downstream targets.

Cache Layers

Do not conflate these caches:

Deleting one does not explain or clear every other. bazel clean targets build outputs/action state; --expunge removes the output base and server. It is not routine cache correctness maintenance.

Disk Cache

build --disk_cache=/path/to/bazel-disk-cache

A disk cache uses remote-cache protocols locally. Put environment-specific paths in the appropriate local/CI config rather than assuming one checked-in absolute path works everywhere.

Repository Cache

The repository cache reuses fetched archives across workspaces when repository downloads specify content checksums. It does not cache compilation/test actions. Missing checksums reduce reproducibility and reuse.

Remote Cache

A remote cache shares action results and content-addressed output blobs across machines. It is safe only when actions are reproducible and action keys cover their real inputs.

Configuration commonly includes a provider endpoint and authentication plus read/write policy. Keep provider-specific flags in named config/imported rc files according to repository policy.

Rollout

1. Establish sandboxed local correctness.
2. Pin Bazel, rulesets, external dependencies, and toolchains.
3. Enable cache reads for a representative target matrix.
4. Compare rebuilt outputs and test results across machines/platforms.
5. Enable writes from a controlled CI population.
6. Expand developer writes only when trust/isolation policy supports it.
7. Monitor hit rate, upload/download volume, error rate, and incorrect-result reports.

Do not measure value solely by “cache hit” lines. Restoring tiny actions can cost more than execution; downloading all outputs can dominate time. Profile end-to-end critical paths.

Read/Write Policy

Consider:

• whether untrusted changes may upload results consumed by protected builds;
• namespace/instance separation across incompatible projects or toolchains;
• retention and eviction;
• read-only credentials for contexts that should not publish;
• how no-cache/local-only tags or provider controls are applied;
• whether test results, stdout/stderr, and artifacts have appropriate visibility.

Remote Execution

Remote execution sends eligible actions to workers implementing the Remote Execution API. It adds compute scaling and a controlled execution environment, but it imposes a stricter contract than cache-only use.

Requirements:

• every input/tool must be uploadable and declared;
• executables must match the execution platform;
• actions cannot depend on local services/files/devices unless modeled;
• outputs must be deterministic and declared;
• platform properties must select the correct worker image/capabilities;
• tests must be self-contained or explicitly routed to a compatible executor.

Adoption Sequence

1. Hermetic local toolchains and sandboxed actions.
2. Shared remote cache with output reproducibility checks.
3. A representative remote-execution config/platform.
4. Route supported action mnemonics/targets; keep documented exceptions local.
5. Compare action/test output with local execution.
6. Tune download mode, concurrency, worker properties, and dynamic execution using profiles.

Dynamic Execution

Dynamic execution races eligible local and remote branches and keeps the first successful result. Use it only after remote execution works. It can improve mixed clean/incremental workloads but consumes extra resources and can reveal local/remote discrepancies.

Do not enable it blindly for every action. Linking, short actions, persistent-worker actions, and high-latency remote paths have different tradeoffs. Profile and select strategies by mnemonic/execution group where supported.

Cache Diagnostics

Unexpected Misses

First confirm the invocations are actually comparable:

• same Bazel and ruleset versions;
• same MODULE.bazel.lock and external contents;
• same rc expansion and flags;
• same target/execution platform/toolchain;
• same source/input content;
• same relevant environment and stamping state;
• compatible remote cache instance/permissions.

Capture compact execution logs:

bazel build //app:target --execution_log_compact_file=/tmp/exec-one.log
bazel build //app:target --execution_log_compact_file=/tmp/exec-two.log

The official execution-log parser can align and compare action records. Differences in inputs, argv, environment, platform, or outputs explain action-key divergence. Compact format is preferred because it is smaller and cheaper than JSON/binary alternatives.

Also inspect:

bazel aquery 'mnemonic("RelevantMnemonic", deps(//app:target))' \
  --include_param_files \
  --output=text

Incorrect Hits or Non-Reproducible Results

Treat this as a correctness incident:

1. Preserve invocation IDs, BEP, action/execution logs, exact source/config/toolchain identities, and both output copies.
2. Rebuild in isolated output roots and with cache reads/writes controlled.
3. Compare action declarations (aquery) and output bytes/metadata.
4. Look for undeclared host tools/files/environment/network and non-deterministic output generation.
5. Fix and regression-test the action/rule.
6. Invalidate/quarantine affected cache entries through the provider's supported mechanism when necessary.

Do not assume “Bazel cache is stale” and normalize the workflow around clean --expunge. The graph/action contract must be corrected.

Remote-Only Failure

Reproduce under the closest local sandbox/container/toolchain environment, but remember a container-only workaround is not a declared toolchain.

Non-Determinism Checklist

• Stable input enumeration and archive ordering
• Normalized timestamps/owners/modes where artifact format permits
• No hostname, username, home, temp, execroot, or workspace absolute path in output
• Fixed locale/timezone/encoding
• Seeded or removed randomness
• No concurrent writes to shared output/cache files
• Pinned compilers, interpreters, SDKs, plugins, and fetched dependencies
• Consistent line endings/path separators across supported platforms
• Deterministic compiler/linker flags and debug-path remapping
• No uncontrolled network response in actions
• Tests isolate ports/processes/state and clean up children

Review Checklist

• All files read by actions are declared artifacts.
• All executables/support files come from tool attributes or toolchains.
• Action environment is deliberate and output-affecting values are tracked.
• Actions write only declared outputs with deterministic contents.
• Runtime resources use data and runfiles APIs.
• Sandbox is enabled for normal validation; local strategy is not a hidden-input workaround.
• Cache layers are identified correctly before clearing anything.
• Remote cache rollout includes reproducibility and trust/isolation policy.
• Remote execution platform/toolchain constraints match actual workers.
• Cache discrepancies use execution logs/aquery/BEP, not only wall-clock or terminal text.

Sources

• Hermeticity: https://bazel.build/concepts/hermeticity
• Sandboxing: https://bazel.build/docs/sandboxing
• Runfiles: https://bazel.build/concepts/runfiles
• Dependencies (srcs/deps/data): https://bazel.build/concepts/dependencies
• Remote caching: https://bazel.build/remote/caching
• Remote execution overview: https://bazel.build/remote/rbe
• Adapting rules for remote execution: https://bazel.build/remote/rules
• Dynamic execution: https://bazel.build/remote/dynamic
• Debugging remote cache hits: https://bazel.build/remote/cache-remote
• Output directory layout: https://bazel.build/remote/output-directories
• Action graph query: https://bazel.build/query/aquery
• Test encyclopedia: https://bazel.build/reference/test-encyclopedia