[build] Add ccache support for C/C++ compilation (SONIC_CONFIG_USE_CCACHE) by rustiqly · Pull Request #25650 · sonic-net/sonic-buildimage

rustiqly · 2026-02-24T19:52:17Z

Description

Add optional ccache integration to dramatically speed up incremental C/C++ builds. When SONIC_CONFIG_USE_CCACHE=y is set, ccache intercepts gcc/g++ calls and caches object files, making recompilation of unchanged source near-instant.

Motivation

The critical build path includes three C++ heavy packages:

libswsscommon: 57 .cpp files, 2m build time
libsairedis: 3m18s build time
swss: 4m49s build time

These packages are on the serial critical path (libswsscommon → libsairedis → swss = ~10m). With a warm ccache, incremental rebuilds (e.g., when only one file changes in swss) drop from minutes to seconds.

Changes

File	Change
`rules/config`	Add `SONIC_CONFIG_USE_CCACHE` knob (default: `n`)
`slave.mk`	Define `CCACHE_ENV`, inject into build commands, add `ccache-stats`/`ccache-clear` targets
`sonic-slave-bookworm/Dockerfile.j2`	Install `ccache` package
`sonic-slave-trixie/Dockerfile.j2`	Install `ccache` package

How it works

ccache is installed in the sonic-slave container (adds ~1MB)
When enabled, /usr/lib/ccache is prepended to PATH — this directory contains symlinks (gcc, g++, cc, c++) that route through ccache
Cache is stored at target/ccache/<distro>/ — persists across builds, per-distro to avoid cross-contamination
CCACHE_COMPILERCHECK=content ensures cache invalidation on compiler upgrades
When disabled (default), CCACHE_ENV is empty — zero impact

Usage

# Enable in config.user (persistent)
echo 'SONIC_CONFIG_USE_CCACHE = y' >> rules/config.user

# Or one-shot
make SONIC_CONFIG_USE_CCACHE=y target/sonic-vs.img.gz

# Check cache statistics
make ccache-stats

# Clear cache
make ccache-clear

Expected Impact

Scenario	Without ccache	With warm ccache
Full clean build	~53m	~53m (first build populates cache)
Rebuild after swss change	~5m (swss)	~10s (only changed files recompile)
Rebuild after no changes	~10m (full critical path)	~30s (all cache hits)

Backward Compatibility

Default n — no behavior change unless explicitly enabled
ccache package is installed but dormant when not enabled
No impact on cross-builds (CCACHE_ENV applied uniformly)
DPKG cache (SONIC_DPKG_CACHE_METHOD) is orthogonal and works alongside ccache

Testing

Full VS build test pending with CI.

rustiqly · 2026-02-24T19:52:21Z

@copilot review

mssonicbld · 2026-02-24T19:52:51Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-24T19:53:04Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-24T22:08:43Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-24T22:08:55Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-24T22:28:09Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-24T22:28:20Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Adds an opt-in SONIC_CONFIG_USE_CCACHE build knob to integrate ccache into the SONiC build environment, aiming to speed up incremental C/C++ rebuilds by caching compilation outputs.

Changes:

Introduces SONIC_CONFIG_USE_CCACHE configuration option (default n).
Installs ccache in the bookworm and trixie sonic-slave images.
Injects a CCACHE_ENV prefix into Debian package build commands and adds ccache-stats / ccache-clear helper targets.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
`rules/config`	Adds `SONIC_CONFIG_USE_CCACHE` knob and documentation comment.
`slave.mk`	Defines `CCACHE_ENV`, prefixes build commands, prints knob value, and adds ccache helper targets.
`sonic-slave-bookworm/Dockerfile.j2`	Installs `ccache` in the bookworm build container.
`sonic-slave-trixie/Dockerfile.j2`	Installs `ccache` in the trixie build container.

Copilot · 2026-02-24T22:32:45Z

slave.mk

+CCACHE_ENV = PATH=/usr/lib/ccache:$$PATH CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) CCACHE_BASEDIR=$(BUILD_WORKDIR) CCACHE_COMPILERCHECK=content CCACHE_UMASK=000
+$(shell mkdir -p $(TARGET_PATH)/ccache/$(BLDENV) && chmod 777 $(TARGET_PATH)/ccache/$(BLDENV))


CCACHE_DIR is set as a relative path ("$(TARGET_PATH)/ccache/$(BLDENV)"). Since the build recipes run sub-builds in other working directories (e.g., make -C ... and pushd $($*_SRC_PATH)), ccache will resolve this relative to the package directory and end up writing to per-package src/.../target/ccache/... instead of the intended top-level target/ccache/..., which also makes ccache-stats misleading. Use an absolute path for CCACHE_DIR (e.g., based on $(PROJECT_ROOT) or $(abspath ...)) so all build steps share the same cache location.

Suggested change

CCACHE_ENV = PATH=/usr/lib/ccache:$$PATH CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) CCACHE_BASEDIR=$(BUILD_WORKDIR) CCACHE_COMPILERCHECK=content CCACHE_UMASK=000

$(shell mkdir -p $(TARGET_PATH)/ccache/$(BLDENV) && chmod 777 $(TARGET_PATH)/ccache/$(BLDENV))

CCACHE_ENV = PATH=/usr/lib/ccache:$$PATH CCACHE_DIR=$(abspath $(TARGET_PATH)/ccache/$(BLDENV)) CCACHE_BASEDIR=$(BUILD_WORKDIR) CCACHE_COMPILERCHECK=content CCACHE_UMASK=000

$(shell mkdir -p $(abspath $(TARGET_PATH)/ccache/$(BLDENV)) && chmod 777 $(abspath $(TARGET_PATH)/ccache/$(BLDENV)))

All addressed in latest push:

CCACHE_DIR now uses $(abspath ...) for a stable absolute path

Added command -v ccache guard that errors early if ccache is missing

Tightened UMASK from 000 to 022

ccache-stats uses the same $(CCACHE_DIR) variable

Hint now says rules/config.user instead of rules/config

Thanks for the thorough review!

Copilot · 2026-02-24T22:32:46Z

slave.mk

+###############################################################################
+ccache-stats :
+ifeq ($(SONIC_CONFIG_USE_CCACHE),y)
+	CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) ccache -s


ccache-stats uses a relative CCACHE_DIR ("$(TARGET_PATH)/ccache/$(BLDENV)"). If builds run from other working directories (e.g., sub-makes in package directories), this may not point at the same cache used during compilation. After switching CCACHE_DIR to an absolute path, update this target to match so the stats reflect the actual cache.

Suggested change

CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) ccache -s

CCACHE_DIR=$(CCACHE_DIR) ccache -s

Copilot · 2026-02-24T22:32:46Z

slave.mk

+ifeq ($(SONIC_CONFIG_USE_CCACHE),y)
+	CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) ccache -s
+else
+	@echo "ccache is not enabled. Set SONIC_CONFIG_USE_CCACHE=y in rules/config"


The message suggests editing rules/config to enable ccache, but user overrides are typically done via rules/config.user (which is included after rules/config) or via the make command line. Consider updating the hint so it doesn't encourage modifying the tracked rules/config file.

Suggested change

@echo "ccache is not enabled. Set SONIC_CONFIG_USE_CCACHE=y in rules/config"

@echo "ccache is not enabled. Set SONIC_CONFIG_USE_CCACHE=y in rules/config.user or pass SONIC_CONFIG_USE_CCACHE=y on the make command line"

Copilot · 2026-02-24T22:32:46Z

slave.mk

+# calls are intercepted by ccache symlinks. Cache is stored under target/ccache/
+# and persists across builds for near-instant recompilation of unchanged files.
+CCACHE_ENV =
+ifeq ($(SONIC_CONFIG_USE_CCACHE),y)


This enables ccache purely via PATH/CCACHE_* env vars, but ccache is only installed in the bookworm/trixie slave images in this PR. On other supported build envs (e.g., bullseye), enabling SONIC_CONFIG_USE_CCACHE=y will silently do nothing for compilation (no /usr/lib/ccache symlinks) and make ccache-stats will fail with ccache: command not found. Either install ccache in the other supported slave images as well, or add a guard that errors out early when ccache isn't available.

Suggested change

ifeq ($(SONIC_CONFIG_USE_CCACHE),y)

ifeq ($(SONIC_CONFIG_USE_CCACHE),y)

ifeq ($(shell command -v ccache >/dev/null 2>&1 || echo notfound),notfound)

$(error SONIC_CONFIG_USE_CCACHE=y but 'ccache' is not installed in the build environment. Install ccache in the slave image or unset SONIC_CONFIG_USE_CCACHE)

endif

Copilot · 2026-02-24T22:32:46Z

slave.mk

+CCACHE_ENV = PATH=/usr/lib/ccache:$$PATH CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) CCACHE_BASEDIR=$(BUILD_WORKDIR) CCACHE_COMPILERCHECK=content CCACHE_UMASK=000
+$(shell mkdir -p $(TARGET_PATH)/ccache/$(BLDENV) && chmod 777 $(TARGET_PATH)/ccache/$(BLDENV))


Setting CCACHE_UMASK=000 and applying chmod 777 to $(TARGET_PATH)/ccache/$(BLDENV) makes the ccache directory world-writable. On multi-user build hosts or shared workspaces, another local user who can traverse the repo can modify cached object files and have them reused in later builds, allowing injection of arbitrary code into built images/packages. Restrict the cache directory to the build user only (e.g., a per-user cache location with a tight umask and non-world-writable permissions) so other accounts cannot tamper with build artifacts.

Suggested change

CCACHE_ENV = PATH=/usr/lib/ccache:$$PATH CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) CCACHE_BASEDIR=$(BUILD_WORKDIR) CCACHE_COMPILERCHECK=content CCACHE_UMASK=000

$(shell mkdir -p $(TARGET_PATH)/ccache/$(BLDENV) && chmod 777 $(TARGET_PATH)/ccache/$(BLDENV))

CCACHE_ENV = PATH=/usr/lib/ccache:$$PATH CCACHE_DIR=$(TARGET_PATH)/ccache/$(USER)/$(BLDENV) CCACHE_BASEDIR=$(BUILD_WORKDIR) CCACHE_COMPILERCHECK=content CCACHE_UMASK=077

$(shell mkdir -p $(TARGET_PATH)/ccache/$(USER)/$(BLDENV) && chmod 700 $(TARGET_PATH)/ccache/$(USER)/$(BLDENV))

mssonicbld · 2026-02-24T22:33:32Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-24T22:33:44Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-25T00:47:18Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T00:47:31Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-25T04:49:13Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T04:49:24Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-25T05:47:27Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T05:47:38Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-25T07:50:17Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T07:50:28Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-25T08:59:37Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T08:59:49Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-25T09:58:24Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T09:58:36Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-02-25T11:59:02Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T11:59:15Z

Azure Pipelines successfully started running 1 pipeline(s).

…ACHE) Add optional ccache integration to speed up incremental C/C++ builds. When SONIC_CONFIG_USE_CCACHE=y is set in rules/config, ccache intercepts gcc/g++ calls via /usr/lib/ccache PATH prepend and caches compilation results under target/ccache/<distro>/. Changes: - rules/config: Add SONIC_CONFIG_USE_CCACHE knob (default: n) - slave.mk: Define CCACHE_ENV with PATH, CCACHE_DIR, CCACHE_BASEDIR, and CCACHE_COMPILERCHECK; inject into dpkg-buildpackage and make build commands; add ccache-stats and ccache-clear targets; print config status in build info - sonic-slave-bookworm/Dockerfile.j2: Install ccache package - sonic-slave-trixie/Dockerfile.j2: Install ccache package Benefits: - First build populates the cache (minimal overhead from ccache lookups) - Subsequent builds with unchanged sources: near-instant recompilation - Key packages on the critical path (libswsscommon 2m, libsairedis 3m, swss 5m) benefit most from cache hits during iterative development - Cache is per-distro (bookworm/trixie) to avoid cross-contamination - CCACHE_COMPILERCHECK=content ensures cache validity across compiler updates Usage: echo 'SONIC_CONFIG_USE_CCACHE = y' >> rules/config.user make ccache-stats # view hit/miss statistics make ccache-clear # wipe cache Signed-off-by: Rustiqly <rustiqly@users.noreply.github.com>

mssonicbld · 2026-02-25T15:58:51Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-02-25T15:59:03Z

Azure Pipelines failed to run 1 pipeline(s).

rustiqly requested review from lguohan, qiluo-msft and xumia as code owners February 24, 2026 19:52

lihuay requested a review from Copilot February 24, 2026 22:27

Copilot started reviewing on behalf of lihuay February 24, 2026 22:27 View session

Copilot AI reviewed Feb 24, 2026

View reviewed changes

rustiqly force-pushed the ccache-support branch from 6faeb06 to e85a3e6 Compare February 25, 2026 00:47

rustiqly force-pushed the ccache-support branch from e85a3e6 to ba589dc Compare February 25, 2026 04:49

rustiqly force-pushed the ccache-support branch from ba589dc to 02a21f5 Compare February 25, 2026 05:46

rustiqly force-pushed the ccache-support branch from 02a21f5 to 2ce38e7 Compare February 25, 2026 07:47

rustiqly force-pushed the ccache-support branch from 2ce38e7 to 0a346c7 Compare February 25, 2026 08:58

rustiqly force-pushed the ccache-support branch from 0a346c7 to 2a307d3 Compare February 25, 2026 09:57

rustiqly force-pushed the ccache-support branch from 2a307d3 to 0768cd0 Compare February 25, 2026 11:58

rustiqly force-pushed the ccache-support branch from 0768cd0 to cb588a6 Compare February 25, 2026 15:58

		CCACHE_ENV = PATH=/usr/lib/ccache:$$PATH CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) CCACHE_BASEDIR=$(BUILD_WORKDIR) CCACHE_COMPILERCHECK=content CCACHE_UMASK=000
		$(shell mkdir -p $(TARGET_PATH)/ccache/$(BLDENV) && chmod 777 $(TARGET_PATH)/ccache/$(BLDENV))

	CCACHE_DIR=$(TARGET_PATH)/ccache/$(BLDENV) ccache -s
	CCACHE_DIR=$(CCACHE_DIR) ccache -s

	@echo "ccache is not enabled. Set SONIC_CONFIG_USE_CCACHE=y in rules/config"
	@echo "ccache is not enabled. Set SONIC_CONFIG_USE_CCACHE=y in rules/config.user or pass SONIC_CONFIG_USE_CCACHE=y on the make command line"

Comments

Conversation

rustiqly commented Feb 24, 2026

Description

Motivation

Changes

How it works

Usage

Expected Impact

Backward Compatibility

Testing

Uh oh!

rustiqly commented Feb 24, 2026

Uh oh!

mssonicbld commented Feb 24, 2026

Uh oh!

azure-pipelines bot commented Feb 24, 2026

Uh oh!

mssonicbld commented Feb 24, 2026

Uh oh!

azure-pipelines bot commented Feb 24, 2026

Uh oh!

mssonicbld commented Feb 24, 2026

Uh oh!

azure-pipelines bot commented Feb 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

rustiqly Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Feb 24, 2026

Uh oh!

azure-pipelines bot commented Feb 24, 2026

Uh oh!

mssonicbld commented Feb 25, 2026

Uh oh!

azure-pipelines bot commented Feb 25, 2026

Uh oh!

mssonicbld commented Feb 25, 2026

Uh oh!

azure-pipelines bot commented Feb 25, 2026

Uh oh!

mssonicbld commented Feb 25, 2026

Uh oh!

azure-pipelines bot commented Feb 25, 2026

Uh oh!

mssonicbld commented Feb 25, 2026

Uh oh!

azure-pipelines bot commented Feb 25, 2026

Uh oh!

mssonicbld commented Feb 25, 2026

Uh oh!

azure-pipelines bot commented Feb 25, 2026

Uh oh!

mssonicbld commented Feb 25, 2026

Uh oh!

azure-pipelines bot commented Feb 25, 2026

Uh oh!

mssonicbld commented Feb 25, 2026

Uh oh!

azure-pipelines bot commented Feb 25, 2026