Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Deduplication and Compression

Nix stores lots of similar data. Two versions of the same package often differ in only a few files, and the same package often ends up in several caches. Celler deduplicates on two levels so it stores this data only once.

NAR-Level Deduplication

Every NAR is identified by its hash. If someone uploads a NAR that the server already has, in any cache, Celler doesn’t store it again. It only adds a reference from the cache to the existing NAR.

This happens on the server and is invisible to the client. By default, the client still has to upload the full NAR to prove that it has the data. See Proof of Possession.

Chunk-Level Deduplication

If chunking is enabled, new NARs are split into chunks with the FastCDC algorithm. FastCDC places chunk boundaries based on the content, so a small change in a file only affects the chunks around the change. Chunks that already exist are not stored again.

When Nix downloads a chunked NAR, cellerd reads its chunks from the storage backend and streams them to the client as one NAR.

NARs smaller than nar-size-threshold are stored as a single chunk; NAR-level deduplication still applies to them.

Chunking is optional and costs performance. See Chunking for the trade-offs.

Why Chunk Whole NARs Instead of Individual Files?

Celler splits the entire uncompressed NAR, ignoring the boundaries of the files inside it.

The NARs that benefit the most from deduplication, such as VSCode or Zoom, often contain thousands of small files. Storing each file separately would mean fetching thousands of objects from the storage backend to rebuild a single NAR, which is slow and expensive. Chunking the whole NAR lets small files share a chunk and keeps the number of chunks manageable. casync takes the same approach.

The Tvix store protocol, in contrast, chunks individual files. Celler’s design is driven by making good use of existing platforms, such as S3, and their practical limits.

Compression

The server compresses each chunk before writing it to the storage backend (zstd by default, see [compression]). Deduplication uses the hash of the uncompressed chunk, together with the compression type. If you change the compression type, new uploads don’t reuse chunks stored with the old type. NAR-level deduplication isn’t affected.

                        ┌───────────────────────────────────►Chunk Hash
                        │
                        │
                        ├───────────────────────────────────►Chunk Size
                        │
                ┌───────┴────┐  ┌──────────┐  ┌───────────┐
 Chunk Stream──►│Chunk Hasher├─►│Compressor├─►│File Hasher├─►File Stream─►S3
                └────────────┘  └──────────┘  └─────┬─────┘
                                                    │
                                                    ├───────►File Hash
                                                    │
                                                    │
                                                    └───────►File Size

Missing or Corrupt Chunks

If a chunk is deleted from the database, every NAR that uses it becomes unavailable and the server responds with an error (503). This fixes itself once someone uploads any NAR that contains the chunk again.

At the moment, Celler can’t detect on its own whether a chunk in the storage backend is corrupt or missing. Telling temporary storage errors apart from permanent ones is difficult.