The "Ceph Tax": The Price of 3x Replication
We love Ceph. It is self-healing, highly available, and theoretically scales forever. But in the enterprise and homelab space alike, Ceph comes with a brutal reality check often referred to as the "Ceph Tax".
To provide that incredible fault tolerance, a standard Ceph deployment uses 3x replication. If a user saves a 1TB video file to your CephFS share, it instantly consumes 3TB of raw storage across your cluster's OSDs (Object Storage Daemons).
For active databases and virtualization, that redundancy is worth every penny. But what happens to that 1TB video file 6 months later? It gets cold. No one watches it. Yet, it sits there, monopolizing 3TB of expensive NVMe or HDD space, forcing you to keep three separate drives spinning, drawing power, generating heat, and taking up IOPS during background scrubbing.
Expanding a Ceph cluster just to hold cold data is financial madness. You have to buy drives in balanced sets, upgrade network switches, and increase your power budget. This is where HuskHoard steps in to permanently cure cold cluster .
Ceph IS the Hot Tier (The Gateway Pattern)
When people first hear about HuskHoard tiering, they assume they need a separate, dedicated "Hot Tier" NVMe drive to sit in front of their Ceph cluster. This is a misconception. In this architecture, Ceph is the Hot Tier.
HuskHoard doesn't replace Ceph; it operates directly inside it via the Gateway Node Pattern.
Because Ceph is distributed, files are chunked into objects across dozens of servers. Linux's fanotify (the kernel feature HuskHoard uses to pause reads and trigger restores) requires a unified POSIX mount point. Therefore, you run HuskHoard on your Gateway Server—the machine that mounts CephFS and serves it to your users via Samba, NFS, or Nextcloud.
The Brute Force Method
The Gateway Offload
The Magic of Native RADOS Deletion
Here is where the technical synergy between HuskHoard and CephFS becomes borderline magical.
When you configure HuskHoard to monitor your CephFS kernel mount (e.g., /mnt/ceph_gateway), it watches for files that haven't been touched in, say, 30 days. When the Janitor thread wakes up, it reads that cold file, writes a compressed copy to your offline Tape Drive or S3 bucket, and then issues a Linux fallocate (hole-punch) command.
The CephFS kernel client intercepts that fallocate command and translates it directly into RADOS object deletions. Ceph literally deletes the underlying 4MB chunks from your OSDs. The 3x replication overhead vanishes instantly.
fallocate(PUNCH_HOLE), which is what actually triggers RADOS object deletion. The CephFS FUSE client handles hole-punching inconsistently and should not be relied on for this workflow.The beauty of this integration is the Metadata Server (MDS). Even though the massive data chunks have been securely deleted from the OSDs, the Ceph MDS keeps the file's metadata perfectly intact. To the user browsing the Samba share, the file still appears to be exactly 100GB. It retains its original creation date, permissions, and location.
To ensure this hole-punching magic works flawlessly, your Gateway node must mount CephFS using the Linux Kernel Client (not FUSE). The kernel client natively supports the fallocate and Extended Attributes (xattr) that HuskHoard relies on to track stubbed files.
If you are a Ceph admin, you might be thinking: "Wait, Ceph deprecated cache tiering. Won't moving cold blocks cause massive latency spikes and mark Placement Groups (PGs) offline like legacy HSMs (e.g., Dell Compellent)?"
HuskHoard is NOT a RADOS-level cache tier. It operates 100% at the POSIX/CephFS layer.
- No Offline PGs: Ceph natively handles sparse files beautifully. A hole-punched file is a healthy, expected POSIX state. From Ceph's perspective, the file is perfectly fine; the RADOS objects are just empty.
- Zero Cluster Latency Spikes: Legacy block-level HSMs paused entire storage controllers to migrate data, spiking latency across the tier. HuskHoard uses
fanotifyto pause only the specific application PID requesting the file. The rest of your cluster and other clients experience zero IO pause or latency impact.
Zero-Friction Access & StreamGate
Offloading data is easy. Getting it back seamlessly is the hard part. If a user double-clicks a stubbed video file six months later, what happens?
HuskHoard provides two distinct recall paths, and it is important not to confuse them.
Path 1 — Standard file access (fanotify). If a user double-clicks a stubbed file from a Samba or NFS share, HuskHoard intercepts the open request via fanotify before the application reaches the empty RADOS objects. The application is paused, HuskHoard fetches the complete file from tape or S3, and writes it in full back to the CephFS mount. Ceph re-creates all of the corresponding RADOS objects on your OSDs in real-time, and the application is unpaused with no error. The file is now fully resident on the hot tier again. The trade-off is recall time: a tape load and seek typically takes 30–90 seconds, so this path is best suited to workloads where occasional recall latency is acceptable.
Path 2 — StreamGate (media players via HTTP). For media servers like Plex or Jellyfin, or players like VLC pointed at HuskHoard's local HTTP gateway (localhost:8080), StreamGate provides a fundamentally different path. The player issues an HTTP Range request for the bytes it needs right now. StreamGate uses a Jump-Table in the Husk Catalog to seek directly to that position on tape or in the S3 object, streams only those bytes over HTTP, and never writes anything back to CephFS. No RADOS objects are recreated. For a 100GB video, starting playback means fetching only the first few megabytes rather than waiting for a full file recall. This is the zero-disk path.
The choice between them is architectural: the fanotify path is fully transparent to every application with no configuration changes required. The StreamGate path requires pointing your media software at an HTTP endpoint instead of a filesystem path, but gives you immediate random access and keeps your Ceph cluster untouched until a deliberate full recall is triggered.
The Economics of Offloading
Let's look at a real-world scenario: a 100TB production dataset where 80% of the data is older than 60 days. In a standard 3x replication Ceph setup, that cold 80TB actually consumes 240TB of raw, spinning, power-hungry cluster storage.
| Architecture (100TB Dataset) | Raw Ceph Space Used | Cold Tier Required | Power & Wear |
|---|---|---|---|
| Standard Ceph (3x Rep) | 300TB (Hot + Cold) | None | High (All OSDs spinning 24/7) |
| Ceph + HuskHoard (80% Offload) | 60TB (Hot Data Only) | 1x 80TB Tape/S3 | Low (OSD count reduced by 80%) |
By deploying HuskHoard on your Gateway node, you reclaim 240TB of Ceph storage overnight. You can literally decommission OSD nodes, lower your rack's thermal output, and dramatically reduce the time it takes Ceph to perform background scrubs and rebalancing.
Bypass the 3x replication penalty for cold data. HuskHoard moves it to a 1:1 offline replica, instantly freeing up massive amounts of raw block storage in your OSDs.
Because the file stubs remain in CephFS, your Metadata Servers (MDS) continue to serve directory listings and searches at lightning speed without waking up cold storage.
Whether you use LTO Tape, sleeping CMR drives, or cloud buckets, your cold data consumes zero local electricity and zero cluster IOPS until the exact moment a user requests it.
The Verdict
Ceph is designed for performance, high availability, and active workloads. It was never meant to be a cheap, deep-freeze archive. By pairing CephFS with HuskHoard, you get the best of both worlds: the indestructible, distributed performance of Ceph for your hot data, and the unbeatable economics of Tape/S3 for your cold data—all seamlessly unified behind a single namespace.
Stop paying the Ceph Tax on data nobody is looking at. Deploy a Gateway, punch those holes, and let your cluster breathe.