The Abstraction Applies Everywhere
In our last post, we established that HuskHoard treats local storage as a presentation layer. It uses fanotify to intercept file requests and pull data from offline tapes or cold disks on-demand. But the logic that powers a local tape robot scales perfectly to the internet.
To HuskHoard's storage engine, AWS S3, Google Cloud Storage, or Backblaze B2 are not special. They are simply treated as "Append-Only" storage mediums with very high latency—conceptually identical to an LTO-9 tape drive. Because HuskHoard natively integrates with Rclone via child-process pipes, any of the 70+ cloud providers supported by Rclone can be mounted as a Husk volume.
LTO Tape Drive (/dev/nst0)
A sequential, append-only medium. Requires physical hardware, robots, and physical space. Retrieving a file means rewinding the tape, seeking to a byte offset, and reading.
Rclone Remote (rclone:s3:bucket)
A sequential, append-only medium (from Husk's perspective). Requires network bandwidth. Retrieving a file means making an HTTP Range request to a specific byte offset, and reading.
Because the engine abstracts the storage backend, you can architect your deployment in two distinct ways: using the cloud purely as a bottomless dump for a local server, or running HuskHoard itself inside the cloud to act as a centralized, global gateway.
Architecture A: The Local Cloud Gateway
In this architecture, HuskHoard runs on a machine inside your office or homelab. Your "Hot Tier" is a fast local NVMe drive or a local ZFS array. Your "Cold Tier" is a cloud bucket.
You work off the local NVMe drive at LAN speeds. When files age past your policy threshold (e.g., 30 days), the Husk Janitor daemon wakes up, compresses the file via `zstd`, and pipes it directly through Rclone into your cloud bucket. The local file is then "stubbed"—reduced to zero bytes on the NVMe drive, but retaining its original size, timestamps, and permissions in the directory listing.
If a user tries to open a stubbed file 6 months later, the fanotify interceptor pauses the application. HuskHoard calculates exactly where that file lives inside the cloud object, spawns rclone cat --offset X --count Y, and streams the decrypted, decompressed bytes directly into the application's memory buffer.
read().The Advantage: You get local NVMe performance for active projects, but your storage capacity is infinite. You aren't paying to keep terabytes of cold data on local spinning rust, and you don't have to manage physical tapes. It's a bottomless, self-managing local NAS.
Architecture B: The Cloud-Native Hub
What if your team is fully remote? You don't have a local office to put an NVMe drive in. In this architecture, HuskHoard runs on a cloud compute instance (like an AWS EC2 or DigitalOcean Droplet).
Here, the EC2 instance has a small, fast EBS (Elastic Block Store) volume mounted as the Hot Tier. The Cold Tier is an S3 bucket (or a cheaper cold-block volume like AWS st1). Users connect to the EC2 instance via VPN, SMB, or HuskHoard's built-in HTTP Streaming Gateway.
As remote users upload or edit files on the EC2 instance, the EBS volume fills up. HuskHoard's emergency spillover policy kicks in:
When EBS hits 80% capacity, HuskHoard aggressively stubs the oldest files, silently migrating them to S3. To the remote users accessing the SMB share, it looks like a 500TB drive that never runs out of space, even though you are only paying for a 500GB EBS volume.
The Advantage: Centralization. The bandwidth required to migrate data from Hot (EBS) to Cold (S3) happens entirely across the cloud provider's internal backbone at gigabit speeds. Users only consume bandwidth when they specifically read or write a file.
Inside the Object: How Husk Writes to the Cloud
If you upload 10,000 small files to Amazon S3, you will be penalized. Cloud providers charge for PUT and GET API calls. If HuskHoard uploaded a 1-to-1 mirror of your filesystem, archiving a node_modules folder would cost a fortune in API calls alone.
To solve this, HuskHoard treats the cloud like a tape drive. It packs files into large, sequentially appended binary objects.
When the Janitor daemon processes the queue, it streams files through Blake3 hashing and Zstd compression, piping the raw byte stream to rclone rcat. Dozens or hundreds of files are packed into a single .bin object. The local Husk Catalog remembers exactly which byte offset inside husk_4096.bin belongs to which file.
When a user opens a stubbed file, HuskHoard doesn't download the 150GB object. It utilizes its StreamGate logic to spawn a targeted rclone cat command:
Rclone translates this into an HTTP Range request. S3 serves only the exact 40KB of compressed data required. HuskHoard decompresses it in RAM and hands it to the application. Zero-Disk Delivery. The local hot tier isn't even touched during a read.
The Economics of the Stub
Let's look at why you would choose either of these architectures from a purely financial perspective.
PUT requests into a steady, single stream. You pay for storage space, not transaction penalties.Choosing Your Flight Path
Both architectures leverage the same core truth: the filesystem is an abstraction.
If your team is physically located in one building and works with heavy assets (video editing, 3D rendering), Architecture A (Local Gateway) is superior. You need the 10Gbps LAN speed of local NVMe for active projects, and pushing cold data to the cloud is simply a hands-free offsite backup that reclaims local space.
If your team is distributed across time zones, or your workload is highly transactional (documents, code, web assets), Architecture B (Cloud-Native Hub) makes more sense. You offload the hardware maintenance entirely to AWS or DigitalOcean, and let HuskHoard aggressively manage the expensive EBS block storage so you don't have to.
In either scenario, the end-user experience is exactly the same. They look at a folder. They see their files. They double-click. HuskHoard figures out the rest.