Picture the request: an auditor needs every login and transaction record from a 20-minute window last March. Your logs for that quarter live in a single compressed archive — 150 GB, sitting in cloud storage or on tape. The records you need amount to a few megabytes.
With conventional compression, you can't get to those few megabytes without first pulling down the whole archive. That's hours of transfer time and a cloud egress bill for data you never wanted. Do it again for the next audit question, and the next.
This is the problem we picked Zstd to solve — specifically, Zstd's ability to split compressed data into independently readable frames.
The Problem with Sequential-Only Compression
Gzip, bzip2, and plain Zstd all produce one continuous compressed stream. Each part of the stream depends on what came before it, so to read byte 50 GB into the archive, a decompressor has to start at byte zero and work its way forward, discarding everything it doesn't need.
Locally, that wastes CPU and time. Over a network or from tape, it's worse: you have to transfer every byte before the one you want, and cloud providers charge you per byte downloaded. The cost of one small lookup scales with the size of the entire archive.
The requirement: reading 5 MB of a 150 GB archive should cost about 5 MB — in transfer time, in egress fees, and in drive wear.
Enter Seekable Frames
Instead of one giant stream, HuskHoard's Archive Worker compresses data into independent Zstd frames — we call them Jump Frames — each covering a fixed slice of the original data (16 MB by default). Because every frame is self-contained, any one of them can be decompressed without touching the others. An index at the head of each archive object records where every frame starts and which uncompressed byte range it covers.
[ Index (TLV) ] [ Frame 0 ] [ Frame 1 ] [ Frame 2 ] ... [ Frame N ]
0–16 MB 16–32 MB 32–48 MB ...
To read from any offset, HuskHoard does three things:
- Binary-searches the index for the frame containing the requested offset.
- Reads only that frame's bytes — with an HTTP
Rangerequest for cloud storage, or a positioned read on tape or disk. - Decompresses that single frame and returns exactly the bytes asked for.
What that means for the audit scenario
✗ Conventional compressed archive
downloaded
Download the whole object, decompress from byte zero, then search for your 20 minutes of records.
✓ HuskHoard Jump Frames
downloaded
Look up the frame(s) covering your time range, fetch just those, decompress just those.
The saving is the ratio between the size of the archive and the size of the slice you need — and for audit work, that ratio is usually enormous. At an illustrative egress rate of $0.09/GB (check your own provider's pricing), pulling a 150 GB archive costs about $13.50 per lookup; pulling a couple of frames costs a fraction of a cent. The time saved is just as dramatic: seconds instead of hours.
Who This Really Matters For
Anyone retaining large volumes of log data that is written constantly, read rarely, and must be kept for years:
- Security teams investigating an incident need one host, one hour, out of months of retained logs.
- Compliance and audit teams answering a regulator's question about a specific account or transaction window.
- Finance and operations reconciling a handful of transactions from a multi-terabyte ledger archive.
- Anyone paying cloud egress on cold data, where every full-archive download is a line item.
Log files are also text-heavy and compress extremely well, so you get the storage savings of compression and the cheap random access of an uncompressed file. You're no longer forced to choose between "small and slow to query" or "fast to query and expensive to store."
Two Ways HuskHoard Reads Just the Part You Need
Partial extraction in HuskHoard works two ways, depending on what's inside the file:
1. Jump Frames — for compressible data like logs, documents, and databases
This is what we've covered above: seekable Zstd frames plus an index. We go deeper on the frame layout and index format in The Frame Up: Why Saving Data as Compressed Frames Works.
2. StreamGate — byte-range streaming, mostly for media
Video and audio files like MP4 and MOV are already compressed, so there's nothing to gain from compressing them again. For these, byte offsets map 1-to-1 between the original file and what's stored, which means a seek is a direct jump with no decompression at all. StreamGate, HuskHoard's local HTTP gateway, answers standard HTTP Range requests so players like mpv, VLC, Plex, and Jellyfin can scrub through a file stored on tape or S3 without recalling the whole thing. Read the full story in our StreamGate post.
Same principle, two paths: never move more bytes than the request actually needs.
Why Zstd, Specifically
Seekable frames are possible with other codecs, but Zstd gives us the combination we wanted for cold storage:
- Strong compression ratios at fast speeds, so archiving doesn't bottleneck on CPU and log data shrinks substantially.
- Frames as a first-class concept, so independent, concatenable frames are a natural fit rather than a hack.
- An open, widely implemented format — which matters for the next point.
The Easy Exit stays intact
Because the payload is standard Zstd verified with BLAKE3, you aren't locked into HuskHoard to read your own archives. With nothing more than dd and zstd, you can carve out a frame and decompress it yourself. That's a deliberate design promise: an audit archive you're legally required to keep for years shouldn't depend on any one vendor's software surviving.
Choosing a frame size
Frame size is a trade-off. Smaller frames mean finer-grained seeks and less data moved per lookup, but slightly lower compression ratios and a larger index. Larger frames compress a bit better but make each lookup pull more data. We default to 16 MB frames because it keeps worst-case egress for any single read capped at one frame while still compressing well, and because it packs efficiently into cloud objects and tape blocks.
If you're sitting on terabytes of security, access, or transaction logs and dreading the next audit, seekable compression is worth a close look. The format is open, the savings scale with the size of your archive, and you never have to pay to move data you didn't need.
Questions or corrections? Open an issue on GitHub or reach us at info@huskhoard.com.