In this article
  1. The Problem with Continuous Streams
  2. Creating Checkpoints: The Jump Frame
  3. Mapping the Void: The TLV Header Index
  4. The Dual-Path Restore: Video vs. Data

The Problem with Continuous Streams

Cold storage mediums—especially tape drives—are fundamentally linear devices. They excel at writing data in massive, continuous streams, but they punish you for trying to jump around. When you throw data compression (like Zstd or gzip) into the mix, you introduce an even harder mathematical barrier.

Compression algorithms build a rolling dictionary based on the data they've already seen. If a data scientist queries a specific time-slice halfway into a 200 GB GRIB or CSV dataset, a compressed archive cannot just start decompressing from the 100 GB mark. Because the compression state relies on the previous 99.99 GB, you are forced to stream and discard half the file just to serve that single query.

Standard Tarball / Archive

Continuous Stream

✗Decompression state is chained globally
✗Seeking requires reading prior data
✗Massive RAM usage for large file scrubs
HuskHoard Formatting

Framed Architecture

✓File broken into independent 16MB frames
✓Decompression state resets at boundaries
✓O(1) memory overhead for partial reads

Creating Checkpoints: The Jump Frame

To eliminate this massive bottleneck, HuskHoard deliberately breaks the compression chain. When a file is archived, it isn't written as a single blob. Instead, it is sliced into discrete Jump Frames.

By default, we use 16MB blocks. When compressing data, HuskHoard forces the Zstd dictionary to flush and reset its state at every 16MB boundary. This means each frame is completely mathematically independent from the previous one.

The Trade-off

Flushing the compression dictionary costs you a tiny amount of compression efficiency (usually less than 1-2%). In exchange, you gain the ability to drop straight into the middle of a colossal file and begin decompressing immediately.

But having independent frames is only half the battle. If a user asks for byte 50,000,000,000, you still need to know exactly which Jump Frame holds that byte, and where that specific frame physically lives on the tape.

Mapping the Void: The TLV Header Index

Normally, archive software maintains a giant external database to track file offsets. HuskHoard has the catalog database, but we firmly believe that the file itself must be the ground source of truth.

To achieve this, we prepend a custom TLV (Type-Length-Value) Header to the beginning of every single file we archive. TLV is an incredibly flexible parsing format. Instead of a rigid C-struct that breaks when you update your software, a TLV stream just says: "Here is the type of data, here is exactly how long it is, and here are the bytes."

// From src/format/tlv.rs — Parsing the Frame Index let tag = reader.read_u16::<LittleEndian>()?; let length = reader.read_u32::<LittleEndian>()?; if tag == TAG_FRAME_INDEX { // We found the Jump Frame Index! let mut index_data = vec![0u8; length as usize]; reader.read_exact(&mut index_data)?; let jump_frames: Vec<FrameOffset> = bincode::deserialize(&index_data)?; return Ok(jump_frames); } else { // Unknown tag? Just skip the bytes and move on. Future-proof! reader.seek(SeekFrom::Current(length as i64))?; }

Inside one of these TLV blocks (specifically TAG_FRAME_INDEX), we embed an array of tuples mapping the uncompressed byte offsets to the compressed byte offsets on the physical media. Because we use TLV, we can slip this index directly into the file header without breaking compatibility with older parsers.

The Dual-Path Restore: Video vs. Data

When an application tries to read a chunk from the middle of a stubbed file, HuskHoard intercepts the read using fanotify. How it handles that read depends entirely on what kind of file you are opening.

1. The Data Engineer Path (Compressible Files)

For highly compressible formats like Parquet, GRIB, CSV, or database dumps, HuskHoard routes the file through the Zstd compression engine and builds the Jump Table. If a Python Pandas script queries offset 50GB, the process looks like this:

1. Intercept read at offset 50GB
→
2. Read TLV Header (Frame Index)
3. Binary search index for 50GB boundary
→
4. Locate target Jump Frame (e.g. Frame 3125)
5. Issue SCSI Tape Seek to physical block
→
6. Read 16MB & Decompress locally

2. The Video Editor Path (Pre-compressed Files)

Media files like .mp4, .mov, and .zip are already heavily compressed by the camera or encoding software. Running them through Zstd wastes CPU cycles and yields zero space savings. HuskHoard recognizes these extensions via the no_compress_extensions config and bypasses the compression engine entirely, writing them as RAW blocks to the tape.

Because the file is written RAW, 1 logical byte equals exactly 1 physical byte. We don't need a Jump Table or a TLV map. If VLC media player asks for byte 50,000,000, HuskHoard performs a mathematically perfect O(1) seek, instructing the SCSI tape drive to literally drive its motors straight to offset 50,000,000.

O(1)
Raw Media Seeks

By skipping compression for MP4s/MOVs, physical byte offsets match logical ones perfectly.

O(log N)
Data Index Lookup

For compressed datasets, finding the right frame takes microseconds via binary search.

16 MB
Max RAM Overhead

We never decompress more than a single frame, ensuring low memory usage for petabyte files.

By splitting the architecture into a Compression Path for data scientists and a Raw Path for media professionals, HuskHoard turns a sequential tape blob into an instantly traversable block device. Whether you interact with a massive dataset or scrub a 4K video clip, it behaves exactly like it was on a local SSD.

We'll dive deeper into how we prevent the filesystem from misreporting these chunk allocations in the next post.