When fastembed silently rotted my worker: anchor your caches, don't trust /tmp

When fastembed silently rotted my worker: anchor your caches, don’t trust /tmp

A research job using fastembed silently failed with a ‘NoSuchFile’ error. The root cause was that the OS temp directory had partially deleted the model cache, leaving metadata inconsistent with the files on disk.

Why This Matters

The ideal model assumes model caches persist indefinitely, but reality is that OS temp directories are ephemeral and actively cleaned. This caused a 90MB model to be partially deleted, leading to a silent failure that could recur intermittently, undermining reliability of unattended workers. The pattern is common across many ML libraries and affects any system that downloads model weights without explicitly anchoring the cache to a stable location.

Key Insights

• Fastembed, like many ML libraries, defaults to tempfile.gettempdir() for caching, which on Windows is actively cleaned by Storage Sense or disk cleanup (blog post, 2026).
• The error ‘Local file sizes do not match the metadata’ indicates partial cache corruption, not a download failure – only a subset of the snapshot files were deleted.
• The author previously solved a similar issue with Qdrant by anchoring all data paths to a single PROJECT_ROOT, preventing two processes from creating separate state directories.
• The fix is a single cache_dir= parameter pointed at a project-relative path, making the cache survive OS cleanup and shared between API and worker processes.

Working Examples

Original initialization that silently used the OS temp directory for caching.

from functools import lru_cache
from pacos.shared.config import settings

Vector = list[float]

@lru_cache(maxsize=1)
def _model():
    from fastembed import TextEmbedding
    return TextEmbedding(settings.embed_model)

Declaring the cache path in config to anchor it to the project root.

# src/pacos/shared/config.py
embed_model: str = "BAAI/bge-small-en-v1.5"
fastembed_cache_path: str = "./fastembed_cache"

Fixed initialization that explicitly sets a project-relative cache directory.

@lru_cache(maxsize=1)
def _model():
    from fastembed import TextEmbedding
    cache_dir = (PROJECT_ROOT / settings.fastembed_cache_path).resolve()
    cache_dir.mkdir(parents=True, exist_ok=True)
    return TextEmbedding(settings.embed_model, cache_dir=str(cache_dir))

Gitignore entry to exclude the regenerable cache from version control.

# .gitignore
qdrant_data/
fastembed_cache/
knowledge_repo/

Practical Applications

• Unattended workers using fastembed for embeddings: if cache_dir is not set, the OS temp directory may be cleaned, causing intermittent failures that appear as corruption.
• Two processes (API + worker) sharing a model cache: if the cache is in a temp directory, each process may have a different copy or one may be cleaned, leading to silent disagreement.
• Any library that downloads model weights on first use (e.g., Hugging Face transformers, spaCy, ONNX runtime): assuming the default cache location is safe can lead to unexpected downtime in production.

References:

• https://dev.to/yogeshchavan2008/when-fastembed-silently-rotted-my-worker-anchor-your-caches-dont-trust-tmp-3ab3