Self-Hosting Vision Models on Datacenter GPUs

Self-Hosting a Vision Model on a Datacenter GPU

The BAGEL-7B-MoT vision model has been successfully self-hosted on a Tesla V100 datacenter GPU. This achievement enables real-time webcam-capture, face-detection, and emotion-reading capabilities.

Why This Matters

The technical reality of self-hosting vision models on datacenter GPUs is far from ideal due to compatibility issues such as lack of bfloat16 support and flash attention. However, with careful configuration and quantization techniques like NF4, it is possible to achieve remarkable performance and accuracy, making such models viable for real-world applications.

Key Insights

• BAGEL-7B-MoT outperforms LLaVA 1.6 7B in terms of speed and description quality, achieving 2-3x faster response times for short descriptions.
• NF4 quantization reduces the model’s weight footprint from 14GB to approximately 4.2GB, enabling it to run on 16GB GPUs like the Tesla V100.
• The MoT architecture of BAGEL-7B-MoT allows for more efficient processing of image tokens, resulting in better descriptions and faster inference times.

Working Examples

Quantization configuration for BAGEL-7B-MoT on Tesla V100

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16, # NOT bfloat16
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)
model = AutoModelForCausalLM.from_pretrained(
    "BAGEL-7B-MoT",
    quantization_config=quantization_config,
    torch_dtype=torch.float16, # NOT bfloat16
    device_map="auto",
)

Practical Applications

• Elyan Labs’ Sophia AI character uses BAGEL-7B-MoT for real-time vision capabilities, demonstrating a practical application of self-hosted vision models in interactive AI systems.
• The use of BAGEL-7B-MoT in Godot games shows potential for enhanced user experiences through AI-powered vision, but may be hindered by compatibility issues and performance optimization challenges.

References:

• https://dev.to/scottcjn/self-hosting-a-vision-model-on-a-datacenter-gpu-bagel-7b-mot-on-a-tesla-v100-4mm4