Upskilling Coding Agents with CUDA Kernel Generation

Upskilling Coding Agents with CUDA Kernel Generation

The upskill tool, developed by Hugging Face, has been used to generate and evaluate agent skills for creating CUDA kernels, resulting in a 45% improvement in model performance. This breakthrough was achieved by leveraging the capabilities of Claude Opus 4.5, a powerful coding agent, to create a skill that can be used by smaller, cheaper models.

Why This Matters

The ability to upskill coding agents has significant implications for the field of artificial intelligence, as it enables the transfer of domain expertise from powerful models to cheaper, more accessible ones. This can lead to substantial cost savings and improved performance in specialized tasks, such as CUDA kernel generation. However, the technical reality is that creating effective skills requires a deep understanding of the task, the model, and the skill creation process, which can be time-consuming and challenging.

Key Insights

• The upskill tool can generate skills that improve model performance by up to 45% in CUDA kernel generation tasks.
• The skill creation process involves using a teacher model, such as Claude Opus 4.5, to generate a skill that can be used by smaller models.
• The upskill tool can evaluate the performance of models with and without skills, providing valuable insights into the effectiveness of the skill.

Working Example

# install upskill
pip install upskill
# generate a skill based on an agent trace
upskill generate "write nvidia kernels" --from ./trace.md
# evaluate models on a skill
upskill eval ./skills/my-skill/ --model haiku --model sonnet

Practical Applications

• Use Case: Companies like Hugging Face can use upskill to create skills for their coding agents, improving performance and reducing costs in specialized tasks like CUDA kernel generation.
• Pitfall: One common anti-pattern is to assume that a skill created for one model will work equally well for another, without proper evaluation and testing.

References:

• https://huggingface.co/blog/upskill
• https://github.com/huggingface/upskill