Hugging Face Introduces Model Integration Contracts
Hugging Face proposes a standard for packaging model tests, aiming to catch bugs before they reach main. The new standard, called Model Integration Contracts, includes a shrunk model, usage examples, and baselines.
Original reporting · huggingface.co
Original: “Transformers Model Integration Contracts”
Read “Transformers Model Integration Contracts” at Hugging Face →How this was written. Content Delta wrote this article with a language model from Hugging Face’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Hugging Face’s report
- Hugging Face introduces Model Integration Contracts, a proposed standard for packaging model tests.
- The standard includes a shrunk model, usage examples, and baselines, and aims to catch bugs before they reach main.
- Hugging Face has developed a CLI, tmic, which creates the contract and builds the fixture.
- The company has used agents to build Model Integration Contracts across model families.
- Hugging Face has published a fleet dataset on the Hub, which pins each Model Integration Contract to a frozen revision.
Hugging Face, a leading AI company, has introduced Model Integration Contracts, a proposed standard for packaging model tests. According to Hugging Face, this standard aims to catch bugs before they reach main, by providing a common recipe for building and testing models.
Hugging Face's Transformers model supports over 750 model types and has a large test suite, with 88,000+ CPU tests on main and 200,000+ GPU tests in a daily run. However, running all these tests on every pull request would be impractical, so the company has developed a way to shrink models and run fast integration tests on a cheap CPU runner.
The shrunk model, called a fixture, is made from a real checkpoint's configuration, with fewer and narrower layers, but keeps every kind of layer and its tokenizer and processor files unchanged. This approach also covers very large models that don't fit on standard GPU runners.
Hugging Face's Model Integration Contracts have three parts: a fixture, a usage path, and baselines. The contract holds if running the usage path on the fixture still produces the baselines. This standard covers an architecture and a usage path, not a single checkpoint, and compatible checkpoints can share a fixture.
The company has developed a CLI, tmic, which creates the contract and builds the fixture. The CLI drafts the usage code and manifest, and CPU and GPU runs measure the candidate baselines. However, changed expectations require human review.
Hugging Face has used agents to build Model Integration Contracts across model families. The company has also published a fleet dataset on the Hub, which pins each Model Integration Contract to a frozen revision, along with the package versions used to record its baselines.
Questions and answers
What is a Model Integration Contract?
A Model Integration Contract is a proposed standard for packaging model tests, which includes a shrunk model, usage examples, and baselines.
How does Hugging Face build Model Integration Contracts?
Hugging Face builds Model Integration Contracts using a CLI, tmic, which creates the contract and builds the fixture, and agents that draft the usage code and manifest.
What is the purpose of Model Integration Contracts?
The purpose of Model Integration Contracts is to catch bugs before they reach main, by providing a common recipe for building and testing models.
