Hugging Face Turns Database Grading Into Rewards
Hugging Face and Microsoft developed ThinkingBox, which evaluates agents' work by comparing final database states, and used it to fine-tune a model, increasing repeatable success. This improvement matters because it shows a way to make models more reliable in real-world applications.
Original reporting · huggingface.co
Original: “Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox”
Read “Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox” at Hugging Face →
How this was written. Content Delta wrote this article with a language model from Hugging Face’s reporting (model: @cf/meta/llama-3.3-70b-instruct-fp8-fast); it describes that reporting rather than adding to it. The original, linked above, is the full story.
Key points from Hugging Face’s report
- Hugging Face and Microsoft developed ThinkingBox, a tool that evaluates agents' work by comparing final database states, and used it to fine-tune a model, increasing repeatable success.
- The fine-tuned model, Qwen3.8-27B, showed a significant increase in repeatable success, with its average success rate rising from 51.70% to 60.78%.
- The improvement in repeatable success is important because it shows that the model is more reliable and consistent in its performance, which is crucial in real-world applications.
- The researchers used a technique called offline policy-relative difficulty to select the tasks used for fine-tuning the model, which helped to improve the model's performance.
- The results of the fine-tuning process were evaluated on a benchmark dataset called ThinkingBox-Bench, which consists of 507 tasks, and the model's performance was compared to other models.
Hugging Face and Microsoft have collaborated on a project to improve the reliability of models in real-world applications, according to Hugging Face. They developed ThinkingBox, a tool that evaluates an agent's work by comparing the final database state with the expected outcome.
ThinkingBox was initially used to catch agents that say they are done when they are not, but its developers soon realized that it could also be used as a reward function for training models. This is because the executable checks used to evaluate the agents can return a direct reward signal.
The researchers used ThinkingBox to fine-tune a model called Qwen3.8-27B, which resulted in a significant increase in repeatable success. The model's average success rate rose from 51.70% to 60.78%, and the number of tasks it completed correctly on all 20 attempts rose from 38 to 107.
The improvement in repeatable success is important because it shows that the model is more reliable and consistent in its performance. This is crucial in real-world applications where models are used to make decisions that can have significant consequences. The researchers used a technique called offline policy-relative difficulty to select the tasks used for fine-tuning the model.
The results of the fine-tuning process were evaluated on a benchmark dataset called ThinkingBox-Bench, which consists of 507 tasks. The model's performance was compared to other models, including Claude Opus 5.5 and GPT-5.4, and the results showed that the fine-tuned Qwen3.8-27B model had improved significantly in terms of repeatable success.
The researchers also found that the improvement in repeatable success came at a small cost in terms of coverage, with the model solving slightly fewer tasks at least once. However, the overall performance of the model was still better than the other models tested, and the researchers believe that this approach has the potential to improve the reliability of models in real-world applications.
The next steps for this research will involve further testing and refinement of the ThinkingBox tool and the fine-tuning process. The researchers also plan to explore the use of ThinkingBox in other applications, such as natural language processing and computer vision.
Questions and answers
What is ThinkingBox?
ThinkingBox is a tool developed by Hugging Face and Microsoft that evaluates an agent's work by comparing the final database state with the expected outcome.
How was the Qwen3.8-27B model fine-tuned?
The Qwen3.8-27B model was fine-tuned using a technique called offline policy-relative difficulty, which involves selecting tasks that are suitable for the model to learn from.
What were the results of the fine-tuning process?
The fine-tuning process resulted in a significant increase in repeatable success, with the model's average success rate rising from 51.70% to 60.78%, and the number of tasks it completed correctly on all 20 attempts rising from 38 to 107.
