As large model applications gradually enter production environments, enterprises no longer care only about whether a model can be called successfully. They also need to continuously understand whether the model call process is stable, whether responses are timely, whether Token consumption is controllable, and whether output results meet expectations. In real business scenarios, models may respond slowly, time out, return interrupted responses, encounter HTTP errors, or show abnormal growth in Token consumption. They may also produce inconsistent results for the same input or fail to return content in the expected structure.Without unified model monitoring capabilities, R&D and operations teams can only rely on logs or user feedback for reactive troubleshooting. This makes it difficult to detect fluctuations in model service quality in time or quantify output quality differences across different models, scenarios, and Prompts.
For model quality monitoring, continuous monitoring can be performed from two dimensions: model service quality and model output quality. For service quality, the system can collect metrics such as Input Token Count, Output Token Count, Reasoning Token Count, Content Token Count, Total Token Count, First Token Time, Reasoning Time, Content Generation Time, Full Response Time, Average Time per Token, Reasoning Speed, Content Speed, and Average Speed. These metrics are used to evaluate model call cost, response efficiency, and generation performance.The system can also monitor Success Rate, HTTP Error Rate, Timeout Rate, Interruption Rate, repeated-call stability, and Cache Hit Rate to help locate issues such as request failures, abnormal responses, cache misses, or unstable repeated-call results. For output quality, the platform can evaluate the correctness, consistency, and structured output accuracy of model answers based on preset standard answers or validation rules, determining whether model results match expected answers, remain stable across multiple calls, and return content according to the specified structure, fields, and values.I. Task Creation
1.
Basic Information
Add the Model Quality Monitoring network type in Task Management, and add Model Service Quality, Model Output Quality, and Model Cost Efficiency (to be released) under task types.
2.
Required Parameters
Package: Default model monitoring package (must be enabled)
Task Name: Custom task name
Model Provider: Currently, models from eight model providers are available for evaluation: DeepSeek, Baidu, Alibaba, Tencent, Zhipu AI, ByteDance, MiniMax, and Moonshot AI.
Model Provider and Model Version: The model version is automatically linked based on the selected model provider.
API Address: Default value; no editing is required.
II. Synthetic Monitoring
1.
Overview Data
Overview data displays the current task overview metrics, including Success Rate (%), First Token Time (s), Full Response Time (s), Average Time per Token (s), HTTP Error Rate (%), Timeout Rate (%), Interruption Rate (%), Repeated-Call Stability (%), and Cache Hit Rate (%).
Users can view task data by dimension: trends, regions, maps, hosts, errors, and scatter data.
2.
Scatter Data
The header of the scatter page displays core metrics: First Token Time, Reasoning Time, Content Generation Time, Average Time per Token, and Reasoning Token Count.
The Bar chart adds First Token Time, Reasoning Time, and Content Generation Time.
In diagnostic data, the platform displays the test question set used for the current issue. In model output quality monitoring, one metric corresponds to one test question.
With model quality monitoring, enterprises can go beyond determining whether a model is available and further determine whether the model is stable, efficient, and reliable in its output. R&D and operations teams can identify performance exceptions, error exceptions, and stability issues in the model call chain in time, and locate where issues occur by using metrics such as Tokens, latency, speed, and success rate.At the same time, output quality metrics help teams evaluate answer quality across different models, Prompts, and business scenarios. They provide quantitative evidence for model selection, Prompt tuning, cache strategy optimization, structured output validation, and release regression testing, reducing business risks caused by unstable model outputs.
