1. Model Quality Monitoring
听云Network
  • 中文版
    • Release Notes
      • 2026年
      • 2025年
      • 2024年
    • 产品介绍
      • 产品概述
        • 产品概述
      • 工作原理
        • 监测原理
        • 监测点部署
      • 产品架构
        • 产品架构
      • 典型场景
        • CDN加速评估及选型
        • 网站对标分析
        • 网站应用压力测试分析
        • 对网站进行性能评估和优化
        • 具备可观测混合压测场景
        • DNS劫持监测-CNAME监测
        • VPN质量监控
        • WIFI质量监控
        • 短信服务质量监控
        • API主动监控
        • 证书监控
        • MTR网络链路监控
        • 企业业务质量监控
        • 端到端场景监测
        • whois域名监控场景
        • IPV6支持率监控
        • 模型质量监控
      • 核心优势
        • 端到端监测
        • 市场地位
        • 数据保障
        • 本土化服务
      • 功能模块
        • 即时监测
          • 即时监测
        • 持续监测
          • 持续监测介绍
          • 监测任务类型
          • 监测结果分析
        • 私有节点监测
          • 私有节点监测
        • 域名分析
          • 域名分析
        • CDN调优
          • CDN调优
        • 劫持监测
          • 劫持监测
        • 警报通知
          • 警报通知
        • 报告服务
          • 报告服务
        • API服务
          • API服务
    • 使用指南
      • 持续监测图表
        • 公共操作
        • 概览
        • 趋势图
        • 地域图
        • 地图
        • 主机图
        • 错误图
        • 散点图
        • 元素
      • 任务类型
        • 监测任务类型介绍
        • 固网业务监测
        • 移动业务监测
        • 模型质量监控业务监测
      • 持续监测
        • 持续监测介绍
        • 新建任务
        • 查询任务
        • 管理任务
        • 绑定任务
        • 节点组管理
      • 即时监测
        • 即时监测介绍
        • 配置即时监测
        • 查看监测结果
      • 系统管理
        • 系统管理介绍
        • 自有点监测
        • 用户管理
        • 使用量查询
        • 授权码管理
        • 地图参数
      • 域名分析
        • 域名分析
      • CDN调优
        • CDN调优介绍
        • CDN评估
        • 算法建议
        • 策略调优
      • 警报
        • 警报介绍
        • 性能警报
        • 错误警报
        • 智能警报
        • 警报日志
        • 警报推送
      • 报告
        • 报告介绍
        • 创建快速报告
        • 创建智能报告
        • 导出报告
        • 图表池
      • API服务
        • API服务
      • MCP 服务
        • 听云 Network MCP 接入指南
        • 听云 Network MCP 工具
      • 其他功能
        • 分享当前页面
        • 移动端快速访问
      • WebRecorder脚本录制器(插件版)
        • 介绍
        • 安装
        • 创建项目及界面展示
        • 脚本录制
        • 配置任务
        • 注意事项
        • 脚本录制器支持的命令
        • 脚本录制器对事务验证码的支持
    • 指标说明
      • PC监控
        • 页面指标
        • 文件指标
        • 事务指标
        • 流媒体指标
        • Ping指标
        • MTR指标
        • 模型质量监控指标
        • 私有协议指标
          • 私有协议指标解释概述
          • Websocket 插件配置说明
          • DIG 插件配置说明
          • Socket(TCP/UDP) 插件配置说明
          • Telnet 插件配置说明
          • FTP 插件配置说明
          • SMTP/IMAP/POP3 插件配置说明
          • MTR 插件配置说明
          • Traceroute 插件配置说明
          • DeepSeek 插件配置说明
          • TCP Send 插件配置说明
          • UDP 插件配置说明
          • HTTP(SM) 插件配置说明
          • HTTP3 插件配置说明
          • OCSP 插件配置说明
          • WIFI 插件配置说明
          • VPN 插件配置说明
      • 移动(H5)监控
        • 页面指标
        • 文件指标
        • 事务指标
        • 流媒体指标
        • Ping指标
        • 短信指标
    • 脚本录制器
      • NBR脚本录制器工具的下载
      • NBR脚本录制器的使用-----简版
    • 常见问题
      • 功能分析
        • 创建任务
        • 创建节点组
        • 高级参数----结束条件
        • 高级参数----诊断
        • 报表无数据之校验节点DNS功能
        • 流媒体监测,配置任务后无数据
        • 平台是否支持查看“指定Cname是否生效”
        • 真机单文件任务下载崩溃
        • 听云检测和浏览器实际访问数据对不上
        • 监测地址写不全导致的报错
        • 绑定Host未生效,出现了列表之外的主机IP
        • 绑定Host未生效,出现了列表之外的主机IP
        • 不同任务类型,相同IP解析结果不同
        • 复制任务,无法做绑定
        • 即时监测IPV6页面级别解释
        • 私有协议Socket(TCP/UDP)
      • 错误分析
        • 打开流媒体失败
        • 打开流媒体失败
        • 打开流媒体失败
        • 页面任务元素出现阻塞时间的分析思路
        • 无法解析域名和服务器地址
        • 无法解析服务器名称或地址
        • 无法解析服务器名称或地址
        • 无法解析服务器名称或地址
        • 无法解析服务器名称或地址
        • 无法与服务器建立连接
        • 无法与服务器建立连接
        • 无法与服务器建立连接
        • 无法与服务器建联连接
        • 与服务器建立连接失败
        • 操作超时
        • 操作超时之Chrome90
        • 任务超时
        • 任务超时
        • 服务器连接被重置
        • 与服务器连接被重置
        • 与服务器连接被重置
        • 应用的证书无效
        • 应用证书无效
        • 应用证书无效
        • 安全频道支持出错
        • 安全频道支持出错
        • 安全频道支持出错
        • 证书错误
        • 私有化平台页面访问报-100错误
        • 404 - Not Found错误
        • HTTP/1.1 403 Forbidden
        • HTTP412错误
        • 元素报表数值对不上
      • 数据分析
        • 如何区分HLS格式是直播还是点播
        • 报表数据与客户服务器端统计不一致
        • DNS时间过长的分析思路
        • 绑定对比任务的消耗点数不同
        • 4G真机监测下客户反馈http via不正确
        • 节点IP定位不准确
        • 私有化平台任务无数据
        • CDN厂商CNAME无响应问题
        • 单文件下载任务, 创建后无数据
        • SSL握手时间长
        • 监控数据量少,和节点组频率对应不上
        • 跨省跨运营商现象分析
        • 流媒体任务,三个任务数据量相差较大
        • 相同文件下载,不同厂商字节数下载不同
        • 单文件下载,配置MD5,下载不全未报错。
        • DNS时间较长问题分析
        • PING任务警告,Request timed out
        • PING任务丢包百分之百
        • 元素瀑布图中有重复元素加载
        • 散点数据和抓包数据对不上
        • 抓包中如何查看传输的内容
        • 任务无数据/数据少
        • 相同URL,额外等待时间影响onload时间
        • 听云监测数据和浏览器访问数据总下载字节数相差较大
        • DNS时间较长
        • 跨运营商访问
        • 页面任务元素出现阻塞时间
        • 抓包中出现TCP ZERO window
      • 报表权限
      • 节点分析
        • 数据访问发现有跨区访问
        • 节点劫持
      • 告警分析
        • 运营商链路故障引起告警
        • 客户应用500,平台未告警
      • 问题百问
        • Network问题Q&A
        • 模型质量监控相关问题Q&A
        • 拨测任务被 Cloudflare 人机验证拦截的处理方法
    • 专题使用指南
      • 模型质量监控
        • 概述
        • 专题使用指南
  • English
    • Release Notes
      • 2026
      • 2025
      • 2024
    • Product Introduction
      • Product Overview
        • Product Overview
      • How It Works
        • Monitoring Principles
        • Probe Deployment
      • Product Architecture
        • Product Architecture
      • Typical Scenarios
        • CDN Acceleration Evaluation and Selection
        • Website Benchmarking Analysis
        • Website Application Stress Test Analysis
        • Website Performance Evaluation and Optimization
        • Observable Hybrid Load Testing Scenario
        • DNS Hijacking Monitoring - CNAME Monitoring
        • VPN Quality Monitoring
        • WIFI Quality Monitoring
        • SMS Service Quality Monitoring
        • API Active Monitoring
        • Certificate Monitoring
        • MTR Network Link Monitoring
        • Enterprise Business Quality Monitoring
        • End-to-End Scenario Monitoring
        • Whois Domain Monitoring Scenario
        • IPv6 Support Rate Monitoring
        • Model Quality Monitoring
      • Core Advantages
        • End-to-End Monitoring
        • Market Position
        • Data Protection
        • Localized Service
      • Feature Modules
        • Instant Testing
          • Instant Testing
        • Synthetic Monitoring
          • Synthetic Monitoring Introduction
          • Monitoring Task Types
          • Monitoring Result Analysis
        • Private Probe Monitoring
          • Private Probe Monitoring
        • Domain Analysis
          • Domain Analysis
        • CDN Tuning
          • CDN Tuning
        • Hijacking Monitoring
          • Hijacking Monitoring
        • Alert Notifications
          • Alert Notifications
        • Report Service
          • Report Service
        • API Service
          • API Service
    • User Guide
      • Synthetic Monitoring Charts
        • Common Operations
        • Overview
        • Trend Chart
        • Region Chart
        • Map
        • Host Chart
        • Error Chart
        • Scatter Plot
        • Elements
      • Task Types
        • Monitoring Task Types Introduction
        • Fixed Network Service Monitoring
        • Mobile Service Monitoring
        • Model Quality Monitoring Service Monitoring
      • Synthetic Monitoring
        • Synthetic Monitoring Introduction
        • Create Task
        • Query Tasks
        • Manage Tasks
        • Bind Task
        • Probe Group Management
      • Instant Testing
        • Instant Testing Introduction
        • Configure Instant Testing
        • View Monitoring Results
      • System Management
        • System Management Introduction
        • Private Probe Monitoring
        • User Management
        • Usage Query
        • Authorization Code Management
        • Map Parameters
      • Domain Analysis
        • Domain Analysis
      • CDN Tuning
        • CDN Tuning Introduction
        • CDN Assessment
        • Algorithm Recommendations
        • Policy Tuning
      • Alerts
        • Alert Introduction
        • Performance Alert
        • Error Alert
        • Intelligent Alert
        • Alert Log
        • Alert Push
      • Reports
        • Report Introduction
        • Create Quick Report
        • Create Smart Report
        • Export Report
        • Chart Pool
      • API Service
        • API Service
      • WebRecorder Script Recorder (Plugin Version)
        • Configure Task
        • Create Project and Interface Display
        • Introduction
        • Script Recording
        • Commands Supported by Script Recorder
        • Script Recorder Support for Transaction Verification Codes
        • Notes
        • Installation
      • Other Features
        • Share Current Page
        • Quick Mobile Access
      • MCP Service
        • Tingyun Network MCP Access Guide
        • Tingyun Network MCP Tools
    • Metric Descriptions
      • PC Monitoring
        • Page Metrics
        • File Metrics
        • Transaction Metrics
        • Streaming Media Metrics
        • Ping Metrics
        • MTR Metrics
        • Model Quality Monitoring Metrics
        • Private Protocol Metrics
          • Private Protocol Metrics Overview
          • WebSocket Plugin Configuration Guide
          • DIG Plugin Configuration Guide
          • Socket(TCP/UDP) Plugin Configuration Guide
          • Telnet Plugin Configuration Guide
          • FTP Plugin Configuration Guide
          • SMTP/IMAP/POP3 Plugin Configuration Guide
          • MTR Plugin Configuration Guide
          • Traceroute Plugin Configuration Guide
          • DeepSeek Plugin Configuration Guide
          • TCP Send Plugin Configuration Guide
          • UDP Plugin Configuration Guide
          • HTTP(SM) Plugin Configuration Guide
          • HTTP3 Plugin Configuration Guide
          • OCSP Plugin Configuration Guide
          • WIFI Plugin Configuration Guide
          • VPN Plugin Configuration Guide
      • Mobile (H5) Monitoring
        • Page Metrics
        • File Metrics
        • Transaction Metrics
        • Streaming Media Metrics
        • Ping Metrics
        • SMS Metrics
    • Script Recorder
      • Download the NBR Script Recorder Tool
      • Using the NBR Script Recorder - Short Version
    • FAQs
      • Feature Analysis
        • Create Task
        • Create Probe Group
        • Advanced Parameters - End Conditions
        • Advanced Parameters - Diagnostics
        • Verify Probe DNS When Report Has No Data
        • No Data After Configuring a Streaming Media Monitoring Task
        • Does the Platform Support Checking Whether a Specified CNAME Takes Effect
        • Real Device Single File Task Download Crash
        • Tingyun Monitoring Does Not Match Actual Browser Access Data
        • Error Caused by Incomplete Monitoring Address
        • Bound Host Does Not Take Effect and Host IP Outside List Appears
        • Bound Host Does Not Take Effect and Host IP Outside List Appears
        • Same IP Has Different Resolution Results Across Task Types
        • Copied Task Cannot Be Bound
        • Instant Testing IPv6 Page-Level Explanation
        • Private Protocol Socket (TCP/ UDP)
      • Error Analysis
        • Failed to open streaming media
        • Failed to open streaming media (2)
        • Failed to open streaming media (3)
        • Troubleshooting Blocking Time for Page Task Elements
        • Unable to Resolve Server Name or Address
        • Unable to Resolve Server Name or Address (2)
        • Unable to Resolve Server Name or Address (3)
        • Unable to Resolve Server Name or Address (4)
        • Unable to Resolve Server Name or Address (5)
        • Unable to Establish Connection with Server
        • Unable to Establish Connection with Server (2)
        • Unable to Establish Connection with Server (3)
        • Unable to Establish Connection with Server (4)
        • Failed to establish connection with server
        • Operation Timeout
        • Chrome 90 Operation Timeout
        • Task timeout
        • Task timeout (2)
        • Server Connection Reset
        • The connection to the server was reset
        • The connection to the server was reset (2)
        • Invalid Application Certificate
        • Invalid Application Certificate (2)
        • Invalid Application Certificate (3)
        • Secure channel support error
        • Secure channel support error (2)
        • Secure channel support error (3)
        • Certificate Error
        • -100 Error When Accessing Private Platform Page
        • 404 - Not Found error
        • HTTP/1.1 403 Forbidden
        • HTTP412 error
        • Element Report Values Do Not Match
      • Data Analysis
        • How to Distinguish Live and VOD HLS Formats
        • Report Data Does Not Match Customer Server Statistics
        • Troubleshooting Long DNS Time
        • Different Point Consumption for Bound Comparison Tasks
        • Incorrect HTTP Via Reported in 4G Real Device Monitoring
        • Inaccurate Probe IP Location
        • No Data for Tasks on Private Platform
        • CDN Vendor CNAME No Response Issue
        • No Data After Creating Single File Download Task
        • Long SSL Handshake Time
        • Low Monitoring Data Volume Does Not Match Probe Group Frequency
        • Cross-Province and Cross-Carrier Phenomenon Analysis
        • Large Data Volume Difference Across Three Streaming Media Tasks
        • Different Vendors Download Different Byte Counts for Same File
        • Single File Download with MD5 Configured Does Not Report Incomplete Download
        • Long DNS Time Analysis
        • Ping Task Warning - Request Timed Out
        • Ping Task 100 Percent Packet Loss
        • Duplicate Elements Loaded in Element Waterfall Chart
        • Scatter Data Does Not Match Packet Capture Data
        • How to View Transmitted Content in Packet Capture
        • No or Insufficient Task Data
        • Additional Wait Time Affects Onload Time for Same URL
        • Large Difference Between Tingyun Monitoring and Browser Total Download Bytes
        • Long DNS Time
        • Cross-Carrier Access
        • Blocking Time Appears for Page Task Elements
        • TCP Zero Window Appears in Packet Capture
      • Probe Analysis
        • Cross-Region Access Found in Data Access
        • Probe Hijacking
      • Alert Analysis
        • Alert Caused by Carrier Link Failure
        • Customer Application 500 with No Platform Alert
      • Q&A
        • Network Q&A
        • Model Quality Monitoring Q&A
        • How to Handle Synthetic Tasks Blocked by Cloudflare Human Verification
      • Report Permissions
    • Topic User Guides
      • Model Quality Monitoring
        • Overview
        • Model Quality Monitoring Guide
  1. Model Quality Monitoring

Model Quality Monitoring Guide

Updated: 2026-09-11
This guide helps task creators and configuration administrators work with platform presets, custom integrations, custom model providers, and test sets. It focuses on field dependencies, submission controls, relationships between list actions, and post-operation verification.

Contents#

1 Before You Begin
2 Create a Task with a Platform Preset
3 Create a Task with a Custom Integration
4 Maintain Custom Model Providers
5 Maintain Custom Test Sets
6 FAQs and Operational Verification
The providers, model records, validation times, and test set times shown in the examples are for illustration only. They do not represent default configuration commitments for a new account.

1 Before You Begin#

This guide is intended for model monitoring task creators and configuration administrators. It explains how to use platform preset models, connect custom models, and maintain model providers and test sets. After configuring a provider and preparing test sets, you can use the same task creation workflow for Model Service Quality or Model Output Quality monitoring.
Check Before You BeginWhat to Prepare
Account and feature accessUse an account that can access Task Management, Global Settings, and Model Quality Monitoring configuration. If an entry or action is unavailable, contact an administrator to verify your permissions.
Package and ProbesMake sure that a valid model monitoring package is available and that you can select the required Probe Groups.
Platform presetIdentify the target model provider, model vendor, and model name in advance. You do not need to enter an API Key on the task page.
Custom integrationPrepare the API protocol, provider name, model vendor, model name, API Address, and corresponding API Key. You can select an existing configuration if one is available.
Custom test setDownload and complete the current template for the target metric. If no custom test set has been uploaded, the platform-provided public test set is used.

1.1 Select a Monitoring Task Type#

Task TypePurposeTest Set Relationship
Model Service QualityObserve the service performance of model APIs.All service quality metrics share one test set.
Model Output QualityEvaluate the correctness, consistency, and structured output accuracy of model responses.Each of the three metrics uses its own test set.
Both task types support Platform Preset and Custom Integration. Model Cost Efficiency is currently marked as pending release and cannot be selected.

1.2 Open the Relevant Page#

OperationNavigation Path
Create a model monitoring taskTask Management > Task Maintenance > Create Task > Model Quality Monitoring
Maintain custom model providersSystem Management > Global Settings > Model Quality Monitoring Configuration > Custom Model Provider Maintenance
Maintain custom test setsSystem Management > Global Settings > Model Quality Monitoring Configuration > Custom Test Set Maintenance

2 Create a Task with a Platform Preset#

2.1 Create a Model Service Quality Task#

1.
Go to Task Management > Task Maintenance and click Create Task.
2.
In Basic Information, set Network Type to Model Quality Monitoring, set Task Type to Model Service Quality, and select Manual Creation.
3.
Under Required Parameters, select an available package and enter a descriptive task name.
4.
Keep Integration Method set to Platform Preset. OpenAI Protocol is selected by default, and Anthropic Protocol is disabled.
5.
Select Model Provider, Model Vendor, and Model Name in order, and verify the automatically populated API Address.
6.
Configure the monitoring schedule, Probe scope, and any required advanced parameters. Then click Save at the bottom of the page. Confirm that the task was created by checking both the success message and the task list.
image.png
Figure 1. Example interface for creating a Model Service Quality task with a platform preset
With a platform preset, the platform supplies the model information. Do not enter a custom API address or credential in an unrelated field. To use your own service, switch to Custom Integration.

2.2 Create a Model Output Quality Task#

1.
From the same task creation page, change Task Type to Model Output Quality.
2.
Keep Integration Method set to Platform Preset, and recheck the package, task name, provider, vendor, model name, and API Address.
3.
To use custom question sets, first maintain separate Correctness, Consistency, and Structured Output Accuracy test sets in Global Settings. Upload a separate file for each metric; one file cannot replace all three test sets.
4.
Configure the monitoring schedule and Probe scope, and check Task Timeout under Advanced Parameters. The current default is 300 seconds for Model Output Quality and 150 seconds for Model Service Quality.
5.
Click Save and check the page response. If field validation fails, correct the indicated field and submit the task again.
image.png
Figure 2. Example interface for creating a Model Output Quality task with a platform preset

2.3 Understand Required-Parameter Dependencies#

FieldHow to Complete ItWhat to Recheck After a Change
PackageSelect a model monitoring package available to the current account.Verify the package status and validity period.
Task NameEnter the name manually.Use a name that distinguishes the provider, model, and monitoring purpose.
Integration MethodSelect Platform Preset or Custom Integration.Recheck the protocol and the three-level model selection after switching methods.
API ProtocolPlatform presets use the OpenAI protocol.Anthropic being disabled is expected in this mode.
Model ProviderSelect a provider from the platform preset list.Available vendors and model names change with the provider.
Model VendorSelect a vendor supported by the current provider.A provider may offer models from multiple vendors.
Model NameSelect a model from the current vendor's model list.Use the full model identifier rather than a similar name.
API AddressAutomatically populated from the selected platform configuration.Recheck the address after changing the provider or model. Do not override it manually.
The Required Parameters section ends at API Address. Monitoring Schedule, Probe Scope, and Advanced Parameters are separate sections that follow it. The current Required Parameters section does not include Test Text Length and does not display an API Key.

2.4 Platform Preset Providers#

DeepSeek, Moonshot AI, MiniMax, Baidu, Alibaba, ByteDance, Zhipu AI, and Tencent.

2.5 Configure the Monitoring Schedule and Probe Scope#

1.
Under Monitoring Schedule, choose whether to run for the package validity period or during a specified time range, and select a monitoring frequency.
2.
To limit execution to specific hours, enable the custom execution schedule and enter the allowed time range.
3.
Under Probe Scope, select a Probe Group and click View to confirm its Probes. To create a Probe Group, use the action provided in this section.
4.
Under Advanced Parameters, review the termination conditions, diagnostics, Probe filters, client behavior, and HTTP parameters as needed. Verify the settings before saving the task.
Frequency, validity period, execution schedule, and Probe Group together determine when and from which Probes the task runs. After changing the task type or integration configuration, recheck these settings instead of submitting the task after reviewing only the required parameters.

3 Create a Task with a Custom Integration#

3.1 Select a Protocol and an Existing Model Configuration#

1.
Open the Model Quality Monitoring task creation page, select Model Service Quality or Model Output Quality, and specify the package and task name.
2.
Change Integration Method to Custom Integration.
3.
Select OpenAI Protocol or Anthropic Protocol according to the target API. The selected protocol determines which model providers are available.
4.
Select Model Provider > Model Vendor > Model Name in order. Each lower-level option is filtered by its parent selection, so do not skip a parent field.
5.
Verify the automatically populated API Address. The task parameter section does not display the API Key, so you do not need to enter the credential again when creating the task.
6.
Complete the monitoring schedule, Probe scope, and advanced parameters. Click Save at the bottom of the task page and verify the result from the page response.
image.png
Figure 3. Example interface for selecting the OpenAI protocol in a custom integration

3.2 Switch to the Anthropic Protocol#

1.
Under Custom Integration, select Anthropic Protocol.
2.
Open Model Provider again, select a configuration that supports the protocol, and then select the vendor and model name.
3.
Verify that the API Address has changed to the address associated with the selected configuration. If the target provider is unavailable, check the API protocol configured for its maintenance record.
image.png
Figure 4. Example interface for selecting the Anthropic protocol in a custom integration
A single provider may have separate records for different protocols, model vendors, or model names. After switching protocols, a previously selected record may no longer be available. Recheck every dependent field.

3.3 Create a Model Provider While Creating a Task#

1.
Click Create Model Provider next to the Model Provider drop-down list under Custom Integration.
2.
In the dialog, enter the API protocol, model provider, model vendor, model name, API Address, and API Key.
3.
Verify that the model name matches an identifier supported by the target service, the API Address matches the protocol, and the API Key can access the model.
4.
Click Save at the bottom of the dialog. In the current version, this entry uses Save, which differs from Test and Save in Global Settings. The primary button may remain disabled until all required fields are complete.
5.
After the provider is saved, return to the task form and confirm that the new configuration appears under Model Provider, Model Vendor, and Model Name. If it is not selected automatically, select it manually and verify the API Address.
6.
Complete the remaining task settings and click Save on the task page. Saving the provider configuration does not save the model monitoring task.
image.png
Figure 5. Example Create Model Provider dialog opened from the task creation page
Enter the API Key only in the creation dialog. It is not displayed in plain text in the task parameter section or maintenance list. Closing or canceling the dialog does not save it. Never put a credential in a task name, provider name, model name, or descriptive text field.

3.4 Troubleshoot a Missing Configuration#

1.
Confirm that the current API protocol is correct and clearly distinguish between OpenAI and Anthropic configurations.
2.
On the Custom Model Provider Maintenance page in Global Settings, search for the target provider, vendor, or model name and verify that the record exists.
3.
Check the record's validation status, API Address, and model name. Validate or edit it as required.
4.
Return to the task creation page, reselect the protocol and the three-level model fields, verify the address, and then save the task.
If an API error occurs or the task cannot be saved, follow the error returned by the current operation. Check an invalid model name, a protocol and API Address mismatch, or insufficient permissions in the relevant configuration or provider system. Do not assume that an API has passed validation merely because it is available in a drop-down list.

4 Maintain Custom Model Providers#

4.1 View and Search the List#

1.
Go to System Management > Global Settings.
2.
Expand Model Quality Monitoring Configuration and select Custom Model Provider Maintenance.
3.
In the search box, enter a provider, model vendor, model name, or API Address keyword. Select a validation status as needed.
4.
Review the filtered records and count. If there are no matches, clear the keyword or restore All Statuses, and then check again.
image.png
Figure 6. Example Custom Model Provider Maintenance list
List FieldPurpose
CheckboxSelect records for batch validation.
Model ProviderIdentifies the API provider, which may offer models from multiple vendors.
Model VendorIdentifies the vendor that owns the current model.
Model NameIdentifies the complete model identifier used by the API.
API AddressIdentifies the destination of model requests.
API ProtocolDistinguishes between the OpenAI and Anthropic protocols.
Validation StatusShows the most recent validation result. Consider the validation time when assessing whether the result is current.
ActionsEdit or delete the corresponding record.
The API Key is not displayed in the list. Do not expose credentials through screenshots, shared spreadsheets, or exported documentation.

4.2 Combine Search and Validation Status Filters#

Validation StatusFilter BehaviorWhen to Use It
All StatusesDisplays all validation statuses that match the keyword.Use for an initial search or when troubleshooting a missing record.
Validation PassedDisplays only matching records that passed validation.Also check the latest validation time. A historical result is not a guarantee of continued validity.
Validation FailedDisplays only matching records that failed validation.Correct the configuration and validate it again. An empty result means there are no matching records under the current filters.
image.png
Figure 7. Example validation status filter
The validation status filter only changes the records displayed; it does not run validation. The current options are Validation Passed and Validation Failed, not the Valid and Invalid labels used in an earlier prototype.

4.3 Validate Selected Records#

1.
Use a keyword and validation status to locate the records you want to check.
2.
Select the corresponding rows. If selecting multiple rows, select them individually. After using the header checkbox, verify the complete selection.
3.
After a row is selected, Validate Selected appears on the toolbar. Click it to start validation and wait for the results.
4.
Check the Validation Status and validation time for each record. For a failed record, review the details or error returned by the page, correct the cause, and validate the record again.
5.
Clear the selection when finished, especially before changing filters or pages, to avoid operating on unintended records.
image.png
Figure 8. Example Validate Selected action displayed after selecting records
Validate Selected checks existing configurations; it does not replace creation, import, or edit operations. A historical Validation Passed status reflects the result at that time. Validate the record again after changing provider-side permissions, the model, or the API. The way failure details are displayed may vary by deployment version; use the information returned by the current operation.

4.4 Create a Model Provider#

1.
Click Create Model Provider to the right of the search controls.
2.
Complete the following six fields, making sure that the API protocol, API Address, and model identifier match.
3.
Click Test and Save and wait for the page response. If a required field is missing or a value is invalid, correct it and submit again. Do not assume the operation succeeded merely because the dialog closed.
4.
Search the list for the new record and verify the vendor, model name, API Address, and validation status.
FieldInstructions
API ProtocolSelect the OpenAI or Anthropic protocol actually supported by the target API.
Model ProviderEnter a descriptive provider name.
Model VendorEnter the vendor that owns the model. Do not confuse this field with Model Provider.
Model NameEnter the full model identifier recognized by the server.
API AddressEnter the complete API Address that matches the protocol and target service.
API KeyEnter a credential that can access the API and model. The key is not displayed in plain text after creation.
image.png
Figure 9. Example Create Model Provider dialog in Global Settings
Test and Save is the primary action in this dialog. Batch validation from the list is a separate action. After creating a provider, check the actual validation status in the list instead of assuming that the status was updated because of the button label.

4.5 Edit an Existing Configuration#

1.
Locate and verify the target record, and then click Edit in the Actions column.
2.
The dialog displays API Protocol, Model Provider, Model Vendor, Model Name, and API Address. The API protocol is locked and cannot be changed while editing. The API Key is not displayed.
3.
Modify the required fields. A message below API Address states, Editing configuration information automatically synchronizes it to tasks. Before making a change, identify the affected tasks. After the change, check the relevant task configurations and results.
4.
Click Test and Save and verify the page response. After returning to the list, recheck the fields and validation status.
image.png
Figure 10. Example edit dialog with a locked protocol and hidden API Key
The current edit dialog does not allow you to change the API Key or API protocol. To replace the key or protocol, ask an administrator to confirm the applicable maintenance procedure. Do not put the key in another editable field. The point at which a configuration change takes effect for running tasks depends on the actual deployment and runtime response.

4.6 Import Providers from a Template#

1.
Click Import Template on the toolbar, and then click Download Template.
2.
Keep the template headers unchanged. For each row, enter the model provider, model vendor, model name, API Address, API Key, and API protocol.
3.
Enter the API protocol as 0 or 1: 0 means OpenAI and 1 means Anthropic. Do not enter the protocol label shown in the UI where the template requires a numeric value.
4.
Check the file type, number of data rows, row size, and total file size, and then click Select File.
5.
Verify the file name, click Start Import, and resolve any errors in the returned result. Selecting a file does not mean that the import succeeded.
6.
After the import, search for the imported configurations and verify the records. To validate them, select the target records and click Validate Selected.
image.png
Figure 11. Example model provider template import interface
LimitRequirement
File formats.xlsx, .xls, or .csv
Data recordsA maximum of 50 records, excluding the header row
Row sizeNo more than 32 KB
File sizeNo more than 1 MB
Template structureDo not change header names. Use 0 or 1 for the API protocol.
The uploaded file contains API Keys, so restrict where it is stored and with whom it is shared. When resolving import errors, use the returned row and field details. Do not upload the entire file again without checking whether some records were already imported. If an Imported in This Batch marker appears, use it to help locate the records and still verify their content against the import result.

4.7 Delete a Record#

1.
Locate the target record and click Delete in the Actions column.
2.
Verify the target in the confirmation popover. Click Cancel or close the popover to keep the record. Click Confirm only after verifying that the record should be deleted.
3.
Use both the page response and the list to confirm that the deletion succeeded, and check whether associated tasks require changes.
Deleting a record does not disable it or revoke its validation status. Before deleting a configuration, confirm whether it is referenced by a task. If deletion is blocked, follow the dependency or permission message. Associated tasks must be deleted before the configuration can be deleted. Do not submit the same operation repeatedly. Deletion rules for referenced configurations and their impact on associated tasks depend on the validation behavior of the current deployment.

5 Maintain Custom Test Sets#

5.1 Select the Target Test Set#

1.
Go to System Management > Global Settings > Model Quality Monitoring Configuration > Custom Test Set Maintenance.
2.
Identify the task type and metric, and then locate the corresponding record. The page always lists four test set categories, with actions available on each row.
3.
Check Last Created and Status to identify the test set you intend to upload, download, enable, disable, or reset.
image.png
Figure 12. Example Custom Test Set Maintenance list
MetricTest Set Scope
Model Service Quality (shared by all metrics)All service quality metrics share this test set.
Model Output Quality - CorrectnessUsed only for the Correctness metric.
Model Output Quality - ConsistencyUsed only for the Consistency metric.
Model Output Quality - Structured Output AccuracyUsed only for the Structured Output Accuracy metric.
Service quality metrics share one test set, whereas each output quality metric has a separate test set. If no custom test set has been uploaded, the official public test set is used. Existing creation times and enablement statuses reflect the current configuration and do not mean that a new account includes preset custom test sets.

5.2 Upload or Replace a Test Set#

1.
Click Upload in the row for the target metric. The page no longer uses a top-level Create Test Set action.
2.
Check the metric shown in the dialog. It is inherited from the selected row. If the wrong metric is shown, cancel and open the dialog again from the correct row.
3.
Click Download Template and complete the question name, test input, expected result, and regular expression notes according to the template. Do not change the headers.
4.
Verify that the file contains no more than 50 records, each row is no larger than 32 KB, and the file is no larger than 1 MB. Save it as .xlsx, .xls, or .csv.
5.
Click Select File, verify the file name, metric, and content, and then click Confirm Upload.
6.
Resolve validation errors based on the page response. After the upload, return to the list and check Last Created and Status. If the test set is still disabled, enable it as required.
image.png
Figure 13. Example metric-specific test set upload and template limits
Template ColumnWhat to Check
Question NameUse a descriptive name for each test question rather than an ambiguous number.
Test InputEnter the content to send to the model.
Expected ResultEnter the expected output according to the selected metric and the downloaded template.
Regular Expression NotesKeep the existing column name and follow the template instructions. If the page does not define a common regular expression syntax or evaluation rule, do not invent one.
The 50-record limit excludes the header row. The 32 KB limit applies to the size of each row, not its character count; multibyte text and line breaks also consume space. Uploaded content must not contain personal data, trade secrets, or other information that should not be sent to a model service.
Maintain the four test set categories separately. Replacing the Correctness test set does not replace the Consistency or Structured Output Accuracy test set. Before replacing an existing file, download a backup. After the upload, check the target metric and status rather than checking only upload progress.

5.3 Download the Current Test Set#

1.
Confirm that a custom test set exists in the row for the target metric.
2.
Click Download in that row, save the file, and verify its content. If Download is unavailable, first confirm that the row contains uploaded data.
3.
Distinguish between Download Template and Download: Download Template in the upload dialog provides a blank file for new data, whereas Download in the list retrieves the current test set.
Downloading does not change whether the test set is enabled. A successful download also does not mean that any other operation has completed.

5.4 Understand Enable and Disable Actions#

Current StatusDownloadEnableDisableReset
No custom test set uploadedUsually unavailableUnavailableUnavailableNo custom content to reset
Uploaded and enabledAvailableDisabledAvailableAvailable
Uploaded and disabledAvailableAvailableDisabledAvailable
1.
To start using an uploaded test set that is disabled, click Enable in its row and verify the updated status.
2.
To stop using a custom test set, click Disable, review and accept the confirmation prompt, and verify the updated status. Cancel if you do not want to continue. Disabling a test set does not clear it; it can be enabled again later.
3.
If a button is unavailable, check the current status first. An enabled record does not need to be enabled again, and a disabled record does not need to be disabled again.
The table describes common relationships between actions. Actual availability depends on the current interface. Do not infer the fallback behavior, when an active task switches to another test set, or the effective time based only on button labels. After a change, verify task results or follow runtime rules confirmed by an administrator.

5.5 Reset to the Platform Test Set#

1.
To clear a custom test set, download a backup first, and then click Reset in the row for the target metric.
2.
Read the confirmation message: After the reset, the uploaded test set will be cleared and the platform-provided test set will be used by default.
3.
Click Cancel if you do not want to clear the test set. Click Confirm only after verifying that the custom content for this metric should be removed.
4.
Use the page response to confirm that the data was cleared, and then check Last Created, Status, and the available actions for the row. To use custom content again, upload a new file from the same metric row.
image.png
Figure 14. Example reset confirmation for a test set
Reset and Disable have different effects. Reset clears the uploaded test set, and the confirmation message explicitly states that the platform-provided test set will be restored. Disable only indicates that the current custom test set is not active. Verify when running tasks switch by checking actual task results.

6 FAQs and Operational Verification#

IssueWhat to Check FirstRecommended Action
Anthropic cannot be selected for a platform presetIs Integration Method set to Platform Preset?Platform presets use OpenAI. Switch to Custom Integration to use an Anthropic configuration.
A custom provider is missing from the drop-down listDo the API protocol and provider configuration match?Select the correct protocol, check the Global Settings record, and then select the vendor and model in order.
Fewer model names are listed than expectedHave you checked the current provider and vendor and scrolled through the complete list?Recheck the parent selections and search for the full name. Do not treat the initially visible options as the complete list.
The API Address is not the expected valueAre the three model fields and maintenance record correct?Verify the selections. To change a custom configuration, edit it on the maintenance page.
The API Key is unavailable while editingIs the dialog in edit mode?This is expected in the current interface. Ask an administrator to confirm the dedicated procedure for replacing a key.
Validate Selected is unavailableHave you selected a record in the list?Select the target record first. Recheck the selection after filtering or changing pages.
Validation failedCheck the current error, protocol, model name, address, and provider-side permissions.Correct the configuration and retry. A previous successful validation does not replace the current result.
Template import or test set upload was rejectedCheck the format, headers, 50-record limit, 32 KB row limit, and 1 MB file size limit.Use the returned error to locate and correct the record. Do not change the template headers.
Enable or Disable is unavailable for a test setCheck the current row status and whether a custom file exists.An enabled test set can only be disabled, and a disabled test set can only be enabled. Upload a file first if none exists.
Restore the platform test setDo you need to retain the existing custom file?Download a backup, reset the target metric, and confirm the operation.

6.1 Post-Operation Checklist#

1.
Task: Confirm that the task type is correct; Integration Method and API Protocol match; Model Provider, Model Vendor, Model Name, and API Address are consistent; and the monitoring schedule and Probe scope meet the requirements.
2.
Provider: Confirm that a newly created or edited record can be found in the list. Validate it again when required and check the validation time. Confirm that no API Key is exposed in a list or documentation.
3.
Test set: Confirm that you operated on the correct metric, the file satisfies all limits, and the status is correct after upload. Before resetting, download a backup and understand the platform-test-set fallback behavior.
4.
Before leaving the page, check the page result and task list. If an error remains, resolve it instead of treating a closed dialog or a selected file as a successful business operation.
上一页
Overview
Built with