| Check Before You Begin | What to Prepare |
|---|---|
| Account and feature access | Use an account that can access Task Management, Global Settings, and Model Quality Monitoring configuration. If an entry or action is unavailable, contact an administrator to verify your permissions. |
| Package and Probes | Make sure that a valid model monitoring package is available and that you can select the required Probe Groups. |
| Platform preset | Identify the target model provider, model vendor, and model name in advance. You do not need to enter an API Key on the task page. |
| Custom integration | Prepare the API protocol, provider name, model vendor, model name, API Address, and corresponding API Key. You can select an existing configuration if one is available. |
| Custom test set | Download and complete the current template for the target metric. If no custom test set has been uploaded, the platform-provided public test set is used. |
| Task Type | Purpose | Test Set Relationship |
|---|---|---|
| Model Service Quality | Observe the service performance of model APIs. | All service quality metrics share one test set. |
| Model Output Quality | Evaluate the correctness, consistency, and structured output accuracy of model responses. | Each of the three metrics uses its own test set. |
| Operation | Navigation Path |
|---|---|
| Create a model monitoring task | Task Management > Task Maintenance > Create Task > Model Quality Monitoring |
| Maintain custom model providers | System Management > Global Settings > Model Quality Monitoring Configuration > Custom Model Provider Maintenance |
| Maintain custom test sets | System Management > Global Settings > Model Quality Monitoring Configuration > Custom Test Set Maintenance |
| Field | How to Complete It | What to Recheck After a Change |
|---|---|---|
| Package | Select a model monitoring package available to the current account. | Verify the package status and validity period. |
| Task Name | Enter the name manually. | Use a name that distinguishes the provider, model, and monitoring purpose. |
| Integration Method | Select Platform Preset or Custom Integration. | Recheck the protocol and the three-level model selection after switching methods. |
| API Protocol | Platform presets use the OpenAI protocol. | Anthropic being disabled is expected in this mode. |
| Model Provider | Select a provider from the platform preset list. | Available vendors and model names change with the provider. |
| Model Vendor | Select a vendor supported by the current provider. | A provider may offer models from multiple vendors. |
| Model Name | Select a model from the current vendor's model list. | Use the full model identifier rather than a similar name. |
| API Address | Automatically populated from the selected platform configuration. | Recheck the address after changing the provider or model. Do not override it manually. |
| List Field | Purpose |
|---|---|
| Checkbox | Select records for batch validation. |
| Model Provider | Identifies the API provider, which may offer models from multiple vendors. |
| Model Vendor | Identifies the vendor that owns the current model. |
| Model Name | Identifies the complete model identifier used by the API. |
| API Address | Identifies the destination of model requests. |
| API Protocol | Distinguishes between the OpenAI and Anthropic protocols. |
| Validation Status | Shows the most recent validation result. Consider the validation time when assessing whether the result is current. |
| Actions | Edit or delete the corresponding record. |
| Validation Status | Filter Behavior | When to Use It |
|---|---|---|
| All Statuses | Displays all validation statuses that match the keyword. | Use for an initial search or when troubleshooting a missing record. |
| Validation Passed | Displays only matching records that passed validation. | Also check the latest validation time. A historical result is not a guarantee of continued validity. |
| Validation Failed | Displays only matching records that failed validation. | Correct the configuration and validate it again. An empty result means there are no matching records under the current filters. |
| Field | Instructions |
|---|---|
| API Protocol | Select the OpenAI or Anthropic protocol actually supported by the target API. |
| Model Provider | Enter a descriptive provider name. |
| Model Vendor | Enter the vendor that owns the model. Do not confuse this field with Model Provider. |
| Model Name | Enter the full model identifier recognized by the server. |
| API Address | Enter the complete API Address that matches the protocol and target service. |
| API Key | Enter a credential that can access the API and model. The key is not displayed in plain text after creation. |
0 or 1: 0 means OpenAI and 1 means Anthropic. Do not enter the protocol label shown in the UI where the template requires a numeric value.| Limit | Requirement |
|---|---|
| File formats | .xlsx, .xls, or .csv |
| Data records | A maximum of 50 records, excluding the header row |
| Row size | No more than 32 KB |
| File size | No more than 1 MB |
| Template structure | Do not change header names. Use 0 or 1 for the API protocol. |
| Metric | Test Set Scope |
|---|---|
| Model Service Quality (shared by all metrics) | All service quality metrics share this test set. |
| Model Output Quality - Correctness | Used only for the Correctness metric. |
| Model Output Quality - Consistency | Used only for the Consistency metric. |
| Model Output Quality - Structured Output Accuracy | Used only for the Structured Output Accuracy metric. |
.xlsx, .xls, or .csv.| Template Column | What to Check |
|---|---|
| Question Name | Use a descriptive name for each test question rather than an ambiguous number. |
| Test Input | Enter the content to send to the model. |
| Expected Result | Enter the expected output according to the selected metric and the downloaded template. |
| Regular Expression Notes | Keep the existing column name and follow the template instructions. If the page does not define a common regular expression syntax or evaluation rule, do not invent one. |
| Current Status | Download | Enable | Disable | Reset |
|---|---|---|---|---|
| No custom test set uploaded | Usually unavailable | Unavailable | Unavailable | No custom content to reset |
| Uploaded and enabled | Available | Disabled | Available | Available |
| Uploaded and disabled | Available | Available | Disabled | Available |
| Issue | What to Check First | Recommended Action |
|---|---|---|
| Anthropic cannot be selected for a platform preset | Is Integration Method set to Platform Preset? | Platform presets use OpenAI. Switch to Custom Integration to use an Anthropic configuration. |
| A custom provider is missing from the drop-down list | Do the API protocol and provider configuration match? | Select the correct protocol, check the Global Settings record, and then select the vendor and model in order. |
| Fewer model names are listed than expected | Have you checked the current provider and vendor and scrolled through the complete list? | Recheck the parent selections and search for the full name. Do not treat the initially visible options as the complete list. |
| The API Address is not the expected value | Are the three model fields and maintenance record correct? | Verify the selections. To change a custom configuration, edit it on the maintenance page. |
| The API Key is unavailable while editing | Is the dialog in edit mode? | This is expected in the current interface. Ask an administrator to confirm the dedicated procedure for replacing a key. |
| Validate Selected is unavailable | Have you selected a record in the list? | Select the target record first. Recheck the selection after filtering or changing pages. |
| Validation failed | Check the current error, protocol, model name, address, and provider-side permissions. | Correct the configuration and retry. A previous successful validation does not replace the current result. |
| Template import or test set upload was rejected | Check the format, headers, 50-record limit, 32 KB row limit, and 1 MB file size limit. | Use the returned error to locate and correct the record. Do not change the template headers. |
| Enable or Disable is unavailable for a test set | Check the current row status and whether a custom file exists. | An enabled test set can only be disabled, and a disabled test set can only be enabled. Upload a file first if none exists. |
| Restore the platform test set | Do you need to retain the existing custom file? | Download a backup, reset the target metric, and confirm the operation. |