Model Operation Monitoring
To monitor the model operations:
- If you are on the Model registry page, click the icon to open the Model Ops page.
- If you are on the model metadata or artifact page, click on the “Model Ops” menu item from the top menu bar.
- The Model Ops shows:
- Total Requests: this indicates the number of API calls
- Total Predictions: Number of predictions done by the model. Each API call may send multiple input sets. Predictions are done for each input and hence this shows the number of predictions made by the model for all the inputs received.
- Median Response Time (ms) Per Request: As the name suggests, this is an aggregate median time of all the requests processed by the API.
- Data Error Rate: Any error detected due to data issues, such as incorrect column types, missing or null, the API will record that as errors. The error rate is calculated by the diving the data related error count by the total number of API requests received.
- System Error Rate: This error rate indicates the error due to any system related issues, such as the serving node being down, network issues, or other system related issue not related to the data.
- Consumers: This indicates the number of API consumers.
- Data Error Per Day: This shows a daily trend of error rates.
- The Model Ops report is specific to a particular model.
- The aggregation is done over a week worth of data and older data are removed to save on disk space. This behavior may be configured from the backend.
Figure 5.6 shows a screenshot of the Model Ops page.
Figure 4.6: Screenshot showing the Model Ops page