> For the complete documentation index, see [llms.txt](https://ibm-developer.gitbook.io/cloudpakfordata-telco-workshop/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ibm-developer.gitbook.io/cloudpakfordata-telco-workshop/telco-workshop/openscale-manual-config/quality-feedback-readme.md).

# OpenScale Manual Config Part 3

Watson OpenScale utilizes several monitors to gather data about machine learning inferences and the GUI tool can then present that data in a form that is useful. In this sub-module we will use a Jupyter notebook to configure the monitor for Quality and enable Feedback logging.

## Steps for OpenScale Quality monitor and Feedback logging

The submodule contains the following steps:

1. [Open the notebook](/cloudpakfordata-telco-workshop/telco-workshop/openscale-manual-config/quality-feedback-readme.md#1-open-the-notebook)
2. [Update credentials](/cloudpakfordata-telco-workshop/telco-workshop/openscale-manual-config/quality-feedback-readme.md#3-update-credentials)
3. [Run the notebook](/cloudpakfordata-telco-workshop/telco-workshop/openscale-manual-config/quality-feedback-readme.md#4-run-the-notebook)
4. [Begin to Explore the Watson OpenScale UI](/cloudpakfordata-telco-workshop/telco-workshop/openscale-manual-config/quality-feedback-readme.md#5-begin-to-explore-the-watson-openscale-ui)

## 2. Open the notebook

If you [Created the Project](https://ibm-developer.gitbook.io/cloudpakfordata-telco-churn-workshop/getting-started/pre-work#create-a-new-project) using the [Customer-Churn-Project.zip](https://github.ibm.com/IBMDeveloper/cp4d-workshop-telco-churn/blob/master/projects/Customer-Churn-Project.zip) file, your notebook will be present in that project, under the `Assets` tab:

TODO: Replace with new image ![Project from zip assets tab](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2F799c9d41e9fd3b14493796bf51d0dbbfb47e4477.png?generation=1595273046954672\&alt=media)

You may now skip to the next step [Update credentials](/cloudpakfordata-telco-workshop/telco-workshop/openscale-manual-config/quality-feedback-readme.md#2-update-credentials)

## Import the notebook (If you are not using the Project Import pre-work steps)

> NOTE: You should probably not need this step, and should only perform it if instructed to.

If, for some reason, you are not using the [Created the Project](https://ibm-developer.gitbook.io/cloudpakfordata-telco-churn-workshop/getting-started/pre-work#create-a-new-project) step in the Pre-work to import [Customer-Churn-Project.zip](https://github.ibm.com/IBMDeveloper/cp4d-workshop-telco-churn/blob/master/projects/Customer-Churn-Project.zip), then you will need to import the notebook file by itself. Use the following steps for that.

At the project overview click the *New Asset* button, and choose *Add notebook*.

![Add a new asset](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2Fffdd624db625f2e72fb17345ffc48a82d5d479bf.png?generation=1623704196725749\&alt=media)

On the next panel select the *From URL* tab, give your notebook a name, provide the following URL, and choose the Python 3.6 environment:

```bash
https://raw.githubusercontent.com/IBM/telco-churn-workshop-cpd/master/notebooks/openscale-quality-feedback.ipynb
```

> The notebook is hosted in the same repo as [the workshop](https://github.com/IBM/telco-churn-workshop-cpd)
>
> * **Notebook**: [openscale-quality-feedback.ipynb](https://github.com/IBM/cloudpakfordata-telco-churn-workshop/tree/7d19e544c8b7e873ddaf5a26d604d5c15ea11b15/notebooks/openscale-quality-feedback.ipynb)
> * **Notebook with output**: [openscale-fairness-explainability-with-output.ipynb](https://github.com/IBM/cloudpakfordata-telco-churn-workshop/tree/7d19e544c8b7e873ddaf5a26d604d5c15ea11b15/notebooks/with-output/openscale-quality-feedback-with-output.ipynb)

TODO: new image ![Add notebook name and URL](https://github.com/IBM/cloudpakfordata-telco-churn-workshop/tree/7d19e544c8b7e873ddaf5a26d604d5c15ea11b15/workshop/images/openscale/openscale-add-notebook-url.png)

When the Jupyter notebook is loaded and the kernel is ready then we can start executing cells.

![Notebook loaded](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2F8b03c89e5a5b4bfb9d265173e3373e012a77ab81.png?generation=1595273046600164\&alt=media)

### 3. Update credentials

#### WOS\_CREDENTIALS

* In the notebook section *2.0* you will add your Cloud Pak for Data platform credentials for the *WOS\_CREDENTIALS*.
* For the `url` field, change `https://w.x.y.z` to use the URL your ICP cluster, i.e something like: `"url": "https://zen-cpd-zen.omid-cp4d-v5-2bef1f4b4097001da9502000c44fc2b2-0001.us-south.containers.appdomain.cloud"`.
* For the `username`, use your Cloud Pak for Data login username.
* For the `password`, user your Cloud Pak for Data login password.

### 4. Run the notebook

> **Important**: *Make sure that you stop the kernel of your notebook(s) when you are done, in order to prevent leaking of memory resources!*

![Stop kernel](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2Fc1d53c6af7e75ea6fcc31ddeaaaaa8cb57782c37.png?generation=1595272995416364\&alt=media)

Spend an minute looking through the sections of the notebook to get an overview. You will run cells individually by highlighting each cell, then either click the `Run` button at the top of the notebook. While the cell is running, an asterisk (`[*]`) will show up to the left of the cell. When that cell has finished executing a sequential number will show up (i.e. `[17]`).

## 5. Begin to Explore the Watson OpenScale UI

We've enabled the monitor for Quality and Feedback logging, now let's explore the results in the OpenScale GUI.

* In the same browser (but a separate tab), open the `Services` tab by clicking the icon in the upper right. Go to the `OpenScale` tile under the `AI` category and click `Open`:

![Deploy OpenScale](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2Fb8d5af08bafe59cbf8111dcaecaa57222721a0e8.png?generation=1595273049794733\&alt=media)

* When the dashboard loads, ***Click*** on the ***'Model Monitors'***  tab and you will see the deployment you configured in the jupyter notebook when you ran it in the previous section:

![Explore OpenScale Model monitors](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2Febac53d2a81350271d108850efb786cb4b61e72d.png?generation=1598362513882591\&alt=media)

Do not worry if the name you see does not match exactly with the screenshot. The deployment name you see will correspond to the variable used in the Jupyter notebook

### Quality Monitor

In our dashboard we can see that we have a choice for a variety of graphs under *Quality*. If we choose *Area under ROC*, where there is a threshold violation in my example, we'll see a limited chart due to the lack of scoring data. (More data will be added later to make this more interesting.

* Click on a time slot to dig deeper into the graph:

![Quality area under ROC](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2F30d105755051f231d539b83c040642e44ec0f785.png?generation=1623704195425035\&alt=media)

We can see statistics for this time slot including *Area under ROC*, *TPR*, *FPR*, *Recall*, *Precision*, and more:

![Quality statitstics](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2F5db6507f07f4ce45b86aad02e3a239e4d2f5a017.png?generation=1623704195697989\&alt=media)

Other time slots can be examined to gather the relevant quality statistics.

*Payload logging* is enabled and will take place automatically when used with Watson Machine Learning. All Scoring request payloads and the returned data will be logged in the datamart.

![Payload logging](https://2515897395-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YTFb_WdaBDnhlXs-q%2Fsync%2F9319e7342233be2c5e70125d9aa71862ba100ab5.png?generation=1623704194083233\&alt=media)

### Recap

In this sub-module we've setup Payload logging and the Quality monitor. Move on to the next submodule to learn about the [Drift monitor](https://github.com/IBM/cloudpakfordata-telco-churn-workshop/tree/7d19e544c8b7e873ddaf5a26d604d5c15ea11b15/workshop/openscale-manual-config/DRIFT.md)