# Deploy and Test Machine Learning Models In this module, we will learn how to deploy our Machine Learning models. By doing so, we make them available for use in production such that applications and business processes can derive insights from them. There are several types of deployments available ([depending on the model framework used](https://www.ibm.com/support/producthub/icpdata/docs/content/SSQNUZ_current/wsj/analyze-data/pm_service_supported_frameworks.html)), of which we will explore: * Online Deployments - Allows you to run the model on data in real-time, as data is received by a web service. * Batch Deployments - Allows you to run the model against data as a batch process. This module is broken up into several sections that explore the different model deployment options as well as the different ways to invoke or consume them. The first section of this lab will build an online deployment and test the model endpoint using both the built in testing tool as well as external testing tools. The remaining sections are optional, they build and test the batch deployment, followed by using the model from a python application. 1. [Online Deployment for a Model](#online-model-deployment) * Create Online model deployment * Test the deployed model web UI * (Optional) Test model using cURL 2. [(Optional) Integrate Model to an External Application](#optional-integrate-model-to-python-flask-application) 3. [(Optional) Update Model Deployment](#optional-update-online-model-deployment) > **Note:** The lab instructions below assume you have completed the pre-work section already, if not, be sure to complete the pre-work first to create a project and a deployment space. ## Online Model Deployment After a model has been created, saved and promoted to our deployment space, we can procceed to dpeloying the model. For this section, we will be creating an online deployment. This type of deployment will make an instance of the model available to make predictions in real time via an API. Although we will use the Cloud Pak for Data UI to deploy the model, the same can be done programmatically. * Navigate to the left-hand (☰) hamburger menu and choose `Analyze` -> `Analytics deployments`: ![Analytics Analyze deployments](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2F07f840562c364be3d2b72966a043cb5fe8a69fcc.png?generation=1603461090214849\&alt=media) * Choose the deployment space you setup previously by clicking on the name of your space. * From your deployment space overview, in the table, click on the model name that you previousely promoted. Next, you can click on the `Create Deployment`. > Note: There may be more than one model listed in them 'Models' section. This can happen if you have run the Jupyter notebook more than once or if you have run through both the Jupyter notebook and AutoAI modules to create models. Although you could select any of the models you see listed in the page, the recommendation is to start with whicever model is available that is using a `spark-mllib_2.3` runtime (this is denoted in the `Software Specification` column). ![Actions Deploy model](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2F6bca3eda3499074455d529cd7f2e69078ac266a6.png?generation=1603827637906104\&alt=media) * On the `Create a deployment` screen, choose `Online` for the `Deployment Type`, give the Deployment a name and optionally a description and click the `Create` button. ![Online Deployment Create](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2F68625bfa19dcef5bf0cea90a80e8609f1a713f62.png?generation=1626367015246140\&alt=media) * The Deployment will show as `In progress` and then switch to `Deployed` when done. ![Status Deployed](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2Ffba7885616d16ea19f85226f4a414ab44401aa51.png?generation=1626367029933420\&alt=media) ### Test Online Model Deployment Cloud Pak for Data offers tools to quickly test out Watson Machine Learning models. We begin with the built-in tooling. * From the Model deployment page, once the deployment status shows as `Deployed`, click on the name of your deployment. The deployment `API reference` tab shows how to use the model using `cURL`, `Java`, `Javascript`, `Python`, and `Scala`. * To get to the built-in test tool, click on the `Test` tab and then click on the `Provide input data as JSON` icon. ![Test deployment with JSON](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2F91c4bd466e1a56f8aabdc389c7fd6cda1ab9b0ac.png?generation=1626366978707939\&alt=media) * Copy and paste the following data objects into the `Body` panel > (**Note:** Make sure the input below is the only content in the field. Do not append it to the default content `{ "input_data": [] }` that may already be in the field. Instead, remove the exsiting content and replace it with the following data.). ```javascript { "input_data": [ { "fields": [ "CheckingStatus", "LoanDuration", "CreditHistory", "LoanPurpose", "LoanAmount", "ExistingSavings", "EmploymentDuration", "InstallmentPercent", "Sex", "OthersOnLoan", "CurrentResidenceDuration", "OwnsProperty", "Age", "InstallmentPlans", "Housing", "ExistingCreditsCount", "Job", "Dependents", "Telephone", "ForeignWorker"], "values": [ [ "no_checking", 13, "credits_paid_to_date", "car_new", 1343, "100_to_500", "1_to_4", 2, "female", "none", 3, "savings_insurance", 46, "none", "own", 2, "skilled", 1, "none", "yes"] ] } ] } ``` * Click the `Predict` button. The model will be called with the input data and the results will display in the `Result` window. Scroll down to the bottom of the result to see the prediction (i.e "Risk" or "No Risk"): ![Testing the deployed model](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2Fe9e48c1ff98d4b8f747d75607c5928c12c51edab.png?generation=1626367036319274\&alt=media) > **Note:** For some deployed models (for example AutoAI based models), you can provide the request payload using a generated form by clicking on the `Provide input using form` icon and providing values for the input fields of the form. If the form is not available for the model you deployed, the icon will not be present or will remain grayed out. ![Input to the fields](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2F258b4b3451e2f8c2a30df115f4eff8266aa335a1.png?generation=1626367006676331\&alt=media) ### (Optional) Test Online Model Deployment using cURL Now that the model is deployed, we can also test it from external applications. One way to invoke the model API is using the cURL command. > **Note for WINDOWS users:** This section uses commands available in unix-based systems (MacOS, Linux, ...). Windows users can use [IBM Cloud Shell](https://cloud.ibm.com/shell) or if you want to run the commands locally, it is recommended to [download gitbash](https://gitforwindows.org/) or enable [Windows Linux Subsystem (WSL)](https://docs.microsoft.com/en-us/windows/wsl/install-win10) if you use Windows 10. If neither option works for you, you will need to adapt the commands for your system to follow this section (for instance, you need to change `export` commands to `set` commands, and find an alternative for `cURL`) * First step is to install the [IBM Cloud CLI](https://cloud.ibm.com/functions/learn/cli). You can follow the instructions in the link to do so. * In order to get access token you need to have `API Key`, that you can get from your IBM cloud account. You can create one by running following command. Remember to save your API Key as they can't be retrieved after they are created. ```bash ibmcloud iam api-key-create ``` ![Model Deployment Endpoint](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2F725803862f164a1668884afd46947594b667c7b3.png?generation=1603827638397075\&alt=media) * Next, in a terminal window, run the following command to get a token to access the API. Replace `` with the api key that you got from running above command. ```bash curl -X POST 'https://iam.cloud.ibm.com/identity/token' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: application/json' --data-urlencode 'grant_type=urn:ibm:params:oauth:grant-type:apikey' --data-urlencode 'apikey=' ``` * A json string will be returned with a value for `accessToken` that will look *similar* to this: ```javascript {"access_token":"AAAAAAAfakeACCESSTOKENNNNNNN","refresh_token":"BBBBBBBBBBBFAKEREFRESHTOKENNNNNNNNNNNNN","token_type":"Bearer","expires_in":3600,"expiration":1601317201,"scope":"ibm openid"} ``` * You will save the access token right after the `access_token` in a temporary environment variable in your terminal. Copy the access token value (without the quotes) in the terminal and then use the following export command to save the "accessToken" to a variable called `WML_AUTH_TOKEN`. ```bash export WML_AUTH_TOKEN= ``` * Back on the model deployment page, gather the `URL` to invoke the deployed model from the *API reference* by copying the `Endpoint`. ![Model Deployment Endpoint](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2F6fdbf831055257bbffd9cc3a6992e1703635c582.png?generation=1626367017646751\&alt=media) * Now save that endpoint to a variable named `URL` in your terminal by exporting it. URL also requires a version query parameter. ```bash export WML_URL= ``` Example of an URL: ```bash export WML_URL="https://us-south.ml.cloud.ibm.com/ml/v4/deployments//predictions?version=2020-09-01" ``` * Now run this curl command from the terminal to invoke the model with the same payload we used previousy: ```bash curl -k -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header "Authorization: Bearer $WML_AUTH_TOKEN" -d '{"input_data": [{"fields": [ "CheckingStatus", "LoanDuration", "CreditHistory", "LoanPurpose", "LoanAmount", "ExistingSavings", "EmploymentDuration", "InstallmentPercent", "Sex", "OthersOnLoan", "CurrentResidenceDuration", "OwnsProperty", "Age", "InstallmentPlans", "Housing", "ExistingCreditsCount", "Job", "Dependents", "Telephone", "ForeignWorker"],"values": [[ "no_checking", 13, "credits_paid_to_date", "car_new", 1343, "100_to_500", "1_to_4", 2, "female", "none", 3, "savings_insurance", 46, "none", "own", 2, "skilled", 1, "none", "yes"]]}]}' $WML_URL ``` * A json string will be returned with the response, including a prediction from the model (i.e a "Risk" or "No Risk" at the end indicating the prediction of this loan representing risk). ## (Optional) Integrate Model to Python Flask Application You can also access the online model deployment directly through the REST API. This allows you to use your model for inference in any of your apps. For this workshop we'll be using a Python Flask application to collect information, score it against the model, and show the results. > **IMPORTANT: This SAMPLE application only runs on python 3.6 and above, so the instructions here are for python 3.6+ only. You will need to have Python 3.6 or later already installed on your machine** *Note: The instructions below assume you have completed the pre-work module and thus have the Git repository already on your machine (cloned or downloaded).* ### Download the frontend application In order to follow along with this section, use one of the following links to get a copy of the front end application that is used in this lab. Use the `[Download]` link to get the file. If the link isn't working for you, try clicking the `[Mirror]` to get it from out backup servers. * Python Application [\[Download\]](http://ibm.biz/ddc-fs-cp4daas-python-app) | [\[Mirror\]](http://ibm.biz/ddc-fs-cp4daas-python-app-mirror) ### Install Dependencies The general recommendation for Python development is to use a virtual environment ([`venv`](https://docs.python.org/3/tutorial/venv.html)). To install and initialize a virtual environment, use the `venv` module on Python 3: * Initialize a virtual environment with [`venv`](https://docs.python.org/3/tutorial/venv.html). Run the following commands in a terminal (or command prompt): ```bash # Create the virtual environment using Python. # Note, it may be named python3 on your system. python -m venv venv # Python 3.X # Source the virtual environment. Use one of the two commands depending on your OS. source venv/bin/activate # Mac or Linux ./venv/Scripts/activate # Windows PowerShell ``` > **TIP** To terminate the virtual environment use the `deactivate` command. * To install the Python requirements, from a terminal (or command prompt) navigate to where you cloned/downloaded the Git repository. Run the following commands: ```bash cd flaskapp pip install -r requirements.txt ``` ### Update Environment Variables It's best practice to store configurable information as environment variables, instead of hard-coding any important information. To reference our model and supply an API key, we'll pass these values in via a file that is read, the key-value pairs in this files are stored as environment variables. * Copy the `env.sample` file to `.env`. ```bash cp env.sample .env ``` * Edit `.env` to and fill in the `MODEL_URL` as well as the `TOKEN_REQUEST_URL`, and `API_TOKEN`. * `MODEL_URL` is your web service URL for scoring which you got from the section above * `TOKEN_REQUEST_URL` is the URL for the identity management service where you can request your access tokens. This should not need to be changed * `API_TOKEN` is the API Key that we generated earlier to authenticate with IBM Cloud > Note: Alternatively, you can fill in the `AUTH_TOKEN` instead of `AUTH_URL`, `AUTH_USERNAME`, and `AUTH_PASSWORD`. You will have generated this token in the section above. However, since tokens expire after a few hours and you would need to restart your app to update the token, this option is not suggested. Instead, if you use the username/password option, the app can generate a new token every time for you so it will always have a non-expired ones. * Here's an example of a completed lines of the .env file. ```bash # 1. Required: Provide your web service URL for scoring. # E.g., MODEL_URL=https:///v4/deployments//predictions MODEL_URL=https://us-south.ml.cloud.ibm.com/ml/v4/deployments/35e4cadc-1024-4102-b782-958ab5c68fc7/predictions?version=2021-02-03 # 2. Required: fill in EITHER section A OR B below: # ### A: Authentication using API_TOKEN # Fill in your API Token. You don't need to update the TOKEN_REQUEST_URL # Example: # TOKEN_REQUEST_URL=https://iam.ng.bluemix.net/identity/token # API_TOKEN= TOKEN_REQUEST_URL=https://iam.ng.bluemix.net/identity/token API_TOKEN=_EbKb6tQWS_e7abzP2-DjWx46dMGt59-XJwE9karbWwT ``` ### Start Application * Start the flask server by running the following command: ```bash python creditriskapp.py ``` * Use your browser to go to and try it out. > **TIP**: Use `ctrl`+`c` to stop the Flask server when you are done. #### Test the application * Either use the default values pre-filled in the input form, or modify the value and then click the `Submit` button. The python application will invoke the predictive model and a risk prediction & probability is returned: ![Get the risk percentage as a result](https://1630346741-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YPhXPJogwds2wYiEl%2Fsync%2F121d9ca817a76f011255241d22cd93847c7cd685.png?generation=1590535385938816\&alt=media) ## (Optional) Update Online Model Deployment We have seen how to deploy our machine learning models so they can be called by external consumers. Specifically, we have seen how to create an online deployment which will make an instance of the model available to make predictions in real time via an API. Over time, these models will eventually need to be updated for any number of reasons. In this section, we will explore how you can update the deployment of models without disrupting consumers of the API. See the documentations for [updating model deployments](https://ibm-developer.gitbook.io/cloudpakfordata-credit-risk-workshop/workshop-faststart/credit-risk-workshop/updatemodeldeployment) for details. ## Conclusion In this section we covered the followings: * Creating and Testing Online Deployments for models. * (Optional) Creating and Testing Batch Deployments for models. * (Optional) Integrating the model deployment in an external application. Taking a predictive model and infusing AI into applications. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://ibm-developer.gitbook.io/cloudpakfordata-credit-risk-workshop/workshop-faststart/credit-risk-workshop/machine-learning-deployment-scoring.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.