> For the complete documentation index, see [llms.txt](https://ibm-developer.gitbook.io/cloudpakforintegration-workshop/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ibm-developer.gitbook.io/cloudpakforintegration-workshop/docs/exercise-event-streaming.md).

# Exercise - Using IBM MQ and IBM Event Streams for near realtime data replication

In this lab you will use IBM MQ and IBM Event Streams to replicate data from a transactional database to a reporting database. The pattern used allows for seamless horizontal scaling to minimize the latency between the time the transaction is committed to the transactional database and when it is available to be queried in the reporting database.

The architecture of the solution you will build is shown below:

![Architecture diagram](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-0f220fe9ec092165dca7b2624b63f2b5376745a0%2Farchitecture.png?alt=media)

* The **portfolio** microservice sits at the center of the application. This microservice:
  * sends completed transactions to an IBM MQ queue.
  * calls the **trade-history** service to get aggregated historical trade data.
* The **Kafka Connect** source uses the Kafka Connect framework and an IBM MQ source to consume the transaction data from IBM MQ and sends it to a topic in IBM Event Streams. By scaling this service horizontally you can decrease the latency between the time the transaction is committed to the transactional database and when it is available to be queried in the reporting database,
* The **Kafka Connect** sink uses the Kafka Connect framework and a Mongodb sink to receive the transaction data from IBM Event Streams and publishes it to the reporting database. By scaling this service horizontally you can decrease the latency between the time the transaction is committed to the transactional database and when it is available to be queried in the reporting database.

This lab is broken up into the following steps:

1. [Uninstall the TraderLite app](#step-1-uninstall-the-traderlite-app)
2. [Create a topic in the Event Streams Management Console](#step-2-create-a-topic-in-the-event-streams-management-console)
3. [Add messaging components to the Trader Lite app](#step-3-add-messaging-components-to-the-trader-lite-app)
4. [Generate some test data with the Trader Lite app](#step-4-generate-some-test-data-with-the-traderlite-app)
5. [Verify your trades have been sent as messages to IBM MQ](#step-5-view-messages-in-ibm-mq)
6. [Start Kafka Replication](#step-6-start-kafka-replication)
7. [Verify transaction data was replicated to the Trade History database](#step-7-verify-transaction-data-was-replicated-to-the-trade-history-database)
8. [Examine the messages sent to your Event Streams topic](#step-8-examine-the-messages-sent-to-your-event-streams-topic)
9. [Summary](#summary)

> **Note:** You can click on any image in the instructions below to zoom in and see more details. When you do that just click on your browser's back button to return to the previous state.

## Step 1: Uninstall the TraderLite app

If you've completed the API Connect and/or the Salesforce integration labs then you will have the app running already. For this lab it's easier to install the app from scratch so you can use the OpenShift GUI environment (as opposed to editing the yaml file of an existing instance) to select all the options needed for this app. If the app is *NOT* installed skip to **Step 2**.

1.1 Go to the OpenShift console of your workshop cluster. Select your ***student1*** project. In the navigation on the left select **Installed Operators** in the **Operators** section and select the **TraderLite Operator**

[![](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/select-traderlite-operator.png)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/select-traderlite-operator.png)

1.2 Click on the **TraderLite app** tab

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-37410c9879b586d5402a6ef5ef107ba9bc8a02f1%2Ftraderlite-crd.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/traderlite-crd.png)

1.3 Click on the 3 periods to the right of the existing TraderLite CRD and select **Delete TraderLite** from the context menu.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-4fa256a4967ffe79211fa01056e395a32e3109a4%2Fselect-traderlite-crd2.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/select-traderlite-crd2.png)

1.4 In the navigation area on the left select **Pods** in the **Workloads** section. You should see that the *traderlite-xxxxx-yyyyy* pods are terminated.

> *Note: You can enter `traderlite` in the search by name input field to filter the pods.*

## Step 2: Create a topic in the Event Streams Management Console

2.1 Go to your Workshop Information page and click on the Event Streams component link. (**Note:** if you no longer have the Workshop Information page available see [these instructions](/cloudpakforintegration-workshop/docs/pre-work.md)).

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-1f2469ecabf2411a189a6f0f908ff38bb2442806%2Fnav-to-es.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/nav-to-es.png)

2.2 If prompted to login enter the credentials on the Workshop Information page

2.3 Click on the **Create a topic** tile

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-d9b8934fa828e696fa41483715b4662839a86e12%2Fcreate-topic.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/create-topic.png)

2.4 Name the topic `student1`. Click **Next**.

2.5 Leave the default for partitions and click **Next**.

2.6 Leave the default for message retention and click **Next**.

2.6 Leave the default for replicas and click **Create topic**.

2.7 You should see your new topic listed.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-a4a34cd3c69877f1108a5b0824430d0bfb42d25c%2Fnew-topic.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/new-topic.png)

## Step 3: Add messaging components to the Trader Lite app

In this section you will install the TraderLite app to start storing transactions as MQ messages, without setting up the KafkaConnect part that will move the transactions out of MQ, into Event Streams and then into MongoDB. This demonstrates how MQ can serve as a reliable store and forward buffer especially during temporary network disruption.

3.1 Go to the OpenShift console of your workshop cluster. Select your ***student1*** project.

3.2 Click on **Installed Operators** (in the **Operators** section) in the left navigation and then click on the **TraderLite Operator** in the list.

[![](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/select-traderlite-operator.png)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/select-traderlite-operator.png)

3.3 Click the **Create Instance** to start the installation of the TraderLite app.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-e6c86b1313675fe503cb1335c78b41fbfac9296a%2Ftraderlite-create-instance.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/traderlite-create-instance.png)

3.4 Name the instance *traderlite*

3.5 Set the following values:

* Under **Kafka Access** select the **student1** topic
* Enable **KafkaIntegration**
* Under **Mqaccess** select the **STUDENT1.QUEUE** queue

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-c64d8a423196aba254146904c7d4a80c11b8923c%2Ftraderlite-create-values.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/traderlite-create-values.png)

3.6 Scroll down and click **Create**

3.7 In the left navigation select **Pods** (in the **Workloads** section) and then wait for all the TraderLite pods to have a status of **Running** and be in the **Ready** state.

> *Note: You will know the traderlite-xxxxx pods are in a ready state when the `Ready` column shows `1/1`.*

## Step 4: Generate some test data with the TraderLite app

4.1 In your OpenShift console click on **Routes** in the left navigation under the **Networking** section and then click on the icon next to the url for the **tradr** app (the UI for TraderLite)

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-b283f8864090971da1282a9ffb7a6cc796fb8321%2Ftraderlite-run-tradr.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/traderlite-run-tradr.png)

4.2 Log in using the username `stock` and the password `trader`

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-086df65a2bd3352f86bced6d635a1a167df5f10f%2Fstock-trader-login.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/stock-trader-login.png)

4.3 Click **Add Client** fill in the form. You must use valid email and phone number formats to avoid validation errors.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YkKVvzKQw7CrP7_D-%2Fsync%2F9348b8c78a827dd7d1a62c5840bef691bac26a0b.png?generation=1618606157708879\&alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/new-client.png)

4.4 Click **Save**

4.5 Click on the **Portfolio ID** of the new client to see the details of the portfolio

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YkKVvzKQw7CrP7_D-%2Fsync%2F0e6ca6a430c0f222af3492d1c4082e7ec9c49c2e.png?generation=1618606158326529\&alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/new-portfolio.png)

4.6 Buy some shares of 2 or 3 different companies and then sell a portion of one of the shares you just bought. To buy shares, click the `Buy Stock` button, then select a company and enter a share amount. To sell shares, click the `Sell stock` button, then select the company symbol and enter the number of shares to sell.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YkKVvzKQw7CrP7_D-%2Fsync%2F99b8a89321182f91670c832757ec9128fa333d06.png?generation=1618606164386466\&alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/a-few-trades.png)

> **Note:** Your ROI will be off because we are not yet replicating the historical trade data that goes in to the calculation of the ROI.

## Step 5: View messages in IBM MQ

5.1 Go to your Workshop Information page and click on the App Connect Designer component link. (**Note:** if you no longer have the Workshop Information page available see [these instructions](/cloudpakforintegration-workshop/docs/pre-work.md)).

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-6965b816219a1f7350102d61d5085f20f76fcf93%2Fnav-to-mq.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/nav-to-mq.png)

5.2 If prompted to login enter the credentials on the Workshop Information page.

5.3 Click on the **Manage QMTRADER** tile

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YkKVvzKQw7CrP7_D-%2Fsync%2F424fd1d974303a547736491b15fffb74e88d9799.png?generation=1623713814573165\&alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/manager-qmtrader-tile.png)

5.4 Click on the **STUDENT1.QUEUE** queue.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-38e1b91fc28ce59d01fccb52d07006c46c062742%2Ftrader-queue.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/trader-queue.png)

5.5 You should see messages for the trades you just executed. The number of messages in the queue will vary based on the number of buy/sell transactions you performed in the previous steps.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-e80475ad4f196fc68bf0adb00ca5e89b34bbbc8b%2Fmq-messages.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/mq-messages.png)

5.6 Keep the browser tab with the MQ web interface open as you'll need it later in the lab.

## Step 6: Start Kafka Replication

In this section you will configure the TraderLite app to start moving the transaction data out of MQ, into Kafka and then into the MongoDB reporting database.

6.1 Go to the OpenShift console of your assigned cluster. In the navigation on the left select **Installed Operators** and select the **TraderLite Operator**

6.2 Click on the **TraderLite app** tab

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-37410c9879b586d5402a6ef5ef107ba9bc8a02f1%2Ftraderlite-crd.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/traderlite-crd.png)

6.3 Click on the 3 periods to the right of the existing TraderLite CRD and select **Edit TraderLite** from the context menu.

6.4 Scroll down to line 151 and set **Kafka Connect enabled** to **true** (leave all the other values unchanged).

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-0badba00707b9845743c9f06c293f926b61e8213%2Fkafka-connect-enabled.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/kafka-connect-enabled.png)

6.5 Click **Save**.

6.6 In the navigation on the left select **Installed Operators** and select the **Event Streams** operator.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-2d6b26a9e53dfea578c3887ae5056d8851aec454%2Fes-operator.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/es-operator.png)

6.7 Click on the **All instances** tab and wait for the *mq-source* and *mongodb-sink* connectors to be in the *Ready* state before continuing.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-70a684a17396f1ea2fa3036b2bc948280a9bc0b6%2Fkc-status.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/kc-status.png)

6.8 Go back to the browser tab with the MQ Console and verify that all the messages have been consumed by the *mq-source* connector. *(Note: You may need to reload this browser tab to see that the messages have been consumed.)*

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-71033553454becaeafae1872671138d0503fbfd0%2Fmq-empty.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/mq-empty.png)

## Step 7: Verify transaction data was replicated to the Trade History database

7.1 Go to the OpenShift console of your workshop cluster. In the navigation on the left select **Routes** in the **Networking** section.

7.2 Copy the link for the *trade-history* microservice and paste it into a new browser tab.

> **Note:** You will get a 404 (Not Found) message if you try to access this URL as is. This is because the *trade-history* microservice requires extra path information.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M0YkKVvzKQw7CrP7_D-%2Fsync%2F0bc4dedbe1bd1aa1c2ed7c0ecf2b5fa95bc14b82.png?generation=1618606165334695\&alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/trade-history.png)

7.3 Append the string `/trades/1000` to the address you pasted - you should get back some JSON with the test transactions that you ran earlier.

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-6df42ea34a86d9ca912e04d1d8714d800024fb99%2Ftrade-history2.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/trade-history2.png)

## Step 8: Examine the messages sent to your Event Streams topic

8.1 Go to your Workshop Information page and click on the Event Streams component link. (**Note:** if you no longer have the Workshop Information page available see [these instructions](/cloudpakforintegration-workshop/docs/pre-work.md)).

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-1f2469ecabf2411a189a6f0f908ff38bb2442806%2Fnav-to-es.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/nav-to-es.png)

8.2 Click on the topics icon

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-872bbb4c87100fc145d5bb5f1969bb31b179d084%2Ftopics-icon.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/topics-icon.png)

8.3 Click on topic **student1**

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-6de5f878241541b55978a59fd7966697b99d33ed%2Ftopic-name.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/topic-name.png)

8.4 Select a message to see it's details

[![](https://1545196524-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M0YkKVvzKQw7CrP7_D-%2Fuploads%2Fgit-blob-04fd451d1f7853801fcdc6b57ea3019d85c81527%2Fmessage-details.png?alt=media)](https://github.com/IBM/cloudpakforintegration-workshop/blob/master/docs/exercise-event-streaming/images/message-details.png)

## Summary

Congratulations ! You successfully completed the following key steps in this lab:

* Configured the Trader Lite app to use MQ
* Deploy Kafka Connect with IBM Event Streams
* Generated transactions in the Trader Lite app and verified that the data is being replicated via MQ and Event Streams