|
| 1 | +Sunday, Oct/04/2020 |
| 2 | +========================================================== |
| 3 | +Custom visualiation of the IBM Streams application results |
| 4 | +========================================================== |
| 5 | +Quick brief |
| 6 | +----------- |
| 7 | +This is somewhat an involved IBM Streams application that has a few |
| 8 | +modular blocks of logic/code. That includes ingesting base64 encoded |
| 9 | +data files, identifying out of sequence events, performing Geofence, |
| 10 | +applying custom written event time based windowing for the analytics and |
| 11 | +incorporating a new way to visualize the results in a full-fledged |
| 12 | +web browser based custom dashboard application. So, learning about the |
| 13 | +full application logic will take a considerable amount of time. However, |
| 14 | +if the goal is simply to learn about how the custom visualization is done |
| 15 | +in the context of this application, then that is relatively simpler. |
| 16 | +One can understand the core visualiation idea and its implementation |
| 17 | +with a minimal effort. That learning can also be applied in other |
| 18 | +IBM Streams applications by focusing only on the final section inside the |
| 19 | +main composite of this application. |
| 20 | + |
| 21 | +Core idea |
| 22 | +--------- |
| 23 | +IBM Streams applications are always developed with a common pattern in mind. |
| 24 | +That is to ingest different kinds of data from disparate data sources such as |
| 25 | +files, databases, queues, network etc. and then perform analytics on the |
| 26 | +ingested data (filtering, transformation, enrichment, correlation, |
| 27 | +pattern detection, scoring, alerting, notification etc.). As part of the |
| 28 | +data analytics, IBM Streams applications continuously generate results. |
| 29 | +Such results can be stored in files, databases, queues etc. in order to be used |
| 30 | +later. At the same time, results can also be visualized in real time in an external |
| 31 | +web browser or a thick client application with custom written dashboard, charts, |
| 32 | +plots, tables etc. The streamsx.websocket toolkit includes an example named |
| 33 | +CustomVisualization that shows how to visualize the results of your Streams application |
| 34 | +in a Web browser application that can fetch the results periodically to present it |
| 35 | +to the user in many possible ways. |
| 36 | + |
| 37 | +High-level concept |
| 38 | +------------------ |
| 39 | +In your Streams application, all it requires is an addition of two operators. |
| 40 | +One of them is a Custom operator that can receive various anlytics results |
| 41 | +produced inside the Streams application and keep stroing them in its in-memory |
| 42 | +state variables (list, map etc.). Then, the WebSocketSource operator available |
| 43 | +in the streamsx.websocket toolkit can be used to provide a bi-directional |
| 44 | +data exchange between the Streams application and the external visualization |
| 45 | +application. That operator can receive HTTP GET and POST requests from the |
| 46 | +external visualization application and then feed it to the result collector |
| 47 | +Custom operator that we discussed above. Based on the request received from the |
| 48 | +external application, the result collector can serve the requested information to |
| 49 | +the visualization application via the response path provided by the WebSocketSource |
| 50 | +operator. In fact, in addition to the application results, it can also serve |
| 51 | +html, css, js and image (png, ico, jpeg) files that are typically needed by a |
| 52 | +full-fledged web browser based application. This technical approach lends itself to |
| 53 | +develop a highly responsive and a highly interactive visualization application. |
| 54 | +A high-level conceptual diagram that explains this idea can be found in a file available |
| 55 | +in this example directory (IBM-Streams-Custom-Visualization-Concept.pdf). |
| 56 | + |
| 57 | +This example in a nutshell |
| 58 | +-------------------------- |
| 59 | +The CustomVisualiation example here represents a real-life use case where the |
| 60 | +Taxi location and Taxi booking details are ingested into IBM Streams. This application |
| 61 | +uses the Geospatial toolkit to identify the exact location (city and district) of the |
| 62 | +taxis and from where the bookings are made. It does detailed analytics to compute |
| 63 | +real-time metrics such as taxi counts (hired, available etc.) taxi booking details (new, |
| 64 | +served, unserved, canceled etc.) and supply/demand alerts. The metrics are computed in |
| 65 | +an event time based window and the results are continuously generated. When this application |
| 66 | +is running, users can access a URL via a browser to visualize the results in a few possible |
| 67 | +ways. In this Streams application's etc/webapp directory, developers can find the necessary |
| 68 | +files that make up the browser-based visualization application. Developers will be required |
| 69 | +to have some level of understanding about html, css and Javascript to make sense out of |
| 70 | +those files. Specific visualization techniques used in the Javascript code employ the |
| 71 | +popular d3js library. In addition to fetching the results from the Streams application for |
| 72 | +real-time dashboarding, the web application also shows how to get the user input from the |
| 73 | +browser and post it to the Streams application for adjusting what kind of results are needed |
| 74 | +for the visualization as well as certain event time based window configuration such as the |
| 75 | +metrics montoring period. To understand this example's inner workings entirely, it will take |
| 76 | +a considerable amount of time. That is because of the very involved application logic |
| 77 | +done for the analytics. But, just to focus on learning how to prepare any Streams |
| 78 | +application for custom visualization, you only have to understand the invocation of a few |
| 79 | +operators used inside the main composite of this application (TaxiMetrics.spl). Those |
| 80 | +operators can be found in a code section that has this comment string: |
| 81 | +START OF VISUALIZING THE RESULTS IN A WEB APP |
| 82 | + |
| 83 | +In addition to reviewing the logic in those two operators, you can start from the |
| 84 | +etc/webapp/html/main.html file and see how the web application logic is built with the |
| 85 | +help of the supporting css and js (JavaScript) files. |
| 86 | + |
| 87 | +If your goal is to only understand the technical ideas and the code logic used for |
| 88 | +visualizing the results generated by this example without really running and seeing how |
| 89 | +this application works end-to-end on IBM Streams and in the web browser, then you can skip |
| 90 | +the sections below. If you want to run this example and see the visualization of the results, |
| 91 | +then you can use the following sections. |
| 92 | + |
| 93 | +External dependencies for this example |
| 94 | +-------------------------------------- |
| 95 | +This example has an external toolkit dependency and an external test data dependency. |
| 96 | + |
| 97 | +a) To build and run this example, an enhanced version of the IBM Streams Geospatial |
| 98 | +toolkit is needed (v3.4.3.1). It is available in the following URL for download from |
| 99 | +a browser or via the wget command. You have to extract it in your Linux machine where |
| 100 | +you plan to compile this example. After extraction, it will require 17MB of disk space. |
| 101 | + |
| 102 | +https://github.com/IBMStreams/streamsx.websocket/releases/download/v1.0.9/com.ibm.streams.geospatial-v3.4.3.1.tar.gz |
| 103 | + |
| 104 | +b) To run this example, we need realistic test data files that contain taxi location and |
| 105 | +taxi booking details for a period of 2.5 days (July/04/2020 to July/06/2020). |
| 106 | +These data files are available inside a compressed file (393MB in size) in the following |
| 107 | +URL for download from a browser or via the wget command. You have to extract it in your |
| 108 | +Linux machine from where you plan to run this example. After extraction, it will require |
| 109 | +1.8GB of disk space. |
| 110 | + |
| 111 | +https://github.com/IBMStreams/streamsx.websocket/releases/download/v1.0.9/Custom-Viz-Test-Data.tar.gz |
| 112 | + |
| 113 | +Building this example |
| 114 | +--------------------- |
| 115 | +a) From the command line of a terminal window using the Makefile: |
| 116 | +You can either directly build it inside the |
| 117 | +streamsx.websocket/samples/CustomVisualization directory or take a copy of that |
| 118 | +entire directory elsewhere and build it from there. You have to ensure that you have |
| 119 | +your TLS/SSL certificate copied as ws-server.pem in the etc sub-directory of this example. |
| 120 | +After that, You have to edit the Makefile provided in this example to point to the correct |
| 121 | +directories for the required toolkit dependencies, save and then run "make clean" and "make". |
| 122 | + |
| 123 | +(OR) |
| 124 | + |
| 125 | +b) Importing into your IBM Streams Studio: |
| 126 | +[Please follow these steps to import and build it correctly.] |
| 127 | + |
| 128 | +i) At first, you have to make sure that you have the required toolkit dependencies |
| 129 | +[websocket, json and geospatial] in your Streams Studio. |
| 130 | + |
| 131 | +ii) In a terminal window, copy recursively the entire |
| 132 | +streamsx.websocket/samples/CustomVisualization directory into your |
| 133 | +Streams Studio workspace directory i.e. cp -ufrp <source-dir> <destination-dir> |
| 134 | + |
| 135 | +iii) You have to ensure that you have your TLS/SSL certificate copied as |
| 136 | +ws-server.pem in the etc sub-directory of this example. |
| 137 | + |
| 138 | +iv) Before importing it, you should rename the Makefile present in this example's |
| 139 | +top-level directory in your Streams Studio workspace to Makefile.org |
| 140 | +i.e. mv Makefile Makefile.org. This step will avoid certain build errors while |
| 141 | +importing it within Streams Studio. |
| 142 | + |
| 143 | +v) From your IBM Streams Studio, you can select File->Import and select |
| 144 | +IBM Streams Studio-->SPL Project and click Next. In the resulting dialog, |
| 145 | +click Browse and select your workspace directory and click OK. In the list of |
| 146 | +SPL projects shown, select CustomVisualization and click Finish. If it prompts |
| 147 | +you to overwrite existing files, select "Yes To All". |
| 148 | + |
| 149 | +vi) Right after the import, it may try an external builder by default and give a |
| 150 | +few errors which you can ignore and clear the console. |
| 151 | + |
| 152 | +vii) In the Project Explorer view, expand the top-level CustomVisualization project and then |
| 153 | +expand the namespace (com.ibm.streamsx.....) under that. Then, right click on the TaxiMetrics |
| 154 | +main composite and select New-->Build Configuration and then enter --c++std=c++11 in the |
| 155 | +Other-->Additional SPL compiler options field and click OK. At this point, it should correctly |
| 156 | +start an internal build as opposed to trying to use the external Makefile. It will take around |
| 157 | +10 minutes for the build to complete. Please ensure that the build is successful. |
| 158 | + |
| 159 | +Running this example |
| 160 | +-------------------- |
| 161 | +You can launch it in Distributed mode either from your Streams Studio or from a |
| 162 | +command line via streamtool. You have to ensure that the following submission time |
| 163 | +parameters are set to the correct values. |
| 164 | + |
| 165 | +TaxiBookingDirectory (e-g: /home/usr1/Custom-Viz-Test-Data/taxiBooking) |
| 166 | +TaxiLocationDirectory (e-g: /home/usr1/Custom-Viz-Test-Data/taxiLocation) |
| 167 | + |
| 168 | +Once this appliation is running healthy, you have to test it quickly before the limited |
| 169 | +amount of test data runs out. Open a broswer tab and give this URL. If the page fails to load, |
| 170 | +you may want to ensure the machine name and port number are correct. It could also fail to |
| 171 | +load until the configured amount of initDelay is completed. You can verify them and |
| 172 | +try to load the page after 15 seconds of initDelay. |
| 173 | + |
| 174 | +http://<YourStreamsMachineNameOrIpAddress>:8081/viz/main.html |
| 175 | + |
| 176 | +You can click the "Select Community" tab and then adjust the monitoring interval to |
| 177 | +5 minutes, select a district (e-g: Bur Dubai), select "Start Monitoring" and click |
| 178 | +on the Confirm button. Now, you can go to the other tabs to visualize different kinds |
| 179 | +of results generated by the Streams application. |
| 180 | +========================================================== |
0 commit comments