Command for spark-submit

[stephenllh]

I have a question regarding the command for spark-submit:

spark-submit \
    --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2 \
    stream_all_events.py

What is the meaning of the packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2?

[ankurchavda]

https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#deploying

It is a required dependency for Kafka and Spark Streaming to work.

[stephenllh]

I see! Can you explain what the colon symbol (:) means?

[ankurchavda]

I guess that is just for separation. It would mean that we want the spark-sql-kafka-0-10_2.12 package for the 3.1.2 version of Spark. So let's say you are running Spark 3.1.1. Then the last value after : will change to 3.1.1.

[stephenllh]

Thanks a lot! I actually have another unrelated question. I have been struggling to find out how EventSim and Kafka work together to create the 4 topics: listen_events, page_view_events, auth_events, status_change_events. Is that hard-coded and only works for the Million Song Dataset? I also saw that each event has different columns, so you assign different Spark schemas for them.

[ankurchavda]

Right. Eventsim is currently coded to generate events for a music streaming website. You can change that but that is written in Scala. So if you have experience with that, why not. The topics are also currently hardcoded. You can check the codebase here

[stephenllh]

I see. How did you check the columns and the data types of each Kafka event before you define the schemas?

[ankurchavda]

From the confluent control center UI. It should be available on port 9021 of your VM. You will have to forward that port to your local machine and open it in the browser.

[stephenllh]

I didn't know you can check those out from the control center. Thanks for being helpful.

[ankurchavda]

No worries. If you check the youtube video I put out a few days back, you should get an idea of how to navigate through the project. You'll find the link in the README

[stephenllh]

Hi. Do you mind answering another question of mine? I hope this is not bothering you.

When I set the environmental variable KAFKA_ADDRESS to be the external IP, the eventsim somehow couldn't create all 4 topics. The logs suggested that it struggles to find the correct IP address. But when I don't set the environmental variable (and thus setting it to localhost by default as per your code), then it works. Do you have any idea why?

[ankurchavda]

Hey, no worries, happy to answer your questions.

I am not sure why this would happen. Eventsim is running on the same VM as Kafka, hence it should pick up directly via that. Eventsim doesn't refer to the KAFKA_ADDRESS variable as such. Have you mentioned the correct IP, and do you have your 9092 port open?

[ankurchavda]

Your Kafka broker will be available at that port for other applications to connect.

[stephenllh]

My mistake. It was opened automatically. (Sorry I deleted the comment after I realized that). Now it works. But let me try again after setting the env variable.

[stephenllh]

Yeah it doesn't work if I set the KAFKA_ADDRESS to the external IP of the VM. It works when I unset the env variable.

This is what I see in the terminal:

06:42:20.083 [kafka-producer-network-thread | producer-3] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Initiating connection to node 34.124.249.114:9092 (id: 1 rack: null) using address /34.124.249.114
06:43:20.086 [main] DEBUG o.a.k.clients.producer.KafkaProducer - [Producer clientId=producer-3] Exception occurred during message send:
org.apache.kafka.common.errors.TimeoutException: Topic page_view_events not present in metadata after 60000 ms.