Pykafka Integration with Python Flask

August, 07-2019

BISP

Pykafka Integration with Python Flask

Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. we can discuss Apache Kafka and how Python programmers can use it for building distributed systems. In this blog we can learn what Kafka is? and how we can help to understand the need for a tool like Kafka, and then get started with it.

Pykafka Integration with Python/Flask

PyKafka is a programmer-friendly Kafka client for Python. It includes Python implementations of Kafka producers and consumers, which are optionally backed by a C extension, built on librdkafka(https://github.com/edenhill/librdkafka)

PyKafka’s primary goal is to provide a similar level of abstraction to the JVM Kafka client(https://github.com/apache/kafka/tree/0.8.2/clients/src/main/java/org/apache/kafka) using idioms familiar to Python programmers and exposing the most Pythonic API possible...

Kafka cluster

Kafka Integration with python

Business Requirement
The main objective of this project is to build a live map of London with real-time updates. We will use apache Kafka, javascript and python(flask Pykafk and JSON)

Solutions:
Note: In this document, we explained step by step integration between Python/flask and Kafka (to show live map) using pykafka.

Steps :

Download java

We can download java from below URL-
(https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)

download Java

Once java is downloaded need to install it..and set the path in the environment variable that is present in advance system settings.

Click on advance system settings.
Click on environment variables.

Install Java

Click on an environment variable, and then click on system variable and set the path of java

Install location_Java

And in user variable click on the path then edit button and set the java path.

Java_home

Once the path is set now you can check the java version by entering the cmd

>> java – version in command prompt.

2) Download Apache Kafka by clicking on the below URL.

https://www.apache.org/dyn/closer.cgi?path=/kafka/2.2.0/kafka_2.12-2.2.0.tgz

HTTP
http://apachemirror.wuchna.com/kafka/2.2.0/kafka_2.12-2.2.0.tgz

Once Kafka has downloaded, unzip it

Open the command prompt and go to the directory where you unzip Kafka folder

And hit the cmd

>>f:kafka_2.12-2.2.0inwindows>kafka-topics.bat

Hit enter, if you get something like this.

install kafka

Congrats you have successfully installed Kafka in your windows.

We need to set the Kafka path in an environment variable for proper use of Kafka server

How to start Kafka in windows:

Step1: Go to the directory where Kafka is installed.

Step2: Make a folder called data

Step3: Under data again create two folders Kafka and another one is zookeeper..we need to this folder for storing logs

Step4: We need to modify the zookeeper path n zookeeper.py file

F:kafka_2.12-2.2.0config under this directory

Kafka_ string

Step5: need to modify the Kafka server path F:kafka_2.12-2.2.0config under server properties.

kafka log

Step6: Need to start zookeeper server by entering the following cmd in cmd prompt

>>F:kafka_2.12-2.2.0inwindows>zookeeper-server-start.bat ../../config/zookeeper.properties

If you see the following screen then your zookeeper server is up to running.

zookeeper server

Step7: Need to start kafka server by entering the following cmd in cmd prompt

>>F:kafka_2.12-2.2.0inwindows>kafka-server-start.bat ../../config/server.properties.

If you see the following screen then your Kafka server is up to running.

running kafka server

Step8: Need to create a topic by run the following cmd in cmd prompt

>>F:kafka_2.12-2.2.0inwindows>kafka-topics.bat --zookeeper 0.0.0.0:2181 --topic test_topic --create -- partitions 1 -- replication-factor 1

Created topic test topic

Step9: start producer by entering the following cmd.

>>F:kafka_2.12-2.2.0inwindows>kafka-console-producer.bat --broker-list localhost:9092 --topic test_topic

>message1

>message2

>message3

Step10: Start a consumer by entering the following cmd.

>>F:kafka_2.12-2.2.0inwindows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test_topic --from-beginning

Local host

Connect to flask:
>>pip install pykafka

Here we are generating live bus location

So we need to generate a map API by hit the below URL https://account.mapbox.com/ once you successfully logged in. you see an access token in your map box dashboard copy it

busdata1.py:

import kafka client

kafka producer

index.html:
index html

Leaf.js:

var mymap = L.map('mapid').setView([51.505, -0.09], 13);

L.tileLayer('https://api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token={accessToken}',

{ attribution: 'Map data © OpenStreetMap

contributors, CC-BY-SA, Imagery © Mapbox',

maxZoom: 18,

id: 'mapbox.streets',

accessToken:

'pk.eyJ1IjoibmlzaHRoYTAzIiwiYSI6ImNqeXBuZGxtdzBhdXczbm9mMHkyMHc4cGEifQ.LMOdABNaOv2m-phFTKjtUQ'

//ENTER YOUR ACCESS TOKEN HERE }).addTo(mymap);

mapMarkers1 = [];

mapMarkers2 = [];

mapMarkers3 = [];

var source = new EventSource('/topic/TOPICNAME'); //ENTER YOUR TOPICNAME HERE

source.addEventListener('message', function(e){

console.log('Message');

obj = JSON.parse(e.data);

console.log(obj);

if(obj.busline == '00001') {

for (var i = 0; i < mapMarkers1.length; i++) {

mymap.removeLayer(mapMarkers1[i]);

}

marker1 = L.marker([obj.latitude, obj.longitude]).addTo(mymap);

mapMarkers1.push(marker1);

}

if(obj.busline == '00002') {

for (var i = 0; i < mapMarkers2.length; i++) {

mymap.removeLayer(mapMarkers2[i]);

}

marker2 = L.marker([obj.latitude, obj.longitude]).addTo(mymap);

mapMarkers2.push(marker2);

}

if(obj.busline == '00003') {

for (var i = 0; i < mapMarkers3.length; i++) {

mymap.removeLayer(mapMarkers3[i]);

}

marker3 = L.marker([obj.latitude, obj.longitude]).addTo(mymap);

mapMarkers3.push(marker3);

}

}, false);

Final Output:

google location

Here is the final output, where we can find a live map of London with real-time updates.

google map

Thanks for reading....

We have Highly-focused study material in our online academia and it's really helpful for our candidates and professionals, our professional online Data science, machine learning courses are a mixture of study materials, real-world scenario, use cases, and hands-on experience and we will be providing an industry project to our candidates or professionals, so that candidates can easily prepare himself for any specific role. We have recommended our experienced experts and real-time professionals for training and guidance out there, as well as they, are sharing his industry experience with candidates.

For more guidance please reach out to us, we can share a real-time experience.

Share: -20 -14 -7