SPTrans API Stream to Kafka

The goal of this project is to experiment with SPTrans API, stream data to kafka and deploy a client to consume it and render the information in a web page.

Kafka

1. Download and start Confluent plataform using docker.

• Clone the confluentinv/cp-all-in-one github repository;
• Navigate to the cp-all-in-one-community directory;
• Start Confluent Platform specifying the '-d' option to run in detached mode:

docker-compose up -d

• Verify that the services are up and running:

docker-compose ps

Pykafka Library

Install pykafka library. Below there are two resources:

• pykafka official documentation
• article "PyKafka: Fast, Pythonic Kafka, at Last!"
• article "3 Libraries You Should Know to Master Apache Kafka in Python"

Start a python console and give it a try. Make sure the docker containers are running.

Also, you can run in isolation the ./project/repository/kafka_repository.py and observe the output in the terminal.