Apache Kafka is an open-source distributed event streaming platform that helps us create and process real-time data streams. You can imagine systems where data is generated every second and needs to be processed, such as traffic management systems, weather management systems, data will be generated from many different places and need to be collected for processing. Using Apache Kafka with high scalability can help us solve such problems.

Apache Kafka is an enterprise messaging system, operating with Pub-Sub messaging architecture. We have a Message Publisher, Message Consumer, and Message Broker. Message Publisher is the place to produce messages and emits messages to Message Broker, Message Consumer is the place to receive, it will subscribe to Message Broker and receive messages from Message Publisher through Message Broker.

Apache Kafka Server will act as a Message Broker.

Apache Kafka provides us with libraries to work with Publishers and Subscribers. In addition, Apache Kafka also has other components such as Kafka Connect, Kafka Streams.

Apache Kafka was built from the ground up by LinkedIn, a social network for jobs, and has been open-source since 2011.