Kafka

It aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka provides reliable, millisecond responses to support both customer-facing applications and connecting downstream systems with real-time data.

Miri Infotech is launching a product which will configure and publish Apache Kafka, to produce free implementations of distributed or otherwise scalable and high availability which is embedded pre-configured tool with Ubuntu and ready-to-launch AMI on Amazon EC2 that contains Hadoop, Hbase and Apache Kafka.

Before going into deep, one must learn that whatever we are using, what good it stands for?

And to understand this, we have two applications that are as follows:

1.  Building real-time streaming data pipelines that reliably get data between systems or applications
2.  Building real-time streaming applications that transform or react to the streams of data

It is one of the most popular tool among the developers around the world as it is easy to pick up and such a platform with 4APIs namely Producer, Consumer, Streams, and Connect.

• The Producer API allows an application to publish a stream of records to one or more Kafka topics.
• The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
• The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
• The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.

Without having the basic knowledge, one cannot deeply understand its nature and how it works. For that we should understand a few basic concepts about Apache Kafka:

• Kafka run as a cluster on one or more servers.
• The Kafka cluster stores streams of records in categories called topics.
• Each record consists of a key, a value, and a timestamp.

Topics and Logs

• Let’s first dive into the core abstraction Kafka provides for a stream of records—The topic.
• A topic is a category or feed name to which records are published. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.
• Kafka stores messages which come from arbitrarily many processes called "producers". The data can thereby be partitioned in different "partitions" within different "topics". Within a partition the messages are indexed and stored together with a timestamp.
• Other processes called "consumers" can query messages from partitions. Kafka runs on a cluster of one or more servers and the partitions can be distributed across cluster nodes.
• Apache Kafka efficiently processes the real-time and streaming data when implemented along with Apache Storm, Apache HBase and Apache Spark. Deployed as a cluster on multiple servers, Kafka handles its entire publish and subscribe messaging system with the help of four APIs, namely, producer API, consumer API, streams API and connector API. Its ability to deliver massive streams of message in a fault-tolerant fashion has made it replace some of the conventional messaging systems like JMS, AMQP, etc.