The “black technology” behind Kafka, the hottest technology in big data development

The “black technology” behind Kafka, the hottest technology in big data development

Kafka is an open source stream processing platform developed by the Apache Software Foundation. It is widely used in data buffering, asynchronous communication, log aggregation, and system decoupling. Compared with other common message systems, Kafka guarantees most of the functional features and also has outstanding performance in terms of high throughput and low latency. This article is different from other articles that introduce the use or implementation of Kafka. It just talks about what “black technology” Kafka uses to make it so outstanding in performance.

Kafka is an open source stream processing platform developed by the Apache Software Foundation. It is widely used in data buffering, asynchronous communication, log aggregation, and system decoupling. Compared with other common message systems, Kafka guarantees most of the functional features and also has outstanding performance in terms of high throughput and low latency. This article is different from other articles that introduce the use or implementation of Kafka. It just talks about what “black technology” Kafka uses to make it so outstanding in performance.

Messages are written sequentially to disk

Most of the disks are still mechanical structures (SSD is not in the scope of discussion). If the message is stored in the disk by random writing, it needs to be addressed by cylinder, head, and sector. Addressing is a “mechanical action”. “It’s also the most time consuming. In order to improve the speed of reading and writing to the hard disk, Kafka uses sequential I/O.

The “black technology” behind Kafka, the hottest technology in big data development

Figure 1 Kafka sequential IO

In the above figure, each partition is a file, and each message is appended to the partition, which belongs to the sequential write disk, so the efficiency is very high. This method has a flaw – there is no way to delete data, so Kafka will not delete data, it will keep all data, each consumer (Consumer) has an offset for each topic to represent The number of pieces of data read.

Regarding the performance of disk sequential read and write and random read and write, refer to a set of test data officially given by Kafka (Raid-5, 7200rpm):

Sequence I/O: 600MB/s

Random I/O: 100KB/s

Therefore, by only doing Sequence I/O, Kafka has greatly improved its performance.

Zero Copy

Consider a scenario where a web program reads the content of a file and transmits it to the network. The core code of the implementation is as follows:

The “black technology” behind Kafka, the hottest technology in big data development

The “black technology” behind Kafka, the hottest technology in big data development

Figure 2 Common read method

Although it is only two calls, it has gone through 4 copies, including 2 cpu copies, and multiple context switches between user mode and kernel mode, which will increase the burden on the cpu, and zero copy is to solve this low effect.

#mmap:

One way to reduce the number of copies is to call mmap() instead of the read() call:

The “black technology” behind Kafka, the hottest technology in big data development

The application calls mmap(), the data on the disk will be copied to the kernel buffer through DMA, and then the operating system will share the kernel buffer with the application, so that the content of the kernel buffer does not need to be copied to the user space . The application calls write() again, the operating system directly copies the contents of the kernel buffer to the socket buffer, and finally sends the data to the network card.

The “black technology” behind Kafka, the hottest technology in big data development

Figure 3 mmap method

Using mmap can reduce a cpu copy, but it also encounters some pitfalls. When your program maps a file, but when the file is truncated by another process (truncate), the write system call will be SIGBUS for accessing an illegal address Signal terminated. Usually, it can be solved by establishing a signal handler for the SIGBUS signal or using file leasing, which will not be repeated here.

# sendfile:

Starting with the 2.1 kernel, Linux introduced sendfile to simplify operations

The “black technology” behind Kafka, the hottest technology in big data development

The “black technology” behind Kafka, the hottest technology in big data development

Figure 4 sendfile method

The sendfile() method causes the DMA engine to copy the file content to a read buffer (DMA copy) and then the kernel copies the data to the socket buffer (cpu copy) and finally to the network card (DMA copy). Using sendfile not only reduces data copying It also reduces the number of context switches, and data transfer always occurs only in the kernel space

At this point, sendfile needs at least one cpu copy, so can this step be omitted? To eliminate all the data copying done by the kernel, we need a network interface that supports gather operations. At the same time, in kernel version 2.4, the socket buffer descriptor was also modified to accommodate the zero-copy requirement. Not only does this approach reduce multiple context switches, it also completely eliminates the cpu copy.

The “black technology” behind Kafka, the hottest technology in big data development

Figure 5 sendfile method (DMA gather)

The sendfile system call uses the DMA engine to copy the file content to the kernel buffer, and then adds the buffer descriptor with the file position and length information to the socket buffer. This step will not copy the data in the kernel to the socket buffer. , the DMA engine will copy the data in the kernel buffer to the protocol engine, avoiding the last CPU copy.

Zero-copy technology is very common. JAVA’s transferTo and transferFrom methods are Zero Copy.

The Links:   SKM400GB12T4 NL6448BC33-20