FYI - Kafka's built-in performance test tool

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

FYI - Kafka's built-in performance test tool

Ofir Manor
comes with source code.
Some basic results from the VM,
  • Write every second 50K-60K messages, each 1KB (total 50MB-60MB)
  • Read every second more than 200K messages, each 1KB.
May help in assessing whether any Kafka-related slowness is Kafka limitation or our implementation.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: <a href="tel:%2B972-54-7801286" value="+972507470820" target="_blank">+972-54-7801286 | Email: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: FYI - Kafka's built-in performance test tool

郭健

It seems an internal page so I cannot access it:

Your email address  doesn't have access to equalum.atlassian.net

 

发件人: Ofir Manor <[hidden email]>
日期: 2017526 星期五 01:12
: dev <[hidden email]>
: FYI - Kafka's built-in performance test tool

 

comes with source code.
Some basic results from the VM,

  • Write every second 50K-60K messages, each 1KB (total 50MB-60MB)
  • Read every second more than 200K messages, each 1KB.

May help in assessing whether any Kafka-related slowness is Kafka limitation or our implementation.

 

   <a href="https://equalum.atlassian.net/wiki/display/EQ/Kafka&#43;performance&#43;test&#43;tool">https://equalum.atlassian.net/wiki/display/EQ/Kafka+performance+test+tool

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: <a href="tel:%2B972-54-7801286" target="_blank">+972-54-7801286 | Email: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: FYI - Kafka's built-in performance test tool

Ofir Manor
Hi,
sorry for that, I sent my original email to this list by mistake (gmail autocomplete fooled me), the page I linked isn't open to buplic.

Anyway, since you are interested, here is the sample commands and output from a VirtualBox image on my laptop.

1. Create a topic
kafka-topics.sh --create   --zookeeper localhost:2181 --partitions 2 --replication-factor 1 --topic perf_test_2_partitions

2. Write two millions messages, each 1KB, at a target rate of 50MB/s:
$  kafka-producer-perf-test.sh --topic perf_test_2_partitions --num-records 2000000 --record-size 1024 --throughput 50000 --producer-props bootstrap.servers=localhost:9092
249934 records sent, 49986.8 records/sec (48.82 MB/sec), 85.7 ms avg latency, 415.0 max latency.
250142 records sent, 50028.4 records/sec (48.86 MB/sec), 2.2 ms avg latency, 33.0 max latency.
250029 records sent, 50005.8 records/sec (48.83 MB/sec), 4.6 ms avg latency, 74.0 max latency.
249997 records sent, 49999.4 records/sec (48.83 MB/sec), 16.2 ms avg latency, 98.0 max latency.
249902 records sent, 49980.4 records/sec (48.81 MB/sec), 5.7 ms avg latency, 95.0 max latency.
249568 records sent, 49893.6 records/sec (48.72 MB/sec), 1.8 ms avg latency, 21.0 max latency.
248190 records sent, 49638.0 records/sec (48.47 MB/sec), 10.9 ms avg latency, 158.0 max latency.
2000000 records sent, 49993.750781 records/sec (48.82 MB/sec), 17.06 ms avg latency, 415.00 ms max latency, 2 ms 50th, 81 ms 95th, 343 ms 99th, 407 ms 99.9th

3. Read ten million messages (each 1KB), at a rate of 231MB/s (it was physical I/O - saw with iostat command that the SSD did work at that rate):
(I ran the previous command several times, so the topic did have 10M messages)
$ kafka-consumer-perf-test.sh --topic perf_test_2_partitions --messages 10000000 --broker-list localhost:9092 --date-format HH:mm:ss:SSS
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
19:32:53:408, 19:33:35:523, 9765.6250, 231.8800, 10000000, 237445.0908

If you / someone has followup questions, might be best to do it off Spark dev list...

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: <a href="tel:%2B972-54-7801286" value="+972507470820" target="_blank">+972-54-7801286 | Email: [hidden email]


On Thu, Jun 1, 2017 at 7:25 AM, 郭健 <[hidden email]> wrote:

It seems an internal page so I cannot access it:

Your email address  doesn't have access to equalum.atlassian.net

 

发件人: Ofir Manor <[hidden email]>
日期: 2017526 星期五 01:12
: dev <[hidden email]>
: FYI - Kafka's built-in performance test tool

 

comes with source code.
Some basic results from the VM,

  • Write every second 50K-60K messages, each 1KB (total 50MB-60MB)
  • Read every second more than 200K messages, each 1KB.

May help in assessing whether any Kafka-related slowness is Kafka limitation or our implementation.

 

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: <a href="tel:%2B972-54-7801286" target="_blank">+972-54-7801286 | Email: [hidden email]


Loading...