java - How do I set the size of messages in Kafka? -

-

i'm using kafka 0.9.0.1. according sources i've found, way set sizes of messages modify following key values in server.properties.

  • message.max.bytes
  • replica.fetch.max.bytes
  • fetch.message.max.bytes

my server.properties file has these settings.

message.max.bytes=10485760 replica.fetch.max.bytes=20971520 fetch.message.max.bytes=10485760

other settings may relevant below.

socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600

however, when attempt send messages payloads of 4 6 mb in size, consumer never gets messages. producer seems send messages without exceptions being thrown. if send smaller payloads (like < 1 mb) consumer receive messages.

any idea on i'm doing wrong in terms of configuration settings?

here example code send message.

producer<string, byte[]> producer = new kafkaproducer<>(getproducerprops()); file dir = new file("/path/to/dir"); for(string s : dir.list()) {   file f = new file(dir, s);   byte[] data = files.readallbytes(f.topath());   payload payload = new payload(data); //a simple pojo store payload   string key = string.valueof(system.currenttimemillis());   byte[] val = kryoutil.tobytes(payload); //custom util use kryo bytes[]   producer.send(new producerrecord<>("test", key, val)); } producer.close();

here example code receive message.

kafkaconsumer consumer = new kafkaconsumer<>(getconsumerprops()); consumer.subscribe(arrays.aslist("test")); while(true) {   consumerrecord<string, byte[]> records = consumer.poll(100);   for(consumerrecord<string, byte[]> record : records) {     long offset = record.offset();     string key = record.key();     byte[] val = record.value();     payload payload = (payload)kryoutil.toobject(val, payload.class); //custom util use kryo deserialize object     system.out.println(       system.format("offset=%d, key=%s", offset, key));   } }

here methods populate properties files producer , consumer.

public static properties getproducerprops() {   properties props = new properties();   props.put("bootstrap.servers", "qc1:9092,qc2:9092,qc3:9092,qc4:9092");   props.put("acks", "all");   props.put("retries", 0);   props.put("batch.size", 16384);   props.put("linger.ms", 1);   props.put("buffer.memory", 33554432);   props.put("compression.type", "snappy");   props.put("max.request.size", 10485760); //need   props.put("key.serializer", "org.apache.kafka.common.serialization.stringserializer");   props.put("value.serializer", "org.apache.kafka.common.serialization.bytearrayserializer");   return props; }  public static properties getconsumerprops() {   properties props = new properties();   props.put("bootstrap.servers", "qc1:9092,qc2:9092,qc3:9092,qc4:9092");   props.put("group.id", "test");   props.put("enable.auto.commit", "true");   props.put("auto.commit.interval.ms", "1000");   props.put("session.timeout.ms", "30000");   props.put("max.partition.fetch.bytes", 10485760); //need   props.put("key.deserializer", "org.apache.kafka.common.serialization.stringdeserializer");   props.put("value.deserializer", "org.apache.kafka.common.serialization.bytearraydeserializer");   return props; }

jane, don't use fetch.message.max.bytes first of because that's property consumer , doesn't go in server.properties file , second because old version of consumer, instead use max.partition.fetch.bytes when create consumer part of properties use instantiate it.


Comments

Post a Comment

Popular posts from this blog

c# - How Configure Devart dotConnect for SQLite Code First? -

-
can configure following sample code ? when run following code, error : no entity framework provider found ado.net provider invariant name 'devart.data.sqlite' setting in machine.config <system.data> <dbproviderfactories> <add name="dotconnect sqlite" invariant="devart.data.sqlite" description="devart dotconnect sqlite" type="devart.data.sqlite.sqliteproviderfactory, devart.data.sqlite, version=4.6.287.0, culture=neutral, publickeytoken=09af7300eec23701" /> </dbproviderfactories> </system.data> after add below block code in app.config <?xml version="1.0"?> <configuration>   <configsections> <!-- more information on entity framework configuration, visit http://go.microsoft.com/fwlink/?linkid=237468 --> <section name="entityframework" type="system.data.entity.internal.configfile.entityframeworksection, entit...
Read more

java - Copying object fields -

-
what best way of copying classes when need have 2 independent variables? have simple class: public class myclass { boolean = false; string b= "not empty"; } do need make method : assign(myclass data ) { a= data.a; b= data.b; } is there automatic method copy (duplicate) objects in java? do need make method : pretty close. instead of making method, should make constructor. such constructors called copy constructor , , create them this: myclass(myclass data) { = data.a; b = data.b; } and create copy of instance, use constructor this: myclass obj1 = new myclass(); myclass obj2 = new myclass(obj1); copy constructor can tedious : using copy constructor create deep-copy can tedious, when class has mutable fields. in case, assignment create copy of reference, , not object itself. have create copy of fields (if want deep copy). can go recursive. a better way create deep copy serialize , deserialize object. why not use clone()? ...
Read more

c++ - Clear the memory after returning a vector in a function -

-
i have function processes , stores lot of data in , returns results vector of class. amount of data stored in function tremendous , want clear storage memory of function after finished job. necessary (does function automatically clear memory) or should clear memory function? update: vector<customers> process(char* const *filename, vector<int> id) { vector<customers> list_of_customers; (perform actions) return list_of_customers; } variables defined locally within function automatically released @ end of function scope. destructors objects called, should free memory allocated objects (if objects coded correctly). example of such object std::vector . anything you've allocated new must have corresponding delete release storage. try avoid doing own allocation , use raii instead, i.e. containers or smart pointers.
Read more