Does this value only affect incoming traffic? If i set this value to say 4MB on zookeeper server as well as zookeeper client and I start my client, will I still get data > 4MB when I do a request for a path /abc/asyncMultiMap/subs. If /subs has data greater than 4MB is the server going to break it up in chunks <= 4MB and send it in pieces to the client?
I am using zookeeper 3.4.6 on both client (via vertx-zookeeper) and server. I see errors on clients where it complains that packet length is greater than 4MB.
java.io.IOException: Packet len4194374 is out of range!
at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
"This is a server-side setting"
This statement is incorrect, jute.maxbuffer is evaluated on client as well by Record implementing classes that receive InputArchive. Each time a field is read and stored into an InputArchive the value is checked against jute.maxbuffer. Eg ClientCnxnSocket.readConnectResult
I investigated it in ZK 3.4
There is no chunking in the response.
This is a server-side setting. You will get this error if the entirety of the response is greater than the jute.maxbuffer setting. This response limit includes the list of children of znodes as well so even if subs does not have a lot of data but has enough children such that their length of their paths exceeds the max buffer size you will get the error.
Related
These are the info logs
"2022-08-18T07:47:14.333850Z 32263817 [Note] [MY-010914] [Server] Aborted connection 32263817 to db: 'unconnected' user: 'xyz' host: '10.x.x.x' (Got an error reading communication packets)."
[Note] [MY-010914] [Server] Got packets out of order"
[Server] Got an error reading communication packets
I don't understand why I am getting this continuously on cloud sql logs also is this the reason why my db got crashed.
There are a lot of reasons that could perhaps lead to the MySQL connection packet issues.According to this thread MySQL Error Reading Communication Packets
MySQL network communication code was written under the assumption that queries are always reasonably short,and therefore can be sent to and processed by the server in one chunk, which is called a packet in MySQL terminology. The server allocates the memory for a temporary buffer to store the packet, and it requests enough to fit it entirely. This architecture requires a precaution to avoid having the server run out of memory---a cap on the size of the packet, which this option accomplishes.
The code of interest in relation to this option is found in sql/net_serv.cc. Take a look at my_net_read(), then follow the call to my_real_read() and pay particular attention to net_realloc().
This variable also limits the length of a result of many string functions. See sql/field.cc and sql/intem_strfunc.cc for details.
According to the MySQL Documentation
You can also get these errors if you send a query to the server that is incorrect or too large.you can increase the query limit by setting the server's max_allowed_packet variable, which has a default value of 1MB. You may also need to increase the maximum packet size on the client end. More information on setting the packet size is given in Section C.5.2.10, “Packet too large”.
An INSERT or REPLACE statement that inserts a great many rows can also cause these sorts of errors. Either one of these statements sends a single request to the server irrespective of the number of rows to be inserted; thus, you can often avoid the error by reducing the number of rows sent per INSERT or REPLACE.
Additionally you may have a look at this link1 & link2
I use Kafka version 2.2.0cp2 through Rest Proxy (in the Docker container). I need the consumer to always read only one message.
I set the value max.poll.records=1 in the file /etc/kafka/consumer.properties as follows:
consumer.max.poll.records=1
OR:
max.poll.records=1
It had no effect.
Setting this value in other configs also did not give any result.
So consumer.properties is not read from the REST Proxy
Assuming consumer properties can be changed, the kafka-rest container env-var would be KAFKA_REST_CONSUMER_MAX_POLL_RECORDS, but that setting only controls the inner poll loop of the Proxy server, not the returned amount of data to the HTTP client...
There would have to be a limit flag given to the API, which does not exist - https://docs.confluent.io/current/kafka-rest/api.html#get--consumers-(string-group_name)-instances-(string-instance)-records
I don't see any consumer poll setting mentioned in the below link
https://docs.confluent.io/current/kafka-rest/config.html
But if you know the average message size you can pass max_bytes as below to control record size
GET
/consumers/testgroup/instances/my_consumer/records?timeout=3000&max_bytes=300000
HTTP/1.1
max_bytes:
The maximum number of bytes of unencoded keys and values that should
be included in the response. This provides approximate control over
the size of responses and the amount of memory required to store the
decoded response. The actual limit will be the minimum of this setting
and the server-side configuration consumer.request.max.bytes. Default
is unlimited
When a fetch request is made - the response size is limited by various Kafka parameters and they are well documented. But my question is - what is the read IO size at the core. A process must be opening the segment file and issue a read() operation and get the data into memory. The question is - what is the size of this read() - is it a fixed number or it is equal to - max.partition.fetch.bytes? If so, then if the partition has sufficient data one read IO will get enough data to feed the consumer for that partition. I tried looking into the source code, but could not figure out this size.
The reason I am doing this is - I am benchmarking my Kafka logs file system and for consumers I want to see at what read IO size the filesystem behaves better and want to see if Kafka fetches/polls show case the same pattern.
Two I/O's to your question: Disk i/o and Network i/o
Disk i/o:
Kafka leverages filesystem for the storage and cache.
If you are looking for the core disk I/O operation size, then it is typical block sizes, and most of the modern operating systems the block size defines by PageCache in general it is upto 4096bytes (ex $getconf PAGESIZE shows size on your server)
In summary: Virtual Memory pages map to Filesystem blocks, which map to Block Device sectors.
Reference code: LogSegment.scala internally uses FileRecord.java which uses FileSystem call.
Network i/o next comes to the Consumer Fetch request,
Most of the time the Consumer FetchRequest fetches from (Hot data) PageCache on particular leader partition broker. Based on your consumer Kafka params (ex,Min/MaxBytes and maxWaitMs) it fills-up the NIC request from PageCache and transmit over the wire.
Reference code: Fetcher.java (ConsumerNetworkClient)Client.send() and wait for NIC response.
This combination of pagecache and sendfile means that on a Kafka cluster where the consumers are mostly caught up and you will see no read activity on the disks whatsoever as they will be serving data entirely from cache. Because Kafka uses zero-copy data transfer (Broker → Consumer)
So, most of the performance tuning(apart from available memory for pagecache and disk i/o) you can play with consumer params like min wait time and max buffer size of packets.
Here are some of the points to consider for performance tuning on Consumption:
You can check the default ConsumerConfig here:
https://github.com/apache/kafka/blob/2.3/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java
And how consumer fetches the data from kafka topics, is well defined in Fetcher.java
final FetchSessionHandler.FetchRequestData data = entry.getValue();
final FetchRequest.Builder request = FetchRequest.Builder
.forConsumer(this.maxWaitMs, this.minBytes, data.toSend())
.isolationLevel(isolationLevel)
.setMaxBytes(this.maxBytes)
.metadata(data.metadata())
.toForget(data.toForget())
.rackId(clientRackId);
https://github.com/apache/kafka/blob/2.3/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L237
It has the default values for each property which is overridden by user input value from the config.
I am using Red Hat JBoss AMQ 7.1.0.GA and testing flow control with producerWindowSize, I used example under amq71Install\examples\features\standard\queue, here is sample jndi.proerties:
# Neither of the following parameter works
#connectionFactory.ConnectionFactory=tcp://192.168.56.11:61616?producerWindowSize=1024
java.naming.provider.url=tcp://192.168.56.11:61616?producerWindowSize=1024
I send 10 messages with total size smaller than 1024 but still can see them arrived on broker, did I miss something or I misunderstood this parameter?
Best regards
Lan
Yes, I believe you've misunderstood this parameter.
The "producerWindowSize" is the number of credits which the client will request from the broker. Each credit corresponds to a byte of data. When the client receives those credits then it will be able to send that number of bytes. In your case the client requests 1024 credits from the broker which it receives, therefore it is able to send 1024 bytes before requesting more credits.
Since you're sending 10 messages with a total size smaller than 1024 then you should expect them to arrive on the broker without any issue.
On Windows and Linux, what are the smallest possible SO_SNDBUF and SO_RCVBUF sizes possible? Is it 1 byte? Does setting these values to 1 achieve the smallest possible? Does the OS delay allocating the RAM until the space is needed?
I realize that this will cause terrible performance for transferring data. I am not trying to transfer data. I am trying to check if a server is listening to a port and if not flag a problem.
$ man 7 socket
SO_SNDBUF
Sets or gets the maximum socket send buffer in bytes. The
kernel doubles this value (to allow space for bookkeeping
overhead) when it is set using setsockopt(2), and this
doubled value is returned by getsockopt(2). The default
value is set by the /proc/sys/net/core/wmem_default file
and the maximum allowed value is set by the /proc/sys/net/core/wmem_max file. The minimum (doubled)
value for this option is 2048.
SO_RCVBUF
Sets or gets the maximum socket receive buffer in bytes. The kernel doubles this value (to allow space for bookkeeping overhead) when it is set using setsockopt(2), and this doubled value is returned by get‐sockopt(2). The default value is set by the /proc/sys/net/core/rmem_default file, and the maximum allowed value is set by the /proc/sys/net/core/rmem_max file. The minimum (doubled) value for this option is 256.
If it's just in the "thousands of ports per hour" as you mentioned in your comment, chances are high your server is already getting an order of magnitude more connections per hour than what your test runner would impose. Just do a "connect", then a "close". Anything else is a micro-optimization.
And if there's any sort of proxy, port mapper, or load balancer involved, then testing the TCP connection itself may not be sufficient. You would want to actually test the application protocol being hosted on that socket. For example, if there is a web server running on port 8000, you should not only make a TCP connection, but actually make an HTTP request and see if you get any kind of response back.