I am currently using RTI DDS and am writing some specifications. However, I don't want any of the topic names in my specifications to over exceed the limit for the topic name in DDS. I tried searching and could not find if any such limit exist. Does anyone happen to know if DDS limits your topic name to a certain length of if it is just to not exceed the limit on the length of a String for whatever language you are programming in?
The OMG standard for DDS (rev 1.2) does not supply an arbitrary limit to Topic name length.
A Topic is identified by its name, which must be unique in the whole Domain.
According to the RTI documentation (5.1.0 Users Guide pdf, Section 5.1.1, page 170), RTI's implementation of the Standard implements an arbitrary limit:
topic_name | Name for the new Topic, must not exceed 255 characters
This appears to be the length max for any Entity (you can name entities in its QoS, so that tools can report human-readable names for which Entity they are reporting on).
While it is true that the DDS API specification does not mention a limit for Topic names. The complementary DDS Wire Protocol specification: Real-Time Publish Subscribe (RTPS) protocol (http://www.omg.org/spec/DDSI-RTPS/2.2), does state that Topic names shall not exceed 256 characters (see Table 9.12).
So the 256 character limit on Topic name lengths imposed by the RTI DDS implementation is not arbitrary. It is precisely what is required to be interoperable with other DDS implementations.
Gerardo
Related
I'm trying to figure out the poll records mechanism for Kafka over SCS in a K8s environment.
What is the recommended way to control max.poll.records?
How can I poll the defined value?
Is it possible to define it once for all channels and then override for a specific channel?
(referring to this comment form documentation):
To avoid repetition, Spring Cloud Stream supports setting values for
all channels, in the format of
spring.cloud.stream.kafka.default.consumer.=. The
following properties are available for Kafka consumers only and must
be prefixed with
spring.cloud.stream.kafka.bindings..consumer..")
Is this path supported: spring.cloud.stream.binding.<channel name>.consumer.configuration?
Is this: spring.cloud.stream.**kafka**.binding.<channel name>.consumer.configuration?
How are conflicts being resolved? Let's say in a case where both spring.cloud.stream.binding... and spring.cloud.stream.**kafka**.binding... are set?
I've tried all mentioned configurations, but couldn't see in the log what is the actual poll.records and frankly the documentation is not entirely clear on the subject.
These are the configurations:
spring.cloud.stream.kafka.default.consumer.configuration.max.poll.records - default if nothing else specified for given channel
spring.cloud.stream.kafka.bindings..consumer.configuration.max.poll.records
I'm developing a router (events proxy) application with spring cloud stream over Kafka, in the functional paradigm. the application consumes from constant input topic, maps and filters the message and then should send it to some topic according to some input fields (only single message at a time, not multiple results).
Is the best way to do it by setting the spring.cloud.stream.sendto.destination header for the output message?
and if so, how should I set the bindings for the producer?
You can also use StreamBridge
With regard to the binding configuration. . .
If they are truly dynamic where you don't know the name of the destination (e.g., may come with message header), there is nothing you can do with regard to configuring it.
If they are semi dynamic where you do know the name(s) and it's a limited set of names, then you can configure then as any other binding.
For example, let's say you are sending to destination foo, than you can use spring.cloud.stream.bindings.foo.....
I am writing kafka consumer using spring-kafka template.
When I am instantiating consumers, Spring kafka takes in parameters like the following.
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, fetchMaxBytes);
props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, maxPartitionFetchBytes);
I read the documentation and it looks like there are lots of other parameters that can be passed as the consumer configs too. Interestingly, each of these parameter has a default value.
My question is
On what basis were these arrived?
Will there be a real-need to change these values, if so what would be those
(IMHO, this is on case by case basis. But still would like to hear
it from experts)
The delivery semantic we have is atleast once.
So, for this (atleast-once) delivery semantic, should these be left
untouched and it would still process high volume of data.
Any pointers or answers would be of great help in clarifying my doubts.
The default values are an attempt to serve most of the use cases around Kafka. However, it would be an illusion to assume that those many different configurations can be set to serve all use cases.
A good starting point to understand the default values is the plain-Kafka ConsumerConfiguration and for Spring its documentation. In the Confluence docs you will also find for each configuration the "Importance". If this importance is set to high, it is recommended to really think about it. I have given some more background on the importance here.
at-least-once
For at least once semantics you want to control the commits of the consumed messages. For this, enable.autto.commit needs to be set to false which is the default value since spring version 2.3). In addition the AckMode is per default set to BATCH which is the basis for a at least once semantics.
So, depending on your Spring version it looks like you can leave the default configuration to achieve at-least-once semantics.
I have the following code.
My goal is to group messages by a given key and a 10 second window. I would like to count the total amount accumulated for a particular key in the particular window.
I read that I need to have caching enabled and also have a cache size declared. I am also forwarding the wall clock to enforce the windowing to kick in and group the elements in two separate groups. You can see what my expectations are for the given code in the two assertions.
Unfortunately this code fails them and it does so in two ways:
it sends a result of the reduction operation each time it is executed as opposed to utilizing the caching on the store and sending a single total value
windows are not respected as can be seen by the output
Can you please explain to me how am I misunderstanding the mechanics of Kafka Streams in this case?
A customer wants to exchange data between his application and our application via ActiveMQ, so we want to create an Interface Specification Document which describes the settings and properties that both applications must use so that they can communicate. We don't know which programming language or API the customer will use; so if the specification is incomplete they might implicitly use settings that we don't expect.
So I'm wondering which settings must be the same on both sides, and which settings can be decided by each application on its own. This is what I have so far:
Must be specified in document:
connector type (openwire, stomp, ...)
connector settings (host name where broker runs, TCP port, user name, password)
message type (TextMessage, BytesMessage...)
payload details (XML with XSDs, JSON with schema, ...)
message encoding (UTF-8), for text payload
use queues, or topics, or durable topics
queue names
is any kind of request/response protocol being used
use single queue for requests and responses (with selectors being used to get correct messages), or use separate queues for requests and responses
how to transfer correlation ID used for correlating requests and responses
message expiration
Must not be specified in document:
ActiveMQ broker version (all versions are compatible, right?)
message compression (it should be transparent?)
What did I miss? Which things should be stated in such a document to ensure that two applications can communicate via ActiveMQ?
What did I miss?
You missed message headers. These can be broken into two categories:
Built-in (JMS) headers
Custom headers
Examples of the built-in headers are things such as JMSMessageID, JMSXGroupID, etc. In some cases, your interface definition will need to include details of whether and how these values will be set. For example, if messages need to be grouped, then any message producer or consumer using the definition will need to be aware of this.
Similarly, if there will any custom headers (common uses include bespoke ordering, source system identification, authorization tokens, etc.) attached to the messages need to be part of any interface definition.
In fact, I would argue that the interface definition only needs to include two things:
a schema definition for the message body, and
any headers + if they are required or optional
Everything else you have listed above is either a deployment or a management concern.
For example, whether a consumer or producer should connect to a queue or topic is a management concern, not an interface concern. The address of the queue/topic is a deployment concern, not an interface concern.