Issues encountered when using HMS Push Kit
While I coded my app using HMS push kit, I met the following Issues:
Crash logs :
subscribe_topic, 907122034: topic exceed
subscribe_topic, 907122035: topic send error
It leads to a few questions:
What is ERROR_TOPIC_EXCEED (2000) really about?
Does it mean that client has subscribed too many times to the specified topic?
A maximum of 2000 topics can be subscribed to for one app instance.
If you exceed this limit, you will get the "topic exceed" error you mentioned. In this case, you should either delete some topics or divide them into batches if you want to use 2000+ topics.
If the client application tries to subscribe to the same topic more than once, no error is returned, it is not ignored, and it re-subscribes normally.
Pls refer to Huawei PUSH KIT and Firebase Cloud Messaging
Related
I have a scenario where I need to call an api to the notification service which will send an email to the user.
Based on the message that appears to the topic I need to consume and call an api.
But if the consumer fails in between due to the nature of the at-least once configuration of the consumer its hard to avoid duplicate processing.
Sorry for newbie question. I Tried to find the many blogs but most of them are using newer version of Kafka but in our infra they are still using older version.
Wondering how can we achieve this behavior?
Links that I referred earlier :
If exactly-once semantics are impossible, what theoretical constraint is Kafka relaxing?
I also saw there documentation where they mention to store the processing result and offset to the same storage. But my scenario is api call lets imagine if the consumer call api and dies in between before committing offset then next poll will get that message and call api. In this way email will be sent twice.
Any help is appreciated.
I got a streaming application which is subscribed to two topics and which publishes a topic. One subscribed Topic comes from a datasource beyond my control and gives me null values, where the shouldn't be one.
So I was thinking of implementing a NUll-Check in this Streaming-Application, but I need to know the latest published message, because at the moment the streaming-app is kind of stateless.
So I would add a statestore to the streaming-app where I can query the latest message.
Is this a legit approach? Are there other approaches to this topic beyond adding a "State" to the streaming app?
If you want to handle the possible null value within the streams app and keep track of the latest published message, then yes, adding a statestore is the appropriate thing to do.
I'm still quite new to kafka, and something happened to me which I don't understand. I have 2 apps. One is using SpringKafkas listeners to consumer binary avro messages and process them. For some debugging purposes I wrote trivial second cli app, which take topic, offset brokers etc. to consume it using just plain kafka classes, without spring kafka. Both were working for a while. Recently something went wrong with the server.
Main app now does not consume any messages. During startup is sometimes(!) prints message like: "The following subscribed topics are not assigned to any members ..." When I enable debugging, I see it's spinning on "Leader for partition ... unavailable for fetching offset, wait for metadata refresh".
Ok, I could understand from that, that "something went wrong" with the cluster. What is puzzling me a lot is, that our cli app has no issues whatever to connect to the cluster and prints all messages from beginning of topic. I reverified multiple times, that both apps uses same brokers, same topic, different groupId. Neither app specifies partition id anywhere.
What could be the cause, that one app have no problemm reading from topic, while another is unable to "connect" and consume neither preexisting nor new messages from topic?
There are several applications which have to be integrated together and they have to exchange Issues. So one of them will get the issue and then do something and later on change the Status of this Issue. And the other applications which could be involved to this Issue should get the new Information. This continues until the Issue reaches the final Status Closed. The Problem is the Issue have to be mapped, because these applications do not all support the same Data Format.
I'm not sure whether to send the whole Issue always or just the new Status as an Event.
How does Kafka Support Data Transformation?
What if my Issue has an attachment?(>5MB)
Thanks for your advice
Yes it does make sense.
Kafka can do transformations through both the Kafka Streams API, and KSQL which is a streaming SQL engine built on top of Kafka Streams.
Typically Kafka is used for smaller messages; one pattern to consider for larger content is to store it in an object store (e.g. S3, or similar depending on your chosen architecture) and reference a pointer to it in your Kafka message.
I'm not sure whether to send the whole Issue always or just the new Status as an Event.
You can do this either way. If you send the whole Issue and then publish all subsequent updates to the same issue as Kafka messages that contain a common kafka message key (perhaps a unique issue ID number) then you can configure your kafka topic as a compacted topic and the brokers will automatically delete any older copies of the data to save disk space.
If you chose to only send deltas (changes) then you need to be careful to have a retention period that’s long enough so that the initial complete record will never expire while the issue is still open and publishing updates. The default retention period is 7 days.
How does Kafka Support Data Transformation?
Yes. In Kafka Connect via Single Message Transforms (SMT), or in Kafka Streams using native Streams code (in Java).
What if my Issue has an attachment?(>5MB)
You can configure kafka for large messages but if they are much larger than 5 or 10 MB then it’s usually better to follow a claim check pattern and store them external to Kafka and just publish a reference link back to the externally stored data so the consumer can retrieve the attachment out of band from Kafka.
I have connected a producer to IBM Message Hub in Bluemix. How could I get a view of topic and its depth in message hub. Is there a web console where i can see messages count?
Thanks
Raj
There isn't really the concept of a topic depth in Kafka, as Kafka is a commit log. You're not reading messages off a queue, you're reading messages from a point in the log, you can specify where in the log you start, and reading a message doesn't remove it from the log. So, the number of messages available is not affected by a read operation, but an individual consumer's place on the log moves.
Partitions per topic can be found from the Message Hub UI that lists the topics names, as well as the retention policy for the topic.
Input/Output rates for each topic are available in the Grafana tool.