After all Kafka brokers restart to upgrade offset.retention.minutes setting (to increase it to 60 days), the Kafka Streams application consuming there were stuck, and the consumer group shows rebalancing:
bin/kafka-consumer-groups.sh --bootstrap-server ${BOOTSTRAP_SERVERS} --describe --group stream-processor | sort
Warning: Consumer group 'stream-processor' is rebalancing.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
customers 0 84263 84288 25 - - -
customers 1 71731 85068 13337 - - -
customers 10 71841 84801 12960 - - -
customers 11 84273 84336 63 - - -
customers 12 84189 84297 108 - - -
customers 13 83969 84051 82 - - -
customers 14 84693 84767 74 - - -
customers 15 84472 84556 84 - - -
customers 2 84175 84239 64 - - -
customers 3 71719 71719 0 - - -
customers 4 71446 84499 13053 - - -
customers 5 84291 84361 70 - - -
customers 6 71700 71700 0 - - -
customers 7 72003 85235 13232 - - -
customers 8 84521 84587 66 - - -
customers 9 71513 71513 0 - - -
customers-intermediate 0 102774 102792 18 - - -
customers-intermediate 1 102931 103028 97 - - -
customers-intermediate 10 102883 102965 82 - - -
customers-intermediate 11 102723 102861 138 - - -
customers-intermediate 12 102797 102931 134 - - -
customers-intermediate 13 102339 102460 121 - - -
customers-intermediate 14 103227 103321 94 - - -
customers-intermediate 15 103231 103366 135 - - -
customers-intermediate 2 102637 102723 86 - - -
customers-intermediate 3 84620 103297 18677 - - -
customers-intermediate 4 102596 102687 91 - - -
customers-intermediate 5 102980 103071 91 - - -
customers-intermediate 6 84385 103058 18673 - - -
customers-intermediate 7 103559 103652 93 - - -
customers-intermediate 8 103243 103312 69 - - -
customers-intermediate 9 84211 102772 18561 - - -
events 15 11958555 15231834 3273279 - - -
events 3 1393386 16534651 15141265 - - -
events 4 1149540 15390069 14240529 - - -
visitors 15 2774874 2778873 3999 - - -
visitors 3 603242 603324 82 - - -
visitors 4 565266 565834 568
The streaming application was restarted too, and afterwards I could see some processing logs for about 20 hours and then stopped processing.
It's been like this for two days. But it is also worth mentioning that all topics you see above have 16 partitions, but some show three of them (visitors, events). However I can describe the topics and they have their partitions well distributed as usual and I can find nothing strange there.
What could have happened?
After application restart, I can see all partitions again, and the applications consuming from topic partitions. However many (most) partitions had lost their offsets. Since I changed the offset.retention.minutes setting, this should not have happened.
events 0 - 14538261 - stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
events 1 49070 13276094 13227024 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 10 - 15593746 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 11 - 15525487 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 12 - 21863908 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 13 - 15810925 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 14 - 13509742 - stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer-69da45eb-b0b2-4ad8-a831-8aecf6849892 /ip1 stream-processor-44fefb47-23b1-4597-8a6b-d7f1c364c316-StreamThread-1-consumer
events 15 11958555 15231834 3273279 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer-a8a6e989-d6c1-472f-aec5-3ae637d87b9e /ip2 stream-processor-01f7ecea-4e50-4505-a8e7-8c536058b7bc-StreamThread-1-consumer
...
Kafka 1.1.0, Kafka Streams 1.0
UPDATE Still happening on 2.1.1.
Related
I've got an AWS EKS environment that i'm just setting up and I'm getting 400 bad request.
Below is my config and I wanted to ask if anyone saw anything I should change.
I can see the requests are getting through the aws NLB as it reaches the nginx ingress controller but I can't see any decent information in the ingress controller logs. will add some below.
I'm terminating 443 at the NLB so sending http/80 to controller...
W0408 01:06:35.428413 7 controller.go:1094] Service "dating/photoapi-prod" does not have any active Endpoint.
W0408 01:06:35.428682 7 controller.go:1094] Service "dating/photoapi-test" does not have any active Endpoint.
192.168.119.209 - - [08/Apr/2022:01:50:55 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - 1d65f0090db1addb14e870d877977bfc
192.168.119.209 - - [08/Apr/2022:01:50:59 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - b853574052dfd56745839f72a6fc5ed1
192.168.90.222 - - [08/Apr/2022:01:50:59 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - c38d59e8ffcb008cf01ab4bb3ea4cd39
192.168.90.222 - - [08/Apr/2022:01:51:00 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - 3ca1bfdbb1f35c7b89d213636a12e864
192.168.119.209 - - [08/Apr/2022:01:51:05 +0000] "\x00" 400 150 "-" "-" 0 0.000 [] [] - - - - 338a2445058641d71c32e9acdf467118
As per error it says the service behind the ingress controller doesn’t have active container running behind it.
Do you have container running behind the services mentioned in error?
in my case, it fixed after remove unused listening port in LB.
I thinks it occured by health check.
USE CASE
I am reading from an already created Kafka topic on which a separate cluster is producing some keys and values. My end goal is to write to the HDFS in the JSON format, for which I have been experimenting with Kafka HDFS Sink 5.3 for sometime now.
The problem I am facing is that I am not able to ingest and write all the records from the topic into the HDFS. As of now if my topic contains hourly data of millions of records I am only able to write in terms of 100K records only.
The following are the configurations I am using for the kafka-connect-standalone.properties and my HDFS quickstart-hdfs.properties
kafka-connect-standalone.properties
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
schema.enable=false
offset.flush.interval.ms=10000
group.id=x-hdfs-consumer-group
consumer.session.timeout.ms=10000
consumer.heartbeat.interval.ms=3000
consumer.request.timeout.ms=1810000
consumer.max.poll.interval.ms=1800000
quickstart-hdfs.properties
name=hdfs-sink-mapr
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=10
topics=topic_name
hdfs.url=maprfs:///hive/x/poc_kafka_connect/
flush.size=20000
errors.tolerance=all
format.class=io.confluent.connect.hdfs.json.JsonFormat
partitioner.class=io.confluent.connect.hdfs.partitioner.TimeBasedPartitioner
timestamp.extractor=RecordField
timestamp.field=timestamp
partition.duration.ms=3600000
path.format='year'=YYYY/'month'=MM/'day'=dd/'hour'=HH
locale=en
timezone=UTC
If I don't use the
errors.tolerance=all property then I am just producing ~500 of records.
As far as the worker logs are considered I am not getting any errors in it, so I am not sure what I am missing.
Since I am relatively new to the Kafka Connector and have been trying things for sometime, I would really appreciate if someone can provide some insights on what I have been doing wrong.
FOLLOW UP QUESTION
The kafka connector also died within 2 days. That is it works fine for almost 2 days but after some time it stopped reading the data and does not produce anything. I am running it in the standalone mode, is this could be the reason? I tried describing the consumer group, it seems like all the consumers has died.
kafka/kafka_2.12-2.3.0/bin/kafka-consumer-groups.sh --bootstrap-server <server>:9092 --describe --group connect-ajay-hdfs-sink-mapr
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
connect-ajay-hdfs-sink-mapr topic_name 21 1186755480 1187487551 732071 - - -
connect-ajay-hdfs-sink-mapr topic_name 12 957021804 957736810 715006 - - -
connect-ajay-hdfs-sink-mapr topic_name 17 957031965 957746941 714976 - - -
connect-ajay-hdfs-sink-mapr topic_name 24 957496491 958212413 715922 - - -
connect-ajay-hdfs-sink-mapr topic_name 0 956991807 957716202 724395 - - -
connect-ajay-hdfs-sink-mapr topic_name 28 956940273 957668689 728416 - - -
connect-ajay-hdfs-sink-mapr topic_name 5 957182822 957899308 716486 - - -
connect-ajay-hdfs-sink-mapr topic_name 3 956974180 957695189 721009 - - -
connect-ajay-hdfs-sink-mapr topic_name 19 956878365 957590196 711831 - - -
connect-ajay-hdfs-sink-mapr topic_name 2 956968023 957685835 717812 - - -
connect-ajay-hdfs-sink-mapr topic_name 16 957010175 957726139 715964 - - -
connect-ajay-hdfs-sink-mapr topic_name 7 956900190 957624746 724556 - - -
connect-ajay-hdfs-sink-mapr topic_name 8 957020325 957739604 719279 - - -
connect-ajay-hdfs-sink-mapr topic_name 22 957064283 957788487 724204 - - -
connect-ajay-hdfs-sink-mapr topic_name 29 957026931 957744496 717565 - - -
connect-ajay-hdfs-sink-mapr topic_name 13 957400623 958129555 728932 - - -
connect-ajay-hdfs-sink-mapr topic_name 6 956892063 957618485 726422 - - -
connect-ajay-hdfs-sink-mapr topic_name 11 957117685 957841645 723960 - - -
connect-ajay-hdfs-sink-mapr topic_name 1 957003873 957734649 730776 - - -
connect-ajay-hdfs-sink-mapr topic_name 18 957007813 957734011 726198 - - -
connect-ajay-hdfs-sink-mapr topic_name 27 957047658 957766131 718473 - - -
connect-ajay-hdfs-sink-mapr topic_name 10 956975729 957689182 713453 - - -
connect-ajay-hdfs-sink-mapr topic_name 15 957046441 957775251 728810 - - -
connect-ajay-hdfs-sink-mapr topic_name 23 957011972 957727996 716024 - - -
connect-ajay-hdfs-sink-mapr topic_name 14 957151628 957881644 730016 - - -
connect-ajay-hdfs-sink-mapr topic_name 4 957118644 957845399 726755 - - -
connect-ajay-hdfs-sink-mapr topic_name 9 957109152 957838497 729345 - - -
connect-ajay-hdfs-sink-mapr topic_name 25 956923833 957646070 722237 - - -
connect-ajay-hdfs-sink-mapr topic_name 26 957026885 957742112 715227 - - -
connect-ajay-hdfs-sink-mapr topic_name 20 957010071 957733605 723534 - - -
In order to get all existing records in the topic to a sink connector, add this to the worker properties and restart connect
consumer.auto.offset.reset=earliest
If you've already started a connector, you'll need to either reset its consumer group, or change the name in the config to create a new group
I have a table with this values:
id - idProduction - historical - idVehicle - km
1 - 200 - 1 - 258 - 100
2 - 200 - 2 - 258 - 100
3 - 200 - 2 - 259 - 150
4 - 200 - 3 - 258 - 120
5 - 200 - 3 - 259 - 170
6 - 300 - 1 - 100 - 80
7 - 100 - 1 - 258 - 140
8 - 300 - 2 - 325 - 50
I need to get the values with the max historical, for all the distinct idProduction. In that case:
4 - 200 - 3 - 258 - 120
5 - 200 - 3 - 259 - 170
7 - 100 - 1 - 258 - 140
8 - 300 - 2 - 325 - 50
It my first work with postgresql, so I don't have to much idea on how to do it, does anyone can help me?
Thank you!
I think with that I can solve my problem, but I'm not sure... :
SELECT id, productions_vehicles.id_production, nu_task_number, id_historical, id_vehicle
FROM productions_vehicles
INNER JOIN
(SELECT id_production, MAX(id_historical) AS idHistorico
FROM productions_vehicles
GROUP BY id_production) topHistorico
ON productions_vehicles.id_production = topHistorico.id_production
AND productions_vehicles.id_historical = topHistorico.idHistorico;
You effectively need two requests, your solution looks good, you can also use the WITH clause to do the first request :
WITH topHistorico (
SELECT id_production, MAX(id_historical) SA idHistorico
FROM productions_vehicles
GROUP BY id_production)
SELECT id, pv.id_production, nu_task_number, id_historical, id_vehicle
FROM production_vehicles pv
INNER JOIN topHistorico th ON pv.id_production = th.id_production AND pv.id_historical = th.idHistorico
PostgreSQL: Documentation: 9.1: WITH Queries (Common Table Expressions)
Photoshop CS6/JSX. I'm changing the opacity of the selected layer by increasing or reducing it by 10. The problems I'm getting:
When reducing the value by 10 I get this secuence of reductions:
100 - 90 - 80 - 71 - 61 - 51 - 41 - 31 - 22 - 12 - 2
When increasing the results are:
0 - 10 - 20 - 31 - 41 - 51 - 61 - 71 - 82 - 92
The code is something like this:
var opc = app.activeDocument.activeLayer.opacity;
desc2.putUnitDouble(cTID('Opct'), cTID('#Prc'), opc - 10.0);
/* or
desc2.putUnitDouble(cTID('Opct'), cTID('#Prc'), opc + 10.0); */
Any idea on how to fix it in order to get only multiples of 10?
Thanks in advance
Math.round() do the trick. First, force the opacity of the layer to be round:
var opc = Math.round(app.activeDocument.activeLayer.opacity)
Now you can change the opacity by adding or substracting the desired value:
app.activeDocument.activeLayer.opacity = opc -10; //or +10
Thanks to Anna Forrest for the help.
Ok so when I run my report in iReport I only get one row as output
100 100 - BA - 7294 - 1 - 3
But when I copy the query created by the report out of the server logs and run it I get 80 rows as output
100 100 - BA - 7294 - 1 - 3
100 101 - BA - 7294 - 1 - 3
100 102 - BA - 7294 - 1 - 3
100 103 - BA - 7294 - 1 - 3
100 104 - BA - 7294 - 1 - 3
100 106 - BA - 7294 - 1 - 3
100 107 - BA - 7294 - 1 - 3
100 108 - BA - 7294 - 1 - 3
100 109 - BA - 7294 - 1 - 3
100 110 - BA - 7294 - 1 - 3
etc...
I have done these kinds of reports a hundred times and this has never happened and I can't seem to find a solution
What can be the cause of this missing data ?
Here is the query I copied from the logs (Its a simple query, nothing fancy)
select
`quallev_qualificationlevel`.`quallev_fieldofstudy` as
`quallev_qualificationlevel_quallev_fieldofstudy`,
`qde_qualificationdescription`.`qde_shortdescription` as `qde_qualificationdescription_qde_shortdescription`,
`qde_qualificationdescription`.`qde_namepurpose_cd_id` as `qde_qualificationdescription_qde_namepurpose_cd_id`,
`cn_campusname`.`cn_campusid` as `cn_campusname_cn_campusid`,
`qde_qualificationdescription`.`qde_langauge_id` as `qde_qualificationdescription_qde_langauge_id`,
`fv_factorvalue_ft`.`fv_1` as `fv_factorvalue_ft_fv_1`,
`fv_factorvalue_ft`.`fv_2` as `fv_factorvalue_ft_fv_2`,
`fv_factorvalue_ft`.`fv_3` as `fv_factorvalue_ft_fv_3`,
`fosfte_fieldofstudyftefactor`.`fosfte_factor_cd_id` as `fosfte_fieldofstudyftefactor_fosfte_factor_cd_id`,
`fv_factorvalue_ft`.`fv_4` as `fv_factorvalue_ft_fv_4`,
`fv_factorvalue_ft`.`fv_5` as `fv_factorvalue_ft_fv_5`,
`fv_factorvalue_ft`.`fv_6` as `fv_factorvalue_ft_fv_6`,
`fv_factorvalue_ft`.`fv_7` as `fv_factorvalue_ft_fv_7`,
`fv_factorvalue_ft`.`fv_8` as `fv_factorvalue_ft_fv_8`,
`fv_factorvalue_ft`.`fv_9` as `fv_factorvalue_ft_fv_9`,
`fos_fieldofstudy`.`fos_code` as `fos_fieldofstudy_fos_code`,
`fosfte_fieldofstudyftefactor`.`fosfte_ftefactorvalue` as `fosfte_fieldofstudyftefactor_fosfte_ftefactorvalue`,
`qual_qualification`.`qual_code` as `qual_qualification_qual_code`,
`oun_organisationunitname`.`oun_ou_id` as `oun_organisationunitname_oun_ou_id`,
`fos_fieldofstudy`.`fos_startdate` as `fos_fieldofstudy_fos_startdate`,
`qh_qualificationhemis`.`qh_weight` as `qh_qualificationhemis_qh_weight`
from `qo_qualificationorganisation`
inner join `quallev_qualificationlevel` on (`qo_qualificationorganisation`.`qo_quallev_id` = `quallev_qualificationlevel`.`quallev_id`)
inner join `org_organisation` on (`org_organisation`.`org_be_id` = `qo_qualificationorganisation`.`qo_org_be_id`)
inner join `oun_organisationunitname` on (`oun_organisationunitname`.`oun_ou_id` = `org_organisation`.`org_ou_id`)
inner join `cn_campusname` on (`org_organisation`.`org_campusid` = `cn_campusname`.`cn_campusid`)
inner join `fos_fieldofstudy` on (`quallev_qualificationlevel`.`quallev_fos_id` = `fos_fieldofstudy`.`fos_id`)
inner join `qual_qualification` on (`quallev_qualificationlevel`.`quallev_qual_id` = `qual_qualification`.`qual_id`)
inner join `qh_qualificationhemis` on (`qh_qualificationhemis`.`qh_qual_id` = `qual_qualification`.`qual_id`)
inner join `fosfte_fieldofstudyftefactor` on (`fosfte_fieldofstudyftefactor`.`fosfte_fos_id` = `fos_fieldofstudy`.`fos_id`)
inner join `qde_qualificationdescription` on (`qde_qualificationdescription`.`qde_qual_id` = `qual_qualification`.`qual_id`)
inner join `fv_factorvalue_ft` on (`fos_fieldofstudy`.`fos_id` = `fv_factorvalue_ft`.`fv_fos_id`)
where `qde_qualificationdescription`.`qde_langauge_id` = 3
and `fos_fieldofstudy`.`fos_startdate` <= '2013-11-08'
and `qde_qualificationdescription`.`qde_namepurpose_cd_id` = 7294
and `cn_campusname`.`cn_campusid` = '1'
and `oun_organisationunitname`.`oun_ou_id` = '11'
and `fosfte_fieldofstudyftefactor`.`fosfte_factor_cd_id` = 7699
group by `quallev_qualificationlevel`.`quallev_fieldofstudy`
Note: I am using iReport 5.0.0 on Win7 Pro 64bit for JasperReports Server 5.0.1 on Tomcat 7
Ok so I finally found the solution
The problem was jasperserver had a default row limit set of 200,000
I raised that limmit to 2,000,000 and BOOOM !!! it works now :)