Found multiple arguments for option bootstrap-server, but you asked for only one - apache-kafka

I am running this command to list the offsets of a topic, but it keeps giving me error.
./kafka-consumer-groups.sh --bootstrap-server my.server.com:9092 --describe --group my-group
I also added the --offsets, but error remains same.
./kafka-consumer-groups.sh --bootstrap-server my.server.com:9092 --describe --offsets --group my-group
It keeps giving me error:
Found multiple arguments for option bootstrap-server, but you asked for only one
Option Description
------ -----------
--all-groups Apply to all consumer groups.
--all-topics Consider all topics assigned to a
group in the `reset-offsets` process.
--bootstrap-server <String: server to REQUIRED: The server(s) to connect to.
connect to>
--by-duration <String: duration> Reset offsets to offset by duration
from current timestamp. Format:
'PnDTnHnMnS'
--command-config <String: command Property file containing configs to be
config property file> passed to Admin Client and Consumer.
--delete Pass in groups to delete topic
partition offsets and ownership
information over the entire consumer
group. For instance --group g1 --
group g2
--delete-offsets Delete offsets of consumer group.
Supports one consumer group at the
time, and multiple topics.
--describe Describe consumer group and list
offset lag (number of messages not
yet processed) related to given
group.
--dry-run Only show results without executing
changes on Consumer Groups.
Supported operations: reset-offsets.
--execute Execute operation. Supported
operations: reset-offsets.
--export Export operation execution to a CSV
file. Supported operations: reset-
offsets.
--from-file <String: path to CSV file> Reset offsets to values defined in CSV
file.
--group <String: consumer group> The consumer group we wish to act on.
--help Print usage information.
--list List all consumer groups.
--members Describe members of the group. This
option may be used with '--describe'
and '--bootstrap-server' options
only.
Example: --bootstrap-server localhost:
9092 --describe --group group1 --
members
--offsets Describe the group and list all topic
partitions in the group along with
their offset lag. This is the
default sub-action of and may be
used with '--describe' and '--
bootstrap-server' options only.
Example: --bootstrap-server localhost:
9092 --describe --group group1 --
offsets
--reset-offsets Reset offsets of consumer group.
Supports one consumer group at the
time, and instances should be
inactive
Has 2 execution options: --dry-run
(the default) to plan which offsets
to reset, and --execute to update
the offsets. Additionally, the --
export option is used to export the
results to a CSV format.
You must choose one of the following
reset specifications: --to-datetime,
--by-period, --to-earliest, --to-
latest, --shift-by, --from-file, --
to-current.
To define the scope use --all-topics
or --topic. One scope must be
specified unless you use '--from-
file'.
--shift-by <Long: number-of-offsets> Reset offsets shifting current offset
by 'n', where 'n' can be positive or
negative.
--state [String] When specified with '--describe',
includes the state of the group.
Example: --bootstrap-server localhost:
9092 --describe --group group1 --
state
When specified with '--list', it
displays the state of all groups. It
can also be used to list groups with
specific states.
Example: --bootstrap-server localhost:
9092 --list --state stable,empty
This option may be used with '--
describe', '--list' and '--bootstrap-
server' options only.
--timeout <Long: timeout (ms)> The timeout that can be set for some
use cases. For example, it can be
used when describing the group to
specify the maximum amount of time
in milliseconds to wait before the
group stabilizes (when the group is
just created, or is going through
some changes). (default: 5000)
--to-current Reset offsets to current offset.
--to-datetime <String: datetime> Reset offsets to offset from datetime.
Format: 'YYYY-MM-DDTHH:mm:SS.sss'
--to-earliest Reset offsets to earliest offset.
--to-latest Reset offsets to latest offset.
--to-offset <Long: offset> Reset offsets to a specific offset.
--topic <String: topic> The topic whose consumer group
information should be deleted or
topic whose should be included in
the reset offset process. In `reset-
offsets` case, partitions can be
specified using this format: `topic1:
0,1,2`, where 0,1,2 are the
partition to be included in the
process. Reset-offsets also supports
multiple topic inputs.
--verbose Provide additional information, if
any, when describing the group. This
option may be used with '--
offsets'/'--members'/'--state' and
'--bootstrap-server' options only.
Example: --bootstrap-server localhost:
9092 --describe --group group1 --
members --verbose
--version Display Kafka version.
I have supplied all the necessary arguments, but why is this hapenning ?

I believe the issue is with your bootstrap-server naming convention my.server.com. Can you try with IP Address instead? Or localhost:9092.

Related

Kafka remove unused consumer groups

I have an application that uses Apache Kafka and creates a new consumer group on every startup. It takes fixed string and adds generated uuid to generate group id. (ex. my_consumer_group_123324234234, my_consumer_group_123324234235 ...). When I shut down the app old consumer groups stays unused until offsets.retention.minutes after kafka doesn't remove them.
I wonder if it is possible to remove unused consumer groups (filtered with name like 'my_consumer_group_*') by script
Yes, it should be possible using the kafka-consumer-groups.sh script included with Kafka.
You could create a script that periodically lists the existing consumer groups
kafka-consumer-groups.sh --bootstrap-server <kafka-servers-addrs> --list
Then describes each one of them
kafka-consumer-groups.sh --bootstrap-server <kafka-servers-addrs> --describe --group <consumer-group>
One option to detect if they are unused is to parse the output to see if it returns:
Consumer group '<consumer-group>' has no active members.
Note that relying on the message could be a bit brittle, since the message could change across Kafka versions, so I'd look for some other more robust approach (e.g. status code that the script returns (if any), initialize your own consumer...)
And then deletes the ones that are unused:
kafka-consumer-groups.sh --bootstrap-server <kafka-server-addrs> --delete --group <consumer-group1> --group <consumer-group2>

How to run two console consumers in the same consumer group?

When I run two instances of Kafka-console-consumers with the exact same properties (using the default one config/consumer.properties), I get same messages on both the instances.
./bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic test1
If both the instances have the same consumer group id, shouldn't Kafka send a given message to only one of the consumers? How to run them as one consumer group?
From kafka docs i found this
The default for console consumer's enable.auto.commit property when no group.id is provided is now set to false. This is to avoid polluting the consumer coordinator cache as the auto-generated group is not likely to be used by other consumers.
But here is the trick, use this command to list all consumer groups across all topics, as you said i have opened four console consumers and i want to check list of consumer groups consuming from that topic
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
Every console consumer start with different group id, this is the reason always consuming from beginning addition of this property (--from-beginning)
ups.sh --bootstrap-server localhost:9092 --list
Note: This will not show information about old Zookeeper-based consumers.
console-consumer-66835
console-consumer-38647
console-consumer-18983
console-consumer-18365
console-consumer-96734
Okay easiest way to set group.id for console consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=test1
Read up Managing Consumer Groups.
The trick is to use --consumer.config config/consumer.properties or --consumer-property group.id=test1 that would specify the group.id explicitly.
./bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic test1 \
--consumer.config config/consumer.properties

How to remove a stale consumer from a kafka broker?

kafka-consumer-groups.sh --bootstrap-server hostname:port --describe --group sub1
Consumer group 'sub1' has no active members.
kafka-consumer-groups.sh --bootstrap-server hostname:port --delete --group sub1
Option '[delete]' is only valid with '[zookeeper]'.
Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.
Also when i try to display my consumer details using zookeeper , It tells consumer "sub1" not available.
kafka-consumer-groups.sh --zookeeper hostname:port --describe --group sub1
Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).
Error: The consumer group 'sub1' does not exist.
Group information for consumers that use Kafka to manage offsets instead of Zookeeper cannot be deleted with built-in tools. If you read the whole warning when trying to execute
kafka-consumer-groups.sh --bootstrap-server hostname:port --delete --group sub1
it clearly mentions why group metadata info cannot be deleted:
Option '[delete]' is only valid with '[zookeeper]'.
Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.

How to shift back the offset of a topic within a stable Kafka consumer group?

I try to shift back the offset of topic within consumer group using following command:
bin/kafka-consumer-groups.sh --bootstrap-server loclahost:9092 --group xxx-0 --topic schedule-changed --reset-offsets --shift-by -2 --execute
(Yes, I'm using kafka version > 1.x) As a result I got a message:
Error: Assignments can only be reset if the group 'xxx-0' is
inactive, but the current state is Stable.
How can I change the state of group from 'stable' to 'inactive'?
The reset-offsets option for kafka-consumer-groups.sh first checks to see if a consumer is active in that group before attempting to shift back your offsets. 'Stable' means you have an active consumer running.
Use the describe-groups option to check on your consumer groups:
bin/kafka-consumer-groups.sh --bootstrap-server $SERVERS --group $GROUP --describe
If you see an entry under 'CONSUMER-ID/HOST/CLIENT-ID' for your topic, that means you still have a consumer running. Once you shut down the app that's keeping the consumer alive your consumer group will be inactive and you'll be able to shift your offsets at will.
I had the same problem, but in contrast to kellanburket's answer there was no consumer running any more. In that case, I had to delete the consumer group:
kafka-consumer-groups --zookeeper a.zookeeper.host:2181 --group the-group-name --delete
In newer versions of kafka, you may need to use:
kafka-consumer-groups --bootstrap-server $SERVERS --group the-group-name --delete

Kafka Connect Offsets. Get/Set?

How do I get, set, or reset the offset of a Kafka Connect connector/task/sink?
I can use the /usr/bin/kafka-consumer-groups tool which runs kafka.admin.ConsumerGroupCommand to see the offsets for all my regular Kafka consumer groups. However, Kafka Connect tasks and groups do not show up with this tool.
Similarly, I can use the zookeeper-shell to connect to Zookeeper and I can see zookeeper entries for regular Kafka consumer groups, but not for Kafka Connect sinks.
As of 0.10.0.0, Connect doesn't provide an API for managing offsets. It's something we want to improve in the future, but not there yet. The ConsumerGroupCommand would be the right tool to manage offsets for Sink connectors. Note that source connector offsets are stored in a special offsets topic for Connect (they aren't like normal Kafka offsets since they are defined by the source system, see offset.storage.topic in the worker configuration docs) and since sink Connectors uses the new consumer, they won't store their offsets in Zookeeper -- all modern clients use native Kafka-based offset storage. The ConsumerGroupCommand can work with these offsets, you just need to pass the --new-consumer option).
You can't set offsets, but you can use kafka-consumer-groups.sh tool to "scroll" the feed forward.
The consumer group of your connector has a name of connect-*CONNECTOR NAME*, but you can double check:
unset JMX_PORT; ./bin/kafka-consumer-groups.sh --bootstrap-server *KAFKA HOSTS* --list
To view current offset:
unset JMX_PORT; ./bin/kafka-consumer-groups.sh --bootstrap-server *KAFKA HOSTS* --group connect-*CONNECTOR NAME* --describe
To move the offset forward:
unset JMX_PORT; ./bin/kafka-console-consumer.sh --bootstrap-server *KAFKA HOSTS* --topic *TOPIC* --max-messages 10000 --consumer-property group.id=connect-*CONNECTOR NAME* > /dev/null
I suppose you can move the offset backward as well by deleting the consumer group first, using --delete flag.
Don't forget to pause and resume your connector via Kafka Connect REST API.
In my case(testing reading files into producer and consume in console, all in local only), I just saw this in producer output:
offset.storage.file.filename=/tmp/connect.offsets
So I wanted to open it but it is binary, with some hardly recognizable characters.
I deleted it(rename it also works), and then I can write into the same file and get the file content from consumer again. You have to restart the console producer to take effect because it attempts to read the offset file, if not there, create a new one, so that the offset is reset.
If you want to reset it without deletion, you can use:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <group-name> --reset-offsets --to-earliest --topic <topic_name>
You can check all group names by:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
and check details of each group:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <group_name> --describe
In production environment, this offset is managed by zookeeper, so more steps (and caution) is needed. You can refer to this page:
https://metabroadcast.com/blog/resetting-kafka-offsets
https://community.hortonworks.com/articles/81357/manually-resetting-offset-for-a-kafka-topic.html
Steps:
kafka-topics --list --zookeeper localhost:2181
kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic vital_signs --time -1 // -1 for largest, -2 for smallest
set /consumers/{yourConsumerGroup}/offsets/{yourFancyTopic}/{partitionId} {newOffset}