Running Artemis since last week with average processing 400-500k
message per day. But now journal directory size is around 110G and Artemis is not able to start. I have changed the log level to trace and found following messages. The below file was created on my machine before a week ago. And till now in journal directory total 17000 file got created
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27588.amq can't be reclaimed because activemq-data-27592.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27593.amq can't be reclaimed because activemq-data-27588.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27577.amq can't be reclaimed because activemq-data-27593.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27578.amq can't be reclaimed because activemq-data-27577.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27579.amq can't be reclaimed because activemq-data-27578.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27580.amq can't be reclaimed because activemq-data-27579.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27603.amq can't be reclaimed because activemq-data-27580.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27605.amq can't be reclaimed because activemq-data-27603.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27606.amq can't be reclaimed because activemq-data-27605.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27607.amq can't be reclaimed because activemq-data-27606.amq has negative values
2019-07-05 03:14:34,872 INFO [org.apache.activemq.artemis.core.journal.impl.Reclaimer] activemq-data-27608.amq can't be reclaimed because activemq-data-27607.amq has negative values
Related
We have a 2 node kafka cluster with both auto.create.topics.enable and delete.topic.enable set to true. Our app reads from a common request topic and responds to a response topic provided by the client in the request payload.
auto.create.topics is set to true as our client has an auto-scale environment wherein a new worker will read from a new response topic. Due to some implementation issues on the client side, there are a lot of topics created which have never been used (end offset is 0) and we are attempting to clean that up.
The problem is that upon deleting the topic, it is being recreated almost immediately. These topics don't have any consumer (as the worker listening to it is already dead).
I have tried the following
Kafka CLI delete command
kafka-topics.sh --zookeeper localhost:2181 --topic <topic-name> --delete
Create a zookeeper node under
/admin/delete_topics/<topic-name>
Both don't seem to work. In the logs, I see that a request for delete was received and the corresponding logs/indexes were deleted. But within a few seconds/minutes, the topic is auto-created. Logs for reference -
INFO [Partition <topic-name>-0 broker=0] No checkpointed highwatermark is found for partition <topic-name>-0 (kafka.cluster.Partition)
INFO Replica loaded for partition <topic-name>-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO Replica loaded for partition <topic-name>-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO [Partition <topic-name>-0 broker=0] <topic-name>-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-4.7a79dfc720624d228d5ee90c8d4c325e-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-4.7a79dfc720624d228d5ee90c8d4c325e-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-4.7a79dfc720624d228d5ee90c8d4c325e-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-4 in /home/ec2-user/data/kafka/0/<topic-name>-4.7a79dfc720624d228d5ee90c8d4c325e-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-2.d32a905f9ace459cb62a530b2c605347-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-2.d32a905f9ace459cb62a530b2c605347-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-2.d32a905f9ace459cb62a530b2c605347-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-2 in /home/ec2-user/data/kafka/0/<topic-name>-2.d32a905f9ace459cb62a530b2c605347-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-3.0670e8aefae5481682d53afcc09bab6a-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-3.0670e8aefae5481682d53afcc09bab6a-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-3.0670e8aefae5481682d53afcc09bab6a-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-3 in /home/ec2-user/data/kafka/0/<topic-name>-3.0670e8aefae5481682d53afcc09bab6a-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-7.ac76d42a39094955abfb9d37951f4fae-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-7.ac76d42a39094955abfb9d37951f4fae-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-7.ac76d42a39094955abfb9d37951f4fae-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-7 in /home/ec2-user/data/kafka/0/<topic-name>-7.ac76d42a39094955abfb9d37951f4fae-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-1.4872c74d579f4553a881114749e08141-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-1.4872c74d579f4553a881114749e08141-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-1.4872c74d579f4553a881114749e08141-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-1 in /home/ec2-user/data/kafka/0/<topic-name>-1.4872c74d579f4553a881114749e08141-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-0.489b7241226341f0a7ffa3d1b9a70e35-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-0.489b7241226341f0a7ffa3d1b9a70e35-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-0.489b7241226341f0a7ffa3d1b9a70e35-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-0 in /home/ec2-user/data/kafka/0/<topic-name>-0.489b7241226341f0a7ffa3d1b9a70e35-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-5.6d659cd119304e1f9a4077265364d05b-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-5.6d659cd119304e1f9a4077265364d05b-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-5.6d659cd119304e1f9a4077265364d05b-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-5 in /home/ec2-user/data/kafka/0/<topic-name>-5.6d659cd119304e1f9a4077265364d05b-delete. (kafka.log.LogManager)
INFO Deleted log /home/ec2-user/data/kafka/0/<topic-name>-6.652d1aec02014a3aa59bd3e14635bd3b-delete/00000000000000000000.log. (kafka.log.LogSegment)
INFO Deleted offset index /home/ec2-user/data/kafka/0/<topic-name>-6.652d1aec02014a3aa59bd3e14635bd3b-delete/00000000000000000000.index. (kafka.log.LogSegment)
INFO Deleted time index /home/ec2-user/data/kafka/0/<topic-name>-6.652d1aec02014a3aa59bd3e14635bd3b-delete/00000000000000000000.timeindex. (kafka.log.LogSegment)
INFO Deleted log for partition <topic-name>-6 in /home/ec2-user/data/kafka/0/<topic-name>-6.652d1aec02014a3aa59bd3e14635bd3b-delete. (kafka.log.LogManager)
INFO [GroupCoordinator 0]: Removed 0 offsets associated with deleted partitions: <topic-name>-0. (kafka.coordinator.group.GroupCoordinator)
INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaFetcherManager)
INFO [ReplicaAlterLogDirsManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaAlterLogDirsManager)
INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaFetcherManager)
INFO [ReplicaAlterLogDirsManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaAlterLogDirsManager)
INFO Log for partition <topic-name>-0 is renamed to /home/ec2-user/data/kafka/0/<topic-name>-0.185c7eda12b749a2999cd39b3f90c738-delete and is scheduled for deletion (kafka.log.LogManager)
INFO Creating topic <topic-name> with configuration {} and initial partition assignment Map(0 -> ArrayBuffer(0, 1)) (kafka.zk.AdminZkClient)
INFO [KafkaApi-0] Auto creation of topic <topic-name> with 1 partitions and replication factor 2 is successful (kafka.server.KafkaApis)
INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(<topic-name>-0) (kafka.server.ReplicaFetcherManager)
INFO [Log partition=<topic-name>-0, dir=/home/ec2-user/data/kafka/0] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
INFO [Log partition=<topic-name>-0, dir=/home/ec2-user/data/kafka/0] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 0 ms (kafka.log.Log)
INFO Created log for partition <topic-name>-0 in /home/ec2-user/data/kafka/0 with properties {compression.type -> producer, message.format.version -> 2.2-IV1, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 86400000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
INFO [Partition <topic-name>-0 broker=0] No checkpointed highwatermark is found for partition <topic-name>-0 (kafka.cluster.Partition)
INFO Replica loaded for partition <topic-name>-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO Replica loaded for partition <topic-name>-0 with initial high watermark 0 (kafka.cluster.Replica)
INFO [Partition <topic-name>-0 broker=0] <topic-name>-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)
Does anyone know the reason behind the topic being re-created when no consumers are listening and no producers are producing to the topic? Could replication be behind it (some race condition perhaps)? We are using Kafka 2.2.
Deleting the log directory for that topic directly seems to work, however, this is cumbersome when there are thousands of topics created. We want to have a cleanup script that does this periodically as due to the auto-scale nature of the client environment, there may be frequent orphaned response topics.
Update
I tried Giorgos' suggestion by disabling auto.create.topics.enable and then deleting the topic. This time the topic did get deleted, but none of my applications through any errors (which leads to me the conclusion that there are no consumers/producers to the said topic).
Further, when auto.create.topics.enable is enabled and the topic is created with a replication-factor=1, the topic does not get re-created after deletion. This leads me to believe that perhaps replication is the culprit. Could this be a bug in Kafka?
Jumped the gun here; turns out something is listening/re-creating these topics from the customer environment.
Even if you've mentioned that no consumer/producer is consuming/producing from the topic, it sounds that this is the case. Maybe you have any connectors running on Kafka Connect that replicate data from/to Kafka?
If you still can't find what is causing the re-creation of the deleted Kafka topics, I would suggest setting auto.create.topics.enable to false (temporarily) so that topics cannot be automatically re-created. Then the process that is causing topic re-creation will normally fail and this is going to be reported in your logs.
I am new to scala/spark (about a week now)
The following code is being run on my 8 core laptop, 64bit, Win10
The dataframe has 1700 rows.
ONE select takes over ten seconds.
Watching the console shows the main hang is at this point:
17/09/02 12:23:46 INFO FileSourceStrategy: Pruning directories with:
The Code
{
val major:String =name.substring(0,name.indexOf("_SCORE"))+"_idx1"
println(major)
val majors = dfMergedDroppedDeleted
.select(col(major))
.collect().toSeq
println(s"got majors ${majors.size}")
}
This should take milliseconds (based on experience with hibernate,r,mysql etc)
I am assuming there is something wrong with my configuration of spark?
Any suggestions would be most welcome.
The full console output up to the hang is as follows:
1637_1636_1716_idx1
1637_1636_1716_idx2
17/09/02 12:23:08 INFO ContextCleaner: Cleaned accumulator 765
17/09/02 12:23:08 INFO ContextCleaner: Cleaned accumulator 763
17/09/02 12:23:08 INFO BlockManagerInfo: Removed broadcast_51_piece0 on 192.168.0.13:62246 in memory (size: 113.7 KB, free: 901.6 MB)
17/09/02 12:23:08 INFO ContextCleaner: Cleaned accumulator 761
17/09/02 12:23:08 INFO ContextCleaner: Cleaned accumulator 764
17/09/02 12:23:08 INFO ContextCleaner: Cleaned accumulator 762
17/09/02 12:23:08 INFO ContextCleaner: Cleaned accumulator 766
17/09/02 12:23:08 INFO BlockManagerInfo: Removed broadcast_50_piece0 on 192.168.0.13:62246 in memory (size: 20.7 KB, free: 901.6 MB)
17/09/02 12:23:08 INFO FileSourceStrategy: Pruning directories with:
Putting the dataframe in cache makes a big difference.
val dfMergedDroppedDeletedCached:DataFrame=dfMergedDroppedDeleted.cache()
However, the caching process itself is slow, so this only pays off if you are performing multiple operations
UPDATE
Credit Ramesh Maharjan to who wrote in a comment:
the time consuming part is not select. select is distributed in nature and would be executed in every local data in executors. The time consuming part is the collect. Collect function collects all the data in the driver node. And that takes a lot of time. Thats why collect is always recommended not to be used and if necessary to use it the minimum.
I have changed the query to be as follows:
val majorstr:String = dfMergedDroppedDeletedCached.filter(dfMergedDroppedDeletedCached(major).isNotNull)
.select(col(major))
.limit(1)
.first().getString(0)
Not exactly Oracle speeds but much faster than using collect
I am trying to load file to my spark and here is my code.
lines = sc.textFile('file:///Users/zhangqing198573/data/weblog_lab.csv')
and the error shows
17/03/04 22:16:32 INFO MemoryStore: Block broadcast_0 stored as values
in memory (estimated size 127.4 KB, free 127.4 KB) 17/03/04 22:16:32
INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory
(estimated size 13.9 KB, free 141.3 KB) 17/03/04 22:16:32 INFO
BlockManagerInfo: Added broadcast_0_piece0 in memory on
localhost:53383 (size: 13.9 KB, free: 511.1 MB) 17/03/04 22:16:32 INFO
SparkContext: Created broadcast 0 from textFile at
NativeMethodAccessorImpl.java:-2.
I dont know whats wrong
The first stage of my spark job is quite simple.
It reads from a big number of files (around 30,000 files and 100GB in total) -> RDD[String]
does a map (to parse each line) -> RDD[Map[String,Any]]
filters -> RDD[Map[String,Any]]
coalesces (.coalesce(100, true))
When running it, I observe a quite peculiar behavior. The number of executors grows until the given limit I specified in spark.dynamicAllocation.maxExecutors (typically 100 or 200 in my application). Then it starts decreasing quickly (at approx. 14000/33428 tasks) and only a few executors remain. They are killed by the drive. When this task is done. The number of executors increases back to its maximum value.
Below is a screenshot of the number of executors at its lowest.
An here is a screenshot of the task summary.
I guess that these executors are killed because they are idle. But, in this case, I do not understand why would they become idle. There remains a lot of task to do in the stage...
Do you have any idea of why it happens?
EDIT
More details about the driver logs when an executor is killed:
16/09/30 12:23:33 INFO cluster.YarnClusterSchedulerBackend: Disabling executor 91.
16/09/30 12:23:33 INFO scheduler.DAGScheduler: Executor lost: 91 (epoch 0)
16/09/30 12:23:33 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 91 from BlockManagerMaster.
16/09/30 12:23:33 INFO storage.BlockManagerMasterEndpoint: Removing block manager BlockManagerId(91, server.com, 40923)
16/09/30 12:23:33 INFO storage.BlockManagerMaster: Removed 91 successfully in removeExecutor
16/09/30 12:23:33 INFO cluster.YarnClusterScheduler: Executor 91 on server.com killed by driver.
16/09/30 12:23:33 INFO spark.ExecutorAllocationManager: Existing executor 91 has been removed (new total is 94)
Logs on the executor
16/09/30 12:26:28 INFO rdd.HadoopRDD: Input split: hdfs://...
16/09/30 12:26:32 INFO executor.Executor: Finished task 38219.0 in stage 0.0 (TID 26519). 2312 bytes result sent to driver
16/09/30 12:27:33 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM
16/09/30 12:27:33 INFO storage.DiskBlockManager: Shutdown hook called
16/09/30 12:27:33 INFO util.ShutdownHookManager: Shutdown hook called
I'm seeing this problem on executors that are killed as a result of an idle timeout. I have an exceedingly demanding computational load, but it's mostly computed in a UDF, invisible to Spark. I believe that there's some spark parameter that can be adjusted.
Try looking through the spark.executor parameters in https://spark.apache.org/docs/latest/configuration.html#spark-properties and see if anything jumps out.
We're running one ZooKeeper server and one Kafka broker on a test machine. We're using ZooKeeper 3.3.5 and Kafka 0.9.0.1.
The message retention period on the broker is set to 4 hours:
log.retention.ms = null
log.retention.bytes = -1
log.retention.hours = 4
log.retention.minutes = null
log.retention.check.interval.ms = 30000
We have a single topic whose retention period is essentially infinite (we set it to 50 years).
At some point we shut down then restarted ZooKeeper, then we shut down and restarted the Kafka broker.
Looking at the logs, it seems that the Kafka broker, upon startup, decided to delete old messages, even though they are well within the 50 years retention period.
First, here's Kafka starting up normally:
[2016-09-15 09:09:24,877] INFO starting (kafka.server.KafkaServer)
[2016-09-15 09:09:24,883] INFO Connecting to zookeeper on 10.0.4.83:2182 (kafka.server.KafkaServer)
[2016-09-15 09:09:25,056] INFO Loading logs. (kafka.log.LogManager)
[2016-09-15 09:09:25,096] INFO Completed load of log ecosystem_sharedmodel-0 with log end offset 102152 (kafka.log.Log)
[2016-09-15 09:09:25,105] INFO Logs loading complete. (kafka.log.LogManager)
[2016-09-15 09:09:25,106] INFO Starting log cleanup with a period of 30000 ms. (kafka.log.LogManager)
[2016-09-15 09:09:25,107] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-09-15 09:09:25,149] INFO Awaiting socket connections on ip-10-0-4-83:9092. (kafka.network.Acceptor)
[2016-09-15 09:09:25,151] INFO [Socket Server on Broker 1], Started 1 acceptor threads (kafka.network.SocketServer)
[2016-09-15 09:09:25,169] INFO [ExpirationReaper-1], Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-09-15 09:09:25,173] INFO [ExpirationReaper-1], Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-09-15 09:09:25,244] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-09-15 09:09:25,250] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-09-15 09:09:25,250] INFO 1 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-09-15 09:09:25,321] INFO [GroupCoordinator 1]: Starting up. (kafka.coordinator.GroupCoordinator)
Then, 30 seconds later, the log cleanup kicks in and deletes all our messages:
[2016-09-15 09:09:55,114] INFO Rolled new log segment for 'ecosystem_sharedmodel-0' in 3 ms. (kafka.log.Log)
[2016-09-15 09:09:55,115] INFO Scheduling log segment 0 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,115] INFO Scheduling log segment 618 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,115] INFO Scheduling log segment 3052 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,116] INFO Scheduling log segment 6050 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,116] INFO Scheduling log segment 73419 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,116] INFO Scheduling log segment 73553 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,116] INFO Scheduling log segment 82663 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,117] INFO Scheduling log segment 85600 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,117] INFO Scheduling log segment 96316 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,117] INFO Scheduling log segment 99030 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:09:55,117] INFO Scheduling log segment 101756 for log ecosystem_sharedmodel-0 for deletion. (kafka.log.Log)
[2016-09-15 09:10:55,116] INFO Deleting segment 618 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,116] INFO Deleting segment 0 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,116] INFO Deleting segment 6050 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,117] INFO Deleting segment 73419 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,117] INFO Deleting segment 3052 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,117] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000073419.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,117] INFO Deleting segment 73553 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,117] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000003052.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,118] INFO Deleting segment 82663 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,118] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000073553.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,118] INFO Deleting segment 85600 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,118] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000082663.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,119] INFO Deleting segment 96316 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,119] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000085600.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,119] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000096316.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,119] INFO Deleting segment 99030 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,119] INFO Deleting segment 101756 from log ecosystem_sharedmodel-0. (kafka.log.Log)
[2016-09-15 09:10:55,120] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000000000.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,121] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000099030.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,121] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000000618.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,121] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000101756.index.deleted (kafka.log.OffsetIndex)
[2016-09-15 09:10:55,123] INFO Deleting index /data/kafka/ecosystem_sharedmodel-0/00000000000000006050.index.deleted (kafka.log.OffsetIndex)
Is this the expected behavior of Kafka? Is there a misunderstanding on our side? A configuration problem?
EDIT: it turned out to be a user error, no bugs on Kafka's side.