what is the difference in specifying group at the consumer
spring.kafka.consumer.group-id
vs specifying at the #KafkaListener?
#KafkaListener(topic="test", group = "test-grp")
See the javadocs for the group property; it has nothing to do with the kafka group.id...
/**
* If provided, the listener container for this listener will be added to a bean
* with this value as its name, of type {#code Collection<MessageListenerContainer>}.
* This allows, for example, iteration over the collection to start/stop a subset
* of containers.
* #return the bean name for the group.
*/
This has been renamed containerGroup in 1.3/2.0.
Those release versions also provide...
/**
* Override the {#code group.id} property for the consumer factory with this value
* for this listener only.
* #return the group id.
* #since 1.3
*/
String groupId() default "";
/**
* When {#link #groupId() groupId} is not provided, use the {#link #id() id} (if
* provided) as the {#code group.id} property for the consumer. Set to false, to use
* the {#code group.id} from the consumer factory.
* #return false to disable.
* #since 1.3
*/
boolean idIsGroup() default true;
Previously, you needed a container factory/consumer factory for each listener; these allow you to use one factory instance and override the group.id.
Related
I am creating a new KafkaMessageListenerContainer using a ContainerProperties.
Using ContainerProperties(String... topics), the Consumer Group looks fine: "state": "STABLE", "isSimpleConsumerGroup": false
Using ContainerProperties(TopicPartitionOffset... topicPartitions), the Consumer Groups is not automatically created. It is finally created when a message is sent but the Consumer Group doesn't look fine: "state": "EMPTY", "isSimpleConsumerGroup": true
What's the difference between them, did I miss something. I am expecting to have the same result using the two different ContainerProperties constructors.
ContainerProperties containerProps = new ContainerProperties(tpo.toArray(new TopicPartitionOffset[tpo.size()]));
containerProps.setGroupId(name);
// ContainerProperties containerProps = new ContainerProperties("poc-topic1",
// "poc-topic2", "poc-topic3");
// containerProps.setGroupId(name);
containerProps.setMessageListener(new TopicMessageListener(name));
DefaultKafkaConsumerFactory<String, Serializable> factory = new DefaultKafkaConsumerFactory<>(
Utils.get().getConsumerProperties());
container = new KafkaMessageListenerContainer<>(factory, containerProps);
// container.setAutoStartup(true);
// container.setBeanName(name);
// container.checkGroupId();
container.start();
That's not correct. The topic subscription causes a consumer group and their partitions distribution between group members.
When you do explicit partition assignment, not consumer group is involved at all.
See more in Apache Kafka docs: https://docs.confluent.io/platform/current/clients/consumer.html#consumer-groups
And respective JavaDocs:
/**
* Manually assign a list of partitions to this consumer. This interface does not allow for incremental assignment
* and will replace the previous assignment (if there is one).
* <p>
* If the given list of topic partitions is empty, it is treated the same as {#link #unsubscribe()}.
* <p>
* Manual topic assignment through this method does not use the consumer's group management
* functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic
* metadata change. Note that it is not possible to use both manual partition assignment with {#link #assign(Collection)}
* and group assignment with {#link #subscribe(Collection, ConsumerRebalanceListener)}.
* <p>
* If auto-commit is enabled, an async commit (based on the old assignment) will be triggered before the new
* assignment replaces the old one.
*
* #param partitions The list of partitions to assign this consumer
* #throws IllegalArgumentException If partitions is null or contains null or empty topics
* #throws IllegalStateException If {#code subscribe()} is called previously with topics or pattern
* (without a subsequent call to {#link #unsubscribe()})
*/
#Override
public void assign(Collection<TopicPartition> partitions) {
and:
/**
* Subscribe to the given list of topics to get dynamically assigned partitions.
* <b>Topic subscriptions are not incremental. This list will replace the current
* assignment (if there is one).</b> It is not possible to combine topic subscription with group management
* with manual partition assignment through {#link #assign(Collection)}.
*
* If the given list of topics is empty, it is treated the same as {#link #unsubscribe()}.
*
* <p>
* This is a short-hand for {#link #subscribe(Collection, ConsumerRebalanceListener)}, which
* uses a no-op listener. If you need the ability to seek to particular offsets, you should prefer
* {#link #subscribe(Collection, ConsumerRebalanceListener)}, since group rebalances will cause partition offsets
* to be reset. You should also provide your own listener if you are doing your own offset
* management since the listener gives you an opportunity to commit offsets before a rebalance finishes.
*
* #param topics The list of topics to subscribe to
* #throws IllegalArgumentException If topics is null or contains null or empty elements
* #throws IllegalStateException If {#code subscribe()} is called previously with pattern, or assign is called
* previously (without a subsequent call to {#link #unsubscribe()}), or if not
* configured at-least one partition assignment strategy
*/
#Override
public void subscribe(Collection<String> topics) {
I have a parent class MapItem, and a child class, MapExhibit. My MapExhibit class has a property, $builiding, which ties the exhibit to a particular MapBuilding entity. When the API calls for the JSON, the $building should not appear for MapExhibit entities in particular.
Note I am using the willdurand/Hateoas bundle
Here is my current setup:
/**
* #ORM\Entity(repositoryClass="App\Repository\Map\MapItemRepository")
* #ORM\InheritanceType("JOINED")
* #ORM\DiscriminatorColumn(name="discr", type="string")
* #ORM\DiscriminatorMap({"item" = "MapItem", "bathroom" = "MapBathroom", "building" = "MapBuilding", "bus" = "MapBus", "emergency" = "MapEmergency", "exhibit" = "MapExhibit", "parking" = "MapParking"})
* #Serializer\XmlRoot("mapItem")
* #Hateoas\Relation("self", href = "expr('/api/mapitems/' ~ object.getId())")
*/
abstract class MapItem
{
/**
* #ORM\Id
* #ORM\GeneratedValue
* #ORM\Column(type="integer")
* #Serializer\XmlAttribute
*/
private $id;
...
}
/**
* #ORM\Entity(repositoryClass="App\Repository\Map\MapExhibitRepository")
*
* #Hateoas\Relation(
* "building",
* exclusion = #Hateoas\Exclusion()
* )
*/
class MapExhibit extends MapItem
{
...
/**
* Many emergency devices can belong to one building.
* #ORM\ManyToOne(targetEntity="MapBuilding", inversedBy="emergencyDevices")
* #ORM\JoinColumn(name="building_id", referencedColumnName="id", nullable=true, onDelete="SET NULL")
*/
private $building;
...
public function getBuilding(): ?MapBuilding
{
return $this->building;
}
}
The result is a JSON object that includes the relation from the parent MapItem
"_links":{"self":{"href":"\/api\/mapitems\/29"}}}]
But also includes the data from the exhibit's building. This part should be ignored.
Take a look on exclusion strategies in documentation.
If you would like to always expose, or exclude certain properties. Then, you can do this with the annotations #ExclusionPolicy, #Exclude, and #Expose.
The default exclusion policy is to exclude nothing. That is, all properties of the object will be serialized. If you only want to expose a few of the properties, then it is easier to change the exclusion policy, and only mark these few properties: (...)
So basically, put #Serializer\Exclude() above private $builidng. Don't forget to add use JMS\Serializer\Annotation as Serializer; on top.
I have Symfony 2.6 application and using doctrine-odm-bundle 3.0.2 (doctrine-odm: 1.1.2, mongodb: 1.4.0).
My document has referenceMany and referenceOne in attributes and when I create new instance of it, fill fields and persist - it goes fine in the begining. I can create few nearly empty documents, with referenced document(s) or without and it works fine. At some point I am trying to add new item in the database and getting an error:
E11000 duplicate key error collection: test.Product index: _id_ dup key: { : 0 }
The message is clear - I can see that there was a document added to the collection with id = 0, therefore second one can't go -> duplicate entry. But why it suddenly starts to return "0" for id? Even though, I checked doctrine_increment_ids collection - counter for id is being incremented. But $product->getId() becomes "0" after persist.
If I drop the database and start all over - it works, I can still add new products in the collection. Let's say I successfully created 12 products. Creating 13th resulting a document with id=0 being persisted in the collection. 14th fails with duplicate error.
Can you please help to troubleshoot or suggest an idea where does it go wrong?
P.S> I am not considering an upgrade of Symfony2 (at this point) neither as doctrine-odm-bundle (it depends on newer Symfony2 as well) - migration efforts are quite high and I am not sure it will fix the issue. First I want to find out the root cause.
// Document Product
/**
* #MongoDB\Document
* #MongoDB\HasLifecycleCallbacks
*/
class Product
{
/** #MongoDB\Id(strategy="ALNUM", type="int") */
protected $id;
/**
* #Gedmo\ReferenceOne(type="entity", class="Entity\User", inversedBy="products", identifier="userId")
*/
protected $user;
/**
* #MongoDB\Field(name="user_id", type="int")
*/
protected $userId;
/**
* #MongoDB\ReferenceMany(
* targetDocument="Picture",
* discriminatorMap={"file" = "File", "picture" = "Picture"},
* discriminatorField="discr",
* defaultDiscriminatorValue="picture"
* )
* #Assert\Valid
*/
protected $pictures;
...
}
// Entity User
/**
* User entity
* #ORM\Entity
* #ORM\Table(name="users")
*/
class User
{
/**
* #ORM\Id
* #ORM\Column(type="integer")
* #ORM\GeneratedValue(strategy="AUTO")
*/
protected $id;
/**
* #var ArrayCollection $textures
*
* #Gedmo\ReferenceMany(type="document", class="Document\Product", mappedBy="user")
*/
protected $products;
...
}
// Document Picture
/**
* #MongoDB\Document
* #MongoDB\InheritanceType("SINGLE_COLLECTION")
* #MongoDB\DiscriminatorField("discr")
* #MongoDB\DiscriminatorMap({"file" = "File", "picture" = "Picture"})
* #MongoDB\DefaultDiscriminatorValue("picture")
* #MongoDB\HasLifecycleCallbacks
*/
class Picture
{
/**
*
* #MongoDB\Id(strategy="ALNUM", type="int")
*/
protected $id;
/**
* #MongoDB\ReferenceOne(targetDocument="Product")
*
* #var Product $product
*/
protected $product;
...
}
Documentation reading always helps (generation strategies). Basically, strategy="ALNUM" and type="int" just can't go together :)
Change strategy to INCREMENT and remove type="int" if you want to have integers in your _id.
Or you can change type to string to continue with _id being an alphanumeric string.
I am using #typedef to define some custom types, with #property items.
When doing the same for a class I can just use {#link MyClass#property} to reference the property.
But for some reasons {#link MyType#property} does not resolve.
Is this a bug or is there a different way to reference such properties?
My usage example:
/**
* #typedef helpers.Column
*
* #property {String} name
*/
/**
* This one resolves correctly: {#link helpers.Column}
*
* This one does not resolve: {#link helpers.Column#name}
*/
A symfony2 application has a Job entity that has as a property of type WebSite.
A simplified representation of this without other properties or methods:
/**
* #ORM\Entity
* #ORM\Table(name="Job")
*/
class Job
{
/**
* #ORM\Id
* #ORM\Column(type="integer")
* #ORM\GeneratedValue(strategy="AUTO")
*/
protected $id;
/**
* #ORM\Column(type="integer", name="website_id", nullable=false)
* #ORM\ManyToOne(targetEntity="Example\ExampleBundle\Entity\WebSite")
*/
protected $website;
}
Symfony/Doctrine is trying to cast the website property to a string when persisting resulting in the error:
Catchable Fatal Error: Object of class
Example\ExampleBundle\Entity\WebSite could not be converted to string
in /vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php line 131
I believe the above #ORM\Column annotation denotes the website property to be an integer. I don't understand why Symfony/Doctrine wishes to try and convert the website property to a string.
Non-ideal workarounds I have tried in an attempt to resolve the matter:
Adding __toString() method to WebSite, returning a string representation of the id property, causes the correct value to ultimately end up in the Job.website_id datafield; this workaround is impractical as I will in the future need __toString() to be available to present a string elsewhere in the application.
Removing the #ORM\Column annotation from the website property; this results in all future diff-generated migrations (php app/console doctrine:migrations:diff) removing the 'not null' aspect of the relevant database field.
Are there any changes I should make to the above annotation to inform Symfony/Doctrine to call the getId() method of WebSite to get what it needs when persisting? Are there any changes I could make to WebSite to achieve the same?
What changes are required to ensure that the above annotation can remain in place such that Doctrine ultimately calls WebSite.getId() to get what it needs when persisting instead of trying to cast the object to a string?
You have to remove the #ORM\Column(type="integer" annotation from the $website property.
To be sure that the website_id column keeps the NOT NULL constraint, add a JoinColumn annotation with nullable=false in it:
/**
* #var Example\ExampleBundle\Entity\WebSite
*
* #ORM\ManyToOne(targetEntity="Example\ExampleBundle\Entity\WebSite")
* #ORM\JoinColumn(name="website_id", referencedColumnName="id", nullable=false)
*/
protected $website;