I have a project built with Spring Cloud GCP, which makes use of a subscription that is configured with Exactly once delivery enabled on GCP Pub/Sub subscription settings, and with Ack deadline set to 5 minutes (also in the GCP console).
Using Spring, I also have access to configure subscriber settings using configuration options (for example: spring.cloud.gcp.pubsub.subscriber.max-ack-extension-period=1800).
My question is twofold:
Which of these configurations (GCP console vs Spring configuration parameters) take effect in case of conflict?
Can I actually have exactly-once delivery semantics with Spring Cloud GCP (given that the feature is marked as having limited support in the docs: https://cloud.google.com/pubsub/docs/exactly-once-delivery)?
You can set the default ack timeout in your PubSub subscription and you can extend it (only for pull subscription) in your code, the max timeout being 10 minutes in any case.
There is no conflict: there is a default configuration, and a programmatic extension in case of congestion for example
The exactly-once delivery is also compliant with Spring Cloud GCP if you use the latest version, which use the latest Java client libraries. Spring Cloud GCP is only a wrapper that reuse the Spring programming framework but the underlying implementation still use the standard JAVA client libraries.
Related
Is it possible to make auto-refresh properties for Spring Cloud clients in a multi-pod environment (Google Kubernetes Engine)?
I found several work arounds:
Using Spring Cloud Bus (too heavy solution).
Running refresh inside code using RefreshEvent and #Schedule it (not recommended by Spring).
Creating a new endpoint in Config Server to perform a refresh on all Spring Cloud clients.
We plan to use Spring Cloud Data Flow on Azure Cloud using Azure EventHub as a messaging binder.
On Azure EventHub, there are hard limits :
100 Namespaces
10 topics per namespaces.
The Spring Cloud Azure Event Hub Stream Binder seems to be able to configure only one namespace, so how can we manage multiple namespaces?
Maybe we should use multiple binders, to have multiple instances of the Spring Cloud Azure Event Hub Stream Binder?
Does anyone have any ideas? or documentation we did not find?
Regards
Rémi
Spring Cloud Data Flow and Spring Cloud Skipper support the concept of "platform accounts". Using that, you can set up multiple accounts, for each namespace or any other K8s clusters even. This opens a lot of flexibility to work around these hard limits in Azure stack.
We have a recipe on multi-platform deployments.
When deploying the streams from SCDF, you'd pick and choose the platform account (aka namespace or other configs), so automatically the deployed stream apps (with Azure binder in the classpath) would be running in different namespaces. Effectively, dodging the limits enforced in Azure.
The provenance tracking of where the apps run and the audit trail is automatically also captured in SCDF, so at any given time, you'd know who did what and in which namespace.
I’m doing a POC to deploy Kafka as an application on Mesos Cluster. I came across these 2 codebases on github. One developed by apache-mesos (github page) & other developed by mesosphere and can run only on DCOS (github page).
Question: Would like to know if there are any differences between DCOS-Kafka & mesos-Kafka in terms of features and extended functionality.
Regarding Mesos-Kafka:
I don’t see active participation on github (and some open issues) for mesos-kafka in the past months. Can I assume that the service is robust enough that I can use in production environment? Any Inputs on this would be helpful.
kakfa-mesos is a package that includes a release of Kafka and a custom mesos scheduler that was meant to work around issues with running Kafka as a stateful service on Marathon. I think post but confluent is useful. It also includes a RESTful api for doing ops tasks and aims to include these features in the future (this is pulled from the article I linked)
Integrating Kafka commands (e.g. kafka-topics, etc) into the Scheduler so it can be used through the CLI and REST API.
Auto-scaling clusters (including auto reassignment of partitions) so that the resources (CPU, RAM, etc.) that brokers are using can be used elsewhere in known valleys of traffic.
Rack-aware partition assignment for fault tolerance.
Hooks so that producers and consumers can also be launched from the Scheduler and managed with the cluster.
Automated partition reassignment based on load and traffic
I haven't used it in a production environment myself but it has the support of Confluent which is a good sign.
DC/OS Kafka on the other hand is a DC/OS service which will probably only be useful if you are already running or plan on running services through Mesosphere's DC/OS. It also includes an API and a CLI management tool but is less ambitious with additional features. It's current feature set includes
Single-command installation for rapid provisioning
Multiple clusters for multiple tenancy with DC/OS
High availability runtime configuration and software updates
Storage volumes for enhanced data durability, known as Mesos Dynamic * * Reservations and Persistent Volumes
Integration with syslog-compatible logging services for diagnostics and troubleshooting
Integration with statsd-compatible metrics services for capacity and performance monitoring
I am new to Spring Cloud. Currently, I want to build a new micro service based on Spring Cloud. It is very easy to build a new Eureka server. But my question is that how to make it high availability ? For example I create two Eureka server and a load balancer. When one of the Eureka server is down, the system still works well. But I don't know to to consist registered information in the two Eureka server.
I have already asked something similar in the spring cloud gitter channel.
Because of the CAP theorem, something as a distributes Service discovery has to decide, either to provide availability, or more consistency, with a trade off to the other one.
in short, by quoting Spencer Gibb:
Eureka favors availability over consistency
so it is very available, while registred services may be not acutal anymore.
As Spencer suggested, if consistency is something you need more then availability, try Consul together with spring cloud consul intead
I'm doing a proof of concept at my company with Spring Cloud Config, using a GIT repository. Since I have a pretty fixed infrastructure, I can't have a RabbitMQ broker for Spring Cloud Bus, therefore I'm thinking of embedding the Spring Cloud Config Server in our apps and calling RefreshEndpoint.refresh() every couple of minutes.
One of the main things I'd like to know is what are the concurrency implications of calling RefreshEndpoint.refresh()? If there is a request already going on, could it have any issues?