Given the following stream definition: "http | json-to-tuple > queue:job:process-data". How can I address the "process-data" deployment properties (like count and consumer concurrency - I'm using rabbitmq)?
I've tried module.*.count=0, but the process-data module always gets deployed in a single container (I've got a cluster with 2 containers). Also, when trying with module.process-data.count=0 spring-xd says that the only available modules are http and json-to-tuple.
I'm using spring-xd 1.2.1.
Thanks
process-data is not a module, it's just a named output channel for json-to-tuple.
It's a mechanism to connect two streams together;
foo | bar | baz
is the same as
foo | bar > xyz
xyz > baz
It gives a name xyz to the second | in the single stream.
Related
I'm trying to setup a basic NATS service on my kubernetes cluster, according to their documentation, here. I executed the following code:
$ helm install test-nats nats/nats
NAME: test-nats
LAST DEPLOYED: Thu Jul 14 13:18:09 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
You can find more information about running NATS on Kubernetes
in the NATS documentation website:
https://docs.nats.io/nats-on-kubernetes/nats-kubernetes
NATS Box has been deployed into your cluster, you can
now use the NATS tools within the container as follows:
kubectl exec -n default -it deployment/test-nats-box -- /bin/sh -l
nats-box:~# nats-sub test &
nats-box:~# nats-pub test hi
nats-box:~# nc test-nats 4222
Thanks for using NATS!
$ kubectl exec -n default -it deployment/test-nats-box -- /bin/sh -l
_ _
_ __ __ _| |_ ___ | |__ _____ __
| '_ \ / _` | __/ __|_____| '_ \ / _ \ \/ /
| | | | (_| | |_\__ \_____| |_) | (_) > <
|_| |_|\__,_|\__|___/ |_.__/ \___/_/\_\
nats-box v0.11.0
test-nats-box-84c48d46f-j7jvt:~#
Now, so far, everything has conformed to their start guide. However, when I try to test the connection, I run into trouble:
test-nats-box-84c48d46f-j7jvt:~# nats-sub test &
test-nats-box-84c48d46f-j7jvt:~# /bin/sh: nats-sub: not found
test-nats-box-84c48d46f-j7jvt:~# nats-pub test hi
/bin/sh: nats-pub: not found
It looks like the commands weren't found but they should have been installed when I did the helm install. What's going on here?
I have reproduced the set-up on my kubernetes cluster and have successfully deployed the nats box and started a client subscriber program in which subscribers listen on subjects, and publishers send messages on specific subjects.
1. Create Subscriber
In a shell or command prompt session, start a client subscriber program.
nats sub < subject>
Here, < subject > is a subject to listen on. It helps to use unique and well thought-through subject strings because you need to ensure that messages reach the correct subscribers even when wildcards are used.
For example:
nats sub msg.test
You should see the message: Listening on [msg.test].
2. Create a Publisher and publish a message
Create a NATS publisher and send a message.
nats pub < subject> < message>
Where < subject> is the subject name and < message> is the text to publish.
For example:
nats pub msg.test nats-message-1
You'll notice that the publisher sends the message and prints: Published [msg.test] : 'NATS MESSAGE'.
The subscriber receives the message and prints: [#1] Received on [msg.test]: 'NATS MESSAGE'.
Here, you have provided the wrong syntax nats-sub and nats-pub which are deprecated. Try using the above commands to give precise results.
I had the same problem. nats-sub and nats-pub seem to be deprecated and you need to use nats sub and nats pub instead.
We are using Chef to manage our infrastructure, and I'm running into an issue where the Splunk TA (Add-on for Kafka) simply refuses to acknowledge I've dropped kafka_credential.conf file in the local directory of the plugin. If I use the "Web UI", it generates an entry properly and it shows up in the add-on configuration.
[root#ip-10-14-1-42 local]# ls
app.conf inputs.conf kafka.conf kafka_credentials.conf
[root#ip-10-14-1-42 local]# grep -nr "" *.conf
app.conf:1:# MANAGED BY CHEF. PLEASE DO NOT MODIFY!
app.conf:2:[install]
app.conf:3:is_configured = 1
inputs.conf:1:# MANAGED BY CHEF. PLEASE DO NOT MODIFY!
inputs.conf:2:[kafka_mod]
inputs.conf:3:interval = 60
inputs.conf:4:start_by_shell = false
inputs.conf:5:
inputs.conf:6:[kafka_mod://my_app]
inputs.conf:7:kafka_cluster = default
inputs.conf:8:kafka_topic = log-my_app
inputs.conf:9:kafka_topic_group = my_app
inputs.conf:10:kafka_partition_offset = earliest
inputs.conf:11:index = main
kafka.conf:1:# MANAGED BY CHEF. PLEASE DO NOT MODIFY!
kafka.conf:2:[global_settings]
kafka.conf:3:log_level = INFO
kafka.conf:4:index = main
kafka.conf:5:use_kv_store = 0
kafka.conf:6:use_multiprocess_consumer = 1
kafka.conf:7:fetch_message_max_bytes = 1048576
kafka_credentials.conf:1:# MANAGED BY CHEF. PLEASE DO NOT MODIFY!
kafka_credentials.conf:2:[default]
kafka_credentials.conf:3:kafka_brokers = 10.14.2.164:9092,10.14.2.194:9092
kafka_credentials.conf:4:kafka_partition_offset = earliest
kafka_credentials.conf:5:index = main
Upon restarting splunk, the add-on is installed, and even the input is created under the Inputs section, but the cluster itself is "not available" and when examining the logs I see this:
2017-08-09 01:40:25,442 INFO pid=29212 tid=MainThread file=kafka_mod.py:main:168 | Start Kafka
2017-08-09 01:40:30,508 INFO pid=29212 tid=MainThread file=kafka_config.py:_get_kafka_clusters:228 | Clusters: {}
2017-08-09 01:40:30,509 INFO pid=29212 tid=MainThread file=kafka_config.py:__init__:188 | No Kafka cluster are configured
It seems like this plugin is only respecting clusters created through the WebUI. That is not going to work as we want to be able to fully configure this through Chef. Short of hacking the REST API, and fudging around with the .py files in the addon directory and forcing a dictionary in, what are my options?
Wondering if anyone has encountered this before.
If I had to guess it is silently rejecting the files because # is not traditionally used for comments in INI files. Try a ; instead.
On my karaf 4.0.8 I've installed the feature pax-jdbc-postgresql. The DataFactory for PostgreSQL is installed:
org.osgi.service.jdbc.DataSourceFactory]
osgi.jdbc.driver.class org.postgresql.Driver
osgi.jdbc.driver.name PostgreSQL JDBC Driver
osgi.jdbc.driver.version PostgreSQL 9.4 JDBC4.1 (build 1203)
service.bundleid 204
service.scope singleton
Using Bundles com.eclipsesource.jaxrs.publisher (184)
I've create the file etc/org.ops4j.datasource-psql-sandbox.cfg:
osgi.jdbc.driver.class=org.postgresql.Driver
osgi.jdbc.driver.name=PostgreSQL
url=jdbc:postgresql://localhost:5432/sandbox
dataSourceName=psql-sandbox
user=sandbox
password=sandbox
After that, I see the confirmation in karaf.log that the file was processed:
2017-02-10 14:54:17,468 | INFO | 41-88b277ae0921) |
DataSourceRegistration | 154 - org.ops4j.pax.jdbc.config -
0.9.0 | Detected config for DataSource psql-sandbox. Tracking DSF with filter
(&(objectClass=org.osgi.service.jdbc.DataSourceFactory)(osgi.jdbc.driver.class=org.postgresql.Driver)(osgi.jdbc.driver.name=PostgreSQL))
However, I see no new DataSource in services list in console. What went wrong? I see no exceptions in log ....
The log message tell you that the config was processed and it is now searching for a suitable DataSourceFactory OSGi service.
The problem in your case is that it does not find such a service. So to debug this you should list all DataSourceFactory services and check their properties.
service:list DataSourceFactory
In my case it shows this:
[org.osgi.service.jdbc.DataSourceFactory]
-----------------------------------------
osgi.jdbc.driver.class = org.postgresql.Driver
osgi.jdbc.driver.name = PostgreSQL JDBC Driver
...
As you see it does not match the filter you see in the log. Generally you should only provide either osgi.jdbc.driver.class or osgi.jdbc.driver.name not both. If you remove the osgi.jdbc.driver.name line the config will work.
There is no error message as the system can not know if the error is transient or not. Basically as soon as you install a matching OSGi service the DataSource will be created.
I am trying to deploy rhosp10, and when getting to "openstack overcloud deploy" phase, I get these errors:
Error: only 0 of 1 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Error: only 0 of 5 requested ironic nodes are tagged to profile compute (for flavor compute)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:compute,boot_option:local
Not enough nodes - available: 0, requested: 6
Configuration has 3 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
However, I configured 1 node to use control profile, and 5 to use compute profile. For example:
[stack#rhosp-1-director ~]$ openstack baremetal node show 4e153e0a-4c7b-4ee9-afb5-9036e263949b|grep prop
| properties | {u'cpu_arch': u'x86_64', u'root_device': {u'serial': u'600508b1001c7b0731bc32edbb3a8369'}, u'cpus': u'48', u'capabilities': u'profile:control,boot_option:local', u'memory_mb': u'131072', u'local_gb': u'744'} |
[stack#rhosp-1-director ~]$ openstack baremetal node show 4989038d-de10-4365-8051-44fd42fd0ec7|grep prop
| properties | {u'cpu_arch': u'x86_64', u'root_device': {u'serial': u'600508b1001c73b9fa55f385cd1a4008'}, u'cpus': u'48', u'capabilities': u'profile:compute,boot_option:local', u'memory_mb': u'131072', u'local_gb': u'744'} |
Another thing is that the following command yields no output:
openstack overcloud profiles list
I am following their manual from https://access.redhat.com/documentation/en/red-hat-openstack-platform/10-beta/single/director-installation-and-usage/#sect-Registering_Nodes_for_the_Overcloud step by step, so don't know what I'm doing wrong.
problem ended up being ironic auto cleaning. introspection never completed ok. not sure why, but disabling auto cleaning in ironic.conf right after undercloud install, followed by a reboot (for all ironic services to update this property), followed by the next steps, was successful (including introspection).
Good morning,
I'm looking for help please.I'm only a beginner.
I am using cxf-dosgi from (DOSGi Apache Karaf Feature distribution)
I want to make transparent use of services between two remote machines. So I have a karaf container on each of these two machines.
I tested this example : to start with two containers karaf hosted on the same machine then I changed the configuration to test with two containers hosted on two different remote machines. And it works great !
So I want to do the same thing to export to export my web services. I am using Spring DM. So I do this on the server side :
<osgi:service id="osgi-service" ref="myservice" interface="org.apache.camel.Endpoint"> <osgi:service-properties> <entry key="name" value="service"/> <entry key="service.exported.interfaces" value="*"/> </osgi:service-properties> </osgi:service>
I did the installations like in the tutorial with cxf dosgi version 1.6 But I get this error:
16:25:53,256 | ERROR | pool-21-thread-3 | w.service.RemoteServiceAdminCore 193 | 184 - cxf-dosgi-ri-dsw-cxf - 1.6.0 | failed to create server for interface org.apache.camel.Endpoint
java.lang.NullPointerException
at java.lang.reflect.Array.newArray(Native Method)[:1.7.0_79]
at java.lang.reflect.Array.newInstance(Array.java:70)[:1.7.0_79]
at org.apache.cxf.aegis.type.TypeUtil.getTypeRelatedClass(TypeUtil.java:259)
at org.apache.cxf.aegis.type.AbstractTypeCreator.createTypeForClass(AbstractTypeCreator.java:108)
at org.apache.cxf.aegis.type.AbstractTypeCreator.createType(AbstractTypeCreator.java:402)
at org.apache.cxf.aegis.type.basic.BeanTypeInfo.getType(BeanTypeInfo.java:192)
at org.apache.cxf.aegis.type.basic.BeanType.getDependencies(BeanType.java:547)
at org.apache.cxf.aegis.databinding.AegisDatabinding.addDependencies(AegisDatabinding.java:394)
at org.apache.cxf.aegis.databinding.AegisDatabinding.addDependencies(AegisDatabinding.java:399)
at org.apache.cxf.aegis.databinding.AegisDatabinding.addDependencies(AegisDatabinding.java:399)
at org.apache.cxf.aegis.databinding.AegisDatabinding.initializeMessage(AegisDatabinding.java:371)
at org.apache.cxf.aegis.databinding.AegisDatabinding.initializeOperation(AegisDatabinding.java:283)
at org.apache.cxf.aegis.databinding.AegisDatabinding.initialize(AegisDatabinding.java:242)
at org.apache.cxf.service.factory.AbstractServiceFactoryBean.initializeDataBindings(AbstractServiceFactoryBean.java:86)
at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.buildServiceFromClass(ReflectionServiceFactoryBean.java:490)
at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:550)
at org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:265)
at org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:102)
at org.apache.cxf.frontend.ServerFactoryBean.create(ServerFactoryBean.java:159)
at org.apache.cxf.dosgi.dsw.handlers.AbstractPojoConfigurationTypeHandler.createServerFromFactory(AbstractPojoConfigurationTypeHandler.java:191)
at org.apache.cxf.dosgi.dsw.handlers.PojoConfigurationTypeHandler.createServer(PojoConfigurationTypeHandler.java:119)
at org.apache.cxf.dosgi.dsw.service.RemoteServiceAdminCore.exportInterfaces(RemoteServiceAdminCore.java:184)
at org.apache.cxf.dosgi.dsw.service.RemoteServiceAdminCore.exportService(RemoteServiceAdminCore.java:140)
at org.apache.cxf.dosgi.dsw.service.RemoteServiceAdminInstance$1.run(RemoteServiceAdminInstance.java:59)
at org.apache.cxf.dosgi.dsw.service.RemoteServiceAdminInstance$1.run(RemoteServiceAdminInstance.java:57)
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_79]
at org.apache.cxf.dosgi.dsw.service.RemoteServiceAdminInstance.exportService(RemoteServiceAdminInstance.java:57)[184:cxf-dosgi-ri-dsw-cxf:1.6.0]
at org.apache.cxf.dosgi.dsw.service.RemoteServiceAdminInstance.exportService(RemoteServiceAdminInstance.java:41)[184:cxf-dosgi-ri-dsw-cxf:1.6.0]
at org.apache.cxf.dosgi.topologymanager.exporter.TopologyManagerExport.exportServiceUsingRemoteServiceAdmin(TopologyManagerExport.java:185)[183:cxf-dosgi-ri-topology-manager:1.6.0]
at org.apache.cxf.dosgi.topologymanager.exporter.TopologyManagerExport.doExportService(TopologyManagerExport.java:168)[183:cxf-dosgi-ri-topology-manager:1.6.0]
at org.apache.cxf.dosgi.topologymanager.exporter.TopologyManagerExport$3.run(TopologyManagerExport.java:143)[183:cxf-dosgi-ri-topology-manager:1.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_79]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_79]
Have you an idea of what is wrong?
Ouch .. what are you doing with 1.4-SNAPSHOT. First it is not a release and second it is quite old.
Another thing that looks suspicious is the service.exported.interfaces=myInterface. It should be a fully qualified java interface name. For the start try service.exported.interfaces=* for it.
You should start with my CXF DOSGi tutorial. The code there should work out of the box. You can then add your changes in the config. So it is easier than to start completely new.