I'm trying to setup Artemis with SSL
my etc/bootstrap.xml file looks like :
<broker xmlns="http://activemq.org/schema">
<web bind="https://0.0.0.0:8161" path="web" keystorePath="keystore.p12" keystorePassword="123" truststorePath="trusstore.p12" truststorePassword="123">
<app url="activemq-branding" war="activemq-branding.war"/>
<app url="artemis-plugin" war="artemis-plugin.war"/>
<app url="console" war="console.war"/>
</web>
</broker>
which I did like it was written in the documentation, but when I run artemis service I'm getting error:
javax.xml.bind.UnmarshalException
- with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 28; columnNumber: 188; cvc-complex-type.3.2.2: Attribute 'keystorePath' is not allowed to appear in element 'web'.]
I cant understand what I did wrong?
JAXB unmarshalling is case sensitive, so you can fix this error replacing the following attribute names: keystorePath > keyStorePath, keystorePassword > keyStorePassword, truststorePath > trustStorePath and truststorePassword > trustStorePassword.
<broker xmlns="http://activemq.org/schema">
<jaas-security domain="activemq"/>
<server configuration="file:/home/dbruscin/Workspace/temp/apache-artemis-2.11.0/broker/etc//broker.xml"/>
<web bind="https://0.0.0.0:8161" path="web" keyStorePath="keystore.p12" keyStorePassword="123" trustStorePath="trusstore.p12" trustStorePassword="123">
<app url="activemq-branding" war="activemq-branding.war"/>
<app url="artemis-plugin" war="artemis-plugin.war"/>
<app url="console" war="console.war"/>
</web>
</broker>
Related
I tried to configure keycloak server with my postgresql db. After many tries with a lot of configs I decided to ask the question.
I have the next configurations:
module.xml which is located in keycloak/modules/org/postgresql/main
Also in that directory I put jdbc-driver with name 'postgresql-42.2.16.jar'
<?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.1" name="org.postgresql">
<resources>
<resource-root path="postgresql-42.2.16.jar"/>
</resources>
<dependencies>
<module name="javax.api"/>
<module name="javax.transaction.api"/>
</dependencies>
</module>
standalone.xml which is located in keycloak/standalone/configuration
In that xml I configured
drivers:
<drivers>
<driver name="postgresql" module="org.postgresql">
<xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
</driver>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
</drivers>
datasources:
<datasources>
<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true">
<connection-url>jdbc:postgres://192.168.99.100:5432/keycloak</connection-url>
<driver>postgresql</driver>
<pool>
<max-pool-size>20</max-pool-size>
</pool>
<security>
<user-name>keycloak</user-name>
<password>password</password>
</security>
</datasource>
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>
...
</datasources>
keycloak server config:
<subsystem xmlns="urn:jboss:domain:keycloak-server:1.1">
<spi name="connectionsJpa">
<provider name="default" enabled="true">
<properties>
<property name="dataSource" value="java:jboss/datasources/KeycloakDS"/>
<property name="initializeEmpty" value="false"/>
<property name="migrationStrategy" value="manual"/>
<property name="migrationExport" value="${jboss.home.dir}/keycloak-database-update.sql"/>
</properties>
</provider>
</spi>
</subsystem>
And when I'm trying to start my keycloak server I got the exception:
17:56:19,859 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "datasources"),
("data-source" => "KeycloakDS")
]) - failure description: {
"WFLYCTL0412: Required services that are not installed:" => ["jboss.jdbc-driver.postgresql"],
"WFLYCTL0180: Services with missing/unavailable dependencies" => [
"jboss.driver-demander.java:jboss/datasources/KeycloakDS is missing [jboss.jdbc-
driver.postgresql]",
"org.wildfly.data-source.KeycloakDS is missing [jboss.jdbc-driver.postgresql]"
]
}
17:56:19,860 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "datasources"),
("data-source" => "KeycloakDS")
]) - failure description: {
"WFLYCTL0412: Required services that are not installed:" => [
"jboss.jdbc-driver.postgresql",
"jboss.jdbc-driver.postgresql"
],
"WFLYCTL0180: Services with missing/unavailable dependencies" => [
"jboss.driver-demander.java:jboss/datasources/KeycloakDS is missing [jboss.jdbc-
driver.postgresql]",
"org.wildfly.data-source.KeycloakDS is missing [jboss.jdbc-driver.postgresql]",
"org.wildfly.data-source.KeycloakDS is missing [jboss.jdbc-driver.postgresql]"
]
}
It's looks like I configured my datasource in wrong way, but I can't find the error. I checked this check list and it doesn't resolve my issue.
Could you please retry with the following connection string:
jdbc:postgresql://192.168.99.100:5432/keycloak
instead of
jdbc:postgres://192.168.99.100:5432/keycloak
Also, consider letting the SPI/connectionsJpa block at defaults (I believe in particular that initializeEmpty should be true) and I don't remember that I had to add/modify this block, so please double-check.
You may also disable or comment the ExampleDS (h2) dataSource.
The below are the config xml wrt infinispan and jgroups. ${cache.clusterName}" is substituted at runtime with C1 [the configured cluster name] in
Infinispan_config.xml. The C2 is not the one configured to be a cluster [though present as a clustername, there is no code that I could figureout from the code base that replace c1 as C2].
background we are running jboss 7 in parallel with jboss5 as part of migration but for the jboss5 to be in sync with jboss 7 seems it needs to be the same cluster
1) Please point any other repercussion of this warning apart from the additional set of logs. App was unstable post running in parallel in prod. The same build worked fine in lower env
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"
xmlns="urn:infinispan:config:4.2">
<global>
<transport clusterName="${cache.clusterName}">
<properties>
<property name="configurationFile" value="jgroups-tcp.xml" />
</properties>
</transport>
</global>
<default>
<locking isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="10000" writeSkewCheck="false"
concurrencyLevel="50" useLockStriping="false" />
<clustering mode="replication">
<stateRetrieval initialRetryWaitTime="1000" timeout="10000" numRetries="10"
alwaysProvideInMemoryState="true" fetchInMemoryState="true" />
<sync />
</clustering>
</default>
jgroups.xml
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
<TCP
bind_addr="${jboss.bind.address}"
bind_port="${jgroups.tcp.port:7800}"
loopback="true"
port_range="30"
recv_buf_size="20000000"
send_buf_size="640000"
discard_incompatible_packets="true"
max_bundle_size="64000"
max_bundle_timeout="30"
enable_bundling="true"
use_send_queues="true"
sock_conn_timeout="300"
enable_diagnostics="false"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="30"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="false"
thread_pool.queue_max_size="100"
thread_pool.rejection_policy="Discard"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="2"
oob_thread_pool.max_threads="30"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="Discard"
/>
<!-- Ergonomics, new in JGroups 2.11, are disabled by default in TCPPING until JGRP-1253 is resolved -->
<TCPPING timeout="10000"
initial_hosts="${jgroups.initial.hosts}"
port_range="0"
num_initial_members="${jgroups.num.initial.members:4}"
ergonomics="false"
/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD timeout="3000" max_tries="3"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK
use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="false"/>
<UNICAST timeout="300,600,1200"/>
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" max_bytes="400000"/>
<pbcast.GMS print_local_addr="false" join_timeout="7000" view_bundling="true"/>
<UFC max_credits="2000000" min_threshold="0.10"/>
<MFC max_credits="2000000" min_threshold="0.10"/>
<FRAG2 frag_size="60000"/>
<pbcast.STREAMING_STATE_TRANSFER/>
<pbcast.FLUSH timeout="0"/>
</config>
I'm trying to deploy an ASP.net 5 website to my local service fabric dev cluster but get an Access Denied exception on every deploy. I know that the problem is with the web service because when I remove it from the deployment my other service is deployed without any error
This exception looks like it is trying to delete a file, but which one and why wouldn't it have access?
6>. 'E:\Github\Flow.Server\Flow.Server.Fabric\Scripts\Deploy-FabricApplication.ps1' -ApplicationPackagePath 'E:\Github\Flow.Server\Flow.Server.Fabric\pkg\Debug' -PublishProfileFile 'E:\Github\Flow.Server\Flow.Server.Fabric\PublishProfiles\Local.xml' -DeployOnly:$true -UnregisterUnusedApplicationVersionsAfterUpgrade $false -OverrideUpgradeBehavior 'None' -OverwriteBehavior 'Always' -ErrorAction Stop
6>Message : Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
6>Data : {}
6>InnerException :
6>TargetSite : Void RunInMTA(System.Action)
6>StackTrace : at System.Fabric.Interop.Utility.RunInMTA(Action action)
6> at System.Fabric.Common.FabricDirectory.Delete(String path, Boolean recursive, Boolean
6> deleteReadOnlyFiles)
6> at Microsoft.ServiceFabric.Powershell.ApplicationCmdletBase.TestApplicationPackage(String
6> applicationPackagePath, Hashtable applicationParameters, String imageStoreConnectionString)
6> at System.Management.Automation.CommandProcessor.ProcessRecord()
6>HelpLink :
6>Source : System.Fabric
6>HResult : -2147024891
Here is my ApplicationManifest
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="Flow.Server.FabricType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="Silo_InstanceCount" DefaultValue="-1" />
</Parameters>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="Flow.Server.Fabric.WebApi" ServiceManifestVersion="1.0.0" />
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="Flow.Server.Fabric.SiloPkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
</ServiceManifestImport>
<DefaultServices>
<Service Name="Flow.Server.Fabric.WebApiService">
<StatelessService ServiceTypeName="Flow.Server.Fabric.WebApiType">
<SingletonPartition />
</StatelessService>
</Service>
<Service Name="Flow.Server.Fabric.Silo">
<StatelessService ServiceTypeName="Flow.Server.Fabric.SiloType" InstanceCount="[Silo_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
</ApplicationManifest>
And here is the ServiceManifest
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="Flow.Server.Fabric.WebApi" Version="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="Flow.Server.Fabric.WebApiType">
<Extensions>
<Extension Name="__GeneratedServiceType__">
<GeneratedNames xmlns="http://schemas.microsoft.com/2015/03/fabact-no-schema">
<DefaultService Name="Flow.Server.Fabric.WebApiService" />
<ServiceEndpoint Name="Flow.Server.Fabric.WebApiTypeEndpoint" />
</GeneratedNames>
</Extension>
</Extensions>
</StatelessServiceType>
</ServiceTypes>
<CodePackage Name="C" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>approot\runtimes\dnx-clr-win-x64.1.0.0-rc1-update1\bin\dnx.exe</Program>
<Arguments>--appbase approot\src\Flow.Server.Fabric.WebApi Microsoft.Dnx.ApplicationHost Microsoft.ServiceFabric.AspNet.Hosting --server Microsoft.AspNet.Server.WebListener</Arguments>
<WorkingFolder>CodePackage</WorkingFolder>
<ConsoleRedirection FileRetentionCount="5" FileMaxSizeInKb="2048" />
</ExeHost>
</EntryPoint>
</CodePackage>
<Resources>
<Endpoints>
<Endpoint Name="Flow.Server.Fabric.WebApiTypeEndpoint" Protocol="http" Type="Input" />
</Endpoints>
</Resources>
</ServiceManifest>
I'm trying to configure both Shibboleth service provider and identity provider on localhost for testing purposes.
The problem is that I'm getting "unable to locate metadata for identity provider" error when trying to access a protected resource.
I've already read all tutorials and discussions that are related to software configuration and this particular error fixing, but nothing works for me.
shibboleth2.xml config for SP:
<SPConfig xmlns="urn:mace:shibboleth:2.0:native:sp:config"
xmlns:conf="urn:mace:shibboleth:2.0:native:sp:config"
xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion"
xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol"
xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata"
clockSkew="180">
<InProcess logger="native.logger">
<ISAPI normalizeRequest="true" safeHeaderNames="true">
<Site id="1" name="localhost" />
</ISAPI>
</InProcess>
<TCPListener address="127.0.0.1" port="1600" acl="127.0.0.1" />
<StorageService type="Memory" id="mem" cleanupInterval="900" />
<SessionCache type="StorageService" StorageService="mem" cacheAssertions="false" cacheAllowance="900" inprocTimeout="900" cleanupInterval="900" />
<ReplayCache StorageService="mem" />
<ArtifactMap artifactTTL="180" />
<RequestMapper type="Native">
<RequestMap>
<Host name="localhost">
<Path name="secure" authType="shibboleth" requireSession="true"/>
</Host>
</RequestMap>
</RequestMapper>
<ApplicationDefaults id="default" policyId="default"
entityID="http://localhost/secure"
homeURL="http://localhost/secure"
signing="false" encryption="false"
REMOTE_USER="eppn persistent-id targeted-id">
<Sessions lifetime="28800" timeout="3600" relayState="ss:mem" handlerURL="/Shibboleth.sso"
checkAddress="false" handlerSSL="false" cookieProps="http">
<SSO entityID="https://bios-hp/idp/shibboleth">
SAML2 SAML1
</SSO>
<Logout>SAML2 Local</Logout>
<Handler type="MetadataGenerator" Location="/Metadata" signing="false"/>
<Handler type="Status" Location="/Status" acl="127.0.0.1 ::1"/>
<Handler type="Session" Location="/Session" showAttributeValues="false"/>
<Handler type="DiscoveryFeed" Location="/DiscoFeed"/>
<md:AssertionConsumerService Location="/SAML2/POST" index="1"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"/>
<md:AssertionConsumerService Location="/SAML2/POST-SimpleSign" index="2"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST-SimpleSign"/>
<md:AssertionConsumerService Location="/SAML2/Artifact" index="3"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Artifact"/>
<md:AssertionConsumerService Location="/SAML2/ECP" index="4"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:PAOS"/>
<md:AssertionConsumerService Location="/SAML/POST" index="5"
Binding="urn:oasis:names:tc:SAML:1.0:profiles:browser-post"/>
<md:AssertionConsumerService Location="/SAML/Artifact" index="6"
Binding="urn:oasis:names:tc:SAML:1.0:profiles:artifact-01"/>
<LogoutInitiator type="Local" Location="/SLO/Logout"/>
<md:SingleLogoutService Location="/SLO/SOAP"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:SOAP"/>
<md:SingleLogoutService Location="/SLO/Redirect" conf:template="bindingTemplate.html"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"/>
<md:SingleLogoutService Location="/SLO/POST" conf:template="bindingTemplate.html"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"/>
<md:SingleLogoutService Location="/SLO/Artifact" conf:template="bindingTemplate.html"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Artifact"/>
<md:ManageNameIDService Location="/NIM/SOAP"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:SOAP"/>
<md:ManageNameIDService Location="/NIM/Redirect" conf:template="bindingTemplate.html"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"/>
<md:ManageNameIDService Location="/NIM/POST" conf:template="bindingTemplate.html"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"/>
<md:ManageNameIDService Location="/NIM/Artifact" conf:template="bindingTemplate.html"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Artifact"/>
<md:ArtifactResolutionService Location="/Artifact/SOAP" index="1"
Binding="urn:oasis:names:tc:SAML:2.0:bindings:SOAP"/>
</Sessions>
<Errors supportContact="root#localhost"
helpLocation="/about.html"
styleSheet="/shibboleth-sp/main.css"/>
<MetadataProvider type="XML" file="C:/opt/shibboleth-sp/etc/shibboleth/idp.metadata.xml"/>
<AttributeExtractor type="XML" validate="true" reloadChanges="false" path="attribute-map.xml"/>
<AttributeResolver type="Query" subjectMatch="true"/>
<AttributeFilter type="XML" validate="true" path="attribute-policy.xml"/>
<CredentialResolver type="File" key="sp-key.pem" certificate="sp-cert.pem"/>
</ApplicationDefaults>
<SecurityPolicyProvider type="XML" validate="true" path="security-policy.xml"/>
<ProtocolProvider type="XML" validate="true" reloadChanges="false" path="protocols.xml"/>
idp.metadata.xml references by SP config:
<?xml version="1.0" encoding="UTF-8"?><EntityDescriptor xmlns="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://bios-hp/idp/shibboleth" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:shibmd="urn:mace:shibboleth:metadata:1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><IDPSSODescriptor protocolSupportEnumeration="urn:mace:shibboleth:1.0 urn:oasis:names:tc:SAML:1.1:protocol urn:oasis:names:tc:SAML:2.0:protocol"><Extensions><shibmd:Scope regexp="false"/></Extensions><KeyDescriptor><ds:KeyInfo><ds:X509Data><ds:X509Certificate>MIID...zY=</ds:X509Certificate></ds:X509Data></ds:KeyInfo></KeyDescriptor><ArtifactResolutionService Binding="urn:oasis:names:tc:SAML:1.0:bindings:SOAP-binding" Location="https://bios-hp:8443/idp/profile/SAML1/SOAP/ArtifactResolution" index="1"/><ArtifactResolutionService Binding="urn:oasis:names:tc:SAML:2.0:bindings:SOAP" Location="https://bios-hp:8443/idp/profile/SAML2/SOAP/ArtifactResolution" index="2"/><NameIDFormat>urn:mace:shibboleth:1.0:nameIdentifier</NameIDFormat><NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:transient</NameIDFormat><SingleSignOnService Binding="urn:mace:shibboleth:1.0:profiles:AuthnRequest" Location="https://bios-hp/idp/profile/Shibboleth/SSO"/><SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://bios-hp/idp/profile/SAML2/POST/SSO"/><SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST-SimpleSign" Location="https://bios-hp/idp/profile/SAML2/POST-SimpleSign/SSO"/><SingleSignOnService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://bios-hp/idp/profile/SAML2/Redirect/SSO"/></IDPSSODescriptor><AttributeAuthorityDescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:1.1:protocol urn:oasis:names:tc:SAML:2.0:protocol"><Extensions><shibmd:Scope regexp="false"/></Extensions><KeyDescriptor><ds:KeyInfo><ds:X509Data><ds:X509Certificate>MIID...SzY=</ds:X509Certificate></ds:X509Data></ds:KeyInfo></KeyDescriptor><AttributeService Binding="urn:oasis:names:tc:SAML:1.0:bindings:SOAP-binding" Location="https://bios-hp:8443/idp/profile/SAML1/SOAP/AttributeQuery"/><AttributeService Binding="urn:oasis:names:tc:SAML:2.0:bindings:SOAP" Location="https://bios-hp:8443/idp/profile/SAML2/SOAP/AttributeQuery"/><NameIDFormat>urn:mace:shibboleth:1.0:nameIdentifier</NameIDFormat><NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:transient</NameIDFormat></AttributeAuthorityDescriptor></EntityDescriptor>
Note, that entityID in both files is equal as its the common reason for my error.
The protected site is hosted on local IIS and is accessible via
http://bios-hp/secure
URL where I get the error.
All possible status URLs for both SP and IdP return successfull results.
Any ideas?
Don't know about your secured path, but it seems that your entityId is http://bios-hp/secure, with a http not https, so I believe your secured path would be a http and shibboleth works with https.
Common errors are:
EntityId typo: but that is not your case
metadata file path mistake: but that too is not your case
SP configuration in IdP side: you got to check that, too.
I'm working with a 10-node Infinispan cluster used as a Hibernate Search backend. Our servers are running TC server 2.5 (tomcat 6.0.32) on Java 1.6_24. We are using jGroups 2.12.1.3 for handling cluster cache writes from each node, and for multicast UDP transport.
When we launch 3+ nodes in our cluster, eventually one of the nodes begins to log replication timeouts. We've observed the same result whether we configure Infinispan for replication or for distribution cache modes. Although the rest of the cluster remains stable, the failing node becomes essentially unsuable for search.
Our configuration:
Infinispan:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.0 http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"
xmlns="urn:infinispan:config:5.0">
<global>
<globalJmxStatistics
enabled="true"
cacheManagerName="HibernateSearch"
allowDuplicateDomains="true" />
<transport
clusterName="HibernateSearch-Infinispan-cluster-MT"
distributedSyncTimeout="50000">
<properties>
<property name="configurationFile" value="infinispan-udp.cfg.xml" />
</properties>
</transport>
<shutdown
hookBehavior="DONT_REGISTER" />
</global>
<default>
<locking
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false" />
<storeAsBinary storeKeysAsBinary="false" storeValuesAsBinary="true"
enabled="false" />
<invocationBatching
enabled="true" />
<clustering
mode="replication">
<stateRetrieval
timeout="60000"
logFlushTimeout="65000"
fetchInMemoryState="true"
alwaysProvideInMemoryState="true" />
<sync
replTimeout="50000" />
<l1 enabled="false" />
</clustering>
<jmxStatistics
enabled="true" />
<eviction
maxEntries="-1"
strategy="NONE" />
<expiration
maxIdle="-1" />
</default>
<namedCache
name="LuceneIndexesMetadata">
<clustering
mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="30000" />
<sync
replTimeout="50000" />
<l1 enabled="false" />
</clustering>
<locking
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false" />
<loaders shared="true" preload="true">
<loader class="org.infinispan.loaders.jdbm.JdbmCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">
<properties>
<property name="location" value="/usr/local/tc/.index/metadata" />
</properties>
</loader>
</loaders>
</namedCache>
<namedCache
name="LuceneIndexesData">
<clustering
mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="30000" />
<sync
replTimeout="50000" />
<l1 enabled="false" />
</clustering>
<locking
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false" />
<loaders shared="true" preload="true">
<loader class="org.infinispan.loaders.jdbm.JdbmCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">
<properties>
<property name="location" value="/usr/local/tc/.index/data" />
</properties>
</loader>
</loaders>
</namedCache>
<namedCache
name="LuceneIndexesLocking">
<clustering
mode="replication">
<stateRetrieval
fetchInMemoryState="true"
logFlushTimeout="30000" />
<sync
replTimeout="50000" />
<l1 enabled="false" />
</clustering>
<locking
lockAcquisitionTimeout="20000"
writeSkewCheck="false"
concurrencyLevel="5000"
useLockStriping="false" />
</namedCache>
jGroups (UDP):
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-2.12.xsd">
<UDP
mcast_addr="${jgroups.udp.mcast_addr:228.10.10.9}"
mcast_port="${jgroups.udp.mcast_port:45599}"
tos="8"
ucast_recv_buf_size="20000000"
ucast_send_buf_size="640000"
mcast_recv_buf_size="25000000"
mcast_send_buf_size="640000"
loopback="true"
discard_incompatible_packets="true"
max_bundle_size="64000"
max_bundle_timeout="30"
ip_ttl="${jgroups.udp.ip_ttl:2}"
enable_bundling="true"
enable_diagnostics="false"
thread_naming_pattern="pl"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="30"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="false"
thread_pool.queue_max_size="100"
thread_pool.rejection_policy="Discard"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="2"
oob_thread_pool.max_threads="30"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="Discard"
/>
And the errors we observe:
10-31-2011 13:53:02 ERROR Hibernate Search: Directory writer-3 interceptors.InvocationContextInterceptor: ISPN000136: Execution error
org.infinispan.util.concurrent.TimeoutException: Replication timeout for tc-cluster-0105-21082
at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:71)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:452)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:132)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:156)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:265)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:252)
at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:235)
at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:228)
at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:116)
at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:79)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:294)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:214)
at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:162)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:114)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)
at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274)
at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:524)
at org.infinispan.CacheSupport.putIfAbsent(CacheSupport.java:74)
at org.infinispan.lucene.locking.BaseLuceneLock.obtain(BaseLuceneLock.java:65)
at org.apache.lucene.store.Lock.obtain(Lock.java:72)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1097)
at org.hibernate.search.backend.Workspace.createNewIndexWriter(Workspace.java:202)
at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:180)
at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:103)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Because this error is so pervasive regardless of our topology or caching mode, we believe we must be misconfigured somewhere. Can anyone recommend a fix?
Turns out we had a version clash between Infinispan and Hibernate Search. If you use Hibernate Search 3.4.1, you must use Infinispan 4.2.1, later versions may not work.