The problem
We're facing the UT000047: The number of parameters exceeded the maximum of 1000 issue.
(We use WilfdFly 16).
What we tried, and worked, but is not acceptable
We followed the solution in UT000047, i.e. increase the max-parameters of the http-listener in the "standalone.xml" file:
<subsystem xmlns="urn:jboss:domain:undertow:8.0">
<server name="default-server">
<http-listener name="default" ... max-parameters="2000" />
This works: no UT000047 any more.
But this solution is rejected by our organization (since it implies to modify all the WF services of our organization, just to make 1 page of 1 application works).
What we'd prefer
We would prefer to increase the max-parameter attribute only for this application (even better: only for the HTML page).
We know that some application-specific configuration can be made through a "WEB-INF/jboss-deployment-structure.xml". Such a solution would be fine for us. But (AFAIK) the max-parameter cannot be defined in this file.
Related
Trying to investigate an issue regarding a keycloak deployment.
From the documentation we're seeing that we should alter this property in the stanealone.xml to allow more
However , after altering the standalone xml to this
</datasource>
<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" use-ccm="true">
<connection-url>jdbc:postgresql://${env.DB_ADDR:postgres}/${env.DB_DATABASE:keycloak}${env.JDBC_PARAMS:}</connection-url>
<driver>postgresql</driver>
<pool>
<max-pool-size>200</max-pool-size>
</pool>
When we allow the JMX connection to the management console I notice this
max pool size showing 19
Is there anything which could be overriding the max-pool-size setting we're using or how would one go about debugging where is derives the max-pool-size if not from the standalone.xml
I don't know why exactly you get this issue but I'll try to help to the best of my abilities.
Keycloak 11 uses Widlfy 20, which offers several options to configure the datasource.
Please try to set
pool-use-strict-min to true
pool-prefill to true
min-pool-size to 200
initial-pool-size to 200
Also, monitor the number of opened connection on your Postgres database, and check if they recoup with the JMX report you got.
Finally, if none of these cause a change in your situation I can only suggest to try another ManagedConnectionPool implementation (see mcp attribute)
I'm in the process of configuring a HA Wildfly cluster for session replication and I'm having trouble figuring out how to add JVM arguments to the individual servers in the domain. The arguments I add in domain.conf are applied to the process controller but not the individual servers. I can change the heap size for the server groups in domain.xml:
but I'm having trouble adding other arguments. Can I use this jvm section to add any argument? Is there another way to add arguments to server groups? Thanks.
You can use the jvm-options attribute on in CLI or you really want to edit the XML you can use <jvm-options/>.
CLI Example:
/server-group=main-server-group/jvm=default:write-attribute(name=jvm-options, value=["-XX:-HeapDumpOnOutOfMemoryError", "-XX:+UseCompressedOops"])
XML Example:
<server-group name="main-server-group" profile="full">
<jvm name="default">
<heap size="64m" max-size="512m"/>
<jvm-options>
<option value="-XX:-HeapDumpOnOutOfMemoryError"/>
<option value="-XX:+UseCompressedOops"/>
</jvm-options>
</jvm>
<socket-binding-group ref="full-sockets"/>
</server-group>
You can also define them at the server level if you need some setting only on a single server in the server group. See the host.xml for an example of that.
We are currently testing to move from Wildfly 8.2.0 to Wildfly 9.0.0.CR1 (or CR2 built from snapshot). The system is a cluster using mod_cluster and is running on VPS what in fact prevents it from using multicast.
On 8.2.0 we have been using the following configuration of the modcluster that works well:
<mod-cluster-config proxy-list="1.2.3.4:10001,1.2.3.5:10001" advertise="false" connector="ajp">
<dynamic-load-provider>
<load-metric type="cpu"/>
</dynamic-load-provider>
</mod-cluster-config>
Unfortunately, on 9.0.0 proxy-list was deprecated and the start of the server will finish with an error. There is a terrible lack of documentation, however after a couple of tries I have discovered that proxy-list was replaced with proxies that are a list of outbound-socket-bindings. Hence, the configuration looks like the following:
<mod-cluster-config proxies="mc-prox1 mc-prox2" advertise="false" connector="ajp">
<dynamic-load-provider>
<load-metric type="cpu"/>
</dynamic-load-provider>
</mod-cluster-config>
And the following should be added into the appropriate socket-binding-group (full-ha in my case):
<outbound-socket-binding name="mc-prox1">
<remote-destination host="1.2.3.4" port="10001"/>
</outbound-socket-binding>
<outbound-socket-binding name="mc-prox2">
<remote-destination host="1.2.3.5" port="10001"/>
</outbound-socket-binding>
So far so good. After this, the httpd cluster starts registering the nodes. However I am getting errors from load balancer. When I look into /mod_cluster-manager, I see a couple of Node REMOVED lines and there are also many many errors like:
ERROR [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000042: Error MEM sending STATUS command to node1/1.2.3.4:10001, configuration will be reset: MEM: Can't read node
In the log of mod_cluster there are the equivalent warnings:
manager_handler STATUS error: MEM: Can't read node
As far as I understand, the problem is that although wildfly/modcluster is able to connect to httpd/mod_cluster, it does not work the other way. Unfortunately, even after an extensive effort I am stuck.
Could someone help with setting mod_cluster for Wildfly 9.0.0 without advertising? Thanks a lot.
I ran into the Node Removed issue to.
I managed to solve it by using the following as instance-id
<subsystem xmlns="urn:jboss:domain:undertow:2.0" instance-id="${jboss.server.name}">
I hope this will help someone else to ;)
There is no need for any unnecessary effort or uneasiness about static proxy configuration. Each WildFly distribution comes with xsd sheets that describe xml subsystem configuration. For instance, with WildFly 9x, it's:
WILDFLY_DIRECTORY/docs/schema/jboss-as-mod-cluster_2_0.xsd
It says:
<xs:attribute name="proxies" use="optional">
<xs:annotation>
<xs:documentation>List of proxies for mod_cluster to register with defined by outbound-socket-binding in socket-binding-group.</xs:documentation>
</xs:annotation>
<xs:simpleType>
<xs:list itemType="xs:string"/>
</xs:simpleType>
</xs:attribute>
The following setup works out of box
Download wildfly-9.0.0.CR1.zip or build with ./build.sh from sources
Let's assume you have 2 boxes, Apache HTTP Server with mod_cluster acting as a load balancing proxy and your WildFly server acting as a worker. Make sure botch servers can access each other on both MCMP enabled VirtualHost's address and port (Apache HTTP Server side) and on WildFly AJP and HTTP connector side. The common mistake is to binf WildFLy to localhost; it then reports its addess as localhost to the Apache HTTP Server residing on a dofferent box, which makes it impossible for it to contact WildFly server back. The communication is bidirectional.
This is my configuration diff from the default wildfly-9.0.0.CR1.zip.
328c328
< <mod-cluster-config advertise-socket="modcluster" connector="ajp" advertise="false" proxies="my-proxy-one">
---
> <mod-cluster-config advertise-socket="modcluster" connector="ajp">
384c384
< <subsystem xmlns="urn:jboss:domain:undertow:2.0" instance-id="worker-1">
---
> <subsystem xmlns="urn:jboss:domain:undertow:2.0">
435c435
< <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:102}">
---
> <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
452,454d451
< <outbound-socket-binding name="my-proxy-one">
< <remote-destination host="10.10.2.4" port="6666"/>
< </outbound-socket-binding>
456c453
< </server>
---
> </server>
Changes explanation
proxies="my-proxy-one", outbound socket binding name; could be more of them here.
instance-id="worker-1", the name of the worker, a.k.a. JVMRoute.
offset -- you could ignore, it's just for my test setup. Offset does not apply to outbound socket bindings.
<outbound-socket-binding name="my-proxy-one"> - IP and port of the VirtualHost in Apache HTTP Server containing EnableMCPMReceive directive.
Conclusion
Generally, these MEM read / node error messages are related to network problems, e.g. WildFly can contact Apache, but Apache cannot contact WildFly back. Last but not least, it could happen that the Apache HTTP Server's configuration uses PersistSlots directive and some substantial enviroment conf change took place, e.g. switch from mpm_prefork to mpm_worker. In this case, MEM Read error messages are not realted to WildFly, but to the cached slotmem files in HTTPD/cache/mod_custer that need to be deleted.
I'm certain it's network in your case though.
After a couple of weeks I got back to the problem and found the solution. The problem was - of course - in configuration and had nothing in common with the particular version of Wildfly. Mode specifically:
There were three nodes in the domain and three servers in each node. All nodes were launched with the following property:
-Djboss.node.name=nodeX
...where nodeX is the name of a particular node. However, it meant that all three servers in the node get the same name, which is exactly what confused the load balancer.
As soon as I have removed this property, everything started to work.
I am setting up CruiseControl.NET and I get the following error message on the webdashboard:
No connection could be made because the target machine actively refused it 127.0.0.1:21234
The Url it is looking for is: tcp://localhost:21234/CruiseManager.rem
However the ccnet website in IIS has its tcp port set to 82.
So I use the following Url to navigate to the webdashboard http://127.0.0.1:82/ccnet/ViewFarmReport.aspx
I tried changing the Tcp port in IIS to 21234 and I get the following error message on the webdashboard:
Tcp channel protocol violation: expecting preamble.
I have also tried opening the port with the following command:
netsh firewall add portopening TCP 21234 CCNET
When I try and start the CCNET service I get the following message
The CruiseControl.NET Server service started then stopped. Some services stop automatically if they have no work to do....
Can anyone help me with this problem please?
EDIT - Adding config file
<cruisecontrol xmlns:cb="urn:ccnet.config.builder">
<cb:define PublishDir="C:\Deploy\Portal2.0Build"/>
<project name="Portal2.0">
<workingDirectory>C:\PortalCruiseControl\Working</workingDirectory>
<artifactDirectory>C:\PortalCruiseControl\Artifacts</artifactDirectory>
<webURL>http://192.168.17.59:82/ccnet</webURL>
<triggers>
<intervalTrigger name="continuous" seconds="10"
buildCondition="IfModificationExists"/>
</triggers>
<sourcecontrol type="svn">
<trunkUrl>https://portal2003.local:8443/svn/portalv2.0/trunk</trunkUrl>
<executable>C:\Program Files (x86)\VisualSVN Server\bin\svn.exe</executable>
<username>ccnet</username>
<password>***</password>
<cleanCopy>true</cleanCopy>
</sourcecontrol>
<tasks>
<msbuild>
<executable>
C:\WINDOWS\microsoft.net\Framework64\v3.5\MSBuild.exe
</executable>
<projectFile>Portal2.0.sln</projectFile>
<buildArgs>
/target:build;publish /p:Configuration=Release /p:MSBuildExtensionsPath=C:\Progra~2\MSBuild /p:MSBuildEmitSolution=1 /p:publishdir=C:\Deploy\Portal2.0Build /verbosity:diag
</buildArgs>
<logger>
C:\Program Files (x86)\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MSBuild.dll
</logger>
</msbuild>
</tasks>
<labeller type="assemblyVersionLabeller">
<major>2</major>
<minor>0</minor>
<incrementOnFailure>false</incrementOnFailure>
</labeller>
<publishers>
<statistics />
<xmllogger />
<package>
<name>ZipFilePublish</name>
<compression>9</compression>
<always>false</always>
<flatten>false</flatten>
<baseDirectory>$(PublishDir)</baseDirectory>
<dynamicValues>
<replacementValue property="name">
<format>C:\Deploy\Builds\PortalBuild{0}.zip</format>
<parameters>
<namedValue name="$CCNetLabel" value="Default" />
</parameters>
</replacementValue>
</dynamicValues>
<files>
<file>*.*</file>
<file>**\*</file>
</files>
</package>
<email from="bla" mailhost="bla" port="25" userName="bla"
password="bla" includeDetails="TRUE" useSSL="FALSE">
<users>
<user name="User1" group="Portal" address=""/>
</users>
<groups>
<group name="Portal">
<notifications>
<notificationType>change</notificationType>
</notifications>
</group>
</groups>
</email>
</publishers>
</project>
The first error message is probably caused by CCNET service not running because of which the web dashboard can't connect to it. It should go away as soon as you fix the ccnet.config so that service starts running.
The second problem ("Ilegal characters in path"; you seem to have already figured out the missing nodes part) is caused by msbuild/executable element. It seems that CC.NET doesn't like whitespace and especially new line characters inside it's value. Replacing:
<executable>
C:\WINDOWS\microsoft.net\Framework64\v3.5\MSBuild.exe
</executable>
with:
<executable>C:\WINDOWS\microsoft.net\Framework64\v3.5\MSBuild.exe</executable>
should fix the problem.
Another hint: when you're having problems with the validity of your ccnet.config file, try using CCValidator.exe (it's in your CruiseControl.NET\server folder). It usually points out the problematic part of the config file quite nicely (although that wasn't the case with "Illegal characters in path" problem - I had to comment out specific parts of the config to find the offending node).
The first message you receive (connection actively refused) makes me think of a firewall which is blocking the port you're using.
The second problem could be anything. It could for instance be an error in your XML configuration (ccnet.config) file. Can you find any pointers in the Windows Eventlog ?
Regarding the 2nd problem: did you try to run the CC.NET server from the command line?
If you've got an error in your XML configuration, this will give you a more meaningful error message.
Which account are you using to run the Windows service?
Have you checked your ccnet's dashboard.config file?
It has the following line in it:
<server name="local" url="tcp://localhost:21234/CruiseManager.rem" ... />
Try changing the port on that to 82 and then restarting the website (you should be just able to add a space to the web.config file and save and IIS will restart the website).
Sounds like you're confusing two different functions:
tcp://localhost:21234
This is the default remoting port for clients like CCTray. This is not used for the IIS web site (dashboard).
Configuration document is likely missing Xml nodes required for properly populating CruiseControl co nfiguration. Missing Xml node (packageList) for required member (ThoughtWorks.CruiseControl.Core.Publishers.Package Publisher.PackageList)
Your example config is missing required packageList node.
A misleading error message. The port really is 21234, not 82. I got the same errors. The fix was to start ccnet.exe from the desktop shortcut to discover that the real problem was illegal code in my ccnet.config file.
After fixing the ccnet.config file, the problem moved on. When attempting to build, the system would not let the subversion client modify the read-only marker files in the checked out repo.
In my case I misprinted project configuration file name in ccnet.config instead of timescheduler.config it were timesheduler. When I fixed file name I was able to run ccnet service.
<cruisecontrol xmlns:cb="urn:ccnet.config.builder">
<cb:include href="definitions.xml" xmlns:cb="urn:ccnet.config.builder"/>
<cb:include href="projects/timescheduler.config" xmlns:cb="urn:ccnet.config.builder"/>
</cruisecontrol>
I have a windows service that uses log4net. We noticed that the service in question was running painfully slow so we attached a debugger to it and stepped through. It appears that each time it tries to write an entry to the log via log4net that it takes anywhere from 10 to 30 seconds before the next line of code can execute. Obviously this adds up...
The service is 2.0 .net
We're using log4Net 1.2.0.30714.
We've tested this on a machine running vista and a machine running win sever 2003 and have seen the same or similar results.
Jeff mentioned a performance problem with Log4Net in Podcast 20. It's possible that you are seeing a similar issue.
It turned out that someone had added an SMPTAppender in a config file which was overriding the one in our app. As a result the errant SMPT server address was unreachable. log4net was trying to log the error for a minute per request and then giving up and going on to the next line of code. Correcting the smtp address fixed the problem.
I have log4net with adonet appender and have not seen any decremental performance of my windows service. what appender are you using?
Check your config file for Log4Net settings. Log4Net can be configured to log to a remote machine, and if the connection is slow, so will be your logging speed.
Well I'm not remoting... this is writing to the log file on the machine it's running on. Here's my appender settings:
<appender name="RollingFileAppender" type="log4net.Appender.RollingFileAppender,log4net">
<file value="D:\\ROPLogFiles\\FileProcessor.txt" />
<appendToFile value="true" />
<datePattern value="yyyyMMdd" />
<rollingStyle value="Date" />
<layout type="log4net.Layout.PatternLayout,log4net">
<param name="ConversionPattern" value="%d [%t] %-5p %c [%x] - %m%n" />
</layout>
<threshold value="INFO" />
</appender>
the default maximum file size is 10mb . if your files are about this size and your file systems is quite full and probably heavily fragmented, it may be possible that the problem lies there. how big are your log files? i encountered similar problems with logfiles at gigabyte size.