Extremely Slow Query Execution After Postgres 9.6 Upgrade - postgresql

I am seeing extremely slow processing times after a Postgres upgrade from 9.5 to 9.6 related to queries Osmosis (the command line Java application for processing OpenStreetMap data). Specifically syncing a local osm database with the master osm repo (Postgis 2.3 from 2.2). The syncing used to happen every day and took about 500 seconds on average. After the upgrade, after 48 hours, it is still not over. I notice very slow (below zero) object processing times (log record below) compared to before. I am fairly certain the problem is with the Postgres update. Can't figure out what to fix. pg_stat_activity gives the following query (currently):
UPDATE ways w
SET linestring = (
SELECT ST_MakeLine(c.geom) AS way_line
FROM (
SELECT n.geom AS geom
FROM nodes n
INNER JOIN way_nodes wn ON n.id = wn.node_id
WHERE (wn.way_id = w.id)
ORDER BY wn.sequence_id ) c )
WHERE w.id IN (
SELECT w.id
FROM ways w
INNER JOIN way_nodes wn ON w.id = wn.way_id
WHERE wn.node_id = $1
GROUP BY w.id )
Log record after the update:
Oct 31, 2017 12:23:59 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.45
Oct 31, 2017 12:24:00 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Oct 31, 2017 12:24:00 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Oct 31, 2017 12:24:00 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Oct 31, 2017 12:24:16 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 26022046 with action Modify, 0.06660006660006661 objects/second.
Oct 31, 2017 4:12:49 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 26022047 with action Modify, 7.292125918680253E-5 objects/second.
Oct 31, 2017 6:54:27 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 26022053 with action Modify, 1.0311411856040729E-4 objects/second.
Oct 31, 2017 9:39:22 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 26022056 with action Modify, 1.0106204077408672E-4 objects/second.
Nov 01, 2017 5:07:11 AM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 26022057 with action Modify, 3.72178939465691E-5 objects/second.
Nov 01, 2017 7:43:20 AM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 26022058 with action Modify, 1.0673723278600615E-4 objects/second.
Log record before the update:
Oct 22, 2017 11:00:01 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.45
Oct 22, 2017 11:00:02 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Oct 22, 2017 11:00:02 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Oct 22, 2017 11:00:02 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Oct 22, 2017 11:00:17 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 557521 with action Modify, 0.06700616456714018 objects/second.
Oct 22, 2017 11:00:22 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 49820266 with action Modify, 58.15347721822542 objects/second.
Oct 22, 2017 11:00:27 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 81804664 with action Modify, 40.64719810576164 objects/second.
Oct 22, 2017 11:00:32 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 109690032 with action Modify, 24.57577530719719 objects/second.
Oct 22, 2017 11:00:38 PM org.openstreetmap.osmosis.core.progress.v0_6.ChangeProgressLogger process
INFO: Processing Node 124684125 with action Modify, 58.35490938060148 objects/second.

FYI, I came across with exactly the same issue: I upgraded pg to 9.6 from 9.4 and queries ran much slower.
I had run only vacuum, nothing changed, but then I ran analyse as #harmic suggested and runs fine now (even faster, thanks to pg 9.6 improvements :) )!

After Postgres Upgrade you need to analyze the database for fast query plane.
To analyze the full database run the below query
ANALYZE VERBOSE --for all database analyze
To analyze only a specific table then use this query
ANALYZE VERBOSE [Table_name] --for particular table
query works in this Supported Versions: Current (13) / 12 / 11 / 10 / 9.6 / 9.5
reference: https://www.postgresql.org/docs/9.1/sql-analyze.html

Related

Tomcat not starting in eclipse - no timeout issue - PFB logs

Tomcat starting from command prompt but not from eclipse.
No problem shown in logs. logs shows server started successfully.
Tomcat staring from command prompt but not from eclipse.
No problem shown in logs. logs shows server started successfully.
Mar 23, 2019 1:53:07 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'source' to 'org.eclipse.jst.j2ee.server:spring-interceptor' did not find a matching property.
Mar 23, 2019 1:53:07 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Server version: Apache Tomcat/8.5.28
Mar 23, 2019 1:53:07 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.base=C:\FILES\Softwares\NewACFEclipse\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0
Mar 23, 2019 1:53:07 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.home=C:\Program Files (x86)\Apache Software Foundation\Tomcat 8.5
Mar 23, 2019 1:53:07 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dwtp.deploy=C:\FILES\Softwares\NewACFEclipse\workspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps
Mar 23, 2019 1:53:07 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djava.endorsed.dirs=C:\Program Files (x86)\Apache Software Foundation\Tomcat 8.5\endorsed
Mar 23, 2019 1:53:07 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dfile.encoding=Cp1252
Mar 23, 2019 1:53:07 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\Program Files (x86)\Java\jre1.8.0_161\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:/Program Files (x86)/Java/jre1.8.0_161/bin/client;C:/Program Files (x86)/Java/jre1.8.0_161/bin;C:/Program Files (x86)/Java/jre1.8.0_161/lib/i386;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\RSA SecurID Token Common;C:\Program Files\Java\jdk1.6.0_24\jre\bin\client;C:\oms\bb\cord9software\ant\bin;O:\ora1106w\BIN;C:\PROGRA~2\CA\SC\etpki\lib;C:\Program Files (x86)\CA\SharedComponents\PEC\bin;C:\Program Files\CA\SharedComponents\PEC\bin;c:\progra~1\ca\sc\etpki\lib;C:\Program Files\Java\jdk1.8.0_161\bin;C:\AmdocsProcessManager\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\wbem;C:\WINDOWS\system32\windowspowershell\v1.0\;c:\program files\provision networks\virtual access client\;c:\oms\hotdeployment;C:\Bea10.3.4\wlserver_10.3;C:\Program Files (x86)\CA\Cryptography\;C:\Program Files (x86)\CA\SCM;O:\ora11202w\jdbc\lib\ojdbc6.jar;C:\Program Files\Perforce;C:\Python27;C:\Python27\Scripts;C:\Python27\GnuWin32\bin;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jdk1.8.0_161\bin;C:\Program Files (x86)\Apache Software Foundation\apache-maven-3.6.0\bin;C:\Users\akashmu\AppData\Local\Microsoft\WindowsApps;;C:\FILES\Softwares\NewACFEclipse;;.]
Mar 23, 2019 1:53:08 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8080"]
Mar 23, 2019 1:53:08 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
INFO: Using a shared selector for servlet write/read
Mar 23, 2019 1:53:08 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-nio-8009"]
Mar 23, 2019 1:53:08 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
INFO: Using a shared selector for servlet write/read
Mar 23, 2019 1:53:08 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1216 ms
Mar 23, 2019 1:53:08 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Catalina]
Mar 23, 2019 1:53:08 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/8.5.28
Mar 23, 2019 1:53:19 PM org.apache.jasper.servlet.TldScanner scanJars
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Mar 23, 2019 1:53:19 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Mar 23, 2019 1:53:19 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-nio-8080"]
Mar 23, 2019 1:53:19 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-nio-8009"]
Mar 23, 2019 1:53:19 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 11105 ms
If you are staring the Tomcat server from the Java app then make sure it is attached to the app. Otherwise, start the server independently and the console logs will print the error if any.

jade web services integration gateway

I have gone through the following url for building webservices in jade http://jade.tilab.com/doc/tutorials/WSIG_Guide.pdf
Last page gives you instructions to run sample examples given by wsig....
When i tried running runMathAgent.bat ...I am getting the following output
Jul 15, 2016 12:57:55 PM jade.core.BaseService init
INFO: Service jade.core.management.AgentManagement initialized
Jul 15, 2016 12:57:55 PM jade.core.BaseService init
INFO: Service jade.core.messaging.Messaging initialized
Jul 15, 2016 12:57:55 PM jade.core.BaseService init
INFO: Service jade.core.resource.ResourceManagement initialized
Jul 15, 2016 12:57:55 PM jade.core.BaseService init
INFO: Service jade.core.mobility.AgentMobility initialized
Jul 15, 2016 12:57:55 PM jade.core.BaseService init
INFO: Service jade.core.event.Notification initialized
Jul 15, 2016 12:57:55 PM jade.core.AgentContainerImpl startBootstrapAgents
> SEVERE: Cannot create agent MathAgent1: Class
> com.tilab.wsig.examples.MathAgent for agent ( agent-identifier :name
> MathAgent1#WSIGTestPlatform ) not found - Caused by:
> com.tilab.wsig.examples.MathAgent
Jul 15, 2016 12:57:55 PM jade.core.AgentContainerImpl joinPlatform
INFO: --------------------------------------
Agent container Container-2#192.168.1.3 is ready.
--------------------------------------------
I finally figured solution for my problem. To run jade web services examples u need to deploy using ant (ant deploy-examples) and configuration includes juddi version of apache. This will solve the problem.
You need to run jade web services examples by deploying the exemples: using ant deploy-examples command.

Apache tomcat server startup increase over 30000 ms in eclipse Kepler

when i run my JSP page on apache tomcat 7 in my eclipse, server output shows up in 30000ms or more some times it never starts up, at first output got in lessthan 700ms,
Any help or suggession on this problem...
sample output...
NFO: Starting Servlet Engine: Apache Tomcat/7.0.12
Mar 18, 2014 10:42:39 PM org.apache.catalina.util.SessionIdGenerator createSecureRandom
INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [156] milliseconds.
Mar 18, 2014 10:42:58 PM org.apache.coyote.AbstractProtocolHandler start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Mar 18, 2014 10:42:58 PM org.apache.coyote.AbstractProtocolHandler start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Mar 18, 2014 10:42:58 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 30553 ms
Consider the following 2 suggestions:
1)As I commented above check if more then one application is running on the same server.This could be the primary reason.May be 3 or 4 applications might be running once you start your tomcat service.
2)May be some library that you are using for your application is creating some problem,for example the library might be initializing some file storage or may be something else.For this you will have to debug and check your stacktraces.

Menu items in Eclipse Juno do not work - unresponsive

I cannot seem to click on much inside of Eclipse Juno. I cannot run my projects, and even when I try to do something like Windows -> Preferences nothing pops up for Preferences, and I get no response. I have tried Refresh, Clean, and restarting both Eclipse and my system. Still nothing. I opened Eclipse via command line and got some output, but have no idea what it means. This is what it gave me:
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: Copy resource /javahg.py to /tmp/1346439710466-0/javahg.py
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: Command server started: /home/jordan/Workspaces/pos
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(22ms) status --modified --added --removed --deleted --unknown --clean
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: Random bytes generated: #CEa68#=[;bCQXECR%&^
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: Copy resource /styles/changesets.style to /tmp/1346439710466-0/styles/changesets.style
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: Copy resource /styles/changesets-eager.style to /tmp/1346439710466-0/styles/changesets-eager.style
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(5ms) parents --debug --style /tmp/1346439710466-0/styles/changesets.style
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(2ms) branch
Aug 31, 2012 3:01:50 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(3ms) resolve --list
Aug 31, 2012 3:01:52 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(19ms) parents --debug --style /tmp/1346439710466-0/styles/changesets.style
Aug 31, 2012 3:01:52 PM com.aragost.javahg.log.JULLogger info
INFO: Copy resource /styles/tags.style to /tmp/1346439710466-0/styles/tags.style
Aug 31, 2012 3:01:52 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(4ms) log --debug --style /tmp/1346439710466-0/styles/tags.style --rev 899668f6b2cdd3383d823a341c746a2a29e1f841
Aug 31, 2012 3:01:52 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(2ms) parents --debug --style /tmp/1346439710466-0/styles/changesets.style
Aug 31, 2012 3:01:53 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(10ms) parents --debug --style /tmp/1346439710466-0/styles/changesets.style
Aug 31, 2012 3:02:24 PM com.aragost.javahg.log.JULLogger info
INFO: Command server stopped: /home/jordan/Workspaces/pos
Aug 31, 2012 3:02:42 PM com.aragost.javahg.log.JULLogger info
INFO: Command server started: /home/jordan/Workspaces/pos
Aug 31, 2012 3:02:42 PM com.aragost.javahg.log.JULLogger info
INFO: runcommand(6ms) parents --debug --style /tmp/1346439710466-0/styles/changesets.style
Aug 31, 2012 3:03:12 PM com.aragost.javahg.log.JULLogger info
INFO: Command server stopped: /home/jordan/Workspaces/pos
Anyone know what is wrong with Eclipse and how I can get my programs running again?
Please open the 'Error Log' view to see what exception is happening when you attempt to view the preferences. Then you can enter a bug against or uninstall whatever plugin is causing the problem.

How do I run a web application using SpringSource Toolsuite?

I'd like to run my Spring application using STS, but I have no idea how to do that. I see the tc server instance, and I can run it, but this does not start my application. I've tried dragging my project workspace onto it, but that hasn't worked either. I've right-clicked and read/checked out all of the options on the context-sensitive menu, but I don't see anything to run my project inside tc server. Does anyone know how?
EDIT:
This is what I get when running the project. I see nothing about my project's war at all. It is included as a "web module" when I check out the tc server settings though. And it does have a web.xml and all of that other good stuff. The project works fine in maven's tomcat goal and inside IDEA. I am just having problems getting it to run in eclipse.
Mar 30, 2012 3:35:51 PM com.springsource.tcserver.security.PropertyDecoder <init>
INFO: tc Runtime property decoder using memory-based key
Mar 30, 2012 3:35:51 PM com.springsource.tcserver.security.PropertyDecoder <init>
INFO: tcServer Runtime property decoder has been initialized in 206 ms
Mar 30, 2012 3:35:52 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Mar 30, 2012 3:35:52 PM com.springsource.tcserver.serviceability.rmi.JmxSocketListener init
INFO: Started up JMX registry on 127.0.0.1:6969 in 65 ms
Mar 30, 2012 3:35:52 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 741 ms
Mar 30, 2012 3:35:52 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Mar 30, 2012 3:35:52 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: VMware vFabric tc Runtime 2.6.3.RELEASE/7.0.23.A.RELEASE
Mar 30, 2012 3:35:52 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor C:\Users\me\Documents\workspace-sts-2.9.0.RELEASE\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\conf\Catalina\localhost\trainingdividend.xml
Mar 30, 2012 3:35:52 PM org.apache.catalina.startup.SetContextPropertiesRule begin
WARNING: [SetContextPropertiesRule]{Context} Setting property 'source' to 'org.eclipse.jst.jee.server:trainingdividend' did not find a matching property.
Mar 30, 2012 3:35:52 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive C:\Users\me\Documents\workspace-sts-2.9.0.RELEASE\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\webapps\insight.war
Mar 30, 2012 3:35:52 PM com.springsource.insight.collection.tcserver.ltw.TomcatWeavingInsightClassLoader start
INFO: Context [localhost|insight] will not be woven
Mar 30, 2012 3:35:53 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Mar 30, 2012 3:36:03 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'Spring MVC Dispatcher Servlet'
Mar 30, 2012 3:36:05 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Mar 30, 2012 3:36:05 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 13468 ms
If you are missing the Run As > Run on Server option, then chances are your project does not have the Dynamic Web Module facet.
I was running into 404 errors in my J2ee project, created using Dynamic WebModules. Initially I thought it was a context problem, as the url that STS was attempting to run was quite wrong. However the project the context was correct. But needed to get rid of the errors shown in the markers tab (which may not stop attempts to run as Server) before I made any progress.
After resolving obvious code, jar,path errors, I then check if I can export a war file correctly. If not, it is usually an issue in the DEPLOYMENT ASSEMBLY portion of the projects properties (at least in my Java project). Specifying the class files and jars from required projects is needed to get the correct war file structure. Once accomplished, bye bye 404s