How to create a xml file per records in spark scala - scala
I have a file that has records like below
1_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745_App.xml|<CAudit><ai2aiinst nT="LevFcf#A0" auNdSTy="Analytics" auNdTy="Identifier" ndNo="1" aId="1" conDes="Levered Free Cash Flow" conCd="LevFcf" aiaGUId="1_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745" aiaId="1" aiKey="2990569588" aiId="14" pEndDt="2013-Dec-31" perCd="A" isYr2Dt="False" ><AudNode aId="1" ndNo="2" auNdTy="Operation" auNdSTy="-" nV="2626287569.000000000000000" ><AudNode aId="1" ndNo="3" auNdTy="Operation" auNdSTy="-" nV="2825849069.000000000000000" ><AudNode aId="1" ndNo="4" auNdTy="Identifier" auNdSTy="Standardized" nT="STD.SEBITDA#A0" nV="3130019939.000000000000000" ><ai2si nV="3130019939.00000" nT="STD.SEBITDA#A0" auNdSTy="Standardized" auNdTy="Identifier" ndNo="4" aId="1" inId="1035" conDes="Earnings before Interest, Taxes, Depreciation & Amortization (EBITDA)" conCd="SEBITDA" stdaGUId="841_107570667_STD_2C68EF2F-AB17-40EF-9095-387DE1D5D745" stdIaId="841" siKey="12004131416271429" siId="413" sLiCurIso="KRW" sCurIso="KRW" stCurIso="KRW" stTyCd="INC" sId="1" pEndDt="2013-Dec-31" pId="2" fId="192730348494" fbId="1" /></AudNode><AudNode aId="1" ndNo="5" auNdTy="Identifier" auNdSTy="Standardized" nT="STD.STAX#A0" nV="304170870.000000000000000" ><ai2si nV="304170870.00000" nT="STD.STAX#A0" auNdSTy="Standardized" auNdTy="Identifier" ndNo="5" aId="1" inId="968" conDes="Income Taxes" conCd="STAX" stdaGUId="807_107570667_STD_2C68EF2F-AB17-40EF-9095-387DE1D5D745" stdIaId="807" siKey="120038112041962629" siId="381" sLiCurIso="KRW" sCurIso="KRW" stCurIso="KRW" stTyCd="INC" sId="1" pEndDt="2013-Dec-31" pId="2" fId="192730348494" fbId="1" /></AudNode></AudNode><AudNode aId="1" ndNo="6" auNdTy="Operation" auNdSTy="SUM" nV="199561500.000000000000000" ><AudNode aId="1" ndNo="7" auNdTy="Identifier" auNdSTy="Standardized" nT="STD.SCEX#A0" nV="199561500.000000000000000" ><ai2si nV="199561500.00000" nT="STD.SCEX#A0" auNdSTy="Standardized" auNdTy="Identifier" ndNo="7" aId="1" inId="888" conDes="Capital Expenditures - Total" conCd="SCEX" stdaGUId="704_107570667_STD_2C68EF2F-AB17-40EF-9095-387DE1D5D745" stdIaId="704" siKey="12002771860094347" siId="277" sLiCurIso="KRW" sCurIso="KRW" stCurIso="KRW" stTyCd="CAS" sId="1" pEndDt="2013-Dec-31" pId="2" fId="192730348494" fbId="1" /></AudNode><AudNode aId="1" ndNo="8" auNdTy="Constant" nV="0.000000000000000" /></AudNode></AudNode></ai2aiinst></CAudit>
3_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745_App.xml|<CAudit><ai2aiinst nT="ExcessCashMargin#A0" auNdSTy="Analytics" auNdTy="Identifier" ndNo="1" aId="3" conDes="Excess Cash Margin - %" conCd="ExcessCashMargin" aiaGUId="3_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745" aiaId="3" aiKey="2990569579" aiId="5" pEndDt="2013-Dec-31" perCd="A" isYr2Dt="False" ><AudNode aId="3" ndNo="2" auNdTy="Operation" auNdSTy="*" nV="2.257160458878393" ><AudNode aId="3" ndNo="8" auNdTy="Identifier" auNdSTy="PseudoFinancialConcept" nT="PERCENTSCALE#A0" nV="100.000000000000000" /><AudNode aId="3" ndNo="3" auNdTy="Operation" auNdSTy="//" nV="0.022571604588784" ><AudNode aId="3" ndNo="7" auNdTy="Identifier" auNdSTy="Standardized" nT="STD.STLR#A0" nV="68201182151.000000000000000" ><ai2si nV="68201182151.00000" nT="STD.STLR#A0" auNdSTy="Standardized" auNdTy="Identifier" ndNo="7" aId="3" inId="990" conDes="Revenue from Business Activities - Total" conCd="STLR" stdaGUId="813_107570667_STD_2C68EF2F-AB17-40EF-9095-387DE1D5D745" stdIaId="813" siKey="12003871970759396" siId="387" sLiCurIso="KRW" sCurIso="KRW" stCurIso="KRW" stTyCd="INC" sId="1" pEndDt="2013-Dec-31" pId="2" fId="192730348494" fbId="1" /></AudNode><AudNode aId="3" ndNo="4" auNdTy="Operation" auNdSTy="-" nV="1539410116.000000000000000" ><AudNode aId="3" ndNo="6" auNdTy="Identifier" auNdSTy="Standardized" nT="STD.SNIC#A0" nV="438846856.000000000000000" ><ai2si nV="438846856.00000" nT="STD.SNIC#A0" auNdSTy="Standardized" auNdTy="Identifier" ndNo="6" aId="3" inId="1055" conDes="Net Income after Minority Interest" conCd="SNIC" stdaGUId="856_107570667_STD_2C68EF2F-AB17-40EF-9095-387DE1D5D745" stdIaId="856" siKey="120043012135950005" siId="430" sLiCurIso="KRW" sCurIso="KRW" stCurIso="KRW" stTyCd="INC" sId="1" pEndDt="2013-Dec-31" pId="2" fId="192730348494" fbId="1" /></AudNode><AudNode aId="3" ndNo="5" auNdTy="Identifier" auNdSTy="Standardized" nT="STD.STLO#A0" nV="1978256972.000000000000000" ><ai2si nV="1978256972.00000" nT="STD.STLO#A0" auNdSTy="Standardized" auNdTy="Identifier" ndNo="5" aId="3" inId="924" conDes="Net Cash Flow from Operating Activities" conCd="STLO" stdaGUId="719_107570667_STD_2C68EF2F-AB17-40EF-9095-387DE1D5D745" stdIaId="719" siKey="12002951348701451" siId="295" sLiCurIso="KRW" sCurIso="KRW" stCurIso="KRW" stTyCd="CAS" sId="1" pEndDt="2013-Dec-31" pId="2" fId="192730348494" fbId="1" /></AudNode></AudNode></AudNode></AudNode></ai2aiinst></CAudit>
5_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745_App.xml|<CAudit><ai2aiinst nT="Cf#A0" auNdSTy="Analytics" auNdTy="Identifier" ndNo="1" aId="5" conDes="Cash Flow" conCd="Cf" aiaGUId="5_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745" aiaId="5" aiKey="2990569577" aiId="3" pEndDt="2013-Dec-31" perCd="A" isYr2Dt="False" ><AudNode aId="5" ndNo="2" auNdTy="Operation" auNdSTy="-" nV="898935497.000000000000000" ><AudNode aId="5" ndNo="6" auNdTy="Constant" nV="0.000000000000000" /><AudNode aId="5" ndNo="3" auNdTy="Operation" auNdSTy="+" nV="898935497.000000000000000" ><AudNode aId="5" ndNo="5" auNdTy="Identifier" auNdSTy="Standardized" nT="STD.STDAE#A0" nV="460088641.000000000000000" ><ai2si nV="460088641.00000" nT="STD.STDAE#A0" auNdSTy="Standardized" auNdTy="Identifier" ndNo="5" aId="5" inId="956" conDes="Depreciation, Depletion & Amortization - Total" conCd="STDAE" stdaGUId="796_107570667_STD_2C68EF2F-AB17-40EF-9095-387DE1D5D745" stdIaId="796" siKey="120036611860540497" siId="366" sLiCurIso="KRW" sCurIso="KRW" stCurIso="KRW" stTyCd="INC" sId="1" pEndDt="2013-Dec-31" pId="2" fId="192730348494" fbId="1" /></AudNode><AudNode aId="5" ndNo="4" auNdTy="Identifier" auNdSTy="Standardized" nT="STD.SIAT#A0" nV="438846856.000000000000000" ><ai2si nV="438846856.00000" nT="STD.SIAT#A0" auNdSTy="Standardized" auNdTy="Identifier" ndNo="4" aId="5" inId="1018" conDes="Net Income after Tax" conCd="SIAT" stdaGUId="831_107570667_STD_2C68EF2F-AB17-40EF-9095-387DE1D5D745" stdIaId="831" siKey="120040511473155197" siId="405" sLiCurIso="KRW" sCurIso="KRW" stCurIso="KRW" stTyCd="INC" sId="1" pEndDt="2013-Dec-31" pId="2" fId="192730348494" fbId="1" /></AudNode></AudNode></AudNode></ai2aiinst></CAudit>
I need to make a xml file for each row .
The name of the xml file wouldl be the first column before |
So in this case i will have 3 xml file like below
1_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745_App.xml
3_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745_App.xml
5_107570667_ANA_2C68EF2F-AB17-40EF-9095-387DE1D5D745_App.xml
And each xml file will contain the record after |.
Like this i will have 500000 rows and i need to create xml file for each row .
First, you need to create a paired RDD containing file name and file content as tuple and then use that paired RDD to write the individual files to disk/hadoop.
You can have a look at the following code snippet:
val input = sparkSession.sparkContext.textFile("<your_input_file>")
val pairedRDD = input.map(row => {
val split = row.split("\\|")
val fileName = split(0)
val fileContent = split(1)
(fileName, fileContent)
})
import org.apache.hadoop.io.NullWritable
import org.apache.spark.HashPartitioner
import org.apache.hadoop.mapred.lib.MultipleTextOutputFormat
class RddMultiTextOutputFormat extends MultipleTextOutputFormat[Any, Any] {
override def generateActualKey(key: Any, value: Any): Any = NullWritable.get()
override def generateFileNameForKeyValue(key: Any, value: Any, name: String): String = key.asInstanceOf[String]
}
pairedRDD.partitionBy(new HashPartitioner(1000)).saveAsHadoopFile("<output_path>", classOf[String], classOf[String], classOf[RddMultiTextOutputFormat])
output:
Related
org.postgresql.util.PSQLException: Large Objects may not be used in auto-commit mode Error with JBPM 4.4
We are using Jbpm 4.4 as our 3rd party Business Process Management tool with Java 6.x. However So far we used it with Oracle DB and it worked well, but now we want to run it with PostgreSQL 12.x version DB. So we integrated postgresql-42.2.19.jre6.jar (JDBC driver) and try to run it. We have encountered below error in the run time. Can anyone suggest what need to be done here to resolve the issue, specially with JBPM 4.4 We have already set <prop key="hibernate.connection.autocommit">false</prop> But that did not resolved our issue. 2021-05-05 06:41:57,670 ERROR [o-8443-exec-154] .AbstractFlushingEventListener portaladmin#10.100.250.41 - Could not synchronize database state with session org.hibernate.exception.GenericJDBCException: could not insert: [org.jbpm.pvm.internal.lob.Lob] at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2295) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2688) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:79) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:167) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:64) [hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:996) [hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1141) [hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) [hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.jbpm.pvm.internal.query.AbstractQuery.execute(AbstractQuery.java:93) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.query.ProcessDefinitionQueryImpl.execute(ProcessDefinitionQueryImpl.java:67) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.query.AbstractQuery.untypedList(AbstractQuery.java:67) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.query.ProcessDefinitionQueryImpl.list(ProcessDefinitionQueryImpl.java:157) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.ProcessDeployer.checkKey(ProcessDeployer.java:133) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.ProcessDeployer.deploy(ProcessDeployer.java:92) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.DeployerManager.deploy(DeployerManager.java:46) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.RepositorySessionImpl.deploy(RepositorySessionImpl.java:62) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.cmd.DeployCmd.execute(DeployCmd.java:47) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.cmd.DeployCmd.execute(DeployCmd.java:33) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.DefaultCommandService.execute(DefaultCommandService.java:42) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.tx.SpringCommandCallback.doInTransaction(SpringCommandCallback.java:45) [jbpm-pvm-4.4.jar:4.4] at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) [spring-tx-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.jbpm.pvm.internal.tx.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:49) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.EnvironmentInterceptor.executeInNewEnvironment(EnvironmentInterceptor.java:53) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.EnvironmentInterceptor.execute(EnvironmentInterceptor.java:40) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.RetryInterceptor.execute(RetryInterceptor.java:56) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.SkipInterceptor.execute(SkipInterceptor.java:43) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.DeploymentImpl.deploy(DeploymentImpl.java:90) [jbpm-pvm-4.4.jar:4.4] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody18(JbpmProcessDefinitionRepository.java:108) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody19$advice(JbpmProcessDefinitionRepository.java:92) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody20(JbpmProcessDefinitionRepository.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody22(JbpmProcessDefinitionRepository.java:106) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody23$advice(JbpmProcessDefinitionRepository.java:80) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess(JbpmProcessDefinitionRepository.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody46(JbpmProcessService.java:178) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody47$advice(JbpmProcessService.java:92) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody48(JbpmProcessService.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody50(JbpmProcessService.java:178) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody51$advice(JbpmProcessService.java:80) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody52(JbpmProcessService.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody53$advice(JbpmProcessService.java:61) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess(JbpmProcessService.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody128(TaskController.java:611) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody129$advice(TaskController.java:58) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody130(TaskController.java:1) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody131$advice(TaskController.java:92) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody132(TaskController.java:1) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody134(TaskController.java:605) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody135$advice(TaskController.java:102) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody136(TaskController.java:1) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody137$advice(TaskController.java:55) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload(TaskController.java:1) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController$$FastClassByCGLIB$$2349406.invoke(<generated>) [cglib-nodep-2.1_3.jar:na] at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149) [cglib-nodep-2.1_3.jar:na] at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67) [spring-security-core-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at com.abc.def.portal.partner.client.task.TaskController$$EnhancerByCGLIB$$4f295537.handleFormUpload(<generated>) [cglib-nodep-2.1_3.jar:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_45] at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_45] at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) [servlet-api.jar:na] at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) [servlet-api.jar:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at com.abc.def.portal.partner.client.security.IncompleteUserProfileFilter.doFilterInternal_aroundBody4(IncompleteUserProfileFilter.java:108) [IncompleteUserProfileFilter.class:na] at com.abc.def.portal.partner.client.security.IncompleteUserProfileFilter.doFilterInternal(IncompleteUserProfileFilter.java:89) [IncompleteUserProfileFilter.class:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at com.abc.def.portal.ui.servlet.SsoRequestHeaderAuthenticationFilter.doFilter_aroundBody2(SsoRequestHeaderAuthenticationFilter.java:63) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.servlet.SsoRequestHeaderAuthenticationFilter.doFilter(SsoRequestHeaderAuthenticationFilter.java:58) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:109) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.partner.client.security.SSOAutoLoginFilter.doFilterInternal_aroundBody0(SSOAutoLoginFilter.java:67) [SSOAutoLoginFilter.class:na] at com.abc.def.portal.partner.client.security.SSOAutoLoginFilter.doFilterInternal(SSOAutoLoginFilter.java:63) [SSOAutoLoginFilter.class:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.ui.csrf.CsrfFilter.doFilterInternal_aroundBody0(CsrfFilter.java:86) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:57) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.ui.csrf.AjaxTimeoutFilter.doFilterInternal_aroundBody0(AjaxTimeoutFilter.java:45) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.csrf.AjaxTimeoutFilter.doFilterInternal(AjaxTimeoutFilter.java:31) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.ui.timing.TimingServletFilter.doFilter_aroundBody2(TimingServletFilter.java:71) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.timing.TimingServletFilter.doFilter(TimingServletFilter.java:63) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.ui.servlet.XFilter.doFilterInternal_aroundBody0(XFilter.java:56) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.servlet.XFilter.doFilterInternal_aroundBody1$advice(XFilter.java:64) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.servlet.XFilter.doFilterInternal(XFilter.java:51) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [catalina.jar:7.0.53] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [catalina.jar:7.0.53] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610) [catalina.jar:7.0.53] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [catalina.jar:7.0.53] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) [catalina.jar:7.0.53] at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) [catalina.jar:7.0.53] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.53] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [catalina.jar:7.0.53] at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) [tomcat-coyote.jar:7.0.53] at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) [tomcat-coyote.jar:7.0.53] at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) [tomcat-coyote.jar:7.0.53] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [na:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [na:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45] Caused by: org.postgresql.util.PSQLException: Large Objects may not be used in auto-commit mode. at org.postgresql.largeobject.LargeObjectManager.createLO(LargeObjectManager.java:284) ~[postgresql-42.2.19.jre6.jar:42.2.19.jre6] at org.postgresql.largeobject.LargeObjectManager.createLO(LargeObjectManager.java:272) ~[postgresql-42.2.19.jre6.jar:42.2.19.jre6] at org.postgresql.jdbc.PgPreparedStatement.createBlob(PgPreparedStatement.java:1159) ~[postgresql-42.2.19.jre6.jar:42.2.19.jre6] at org.postgresql.jdbc.PgPreparedStatement.setBlob(PgPreparedStatement.java:1200) ~[postgresql-42.2.19.jre6.jar:42.2.19.jre6] at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.setBlob(NewProxyPreparedStatement.java:495) ~[c3p0-0.9.1.2.jar:0.9.1.2] at org.hibernate.type.BlobType.set(BlobType.java:72) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.type.BlobType.nullSafeSet(BlobType.java:140) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate(AbstractEntityPersister.java:2025) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2271) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] ... 160 common frames omitted
Though jbpm 4.4 is a very old version (currently at 7.54), try to update your schema and use bytea type for postgresql large objects. If you're using JTA datasource, auto-commit setting is always true and it can not be changed. Try to change it to an xa-datasource
Solution: I was able to find out a solution for this. I have changed the DB column to Bytea type in PostgreSQL and change the JBPM 4.4 implementation to use byte[] over java.sql.Blob (in org.jbpm.pvm.internal.lob.Lob class).
is there any way get each item of list without using for loop?
is there any way get each item of list without using for loop ?
i.e. Given data
+----+---------+-------------+----------+-----------+
| id| date| revenue |con_dist_1| con_dist_2|
+----+---------+-------------+----------+-----------+
|3310|1/15/2018| 0.010680705| 6|0.019875458|
|3310|1/15/2018| 0.006628853| 4|0.816039063|
|3310|1/15/2018| 0.01378215| 4|0.082049528|
|3310|1/15/2018| 0.010680705| 6|0.019875458|
|3310|1/15/2018| 0.006628853| 4|0.816039063|
|3310|1/15/2018| 0.01378215| 4|0.082049528|
|3310|1/15/2018| 0.010680705| 6|0.019875458|
|3310|1/15/2018| 0.010680705| 6|0.019875458|
|3310|1/15/2018| 0.014933087| 5|0.034681906|
|3310|1/15/2018| 0.014448282| 3|0.082049528|
+----+---------+-------------+----------+-----------+
val col_list = Array("con_dist_1","con_dist_2")
val median_col_list = partitioned_data.stat.approxQuantile(col_list, Array(0.0,0.1,0.5),0.0)
val percentile_0 = 0;
val percentile_10 = 1;
val Q0 = median_col_list(col_list.indexOf("con_dist_1"))(percentile_0)
val Q10 =median_col_list(col_list.indexOf("con_dist_1"))(percentile_10)
Without looping col_list , is there any way calculate percentile_0 &
percentile_10 for each item in col_list , i mean parallelly ...using
map or something ???
I will answer "how do you calculate multiple (approximate) percentiles in one pass, for multiple columns"?
According to DataFrameStatFunctions docs, the signature
approxQuantile(cols: Array[String], probabilities: Array[Double], relativeError: Double): Array[Array[Double]]
has been available (only) since 2.2.0.
If you're using an older version of Spark, it won't have this signature, and it won't be as easy to do this computation.
This is an example from your data, using Spark 2.4.0.
val df = Seq((3310,"1/15/2018",0.010680705,6,0.019875458),(3310,"1/15/2018",0.006628853,4,0.816039063),(3310,"1/15/2018",0.01378215,4,0.082049528),(3310,"1/15/2018",0.010680705,6,0.019875458),(3310,"1/15/2018",0.006628853,4,0.816039063),(3310,"1/15/2018",0.01378215,4,0.082049528),(3310,"1/15/2018",0.010680705,6,0.019875458),(3310,"1/15/2018",0.010680705,6,0.019875458),(3310,"1/15/2018",0.014933087,5,0.034681906),(3310,"1/15/2018",0.014448282,3,0.082049528)).toDF("id","date","revenue","con_dist_1","con_dist_2")
df.stat.approxQuantile(Array("con_dist_1", "con_dist_2"), Array(0.1, 0.5), 0)
Output (first dimension is column and second is the requested percentile, so for example, the 10th percentile of con_dist_1 is 3.0):
Array[Array[Double]] = Array(Array(3.0, 4.0), Array(0.019875458, 0.034681906))
Create trendline for a plot of a set of disordered data?
I have a totally random set of data with around 500 inputs obtained from a physics experiment in excel format. I plotted it successfully but I am not able to create a trendline for my plot. I need to find the trendline for this data.
excel data can be found from this google drive link.
Alternatively, these are the first 2 columns of the Excel file:
0_x 0_y
0.321816 -82.594828
1.595041 -85.396122
2.868266 -82.085502
3.632201 -84.632133
4.905426 -83.868143
6.178651 -84.122807
7.197231 -85.905448
7.961166 -84.37747
8.470457 -85.141459
9.743682 -82.340165
10.252972 -84.886796
11.271552 -82.594828
13.054067 -84.886796
13.054067 -83.104154
14.327292 -84.632133
15.345872 -86.160112
16.364452 -84.886796
17.383032 -86.160112
18.401612 -83.868143
19.165547 -85.905448
20.438773 -82.849491
21.457353 -85.396122
22.221288 -83.104154
22.475933 -85.650785
23.494513 -85.650785
24.513093 -84.632133
25.277028 -85.141459
26.550253 -84.122807
27.314188 -85.141459
28.587413 -82.849491
29.860638 -84.886796
31.133864 -82.340165
31.897799 -84.886796
32.916379 -83.868143
33.934959 -84.37747
34.698894 -84.886796
35.717474 -84.122807
37.245344 -85.141459
38.263924 -83.104154
39.282504 -84.122807
40.301084 -81.576175
40.810374 -84.122807
42.0836 -82.085502
43.356825 -84.886796
44.12076 -85.396122
45.13934 -85.141459
45.903275 -86.669438
47.1765 -84.632133
47.940435 -85.396122
48.70437 -83.358817
49.977595 -85.396122
50.486885 -82.085502
51.25082 -84.632133
52.2694 -83.868143
52.524045 -83.868143
53.287981 -85.905448
55.070496 -84.37747
56.343721 -86.160112
57.107656 -84.122807
57.871591 -86.160112
58.635526 -83.61348
59.144816 -85.905448
60.163396 -84.122807
61.181976 -85.396122
62.709846 -86.160112
62.964491 -85.396122
64.492362 -86.669438
65.001652 -85.141459
65.765587 -85.905448
66.784167 -84.122807
68.566682 -86.160112
69.585262 -83.358817
70.349197 -85.650785
71.113132 -84.37747
72.386357 -84.632133
73.150292 -85.141459
73.914227 -83.868143
74.932807 -85.141459
75.951388 -82.594828
77.224613 -85.396122
78.752483 -82.340165
79.516418 -85.141459
80.280353 -84.122807
81.553578 -85.905448
82.317513 -85.650785
82.826803 -84.886796
83.845383 -85.905448
83.336093 -83.868143
81.808223 -85.905448
80.534998 -83.104154
79.516418 -85.905448
78.497838 -83.868143
76.969968 -85.396122
75.696743 -86.160112
74.678162 -85.396122
73.659582 -86.669438
72.641002 -85.141459
71.367777 -86.160112
69.585262 -83.61348
68.566682 -86.160112
67.548102 -84.122807
66.274877 -86.160112
65.765587 -85.650785
65.001652 -85.905448
63.728426 -86.414775
61.945911 -84.632133
60.927331 -86.160112
60.418041 -83.61348
59.144816 -85.905448
58.380881 -82.594828
57.362301 -84.886796
56.089076 -82.849491
55.325141 -85.141459
54.561206 -85.141459
53.797271 -83.61348
52.524045 -85.141459
51.505465 -83.358817
50.486885 -85.141459
49.21366 -83.104154
48.19508 -84.886796
47.431145 -82.340165
46.15792 -84.886796
45.13934 -84.37747
44.12076 -85.396122
42.847535 -85.141459
41.319664 -84.886796
40.301084 -85.396122
39.537149 -84.632133
38.009279 -85.905448
37.245344 -82.849491
35.972119 -85.396122
34.953539 -83.868143
34.189604 -85.141459
32.152444 -84.632133
31.388509 -84.632133
30.115283 -86.160112
29.351348 -83.61348
28.078123 -85.396122
27.568833 -82.594828
26.804898 -85.396122
25.786318 -82.085502
24.513093 -85.141459
23.494513 -85.141459
22.730578 -84.37747
21.711998 -84.632133
20.438773 -84.122807
19.674838 -84.632133
18.401612 -82.594828
17.383032 -84.886796
16.619097 -82.594828
15.855162 -84.122807
15.091227 -83.61348
13.818002 -84.122807
12.799422 -84.122807
11.780842 -84.122807
11.016907 -84.886796
9.743682 -82.849491
8.215811 -84.632133
7.706521 -81.830838
6.942586 -84.632133
5.414716 -83.868143
4.141491 -85.396122
3.122911 -85.396122
2.358976 -84.886796
1.085751 -85.905448
0.321816 -84.37747
-0.696764 -85.905448
-1.969989 -83.358817
-2.98857 -85.650785
-4.51644 -82.594828
-5.53502 -85.905448
-6.808245 -85.396122
-7.57218 -86.160112
-8.59076 -86.414775
-10.11863 -85.141459
-10.62792 -86.669438
-11.901145 -84.122807
-12.919725 -86.669438
-13.17437 -83.61348
-13.938306 -85.650785
-15.211531 -84.122807
-15.975466 -85.396122
-17.248691 -86.414775
-18.267271 -85.141459
-19.285851 -86.160112
-20.049786 -85.141459
-21.323011 -84.886796
-22.341591 -82.594828
-23.105526 -85.141459
-24.124106 -82.085502
-25.397332 -84.632133
-26.161267 -84.632133
-27.179847 -84.632133
-27.434492 -85.905448
-28.707717 -84.632133
-29.471652 -85.905448
-30.999522 -83.358817
-32.527392 -85.396122
-33.291327 -82.340165
-34.309907 -85.396122
-35.073842 -83.61348
-35.583132 -85.141459
-37.620293 -85.396122
-38.638873 -84.122807
-39.402808 -85.650785
-40.421388 -84.122807
-41.694613 -86.160112
-43.222483 -82.849491
-44.241063 -86.160112
-45.259643 -83.868143
-46.023578 -85.650785
-47.551449 -85.905448
-48.315384 -85.141459
-49.588609 -85.141459
-50.607189 -84.122807
-51.625769 -85.141459
-52.644349 -82.340165
-53.662929 -84.886796
-54.426864 -82.594828
-54.936154 -84.37747
-56.209379 -83.104154
-56.718669 -83.868143
-57.737249 -84.37747
-59.26512 -83.104154
-60.029055 -84.37747
-61.30228 -81.830838
-62.066215 -83.868143
-62.32086 -82.085502
-63.594085 -84.122807
-64.612665 -82.849491
-66.140535 -84.122807
-66.649825 -84.122807
-68.177695 -83.868143
-69.45092 -85.396122
-70.214856 -83.61348
-71.233436 -85.141459
-72.252016 -82.594828
-73.015951 -84.886796
-74.289176 -82.594828
-76.071691 -84.632133
-76.326336 -85.650785
-77.854206 -84.122807
-78.618141 -86.160112
-79.891366 -85.905448
-80.655301 -86.414775
-81.928527 -83.868143
-82.692462 -86.414775
-83.965687 -83.358817
-84.984267 -85.141459
-86.002847 -83.868143
-86.766782 -84.122807
-87.785362 -84.886796
-88.803942 -83.868143
-89.313232 -85.141459
-90.331812 -83.358817
-91.350392 -85.650785
-92.368973 -82.594828
-93.132908 -84.37747
-94.406133 -83.104154
-95.424713 -84.632133
-96.188648 -85.141459
-96.697938 -84.37747
-97.461873 -85.141459
-98.480453 -84.37747
-99.499033 -85.650785
-100.517613 -83.104154
-101.790838 -85.650785
-102.554773 -83.358817
-103.573354 -85.650785
-104.591934 -85.396122
-105.865159 -85.396122
-107.647674 -85.650785
-108.156964 -84.632133
-108.920899 -85.905448
-110.703414 -83.358817
-111.721994 -86.414775
-112.485929 -83.868143
-113.759154 -85.905448
-114.52309 -85.141459
-115.54167 -86.414775
-116.305605 -86.414775
-117.57883 -85.396122
-118.08812 -86.414775
-118.852055 -84.886796
-120.12528 -86.160112
-120.889215 -83.358817
-121.907795 -85.905448
-122.926375 -83.868143
-123.69031 -85.396122
-124.963535 -85.396122
-125.727471 -84.632133
-126.746051 -85.905448
-127.764631 -83.868143
-128.783211 -84.632133
-129.037856 -82.594828
-130.056436 -85.141459
-131.075016 -82.085502
-132.093596 -84.37747
-133.112176 -84.37747
-133.876111 -84.37747
-135.149336 -84.632133
-136.677207 -83.104154
-137.441142 -85.650785
-138.459722 -83.61348
-139.223657 -84.886796
-140.496882 -82.340165
-141.515462 -84.632133
-142.534042 -83.358817
-143.297977 -84.37747
-144.825847 -85.141459
-145.844427 -84.886796
-146.863007 -86.160112
-147.881588 -84.37747
-148.900168 -85.396122
-149.918748 -83.358817
-150.682683 -85.650785
-151.701263 -83.358817
-152.719843 -85.141459
-153.738423 -85.396122
-153.993068 -84.122807
-155.011648 -84.886796
-156.030228 -84.37747
-157.303453 -84.632133
-158.067388 -82.340165
-159.340614 -85.396122
-159.849904 -82.085502
-159.595259 -84.122807
-158.576678 -82.849491
-157.303453 -84.122807
-156.030228 -84.37747
-155.011648 -83.358817
-153.229133 -84.37747
-152.210553 -82.849491
-151.446618 -84.632133
-149.664103 -81.321512
-148.645523 -84.886796
-147.881588 -82.594828
-146.608362 -84.37747
-145.589782 -84.886796
-144.571202 -84.122807
-143.552622 -85.396122
-143.043332 -84.122807
-142.024752 -85.396122
-140.751527 -83.358817
-139.478302 -85.905448
-138.459722 -83.358817
-137.441142 -85.905448
-136.677207 -85.141459
-135.403981 -85.141459
-134.640046 -85.905448
-134.130756 -84.632133
-132.857531 -84.632133
-131.838951 -83.868143
-130.820371 -85.905448
-129.547146 -82.594828
-128.783211 -85.650785
-127.764631 -83.868143
-125.982116 -85.141459
-124.963535 -85.650785
-123.944955 -84.632133
-122.926375 -85.396122
-121.907795 -83.868143
-120.889215 -85.141459
-119.870635 -82.594828
-118.59741 -85.141459
-117.833475 -82.849491
-116.305605 -85.905448
-114.777735 -85.141459
-114.013799 -84.632133
-113.504509 -85.141459
-112.740574 -83.868143
-111.467349 -85.396122
-110.703414 -83.868143
-109.684834 -85.650785
-108.666254 -83.104154
-107.647674 -86.160112
-106.629094 -84.37747
-105.355869 -85.396122
-104.337289 -85.905448
-103.318709 -85.141459
-102.300128 -86.160112
-102.045483 -83.868143
-101.026903 -86.669438
-100.008323 -83.868143
-98.735098 -86.669438
-97.716518 -84.122807
-96.952583 -85.905448
-95.934003 -86.669438
-94.406133 -85.905448
-93.387553 -85.905448
-92.368973 -84.632133
-91.350392 -85.905448
-90.331812 -83.358817
-89.313232 -85.650785
-88.294652 -83.104154
-87.530717 -85.141459
-86.512137 -83.358817
-85.238912 -84.632133
-84.474977 -84.886796
-83.201752 -84.122807
-82.437817 -84.886796
-80.655301 -84.122807
-79.891366 -85.141459
-78.618141 -82.340165
-77.599561 -84.886796
-76.835626 -83.358817
-75.817046 -85.141459
-75.053111 -85.141459
-73.779886 -84.37747
-73.015951 -86.160112
-72.506661 -83.868143
-71.488081 -85.905448
-70.469501 -82.085502
-69.196275 -85.141459
-67.92305 -83.104154
-67.41376 -85.396122
-66.649825 -85.141459
-65.3766 -85.141459
-65.121955 -85.905448
-64.103375 -84.122807
-63.084795 -84.886796
-62.066215 -83.358817
-61.047635 -85.141459
-60.029055 -82.085502
-59.010475 -84.886796
-58.246539 -83.358817
-56.973314 -84.886796
-55.700089 -85.141459
-54.681509 -83.868143
-53.408284 -84.37747
-52.644349 -83.104154
-51.880414 -84.37747
-50.861834 -81.830838
-49.588609 -84.122807
-48.824674 -81.830838
-47.806094 -84.37747
-46.787513 -84.632133
-45.259643 -84.886796
-43.986418 -84.886796
-43.222483 -83.61348
-42.203903 -85.905448
-41.185323 -83.61348
-40.166743 -86.160112
-39.148163 -83.104154
-38.384228 -85.650785
-37.365648 -85.141459
-36.856358 -84.886796
-35.837777 -84.886796
-35.073842 -83.358817
-33.800617 -84.886796
-33.036682 -82.594828
-31.763457 -85.905448
-31.508812 -83.104154
-30.490232 -86.414775
-29.471652 -84.122807
-28.453072 -86.160112
-27.434492 -86.414775
-26.415912 -85.396122
-25.397332 -86.414775
-24.378751 -84.632133
-23.105526 -86.414775
-21.832301 -83.868143
-21.068366 -85.650785
-19.540496 -83.868143
-19.031206 -85.141459
-18.012626 -85.141459
-16.994046 -84.632133
-15.466176 -85.905448
-14.956886 -83.868143
-13.68366 -85.141459
-12.919725 -82.849491
-11.901145 -84.886796
-10.62792 -82.340165
-9.863985 -85.141459
-8.336115 -83.61348
-7.57218 -84.886796
-6.04431 -86.414775
-5.53502 -85.141459
-5.02573 -85.396122
-4.00715 -83.868143
-2.479279 -85.396122
-1.206054 -82.849491
-0.696764 -85.396122
-0.187474 -83.868143
1.085751 -86.160112
1.849686 -86.160112
3.122911 -85.905448
4.141491 -86.414775
4.905426 -84.632133
5.924006 -86.160112
6.178651 -83.358817
7.197231 -85.905448
8.215811 -82.594828
9.234392 -85.141459
10.252972 -84.632133
11.271552 -84.632133
13.054067 -85.396122
13.563357 -84.122807
14.836582 -85.141459
15.855162 -83.868143
I tried polyval, polyfit, Basic fitting (plot->tootls->basic fitting)
datasheet=xlsread('08_08_2019-11_08_58_.xlsx','Worksheet');
x=datasheet(:,1);
y=datasheet(:,2);
plot(x,y)
What exactly do you understand under trendline? A linear line p1*x+p2 can be achieved by the following code, which assumes that you have the curve fitting toolbox installed. The last part concerning polyfit() does not need this toolbox.
If you check these (fit) and these (fitoptions) docs, you'll find out that also other options, for quadratic polynomials, exponentials or fourier/sin-cos functions (and many more) are available when using the toolbox. The xand y of the data you posted looks like sinusoidal to me, so I'd go with the second option below.
Linear fit p1*x+p2
data = xlsread('08_08_2019-11_08_58_.xlsx');
x=data(:,1);
y=data(:,2);
[xData, yData] = prepareCurveData(x, y);
% Set up fittype and options.
ft = fittype('poly1')
% Fit model to data.
[fitresult, gof] = fit(xData, yData, ft);
% Plot fit with data.
figure
h = plot(fitresult, xData, yData);
legend(h, 'data', 'linear 1st order polynomial', 'Location', 'NorthEast');
xlabel('x'); ylabel('y'); grid on
% Extract the equation(ft) and the coefficients
coeffnames(fitresult)
coeffvalues(fitresult)
Using a Nonlinear Least Squares to fit a Fourier model a0 + a1*cos(x*w) + b1*sin(x*w)
data = xlsread('08_08_2019-11_08_58_.xlsx');
x=data(:,1);
y=data(:,2);
[xData, yData] = prepareCurveData(x, y);
% Set up fittype and options.
ft = fittype('fourier1')
opts = fitoptions('Method', 'NonlinearLeastSquares');
% Fit model to data.
[fitresult, gof] = fit(xData, yData, ft, opts);
% Plot fit with data.
figure
h = plot(fitresult, xData, yData);
legend(h, 'data', 'Fourier model fit', 'Location', 'NorthEast');
xlabel('x'); ylabel('y'); grid on
% Extract the equation(ft) and the coefficients
coeffnames(fitresult)
coeffvalues(fitresult)
Using the polyfit() and polyval() functions for p1*x+p2:
% using polyfit and polyval (no toolbox needed)
data = xlsread('08_08_2019-11_08_58_.xlsx');
x=data(:,1);
y=data(:,2);
p = polyfit(x, y, 1)
pp = polyval(p, x);
h = plot(x, y, 'o', x, pp);
legend(h, 'data', 'linear 1st order polynomial', 'Location', 'NorthEast');
xlabel('x'); ylabel('y'); grid on
Without Curve fitting toolbox, using your mentioned in-plot method
data = xlsread('08_08_2019-11_08_58_.xlsx');
x=data(:,1);
y=data(:,2);
plot(x, y)
In the figure window: Tools -> Basic Fitting -> linear/polynomial
also seems to work for your given case.
How to retreive records from the table based on input 'id' using Python?
I have table1 which is like ,
project_id | metric | design_target | median_property | design_project
------------+----------------------------------------+---------------+-----------------+----------------
300 | ENERGY STAR score (1-100) | Not Available | 50 | Not Available
300 | Source EUI (kBtu/ft\u00b2) | 35.4 | 141.4 | Not Available
300 | Site EUI (kBtu/ft\u00b2) | 15.8 | 63.1 | Not Available
300 | Source Energy Use (kBtu) | 3,536.0 | 14,144.1 | Not Available
300 | Site Energy Use (kBtu) | 1,578.7 | 6,314.9 | Not Available
300 | Energy Cost ($) | 34.61 | 138.44 | Not Available
300 | Total GHG Emissions (Metric Tons CO2e) | 0.2 | 0.6 | 0.0
400 | ENERGY STAR score (1-100) | Not Available | 50 | Not Available
400 | Source EUI (kBtu/ft\u00b2) | 35.4 | 141.4 | Not Available
400 | Site EUI (kBtu/ft\u00b2) | 15.8 | 63.1 | Not Available
400 | Source Energy Use (kBtu) | 3,536.0 | 14,144.1 | Not Available
400 | Site Energy Use (kBtu) | 1,578.7 | 6,314.9 | Not Available
400 | Energy Cost ($) | 34.61 | 138.44 | Not Available
400 | Total GHG Emissions (Metric Tons CO2e) | 0.2 | 0.6 | 0.0
I have table2 which is like,
project_id | energy_type | uses | target | title
------------+-------------------------------------------------------------------------------------------------------------------------+------+----------------------------------+------------------------------
300 | {"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"} | | Target % Better than Median: 75 | About this Property's Design
400 | {"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"} | | Target % Better than Median: 75 | About this Property's Design
I have table3 which is like,
project_id | your_design_score
------------+-------------------
300 | N/A
400 | N/A
I am joining table based project_id using python,
import psycopg2
con = psycopg2.connect(dbname="db",
user="postgres", host="localhost",
password="")
d1 = "select t1.project_id,t1.Metric, t1.Design_Target, t1.Median_Property, t1.Design_Project, t2.Energy_Type, t2.Uses, t2.Target, t2.Title, t3.your_design_score from metric_comparison t1, property_design t2, design_score t3 where t1.project_id = t2.project_id and t2.project_id = t3.project_id"
cursor.execute(d1)
d2 = cursor.fetchall()
print d2
con.commit()
I have got the output like ,
[('300', 'ENERGY STAR score (1-100)', 'Not Available', '50', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('300', 'Source EUI (kBtu/ft\\u00b2)', '35.4', '141.4', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('300', 'Site EUI (kBtu/ft\\u00b2)', '15.8', '63.1', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('300', 'Source Energy Use (kBtu)', '3,536.0', '14,144.1', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('300', 'Site Energy Use (kBtu)', '1,578.7', '6,314.9', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('300', 'Energy Cost ($)', '34.61', '138.44', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('300', 'Total GHG Emissions (Metric Tons CO2e)', '0.2', '0.6', '0.0', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('400', 'ENERGY STAR score (1-100)', 'Not Available', '50', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('400', 'Source EUI (kBtu/ft\\u00b2)', '35.4', '141.4', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('400', 'Site EUI (kBtu/ft\\u00b2)', '15.8', '63.1', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('400', 'Source Energy Use (kBtu)', '3,536.0', '14,144.1', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('400', 'Site Energy Use (kBtu)', '1,578.7', '6,314.9', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('400', 'Energy Cost ($)', '34.61', '138.44', 'Not Available', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A'), ('400', 'Total GHG Emissions (Metric Tons CO2e)', '0.2', '0.6', '0.0', '{"Energy Not Entered","Assumed Mix Based on State & Property Type:","","Electric - Grid (56.9%)","Natural Gas (43.1%)"}', None, ' Target % Better than Median: 75', "About this Property's Design", 'N/A')]
Now am trying to display the records based on project_id input,
PROJECT_ID = '300' #declared in python code
Based on the input of PROJECT_ID, i have to display the records from the above output.
I have to display the records where the project_id is matched to PROJECT_ID.
Filter at the SQL side passing the paremeter in the execute method: d1 = """ select project_id, t1.Metric, t1.Design_Target, t1.Median_Property, t1.Design_Project, t2.Energy_Type, t2.Uses, t2.Target, t2.Title, t3.your_design_score from metric_comparison t1 inner join property_design t2 using (project_id) inner join design_score t3 using (project_id) where project_id = %s """ project_id = '300' cursor.execute(d1, [project_id])
NTP Audit - failed adjtimex syscall?
As part of a new PCI-DSS server deployment I am in the process of configuring a fully auditable NTP time change history. All is working as expected however I am now seeing audit logs written every single second relating to time change operations. After a lot of searching I'm still no closer to understanding what is going on. The issue shows itself in /var/log/messages where an audit message is being written continuously. My research suggests that the syscall "exit=5" message means that the clock was not properly synchronised: adjtimex() syscall response "#define TIME_BAD 5 /* clock not synchronized */". So, in summary it appears that the clock is synced correctly (as far as my understanding takes me) however it is constantly changing - unexpected behaviour with the polling interval set at the default 64s. Is anyone able to offer suggestions? I've included as much detail as I can muster below: Audit time rules: [09:31] callum pci-fram-ipa1 ~ $ sudo cat /etc/audit/rules.d/audit_time_rules.rules -a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time-change -a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time-change -a always,exit -F arch=b64 -S clock_settime -k time-change -a always,exit -F arch=b32 -S clock_settime -k time-change -w /etc/localtime -p wa -k time-change System time vs clock time: [09:14] callum pci-fram-ipa1 ~ $ sudo clock;date Thu 05 Jan 2017 09:14:01 GMT -0.500708 seconds Thu 5 Jan 09:14:01 GMT 2017 Example audit output: [09:15] callum pci-fram-ipa1 ~ $ sudo tail -f /var/log/messages|grep time Jan 5 09:15:25 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607725.390:2328215): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:26 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607726.390:2328216): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:27 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607727.390:2328217): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:28 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607728.390:2328218): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:29 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607729.390:2328219): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:30 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607730.390:2328220): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:31 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607731.390:2328221): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:32 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607732.390:2328222): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:33 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607733.390:2328223): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Jan 5 09:15:34 pci-fram-ipa1 audispd: node=pci-fram-ipa1.x.net type=SYSCALL msg=audit(1483607734.390:2328224): arch=c000003e syscall=159 success=yes exit=5 a0=7ffe85ddc320 a1=7ffe85ddc410 a2=861 a3=2 items=0 ppid=1 pid=11479 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm="ntpd" exe="/usr/sbin/ntpd" subj=system_u:system_r:ntpd_t:s0 key="time-change" Sync stats: [09:15] callum pci-fram-ipa1 ~ $ sudo ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== *neon.trippett.o 131.188.3.221 2 u 112 256 377 17.924 -0.704 0.252 +uno.alvm.me 193.79.237.14 2 u 196 256 377 19.737 0.505 0.436 +greenore.zeip.e 140.203.204.77 2 u 165 256 377 19.616 0.019 0.252 +devrandom.pl 87.124.126.49 3 u 124 256 377 19.675 0.371 0.572 Additional info: [09:17] callum pci-fram-ipa1 ~ $ ntpdc -c sysinfo system peer: neon.trippett.org system peer mode: client leap indicator: 00 stratum: 3 precision: -23 root distance: 0.03258 s root dispersion: 0.04211 s reference ID: [178.62.6.103] reference time: dc188cec.d9ea15c5 Thu, Jan 5 2017 9:14:20.851 system flags: auth ntp stats jitter: 0.000320 s stability: 0.000 ppm broadcastdelay: 0.000000 s authdelay: 0.000000 s
This sounds like this could be expected behavior, based upon how often NTP skews a clock From NTP documentation: 5.1.3.2. How frequently will the System Clock be updated? As time should be a continuous and steady stream, ntpd updates the clock in small quantities. However, to keep up with clock errors, such corrections have to be applied frequently. If adjtime() is used, ntpd will update the system clock every second(I know this is not adjtimex, but adjtimex can function just as adjtime in the ADJ_OFFSET_SINGLESHOT mode: see adjtimex man page). If ntp_adjtime() is available, the operating system can compensate clock errors automatically, requiring only infrequent updates. See also Section 5.2 and Q: 5.1.6.1.. The polling interval has nothing to do with this though. It's instead how often the upstream(lower time stratum) time server is "queried" for reference. If the problem is that you're seeing the audit entries and you don't wish to see them for the ntp user - and you only want to see nefarious time skews, then follow the advice from this link, and exclude the ntp uid/auid. Also, from the adjtimex man page, it seems that the TIME_BAD error you see may not mean that the time was never correctly slewed: TIME_ERROR The system clock is not synchronized to a reliable server. This value is returned when any of the following holds true: * Either STA_UNSYNC or STA_CLOCKERR is set. * STA_PPSSIGNAL is clear and either STA_PPSFREQ or STA_PPSTIME is set. * STA_PPSTIME and STA_PPSJITTER are both set. * STA_PPSFREQ is set and either STA_PPSWANDER or STA_PPSJITTER is set. The symbolic name TIME_BAD is a synonym for TIME_ERROR, provided for backward compatibility.