inverse distance weighting interpolation on scatter data in matlab - matlab
I want to use IDW interpolation technique on my data set. As usual consist on text files. Name Output_00.text to Output_23.text in a folder.
Each text file consist on three columns. Latitude, Longitude and Temperature values. 3rd Column(Temperature column) contain -9999.000 value encoded as missing or NaN value.
I want to interpolate these NaN/missing value in temperature through inverse distance weighting interpolation technique.
Here is what I am trying,but not idea how to use it
function[Vint]=IDW(xc,yc,vc,x,y,e,r1,r2)
%%% INPUTS
%xc = stations x coordinates (columns) [vector]
%yc = stations y coordinates (rows) [vector]
%vc = variable values on the point [xc yc]
%x = interpolation points x coordinates [vector]
%y = interpolation points y coordinates [vector]
%e = distance weight
%r1 --- 'fr' = fixed radius ; 'ng' = neighbours
%r2 --- radius lenght if r1 == 'fr' / number of neighbours if r1 =='ng'
%%% OUTPUTS
%Vint --- Matrix [length(y),length(x)] with interpolated variable values
%%% EXAMPLES
%%% --> V_spa=IDW(x1,y1,v1,x,y,-2,'ng',length(x1));
% Simone Fatichi -- simonef#dicea.unifi.it
% Copyright 2009
% $Date: 2009/06/19 $
% $Updated 2012/02/24 $
Vint=zeros(length(y),length(x));
xc=reshape(xc,1,length(xc));
yc=reshape(yc,1,length(yc));
vc=reshape(vc,1,length(vc));
if strcmp(r1,'fr')
if (r2<=0)
disp('Error: Radius must be positive')
return
end
for i=1:length(x)
for j=1:length(y)
D=[]; V=[]; wV =[]; vcc=[];
D= sqrt((x(i)-xc).^2 +(y(j)-yc).^2);
if min(D)==0
disp('Error: One or more stations have the coordinates of an interpolation point')
return
end
vcc=vc(D<r2); D=D(D<r2);
V = vcc.*(D.^e);
wV = D.^e;
if isempty(D)
V=NaN;
else
V=sum(V)/sum(wV);
end
Vint(j,i)=V;
end
end
else
if (r2 > length(vc)) || (r2<1)
disp('Error: Number of neighbours not congruent with data')
return
end
for i=1:length(x)
for j=1:length(y)
D=[]; V=[]; wV =[];vcc=[];
D= sqrt((x(i)-xc).^2 +(y(j)-yc).^2);
if min(D)==0
disp('Error: One or more stations have the coordinates of an interpolation point')
return
end
[D,I]=sort(D);
vcc=vc(I);
V = vcc(1:r2).*(D(1:r2).^e);
wV = D(1:r2).^e;
V=sum(V)/sum(wV);
Vint(j,i)=V;
end
end
end
return
I want to modify this code so that it read all text file step by step and interpolate NaN values with IDW interpolation technique and save each text file with IDW_00.text to IDW_23.text in the same folder.
My data set looks like this.
21.500 60.500 295.867
21.500 61.500 295.828
21.500 62.500 295.828
21.500 63.500 295.867
21.500 64.500 296.102
21.500 65.500 296.234
21.500 66.500 296.352
21.500 67.500 296.336
21.500 68.500 296.305
21.500 69.500 298.281
21.500 70.500 301.828
21.500 71.500 302.094
21.500 72.500 299.469
21.500 73.500 301.711
21.500 74.500 -9999.000
21.500 75.500 -9999.000
21.500 76.500 -9999.000
21.500 77.500 -9999.000
21.500 78.500 -9999.000
22.500 60.500 295.477
22.500 61.500 295.484
22.500 62.500 295.516
22.500 63.500 295.547
22.500 64.500 295.852
22.500 65.500 295.859
22.500 66.500 295.852
22.500 67.500 295.711
22.500 68.500 295.969
22.500 69.500 298.562
22.500 70.500 300.828
22.500 71.500 302.352
22.500 72.500 300.570
22.500 73.500 301.383
22.500 74.500 -9999.000
22.500 75.500 -9999.000
22.500 76.500 -9999.000
22.500 77.500 -9999.000
22.500 78.500 -9999.000
23.500 60.500 294.906
23.500 61.500 294.898
23.500 62.500 295.000
23.500 63.500 295.078
23.500 64.500 295.297
23.500 65.500 295.359
23.500 66.500 295.297
23.500 67.500 295.312
23.500 68.500 296.664
23.500 69.500 298.781
23.500 70.500 299.211
23.500 71.500 300.109
23.500 72.500 301.000
23.500 73.500 301.594
23.500 74.500 302.000
23.500 75.500 -9999.000
23.500 76.500 -9999.000
23.500 77.500 -9999.000
23.500 78.500 -9999.000
24.500 60.500 294.578
24.500 61.500 294.516
24.500 62.500 294.734
24.500 63.500 294.789
24.500 64.500 294.844
24.500 65.500 294.562
24.500 66.500 294.734
24.500 67.500 296.367
24.500 68.500 297.438
24.500 69.500 298.531
24.500 70.500 298.453
24.500 71.500 299.195
24.500 72.500 300.062
24.500 73.500 -9999.000
24.500 74.500 -9999.000
24.500 75.500 -9999.000
24.500 76.500 -9999.000
24.500 77.500 -9999.000
24.500 78.500 -9999.000
25.500 60.500 296.258
25.500 61.500 296.391
25.500 62.500 296.672
25.500 63.500 296.398
25.500 64.500 295.773
25.500 65.500 295.812
25.500 66.500 296.609
25.500 67.500 297.977
25.500 68.500 297.109
25.500 69.500 297.828
25.500 70.500 298.430
25.500 71.500 298.836
25.500 72.500 298.703
25.500 73.500 -9999.000
25.500 74.500 -9999.000
25.500 75.500 -9999.000
25.500 76.500 -9999.000
25.500 77.500 -9999.000
25.500 78.500 299.023
26.500 60.500 -9999.000
26.500 61.500 298.266
26.500 62.500 296.773
26.500 63.500 -9999.000
26.500 64.500 -9999.000
26.500 65.500 -9999.000
26.500 66.500 297.250
26.500 67.500 296.188
26.500 68.500 295.938
26.500 69.500 296.906
26.500 70.500 297.828
26.500 71.500 299.312
26.500 72.500 299.359
26.500 73.500 -9999.000
26.500 74.500 -9999.000
26.500 75.500 -9999.000
26.500 76.500 -9999.000
26.500 77.500 298.875
26.500 78.500 296.773
27.500 60.500 -9999.000
27.500 61.500 -9999.000
27.500 62.500 -9999.000
27.500 63.500 -9999.000
27.500 64.500 -9999.000
27.500 65.500 -9999.000
27.500 66.500 -9999.000
27.500 67.500 295.352
27.500 68.500 295.148
27.500 69.500 295.750
27.500 70.500 295.750
27.500 71.500 296.070
27.500 72.500 295.227
27.500 73.500 -9999.000
27.500 74.500 -9999.000
27.500 75.500 -9999.000
27.500 76.500 -9999.000
27.500 77.500 -9999.000
27.500 78.500 296.609
28.500 60.500 -9999.000
28.500 61.500 -9999.000
28.500 62.500 -9999.000
28.500 63.500 -9999.000
28.500 64.500 -9999.000
28.500 65.500 -9999.000
28.500 66.500 -9999.000
28.500 67.500 295.773
28.500 68.500 295.375
28.500 69.500 295.438
28.500 70.500 294.664
28.500 71.500 294.906
28.500 72.500 294.812
28.500 73.500 295.805
28.500 74.500 -9999.000
28.500 75.500 -9999.000
28.500 76.500 -9999.000
28.500 77.500 -9999.000
28.500 78.500 -9999.000
29.500 60.500 -9999.000
29.500 61.500 -9999.000
29.500 62.500 -9999.000
29.500 63.500 -9999.000
29.500 64.500 -9999.000
29.500 65.500 -9999.000
29.500 66.500 -9999.000
29.500 67.500 295.719
29.500 68.500 296.797
29.500 69.500 293.375
29.500 70.500 294.305
29.500 71.500 294.070
29.500 72.500 293.750
29.500 73.500 295.539
29.500 74.500 -9999.000
29.500 75.500 -9999.000
29.500 76.500 -9999.000
29.500 77.500 -9999.000
29.500 78.500 -9999.000
30.500 60.500 -9999.000
30.500 61.500 -9999.000
30.500 62.500 -9999.000
30.500 63.500 -9999.000
30.500 64.500 -9999.000
30.500 65.500 -9999.000
30.500 66.500 -9999.000
30.500 67.500 -9999.000
30.500 68.500 -9999.000
30.500 69.500 -9999.000
30.500 70.500 293.320
30.500 71.500 292.930
30.500 72.500 293.570
30.500 73.500 294.648
30.500 74.500 295.383
30.500 75.500 -9999.000
30.500 76.500 -9999.000
30.500 77.500 -9999.000
30.500 78.500 -9999.000
31.500 60.500 -9999.000
31.500 61.500 -9999.000
31.500 62.500 -9999.000
31.500 63.500 -9999.000
31.500 64.500 -9999.000
31.500 65.500 -9999.000
31.500 66.500 -9999.000
31.500 67.500 -9999.000
31.500 68.500 -9999.000
31.500 69.500 -9999.000
31.500 70.500 293.992
31.500 71.500 293.422
31.500 72.500 294.438
31.500 73.500 294.141
31.500 74.500 -9999.000
31.500 75.500 -9999.000
31.500 76.500 -9999.000
31.500 77.500 -9999.000
31.500 78.500 -9999.000
32.500 60.500 -9999.000
32.500 61.500 -9999.000
32.500 62.500 -9999.000
32.500 63.500 -9999.000
32.500 64.500 -9999.000
32.500 65.500 -9999.000
32.500 66.500 -9999.000
32.500 67.500 -9999.000
32.500 68.500 -9999.000
32.500 69.500 -9999.000
32.500 70.500 -9999.000
32.500 71.500 294.312
32.500 72.500 294.812
32.500 73.500 -9999.000
32.500 74.500 -9999.000
32.500 75.500 -9999.000
32.500 76.500 -9999.000
32.500 77.500 -9999.000
32.500 78.500 -9999.000
33.500 60.500 -9999.000
33.500 61.500 -9999.000
33.500 62.500 -9999.000
33.500 63.500 -9999.000
33.500 64.500 -9999.000
33.500 65.500 -9999.000
33.500 66.500 -9999.000
33.500 67.500 -9999.000
33.500 68.500 -9999.000
33.500 69.500 -9999.000
33.500 70.500 -9999.000
33.500 71.500 -9999.000
33.500 72.500 -9999.000
33.500 73.500 -9999.000
33.500 74.500 -9999.000
33.500 75.500 -9999.000
33.500 76.500 -9999.000
33.500 77.500 -9999.000
33.500 78.500 -9999.000
34.500 60.500 -9999.000
34.500 61.500 -9999.000
34.500 62.500 -9999.000
34.500 63.500 -9999.000
34.500 64.500 -9999.000
34.500 65.500 -9999.000
34.500 66.500 -9999.000
34.500 67.500 -9999.000
34.500 68.500 -9999.000
34.500 69.500 -9999.000
34.500 70.500 -9999.000
34.500 71.500 -9999.000
34.500 72.500 -9999.000
34.500 73.500 -9999.000
34.500 74.500 -9999.000
34.500 75.500 -9999.000
34.500 76.500 -9999.000
34.500 77.500 -9999.000
34.500 78.500 -9999.000
35.500 60.500 -9999.000
35.500 61.500 -9999.000
35.500 62.500 -9999.000
35.500 63.500 -9999.000
35.500 64.500 -9999.000
35.500 65.500 -9999.000
35.500 66.500 -9999.000
35.500 67.500 -9999.000
35.500 68.500 -9999.000
35.500 69.500 -9999.000
35.500 70.500 -9999.000
35.500 71.500 -9999.000
35.500 72.500 -9999.000
35.500 73.500 -9999.000
35.500 74.500 -9999.000
35.500 75.500 -9999.000
35.500 76.500 -9999.000
35.500 77.500 -9999.000
35.500 78.500 -9999.000
36.500 60.500 276.742
36.500 61.500 274.406
36.500 62.500 -9999.000
36.500 63.500 -9999.000
36.500 64.500 -9999.000
36.500 65.500 272.219
36.500 66.500 273.023
36.500 67.500 275.875
36.500 68.500 -9999.000
36.500 69.500 -9999.000
36.500 70.500 -9999.000
36.500 71.500 -9999.000
36.500 72.500 -9999.000
36.500 73.500 -9999.000
36.500 74.500 -9999.000
36.500 75.500 -9999.000
36.500 76.500 -9999.000
36.500 77.500 -9999.000
36.500 78.500 -9999.000
37.500 60.500 277.406
37.500 61.500 277.547
37.500 62.500 276.375
37.500 63.500 275.484
37.500 64.500 276.820
37.500 65.500 275.312
37.500 66.500 274.875
37.500 67.500 275.875
37.500 68.500 -9999.000
37.500 69.500 -9999.000
37.500 70.500 -9999.000
37.500 71.500 -9999.000
37.500 72.500 -9999.000
37.500 73.500 -9999.000
37.500 74.500 -9999.000
37.500 75.500 -9999.000
37.500 76.500 -9999.000
37.500 77.500 -9999.000
37.500 78.500 -9999.000
Please help and a lot of thanks for this kind assistance
Related
org.postgresql.util.PSQLException: Large Objects may not be used in auto-commit mode Error with JBPM 4.4
We are using Jbpm 4.4 as our 3rd party Business Process Management tool with Java 6.x. However So far we used it with Oracle DB and it worked well, but now we want to run it with PostgreSQL 12.x version DB. So we integrated postgresql-42.2.19.jre6.jar (JDBC driver) and try to run it. We have encountered below error in the run time. Can anyone suggest what need to be done here to resolve the issue, specially with JBPM 4.4 We have already set <prop key="hibernate.connection.autocommit">false</prop> But that did not resolved our issue. 2021-05-05 06:41:57,670 ERROR [o-8443-exec-154] .AbstractFlushingEventListener portaladmin#10.100.250.41 - Could not synchronize database state with session org.hibernate.exception.GenericJDBCException: could not insert: [org.jbpm.pvm.internal.lob.Lob] at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2295) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2688) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:79) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:167) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:64) [hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:996) [hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1141) [hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) [hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.jbpm.pvm.internal.query.AbstractQuery.execute(AbstractQuery.java:93) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.query.ProcessDefinitionQueryImpl.execute(ProcessDefinitionQueryImpl.java:67) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.query.AbstractQuery.untypedList(AbstractQuery.java:67) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.query.ProcessDefinitionQueryImpl.list(ProcessDefinitionQueryImpl.java:157) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.ProcessDeployer.checkKey(ProcessDeployer.java:133) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.ProcessDeployer.deploy(ProcessDeployer.java:92) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.DeployerManager.deploy(DeployerManager.java:46) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.RepositorySessionImpl.deploy(RepositorySessionImpl.java:62) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.cmd.DeployCmd.execute(DeployCmd.java:47) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.cmd.DeployCmd.execute(DeployCmd.java:33) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.DefaultCommandService.execute(DefaultCommandService.java:42) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.tx.SpringCommandCallback.doInTransaction(SpringCommandCallback.java:45) [jbpm-pvm-4.4.jar:4.4] at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) [spring-tx-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.jbpm.pvm.internal.tx.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:49) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.EnvironmentInterceptor.executeInNewEnvironment(EnvironmentInterceptor.java:53) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.EnvironmentInterceptor.execute(EnvironmentInterceptor.java:40) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.RetryInterceptor.execute(RetryInterceptor.java:56) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.svc.SkipInterceptor.execute(SkipInterceptor.java:43) [jbpm-pvm-4.4.jar:4.4] at org.jbpm.pvm.internal.repository.DeploymentImpl.deploy(DeploymentImpl.java:90) [jbpm-pvm-4.4.jar:4.4] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody18(JbpmProcessDefinitionRepository.java:108) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody19$advice(JbpmProcessDefinitionRepository.java:92) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody20(JbpmProcessDefinitionRepository.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody22(JbpmProcessDefinitionRepository.java:106) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess_aroundBody23$advice(JbpmProcessDefinitionRepository.java:80) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessDefinitionRepository.deployProcess(JbpmProcessDefinitionRepository.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody46(JbpmProcessService.java:178) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody47$advice(JbpmProcessService.java:92) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody48(JbpmProcessService.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody50(JbpmProcessService.java:178) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody51$advice(JbpmProcessService.java:80) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody52(JbpmProcessService.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess_aroundBody53$advice(JbpmProcessService.java:61) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.processes.jbpm.JbpmProcessService.deployProcess(JbpmProcessService.java:1) [com.abc.def.portal.processes-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody128(TaskController.java:611) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody129$advice(TaskController.java:58) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody130(TaskController.java:1) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody131$advice(TaskController.java:92) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody132(TaskController.java:1) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody134(TaskController.java:605) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody135$advice(TaskController.java:102) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody136(TaskController.java:1) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload_aroundBody137$advice(TaskController.java:55) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController.handleFormUpload(TaskController.java:1) [TaskController.class:na] at com.abc.def.portal.partner.client.task.TaskController$$FastClassByCGLIB$$2349406.invoke(<generated>) [cglib-nodep-2.1_3.jar:na] at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149) [cglib-nodep-2.1_3.jar:na] at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67) [spring-security-core-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) [spring-aop-3.1.2.RELEASE.jar:3.1.2.RELEASE] at com.abc.def.portal.partner.client.task.TaskController$$EnhancerByCGLIB$$4f295537.handleFormUpload(<generated>) [cglib-nodep-2.1_3.jar:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_45] at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_45] at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) [spring-webmvc-3.1.2.RELEASE.jar:3.1.2.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) [servlet-api.jar:na] at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) [servlet-api.jar:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at com.abc.def.portal.partner.client.security.IncompleteUserProfileFilter.doFilterInternal_aroundBody4(IncompleteUserProfileFilter.java:108) [IncompleteUserProfileFilter.class:na] at com.abc.def.portal.partner.client.security.IncompleteUserProfileFilter.doFilterInternal(IncompleteUserProfileFilter.java:89) [IncompleteUserProfileFilter.class:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at com.abc.def.portal.ui.servlet.SsoRequestHeaderAuthenticationFilter.doFilter_aroundBody2(SsoRequestHeaderAuthenticationFilter.java:63) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.servlet.SsoRequestHeaderAuthenticationFilter.doFilter(SsoRequestHeaderAuthenticationFilter.java:58) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:109) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.partner.client.security.SSOAutoLoginFilter.doFilterInternal_aroundBody0(SSOAutoLoginFilter.java:67) [SSOAutoLoginFilter.class:na] at com.abc.def.portal.partner.client.security.SSOAutoLoginFilter.doFilterInternal(SSOAutoLoginFilter.java:63) [SSOAutoLoginFilter.class:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.ui.csrf.CsrfFilter.doFilterInternal_aroundBody0(CsrfFilter.java:86) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:57) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.ui.csrf.AjaxTimeoutFilter.doFilterInternal_aroundBody0(AjaxTimeoutFilter.java:45) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.csrf.AjaxTimeoutFilter.doFilterInternal(AjaxTimeoutFilter.java:31) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.ui.timing.TimingServletFilter.doFilter_aroundBody2(TimingServletFilter.java:71) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.timing.TimingServletFilter.doFilter(TimingServletFilter.java:63) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at com.abc.def.portal.ui.servlet.XFilter.doFilterInternal_aroundBody0(XFilter.java:56) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.servlet.XFilter.doFilterInternal_aroundBody1$advice(XFilter.java:64) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at com.abc.def.portal.ui.servlet.XFilter.doFilterInternal(XFilter.java:51) [com.abc.def.portal.ui-2.1.NOPSE19C.1.jar:na] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) [spring-web-3.1.2.RELEASE.jar:3.1.2.RELEASE] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.53] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.53] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [catalina.jar:7.0.53] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [catalina.jar:7.0.53] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610) [catalina.jar:7.0.53] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [catalina.jar:7.0.53] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) [catalina.jar:7.0.53] at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) [catalina.jar:7.0.53] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.53] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [catalina.jar:7.0.53] at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) [tomcat-coyote.jar:7.0.53] at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) [tomcat-coyote.jar:7.0.53] at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) [tomcat-coyote.jar:7.0.53] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [na:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [na:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45] Caused by: org.postgresql.util.PSQLException: Large Objects may not be used in auto-commit mode. at org.postgresql.largeobject.LargeObjectManager.createLO(LargeObjectManager.java:284) ~[postgresql-42.2.19.jre6.jar:42.2.19.jre6] at org.postgresql.largeobject.LargeObjectManager.createLO(LargeObjectManager.java:272) ~[postgresql-42.2.19.jre6.jar:42.2.19.jre6] at org.postgresql.jdbc.PgPreparedStatement.createBlob(PgPreparedStatement.java:1159) ~[postgresql-42.2.19.jre6.jar:42.2.19.jre6] at org.postgresql.jdbc.PgPreparedStatement.setBlob(PgPreparedStatement.java:1200) ~[postgresql-42.2.19.jre6.jar:42.2.19.jre6] at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.setBlob(NewProxyPreparedStatement.java:495) ~[c3p0-0.9.1.2.jar:0.9.1.2] at org.hibernate.type.BlobType.set(BlobType.java:72) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.type.BlobType.nullSafeSet(BlobType.java:140) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate(AbstractEntityPersister.java:2025) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2271) ~[hibernate-core-3.3.1.GA.jar:3.3.1.GA] ... 160 common frames omitted
Though jbpm 4.4 is a very old version (currently at 7.54), try to update your schema and use bytea type for postgresql large objects. If you're using JTA datasource, auto-commit setting is always true and it can not be changed. Try to change it to an xa-datasource
Solution: I was able to find out a solution for this. I have changed the DB column to Bytea type in PostgreSQL and change the JBPM 4.4 implementation to use byte[] over java.sql.Blob (in org.jbpm.pvm.internal.lob.Lob class).
Unable to write spark dataframe to a parquet file format to C drive in PySpark
I am using the following command to try to write a spark (2.4.4 using Ananaconda 3 Jupyter Notebook) dataframe to a parquet file in Pyspark and get a very strange error message that I cannot resolve. I would appreciate any insights any has.
df.write.mode("overwrite").parquet("test/")
Error message is as follows:
--------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
<ipython-input-37-2b4a1d75a5f6> in <module>()
1 # df.write.partitionBy("AB").parquet("C:/test.parquet",mode='overwrite')
----> 2 df.write.mode("overwrite").parquet("test/")
3 # df.write.mode('SaveMode.Overwrite').parquet("C:/test.parquet")
C:\spark-2.4.4-bin-hadoop2.7\python\pyspark\sql\readwriter.py in parquet(self, path, mode, partitionBy, compression)
841 self.partitionBy(partitionBy)
842 self._set_opts(compression=compression)
--> 843 self._jwrite.parquet(path)
844
845 #since(1.6)
C:\spark-2.4.4-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py in __call__(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:
C:\spark-2.4.4-bin-hadoop2.7\python\pyspark\sql\utils.py in deco(*a, **kw)
61 def deco(*a, **kw):
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()
C:\spark-2.4.4-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
--> 328 format(target_id, ".", name), value)
329 else:
330 raise Py4JError(
Py4JJavaError: An error occurred while calling o862.parquet.
: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:198)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:566)
at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 52.0 failed 1 times, most recent failure: Lost task 0.0 in stage 52.0 (TID 176, localhost, executor driver): java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\583621\OneDrive - Booz Allen Hamilton\Personal\Teaching\PySpark Essentials for Data Scientists\PySpark DataFrame Essentials\test\_temporary\0\_temporary\attempt_20191206164455_0052_m_000000_176\part-00000-2cd01dbe-9e3f-44a5-88e1-e904822024c2-c000.snappy.parquet
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:248)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:390)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:349)
at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:151)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:167)
... 32 more
Caused by: java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\583621\OneDrive - Booz Allen Hamilton\Personal\Teaching\PySpark Essentials for Data Scientists\PySpark DataFrame Essentials\test\_temporary\0\_temporary\attempt_20191206164455_0052_m_000000_176\part-00000-2cd01dbe-9e3f-44a5-88e1-e904822024c2-c000.snappy.parquet
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:248)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:390)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:349)
at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:37)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:151)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:236)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:170)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
... 1 more
1
# Now something a bit more complicated: Read in a full parquet
2
parquet
You need to set a Hadoop home.
You can get the WINUTILS.EXE binary from a Hadoop redistribution. There is a repository of this for some Hadoop versions on github.
Then
1) Either You can Set the environment variable %HADOOP_HOME% to point to the directory above the BIN dir containing WINUTILS.EXE.
2)or Configure in code as
import sys
import os
os.environ['HADOOP_HOME'] = "C:/Mine/Spark/hadoop-2.6.0"
sys.path.append("C:/Mine/Spark/hadoop-2.6.0/bin")
Hope this helps !
Pyspark : Replace diacritic characters in dataframe column
I am trying to replace all diacritic characters in a pyspark column. I have a list of tuples with the 1st element being the character to replace, and the 2nd element is character to replace it with. This works perfectly if I call one tuple at a time. This is my code that works
def Str_Substitution(data_prep_output):
# Load Data
DATA_INPUT_DF = data_prep_output
# Set up character substitutions, use '?' if character is not in this list
chr_replacements = [
(u'Á', 'A'), (u'Ă', 'A'), (u'Ắ', 'A'), (u'Ặ', 'A'), (u'Ằ', 'A'), (u'Ẳ', 'A'), (u'Ẵ', 'A'), \
(u'Ǎ', 'A'), (u'Â', 'A'), (u'Ấ', 'A'), (u'Ậ', 'A'), (u'Ầ', 'A'), (u'Ẩ', 'A'), (u'Ẫ', 'A'), \
(u'Ä', 'A'), (u'Ǟ', 'A'), (u'Ȧ', 'A'), (u'Ǡ', 'A'), (u'Ạ', 'A'), (u'Ȁ', 'A'), (u'À', 'A'), \
(u'Ả', 'A'), (u'Ȃ', 'A'), (u'Ā', 'A'), (u'Ą', 'A'), (u'Å', 'A'), (u'Ǻ', 'A'), (u'Ḁ', 'A'), \
(u'Ⱥ', 'A'), (u'Ã', 'A'), (u'Ꜳ', 'AA'), (u'Æ', 'AE'), (u'Ǽ', 'AE'), (u'Ǣ', 'AE'), (u'Ꜵ', 'AO'), \
(u'Ꜷ', 'AU'), (u'Ꜹ', 'AV'), (u'Ꜻ', 'AV'), (u'Ꜽ', 'AY'), (u'Ḃ', 'B'), (u'Ḅ', 'B'), (u'Ɓ', 'B'), \
(u'Ḇ', 'B'), (u'Ƀ', 'B'), (u'Ƃ', 'B'), (u'Ć', 'C'), (u'Č', 'C'), (u'Ç', 'C'), (u'Ḉ', 'C'), \
(u'Ĉ', 'C'), (u'Ċ', 'C'), (u'Ƈ', 'C'), (u'Ȼ', 'C'), (u'Ď', 'D'), (u'Ḑ', 'D'), (u'Ḓ', 'D'), \
(u'Ḋ', 'D'), (u'Ḍ', 'D'), (u'Ɗ', 'D'), (u'Ḏ', 'D'), (u'Dz', 'D'), (u'Dž', 'D'), (u'Đ', 'D'), \
(u'Ƌ', 'D'), (u'DZ', 'DZ'), (u'DŽ', 'DZ'), (u'É', 'E'), (u'Ĕ', 'E'), (u'Ě', 'E'), (u'Ȩ', 'E'), \
(u'Ḝ', 'E'), (u'Ê', 'E'), (u'Ế', 'E'), (u'Ệ', 'E'), (u'Ề', 'E'), (u'Ể', 'E'), (u'Ễ', 'E'), \
(u'Ḙ', 'E'), (u'Ë', 'E'), (u'Ė', 'E'), (u'Ẹ', 'E'), (u'Ȅ', 'E'), (u'È', 'E'), (u'Ẻ', 'E'), \
(u'Ȇ', 'E'), (u'Ē', 'E'), (u'Ḗ', 'E'), (u'Ḕ', 'E'), (u'Ę', 'E'), (u'Ɇ', 'E'), (u'Ẽ', 'E'), \
(u'Ḛ', 'E'), (u'Ꝫ', 'ET'), (u'Ḟ', 'F'), (u'Ƒ', 'F'), (u'Ǵ', 'G'), (u'Ğ', 'G'), (u'Ǧ', 'G'), \
(u'Ģ', 'G'), (u'Ĝ', 'G'), (u'Ġ', 'G'), (u'Ɠ', 'G'), (u'Ḡ', 'G'), (u'Ǥ', 'G'), (u'Ḫ', 'H'), \
(u'Ȟ', 'H'), (u'Ḩ', 'H'), (u'Ĥ', 'H'), (u'Ⱨ', 'H'), (u'Ḧ', 'H'), (u'Ḣ', 'H'), (u'Ḥ', 'H'), \
(u'Ħ', 'H'), (u'Í', 'I'), (u'Ĭ', 'I'), (u'Ǐ', 'I'), (u'Î', 'I'), (u'Ï', 'I'), (u'Ḯ', 'I'), \
(u'İ', 'I'), (u'Ị', 'I'), (u'Ȉ', 'I'), (u'Ì', 'I'), (u'Ỉ', 'I'), (u'Ȋ', 'I'), (u'Ī', 'I'), \
(u'Į', 'I'), (u'Ɨ', 'I'), (u'Ĩ', 'I'), (u'Ḭ', 'I'), (u'Ꝺ', 'D'), (u'Ꝼ', 'F'), (u'Ᵹ', 'G'), \
(u'Ꞃ', 'R'), (u'Ꞅ', 'S'), (u'Ꞇ', 'T'), (u'Ꝭ', 'IS'), (u'Ĵ', 'J'), (u'Ɉ', 'J'), (u'Ḱ', 'K'), \
(u'Ǩ', 'K'), (u'Ķ', 'K'), (u'Ⱪ', 'K'), (u'Ꝃ', 'K'), (u'Ḳ', 'K'), (u'Ƙ', 'K'), (u'Ḵ', 'K'), \
(u'Ꝁ', 'K'), (u'Ꝅ', 'K'), (u'Ĺ', 'L'), (u'Ƚ', 'L'), (u'Ľ', 'L'), (u'Ļ', 'L'), (u'Ḽ', 'L'), \
(u'Ḷ', 'L'), (u'Ḹ', 'L'), (u'Ⱡ', 'L'), (u'Ꝉ', 'L'), (u'Ḻ', 'L'), (u'Ŀ', 'L'), (u'Ɫ', 'L'), \
(u'Lj', 'L'), (u'Ł', 'L'), (u'LJ', 'LJ'), (u'Ḿ', 'M'), (u'Ṁ', 'M'), (u'Ṃ', 'M'), (u'Ɱ', 'M'), \
(u'Ń', 'N'), (u'Ň', 'N'), (u'Ņ', 'N'), (u'Ṋ', 'N'), (u'Ṅ', 'N'), (u'Ṇ', 'N'), (u'Ǹ', 'N'), \
(u'Ɲ', 'N'), (u'Ṉ', 'N'), (u'Ƞ', 'N'), (u'Nj', 'N'), (u'Ñ', 'N'), (u'NJ', 'NJ'), (u'Ó', 'O'), \
(u'Ŏ', 'O'), (u'Ǒ', 'O'), (u'Ô', 'O'), (u'Ố', 'O'), (u'Ộ', 'O'), (u'Ồ', 'O'), (u'Ổ', 'O'), \
(u'Ỗ', 'O'), (u'Ö', 'O'), (u'Ȫ', 'O'), (u'Ȯ', 'O'), (u'Ȱ', 'O'), (u'Ọ', 'O'), (u'Ő', 'O'), \
(u'Ȍ', 'O'), (u'Ò', 'O'), (u'Ỏ', 'O'), (u'Ơ', 'O'), (u'Ớ', 'O'), (u'Ợ', 'O'), (u'Ờ', 'O'), \
(u'Ở', 'O'), (u'Ỡ', 'O'), (u'Ȏ', 'O'), (u'Ꝋ', 'O'), (u'Ꝍ', 'O'), (u'Ō', 'O'), (u'Ṓ', 'O'), \
(u'Ṑ', 'O'), (u'Ɵ', 'O'), (u'Ǫ', 'O'), (u'Ǭ', 'O'), (u'Ø', 'O'), (u'Ǿ', 'O'), (u'Õ', 'O'), \
(u'Ṍ', 'O'), (u'Ṏ', 'O'), (u'Ȭ', 'O'), (u'Ƣ', 'OI'), (u'Ꝏ', 'OO'), (u'Ɛ', 'E'), (u'Ɔ', 'O'), \
(u'Ȣ', 'OU'), (u'Ṕ', 'P'), (u'Ṗ', 'P'), (u'Ꝓ', 'P'), (u'Ƥ', 'P'), (u'Ꝕ', 'P'), (u'Ᵽ', 'P'), \
(u'Ꝑ', 'P'), (u'Ꝙ', 'Q'), (u'Ꝗ', 'Q'), (u'Ŕ', 'R'), (u'Ř', 'R'), (u'Ŗ', 'R'), (u'Ṙ', 'R'), \
(u'Ṛ', 'R'), (u'Ṝ', 'R'), (u'Ȑ', 'R'), (u'Ȓ', 'R'), (u'Ṟ', 'R'), (u'Ɍ', 'R'), (u'Ɽ', 'R'), \
(u'Ꜿ', 'C'), (u'Ǝ', 'E'), (u'Ś', 'S'), (u'Ṥ', 'S'), (u'Š', 'S'), (u'Ṧ', 'S'), (u'Ş', 'S'), \
(u'Ŝ', 'S'), (u'Ș', 'S'), (u'Ṡ', 'S'), (u'Ṣ', 'S'), (u'Ṩ', 'S'), (u'Ť', 'T'), (u'Ţ', 'T'), \
(u'Ṱ', 'T'), (u'Ț', 'T'), (u'Ⱦ', 'T'), (u'Ṫ', 'T'), (u'Ṭ', 'T'), (u'Ƭ', 'T'), (u'Ṯ', 'T'), \
(u'Ʈ', 'T'), (u'Ŧ', 'T'), (u'Ɐ', 'A'), (u'Ꞁ', 'L'), (u'Ɯ', 'M'), (u'Ʌ', 'V'), (u'Ꜩ', 'TZ'), \
(u'Ú', 'U'), (u'Ŭ', 'U'), (u'Ǔ', 'U'), (u'Û', 'U'), (u'Ṷ', 'U'), (u'Ü', 'U'), (u'Ǘ', 'U'), \
(u'Ǚ', 'U'), (u'Ǜ', 'U'), (u'Ǖ', 'U'), (u'Ṳ', 'U'), (u'Ụ', 'U'), (u'Ű', 'U'), (u'Ȕ', 'U'), \
(u'Ù', 'U'), (u'Ủ', 'U'), (u'Ư', 'U'), (u'Ứ', 'U'), (u'Ự', 'U'), (u'Ừ', 'U'), (u'Ử', 'U'), \
(u'Ữ', 'U'), (u'Ȗ', 'U'), (u'Ū', 'U'), (u'Ṻ', 'U'), (u'Ų', 'U'), (u'Ů', 'U'), (u'Ũ', 'U'), \
(u'Ṹ', 'U'), (u'Ṵ', 'U'), (u'Ꝟ', 'V'), (u'Ṿ', 'V'), (u'Ʋ', 'V'), (u'Ṽ', 'V'), (u'Ꝡ', 'VY'), \
(u'Ẃ', 'W'), (u'Ŵ', 'W'), (u'Ẅ', 'W'), (u'Ẇ', 'W'), (u'Ẉ', 'W'), (u'Ẁ', 'W'), (u'Ⱳ', 'W'), \
(u'Ẍ', 'X'), (u'Ẋ', 'X'), (u'Ý', 'Y'), (u'Ŷ', 'Y'), (u'Ÿ', 'Y'), (u'Ẏ', 'Y'), (u'Ỵ', 'Y'), \
(u'Ỳ', 'Y'), (u'Ƴ', 'Y'), (u'Ỷ', 'Y'), (u'Ỿ', 'Y'), (u'Ȳ', 'Y'), (u'Ɏ', 'Y'), (u'Ỹ', 'Y'), \
(u'Ź', 'Z'), (u'Ž', 'Z'), (u'Ẑ', 'Z'), (u'Ⱬ', 'Z'), (u'Ż', 'Z'), (u'Ẓ', 'Z'), (u'Ȥ', 'Z'), \
(u'Ẕ', 'Z'), (u'Ƶ', 'Z'), (u'IJ', 'IJ'), (u'Œ', 'OE'), (u'ᴀ', 'A'), (u'ᴁ', 'AE'), (u'ʙ', 'B'), \
(u'ᴃ', 'B'), (u'ᴄ', 'C'), (u'ᴅ', 'D'), (u'ᴇ', 'E'), (u'ꜰ', 'F'), (u'ɢ', 'G'), (u'ʛ', 'G'), \
(u'ʜ', 'H'), (u'ɪ', 'I'), (u'ʁ', 'R'), (u'ᴊ', 'J'), (u'ᴋ', 'K'), (u'ʟ', 'L'), (u'ᴌ', 'L'), \
(u'ᴍ', 'M'), (u'ɴ', 'N'), (u'ᴏ', 'O'), (u'ɶ', 'OE'), (u'ᴐ', 'O'), (u'ᴕ', 'OU'), (u'ᴘ', 'P'), \
(u'ʀ', 'R'), (u'ᴎ', 'N'), (u'ᴙ', 'R'), (u'ꜱ', 'S'), (u'ᴛ', 'T'), (u'ⱻ', 'E'), (u'ᴚ', 'R'), \
(u'ᴜ', 'U'), (u'ᴠ', 'V'), (u'ᴡ', 'W'), (u'ʏ', 'Y'), (u'ᴢ', 'Z'), (u'á', 'a'), (u'ă', 'a'), \
(u'ắ', 'a'), (u'ặ', 'a'), (u'ằ', 'a'), (u'ẳ', 'a'), (u'ẵ', 'a'), (u'ǎ', 'a'), (u'â', 'a'), \
(u'ấ', 'a'), (u'ậ', 'a'), (u'ầ', 'a'), (u'ẩ', 'a'), (u'ẫ', 'a'), (u'ä', 'a'), (u'ǟ', 'a'), \
(u'ȧ', 'a'), (u'ǡ', 'a'), (u'ạ', 'a'), (u'ȁ', 'a'), (u'à', 'a'), (u'ả', 'a'), (u'ȃ', 'a'), \
(u'ā', 'a'), (u'ą', 'a'), (u'ᶏ', 'a'), (u'ẚ', 'a'), (u'å', 'a'), (u'ǻ', 'a'), (u'ḁ', 'a'), \
(u'ⱥ', 'a'), (u'ã', 'a'), (u'ꜳ', 'aa'), (u'æ', 'ae'), (u'ǽ', 'ae'), (u'ǣ', 'ae'), \
(u'ꜵ', 'ao'), (u'ꜷ', 'au'), (u'ꜹ', 'av'), (u'ꜻ', 'av'), (u'ꜽ', 'ay'), (u'ḃ', 'b'), \
(u'ḅ', 'b'), (u'ɓ', 'b'), (u'ḇ', 'b'), (u'ᵬ', 'b'), (u'ᶀ', 'b'), (u'ƀ', 'b'), (u'ƃ', 'b'), \
(u'ɵ', 'o'), (u'ć', 'c'), (u'č', 'c'), (u'ç', 'c'), (u'ḉ', 'c'), (u'ĉ', 'c'), (u'ɕ', 'c'), \
(u'ċ', 'c'), (u'ƈ', 'c'), (u'ȼ', 'c'), (u'ď', 'd'), (u'ḑ', 'd'), (u'ḓ', 'd'), (u'ȡ', 'd'), \
(u'ḋ', 'd'), (u'ḍ', 'd'), (u'ɗ', 'd'), (u'ᶑ', 'd'), (u'ḏ', 'd'), (u'ᵭ', 'd'), (u'ᶁ', 'd'), \
(u'đ', 'd'), (u'ɖ', 'd'), (u'ƌ', 'd'), (u'ı', 'i'), (u'ȷ', 'j'), (u'ɟ', 'j'), (u'ʄ', 'j'), \
(u'dz', 'dz'), (u'dž', 'dz'), (u'é', 'e'), (u'ĕ', 'e'), (u'ě', 'e'), (u'ȩ', 'e'), (u'ḝ', 'e'), \
(u'ê', 'e'), (u'ế', 'e'), (u'ệ', 'e'), (u'ề', 'e'), (u'ể', 'e'), (u'ễ', 'e'), (u'ḙ', 'e'), \
(u'ë', 'e'), (u'ė', 'e'), (u'ẹ', 'e'), (u'ȅ', 'e'), (u'è', 'e'), (u'ẻ', 'e'), (u'ȇ', 'e'), \
(u'ē', 'e'), (u'ḗ', 'e'), (u'ḕ', 'e'), (u'ⱸ', 'e'), (u'ę', 'e'), (u'ᶒ', 'e'), (u'ɇ', 'e'), \
(u'ẽ', 'e'), (u'ḛ', 'e'), (u'ꝫ', 'et'), (u'ḟ', 'f'), (u'ƒ', 'f'), (u'ᵮ', 'f'), (u'ᶂ', 'f'), \
(u'ǵ', 'g'), (u'ğ', 'g'), (u'ǧ', 'g'), (u'ģ', 'g'), (u'ĝ', 'g'), (u'ġ', 'g'), (u'ɠ', 'g'), \
(u'ḡ', 'g'), (u'ᶃ', 'g'), (u'ǥ', 'g'), (u'ḫ', 'h'), (u'ȟ', 'h'), (u'ḩ', 'h'), (u'ĥ', 'h'), \
(u'ⱨ', 'h'), (u'ḧ', 'h'), (u'ḣ', 'h'), (u'ḥ', 'h'), (u'ɦ', 'h'), (u'ẖ', 'h'), (u'ħ', 'h'), \
(u'ƕ', 'hv'), (u'í', 'i'), (u'ĭ', 'i'), (u'ǐ', 'i'), (u'î', 'i'), (u'ï', 'i'), (u'ḯ', 'i'), \
(u'ị', 'i'), (u'ȉ', 'i'), (u'ì', 'i'), (u'ỉ', 'i'), (u'ȋ', 'i'), (u'ī', 'i'), (u'į', 'i'), \
(u'ᶖ', 'i'), (u'ɨ', 'i'), (u'ĩ', 'i'), (u'ḭ', 'i'), (u'ꝺ', 'd'), (u'ꝼ', 'f'), (u'ᵹ', 'g'), \
(u'ꞃ', 'r'), (u'ꞅ', 's'), (u'ꞇ', 't'), (u'ꝭ', 'is'), (u'ǰ', 'j'), (u'ĵ', 'j'), (u'ʝ', 'j'), \
(u'ɉ', 'j'), (u'ḱ', 'k'), (u'ǩ', 'k'), (u'ķ', 'k'), (u'ⱪ', 'k'), (u'ꝃ', 'k'), (u'ḳ', 'k'), \
(u'ƙ', 'k'), (u'ḵ', 'k'), (u'ᶄ', 'k'), (u'ꝁ', 'k'), (u'ꝅ', 'k'), (u'ĺ', 'l'), (u'ƚ', 'l'), \
(u'ɬ', 'l'), (u'ľ', 'l'), (u'ļ', 'l'), (u'ḽ', 'l'), (u'ȴ', 'l'), (u'ḷ', 'l'), (u'ḹ', 'l'), \
(u'ⱡ', 'l'), (u'ꝉ', 'l'), (u'ḻ', 'l'), (u'ŀ', 'l'), (u'ɫ', 'l'), (u'ᶅ', 'l'), (u'ɭ', 'l'), \
(u'ł', 'l'), (u'lj', 'lj'), (u'ſ', 's'), (u'ẜ', 's'), (u'ẛ', 's'), (u'ẝ', 's'), (u'ḿ', 'm'), \
(u'ṁ', 'm'), (u'ṃ', 'm'), (u'ɱ', 'm'), (u'ᵯ', 'm'), (u'ᶆ', 'm'), (u'ń', 'n'), (u'ň', 'n'), \
(u'ņ', 'n'), (u'ṋ', 'n'), (u'ȵ', 'n'), (u'ṅ', 'n'), (u'ṇ', 'n'), (u'ǹ', 'n'), (u'ɲ', 'n'), \
(u'ṉ', 'n'), (u'ƞ', 'n'), (u'ᵰ', 'n'), (u'ᶇ', 'n'), (u'ɳ', 'n'), (u'ñ', 'n'), (u'nj', 'nj'), \
(u'ó', 'o'), (u'ŏ', 'o'), (u'ǒ', 'o'), (u'ô', 'o'), (u'ố', 'o'), (u'ộ', 'o'), (u'ồ', 'o'), \
(u'ổ', 'o'), (u'ỗ', 'o'), (u'ö', 'o'), (u'ȫ', 'o'), (u'ȯ', 'o'), (u'ȱ', 'o'), (u'ọ', 'o'), \
(u'ő', 'o'), (u'ȍ', 'o'), (u'ò', 'o'), (u'ỏ', 'o'), (u'ơ', 'o'), (u'ớ', 'o'), (u'ợ', 'o'), \
(u'ờ', 'o'), (u'ở', 'o'), (u'ỡ', 'o'), (u'ȏ', 'o'), (u'ꝋ', 'o'), (u'ꝍ', 'o'), (u'ⱺ', 'o'), \
(u'ō', 'o'), (u'ṓ', 'o'), (u'ṑ', 'o'), (u'ǫ', 'o'), (u'ǭ', 'o'), (u'ø', 'o'), (u'ǿ', 'o'), \
(u'õ', 'o'), (u'ṍ', 'o'), (u'ṏ', 'o'), (u'ȭ', 'o'), (u'ƣ', 'oi'), (u'ꝏ', 'oo'), (u'ɛ', 'e'), \
(u'ᶓ', 'e'), (u'ɔ', 'o'), (u'ᶗ', 'o'), (u'ȣ', 'ou'), (u'ṕ', 'p'), (u'ṗ', 'p'), (u'ꝓ', 'p'), \
(u'ƥ', 'p'), (u'ᵱ', 'p'), (u'ᶈ', 'p'), (u'ꝕ', 'p'), (u'ᵽ', 'p'), (u'ꝑ', 'p'), (u'ꝙ', 'q'), \
(u'ʠ', 'q'), (u'ɋ', 'q'), (u'ꝗ', 'q'), (u'ŕ', 'r'), (u'ř', 'r'), (u'ŗ', 'r'), (u'ṙ', 'r'), \
(u'ṛ', 'r'), (u'ṝ', 'r'), (u'ȑ', 'r'), (u'ɾ', 'r'), (u'ᵳ', 'r'), (u'ȓ', 'r'), (u'ṟ', 'r'), \
(u'ɼ', 'r'), (u'ᵲ', 'r'), (u'ᶉ', 'r'), (u'ɍ', 'r'), (u'ɽ', 'r'), (u'ↄ', 'c'), (u'ꜿ', 'c'), \
(u'ɘ', 'e'), (u'ɿ', 'r'), (u'ś', 's'), (u'ṥ', 's'), (u'š', 's'), (u'ṧ', 's'), (u'ş', 's'), \
(u'ŝ', 's'), (u'ș', 's'), (u'ṡ', 's'), (u'ṣ', 's'), (u'ṩ', 's'), (u'ʂ', 's'), (u'ᵴ', 's'), \
(u'ᶊ', 's'), (u'ȿ', 's'), (u'ɡ', 'g'), (u'ᴑ', 'o'), (u'ᴓ', 'o'), (u'ᴝ', 'u'), (u'ť', 't'), \
(u'ţ', 't'), (u'ṱ', 't'), (u'ț', 't'), (u'ȶ', 't'), (u'ẗ', 't'), (u'ⱦ', 't'), (u'ṫ', 't'), \
(u'ṭ', 't'), (u'ƭ', 't'), (u'ṯ', 't'), (u'ᵵ', 't'), (u'ƫ', 't'), (u'ʈ', 't'), (u'ŧ', 't'), \
(u'ᵺ', 'th'), (u'ɐ', 'a'), (u'ᴂ', 'ae'), (u'ǝ', 'e'), (u'ᵷ', 'g'), (u'ɥ', 'h'), (u'ʮ', 'h'), \
(u'ʯ', 'h'), (u'ᴉ', 'i'), (u'ʞ', 'k'), (u'ꞁ', 'l'), (u'ɯ', 'm'), (u'ɰ', 'm'), (u'ᴔ', 'oe'), \
(u'ɹ', 'r'), (u'ɻ', 'r'), (u'ɺ', 'r'), (u'ⱹ', 'r'), (u'ʇ', 't'), (u'ʌ', 'v'), (u'ʍ', 'w'), \
(u'ʎ', 'y'), (u'ꜩ', 'tz'), (u'ú', 'u'), (u'ŭ', 'u'), (u'ǔ', 'u'), (u'û', 'u'), (u'ṷ', 'u'), \
(u'ü', 'u'), (u'ǘ', 'u'), (u'ǚ', 'u'), (u'ǜ', 'u'), (u'ǖ', 'u'), (u'ṳ', 'u'), (u'ụ', 'u'), \
(u'ű', 'u'), (u'ȕ', 'u'), (u'ù', 'u'), (u'ủ', 'u'), (u'ư', 'u'), (u'ứ', 'u'), (u'ự', 'u'), \
(u'ừ', 'u'), (u'ử', 'u'), (u'ữ', 'u'), (u'ȗ', 'u'), (u'ū', 'u'), (u'ṻ', 'u'), (u'ų', 'u'), \
(u'ᶙ', 'u'), (u'ů', 'u'), (u'ũ', 'u'), (u'ṹ', 'u'), (u'ṵ', 'u'), (u'ᵫ', 'ue'), (u'ꝸ', 'um'), \
(u'ⱴ', 'v'), (u'ꝟ', 'v'), (u'ṿ', 'v'), (u'ʋ', 'v'), (u'ᶌ', 'v'), (u'ⱱ', 'v'), (u'ṽ', 'v'), \
(u'ꝡ', 'vy'), (u'ẃ', 'w'), (u'ŵ', 'w'), (u'ẅ', 'w'), (u'ẇ', 'w'), (u'ẉ', 'w'), (u'ẁ', 'w'), \
(u'ⱳ', 'w'), (u'ẘ', 'w'), (u'ẍ', 'x'), (u'ẋ', 'x'), (u'ᶍ', 'x'), (u'ý', 'y'), (u'ŷ', 'y'), \
(u'ÿ', 'y'), (u'ẏ', 'y'), (u'ỵ', 'y'), (u'ỳ', 'y'), (u'ƴ', 'y'), (u'ỷ', 'y'), (u'ỿ', 'y'), \
(u'ȳ', 'y'), (u'ẙ', 'y'), (u'ɏ', 'y'), (u'ỹ', 'y'), (u'ź', 'z'), (u'ž', 'z'), (u'ẑ', 'z'), \
(u'ʑ', 'z'), (u'ⱬ', 'z'), (u'ż', 'z'), (u'ẓ', 'z'), (u'ȥ', 'z'), (u'ẕ', 'z'), (u'ᵶ', 'z'), \
(u'ᶎ', 'z'), (u'ʐ', 'z'), (u'ƶ', 'z'), (u'ɀ', 'z'), (u'ff', 'ff'), (u'ffi', 'ffi'), (u'ffl', 'ffl'), \
(u'fi', 'fi'), (u'fl', 'fl'), (u'ij', 'ij'), (u'œ', 'oe'), (u'st', 'st'), (u'ₐ', 'a'), (u'ₑ', 'e'), \
(u'ᵢ', 'i'), (u'ⱼ', 'j'), (u'ₒ', 'o'), (u'ᵣ', 'r'), (u'ᵤ', 'u'), (u'ᵥ', 'v'), (u'ₓ', 'x'), \
('[^\x00-\x7F]', '?')]
sub_str = chr_replacements[14]
# Use regexp_replace to substitute characters
DATA_INPUT_DF = DATA_INPUT_DF.withColumn("Inc_Summary", F.regexp_replace(DATA_INPUT_DF.Inc_Summary, sub_str[0], sub_str[1]))
sub_str = chr_replacements[66]
# Use regexp_replace to substitute characters
DATA_INPUT_DF = DATA_INPUT_DF.withColumn("Inc_Summary", F.regexp_replace(DATA_INPUT_DF.Inc_Summary, sub_str[0], sub_str[1]))
return DATA_INPUT_DF
When I try to loop over all the tuples in my list I get a StackOverflow error
for sub_str in chr_replacements:
# Use regexp_replace to substitute characters
DATA_INPUT_DF = DATA_INPUT_DF.withColumn("Inc_Summary", F.regexp_replace(DATA_INPUT_DF.Inc_Summary, sub_str[0], sub_str[1]))
I've tried all the various ways of doing this I could find by Googling this, but for me none work
What am i doing wrong?
Bilinear and bicubic interpolation of 2D matrix in matlab
I am trying to apply Bilinear and bicubic interpolation on my data set which is stored in text files. These text files are in a folder and are named output_00.text to output_23.text. Each text file consist of three columns. First is Latitude, second is longitude and third column is temperature value at this latitude and longitude (position over earth). Temperature column contain -9999.000 as not a number or NaN values. This NaN value appears for random rows in each file. I want to interpolate these NaN values with bilinear/ bicubic interpolation technique.This code will read each text file and interpolate it with bilinear method and save it with method_00.text. My one text file look like this 21.500 60.500 295.867 21.500 61.500 295.828 21.500 62.500 295.828 21.500 63.500 295.867 21.500 64.500 296.102 21.500 65.500 296.234 21.500 66.500 296.352 21.500 67.500 296.336 21.500 68.500 296.305 21.500 69.500 298.281 21.500 70.500 301.828 21.500 71.500 302.094 21.500 72.500 299.469 21.500 73.500 301.711 21.500 74.500 -9999.000 21.500 75.500 -9999.000 21.500 76.500 -9999.000 21.500 77.500 -9999.000 21.500 78.500 -9999.000 22.500 60.500 295.477 22.500 61.500 295.484 22.500 62.500 295.516 22.500 63.500 295.547 22.500 64.500 295.852 22.500 65.500 295.859 22.500 66.500 295.852 22.500 67.500 295.711 22.500 68.500 295.969 22.500 69.500 298.562 22.500 70.500 300.828 22.500 71.500 302.352 22.500 72.500 300.570 22.500 73.500 301.383 22.500 74.500 -9999.000 22.500 75.500 -9999.000 22.500 76.500 -9999.000 22.500 77.500 -9999.000 22.500 78.500 -9999.000 23.500 60.500 294.906 23.500 61.500 294.898 23.500 62.500 295.000 23.500 63.500 295.078 23.500 64.500 295.297 23.500 65.500 295.359 23.500 66.500 295.297 23.500 67.500 295.312 23.500 68.500 296.664 23.500 69.500 298.781 23.500 70.500 299.211 23.500 71.500 300.109 23.500 72.500 301.000 23.500 73.500 301.594 23.500 74.500 302.000 23.500 75.500 -9999.000 23.500 76.500 -9999.000 23.500 77.500 -9999.000 23.500 78.500 -9999.000 24.500 60.500 294.578 24.500 61.500 294.516 24.500 62.500 294.734 24.500 63.500 294.789 24.500 64.500 294.844 24.500 65.500 294.562 24.500 66.500 294.734 24.500 67.500 296.367 24.500 68.500 297.438 24.500 69.500 298.531 24.500 70.500 298.453 24.500 71.500 299.195 24.500 72.500 300.062 24.500 73.500 -9999.000 24.500 74.500 -9999.000 24.500 75.500 -9999.000 24.500 76.500 -9999.000 24.500 77.500 -9999.000 24.500 78.500 -9999.000 25.500 60.500 296.258 25.500 61.500 296.391 25.500 62.500 296.672 25.500 63.500 296.398 25.500 64.500 295.773 25.500 65.500 295.812 25.500 66.500 296.609 25.500 67.500 297.977 25.500 68.500 297.109 25.500 69.500 297.828 25.500 70.500 298.430 25.500 71.500 298.836 25.500 72.500 298.703 25.500 73.500 -9999.000 25.500 74.500 -9999.000 25.500 75.500 -9999.000 25.500 76.500 -9999.000 25.500 77.500 -9999.000 25.500 78.500 299.023 26.500 60.500 -9999.000 26.500 61.500 298.266 26.500 62.500 296.773 26.500 63.500 -9999.000 26.500 64.500 -9999.000 26.500 65.500 -9999.000 26.500 66.500 297.250 26.500 67.500 296.188 26.500 68.500 295.938 26.500 69.500 296.906 26.500 70.500 297.828 26.500 71.500 299.312 26.500 72.500 299.359 26.500 73.500 -9999.000 26.500 74.500 -9999.000 26.500 75.500 -9999.000 26.500 76.500 -9999.000 26.500 77.500 298.875 26.500 78.500 296.773 27.500 60.500 -9999.000 27.500 61.500 -9999.000 27.500 62.500 -9999.000 27.500 63.500 -9999.000 27.500 64.500 -9999.000 27.500 65.500 -9999.000 27.500 66.500 -9999.000 27.500 67.500 295.352 27.500 68.500 295.148 27.500 69.500 295.750 27.500 70.500 295.750 27.500 71.500 296.070 27.500 72.500 295.227 27.500 73.500 -9999.000 27.500 74.500 -9999.000 27.500 75.500 -9999.000 27.500 76.500 -9999.000 27.500 77.500 -9999.000 27.500 78.500 296.609 28.500 60.500 -9999.000 28.500 61.500 -9999.000 28.500 62.500 -9999.000 28.500 63.500 -9999.000 28.500 64.500 -9999.000 28.500 65.500 -9999.000 28.500 66.500 -9999.000 28.500 67.500 295.773 28.500 68.500 295.375 28.500 69.500 295.438 28.500 70.500 294.664 28.500 71.500 294.906 28.500 72.500 294.812 28.500 73.500 295.805 28.500 74.500 -9999.000 28.500 75.500 -9999.000 28.500 76.500 -9999.000 28.500 77.500 -9999.000 28.500 78.500 -9999.000 29.500 60.500 -9999.000 29.500 61.500 -9999.000 29.500 62.500 -9999.000 29.500 63.500 -9999.000 29.500 64.500 -9999.000 29.500 65.500 -9999.000 29.500 66.500 -9999.000 29.500 67.500 295.719 29.500 68.500 296.797 29.500 69.500 293.375 29.500 70.500 294.305 29.500 71.500 294.070 29.500 72.500 293.750 29.500 73.500 295.539 29.500 74.500 -9999.000 29.500 75.500 -9999.000 29.500 76.500 -9999.000 29.500 77.500 -9999.000 29.500 78.500 -9999.000 30.500 60.500 -9999.000 30.500 61.500 -9999.000 30.500 62.500 -9999.000 30.500 63.500 -9999.000 30.500 64.500 -9999.000 30.500 65.500 -9999.000 30.500 66.500 -9999.000 30.500 67.500 -9999.000 30.500 68.500 -9999.000 30.500 69.500 -9999.000 30.500 70.500 293.320 30.500 71.500 292.930 30.500 72.500 293.570 30.500 73.500 294.648 30.500 74.500 295.383 30.500 75.500 -9999.000 30.500 76.500 -9999.000 30.500 77.500 -9999.000 30.500 78.500 -9999.000 31.500 60.500 -9999.000 31.500 61.500 -9999.000 31.500 62.500 -9999.000 31.500 63.500 -9999.000 31.500 64.500 -9999.000 31.500 65.500 -9999.000 31.500 66.500 -9999.000 31.500 67.500 -9999.000 31.500 68.500 -9999.000 31.500 69.500 -9999.000 31.500 70.500 293.992 31.500 71.500 293.422 31.500 72.500 294.438 31.500 73.500 294.141 31.500 74.500 -9999.000 31.500 75.500 -9999.000 31.500 76.500 -9999.000 31.500 77.500 -9999.000 31.500 78.500 -9999.000 32.500 60.500 -9999.000 32.500 61.500 -9999.000 32.500 62.500 -9999.000 32.500 63.500 -9999.000 32.500 64.500 -9999.000 32.500 65.500 -9999.000 32.500 66.500 -9999.000 32.500 67.500 -9999.000 32.500 68.500 -9999.000 32.500 69.500 -9999.000 32.500 70.500 -9999.000 32.500 71.500 294.312 32.500 72.500 294.812 32.500 73.500 -9999.000 32.500 74.500 -9999.000 32.500 75.500 -9999.000 32.500 76.500 -9999.000 32.500 77.500 -9999.000 32.500 78.500 -9999.000 33.500 60.500 -9999.000 33.500 61.500 -9999.000 33.500 62.500 -9999.000 33.500 63.500 -9999.000 33.500 64.500 -9999.000 33.500 65.500 -9999.000 33.500 66.500 -9999.000 33.500 67.500 -9999.000 33.500 68.500 -9999.000 33.500 69.500 -9999.000 33.500 70.500 -9999.000 33.500 71.500 -9999.000 33.500 72.500 -9999.000 33.500 73.500 -9999.000 33.500 74.500 -9999.000 33.500 75.500 -9999.000 33.500 76.500 -9999.000 33.500 77.500 -9999.000 33.500 78.500 -9999.000 34.500 60.500 -9999.000 34.500 61.500 -9999.000 34.500 62.500 -9999.000 34.500 63.500 -9999.000 34.500 64.500 -9999.000 34.500 65.500 -9999.000 34.500 66.500 -9999.000 34.500 67.500 -9999.000 34.500 68.500 -9999.000 34.500 69.500 -9999.000 34.500 70.500 -9999.000 34.500 71.500 -9999.000 34.500 72.500 -9999.000 34.500 73.500 -9999.000 34.500 74.500 -9999.000 34.500 75.500 -9999.000 34.500 76.500 -9999.000 34.500 77.500 -9999.000 34.500 78.500 -9999.000 35.500 60.500 -9999.000 35.500 61.500 -9999.000 35.500 62.500 -9999.000 35.500 63.500 -9999.000 35.500 64.500 -9999.000 35.500 65.500 -9999.000 35.500 66.500 -9999.000 35.500 67.500 -9999.000 35.500 68.500 -9999.000 35.500 69.500 -9999.000 35.500 70.500 -9999.000 35.500 71.500 -9999.000 35.500 72.500 -9999.000 35.500 73.500 -9999.000 35.500 74.500 -9999.000 35.500 75.500 -9999.000 35.500 76.500 -9999.000 35.500 77.500 -9999.000 35.500 78.500 -9999.000 36.500 60.500 276.742 36.500 61.500 274.406 36.500 62.500 -9999.000 36.500 63.500 -9999.000 36.500 64.500 -9999.000 36.500 65.500 272.219 36.500 66.500 273.023 36.500 67.500 275.875 36.500 68.500 -9999.000 36.500 69.500 -9999.000 36.500 70.500 -9999.000 36.500 71.500 -9999.000 36.500 72.500 -9999.000 36.500 73.500 -9999.000 36.500 74.500 -9999.000 36.500 75.500 -9999.000 36.500 76.500 -9999.000 36.500 77.500 -9999.000 36.500 78.500 -9999.000 37.500 60.500 277.406 37.500 61.500 277.547 37.500 62.500 276.375 37.500 63.500 275.484 37.500 64.500 276.820 37.500 65.500 275.312 37.500 66.500 274.875 37.500 67.500 275.875 37.500 68.500 -9999.000 37.500 69.500 -9999.000 37.500 70.500 -9999.000 37.500 71.500 -9999.000 37.500 72.500 -9999.000 37.500 73.500 -9999.000 37.500 74.500 -9999.000 37.500 75.500 -9999.000 37.500 76.500 -9999.000 37.500 77.500 -9999.000 37.500 78.500 -9999.000 On google i found a tool for bilinear interpolation over a image or matrix. For the time shake this code can be modify to my requirement. But How? function interVal=interpImg(img,yx,zpad) % BiLinear interpolation using 4 pixels around the target location with ceil convention % RGB = 1 for gray scale images. % img can be a single layer matrix or a RGB layer colored image % yx =[y_value, x_value]; It can be either horizontal or vertical vector % % zpad is a boolean variable. if true, zeros are used for pixel values % outside of the given img. If false, the nearest edge value is repeated. % % Example: % [m,n]=meshgrid(1:3);img=[m+n] % --> 2 3 4 % 3 4 5 % 4 5 6 % interpImg(img,[2.4,2.2]) % --> 4.6 % % Disi A, Sep,16th,2013 % adis#mit.edu if nargin<4,RGB=ndims(img);RGB(RGB<3)=1; end if nargin<3,zpad=true; end yx0=floor(yx); wt=yx-yx0; wtConj=1-wt; interTop=wtConj(2)*pixLookup(img,yx0(1),yx0(2),zpad,RGB)+wt(2)*pixLookup(img,yx0(1),yx(2),zpad,RGB); interBtm=wtConj(2)*pixLookup(img,yx(1),yx0(2),zpad,RGB)+wt(2)*pixLookup(img,yx(1),yx(2),zpad,RGB); interVal=wtConj(1)*interTop+wt(1)*interBtm; end function pixVal=pixLookup(img,y,x,zpad,RGB) % This helper function looks up a pixel value from a given input image % img is the input image (RGB or Grayscale) % yx is the coordinate and repEdge tells the condition for pixel values out % side of img (Use round up convention) % For grayscale use RGB =1 if nargin<4,RGB=3;end pixVal=zeros(1,1,RGB); %Initialize the pixel if nargin<3 zpad=true; %pad with black pixels end if RGB==3 [ROW,COL,~]=size(img); else [ROW,COL]=size(img); end % If the pixel value is outside of image given if (x<=0)||(x>COL)||(y<=0)||(y>ROW) if zpad pixVal(:)=0; else y0=y;x0=x; y0(y0<1)=1; x0(x0<1)=1; y0(y0>ROW)=ROW;x0(x0>COL)=COL; pixVal=img(y0,x0,:); end else pixVal=img(ceil(y),ceil(x),:); end end . Link of this tool is here http://www.mathworks.com/matlabcentral/fileexchange/43533-bilinear-interpolation-of-an-image-or-matrix
You're going to want to read in the data, identify all the rows where the last column is unknown. Then using the "good" data points you can construct a 2D interpolant (f(x,y)) to sample at the unknown points.
You'll have to use griddata rather than interp2 since your data is scattered. You'll want to use the linear interpolation method (the default).
The code below should achieve this result.
% Read in data from file into an M x 3 matrix
fid = fopen('data.txt', 'rb');
data = textscan(fid, '%f %f %f');
data = cat(2, data{:});
% Find rows where third column is unknown
isUnknown = data(:,3) == -9999;
isKnown = ~isUnknown;
% Perform scattered interpolation AT unknown location USING known locations
newValues = griddata(data(isKnown,1), data(isKnown,2), data(isKnown,3), ...
data(isUnknown,1), data(isUnknown, 2), 'linear');
% Replace the unknown values with the interpolated values
data(isUnknown, 3) = newValues;
If we visualize this we will see the following.
Unfortunately, a lot of your empty data values actually lie outside of the range of the data that you know. griddata will fill these in with NaNs. In order to fill these in, you can't just rely on interpolation but rather must try to use extrapolation.
If you actually want to do this (rather than leaving them as NaN's), you will want to use scatteredInterpolant with linear extrapolation.
interpolant = scatteredInterpolant(data(isKnown,1:2), data(isKnown,3), 'linear', 'linear');
newValues = interpolant(data(isUnknown,1:2));
data(isUnknown,3) = newValues;
Now you can see that those values have been extrapolated.
You should choose whichever method is more appropriate for your use case.
scatter data interpolation in matlab 2013
I want to use scatter data interpolation for missing or NaN values in my temperature data? I have text files, each consist on three column
First column is latitude
Second column is longitude
Third column consist on temperature value
There is -9999.000 value for temperature column which representing missing or NaN data. I want to interpolate this values from remaining known values. I want to use F=scatteredInterpolant(x,y,v)
Where x, y are coordinates of sample points and v is corresponding value of these sample point. After making F(q), where q is query point. Which will be represent NaN or missing value of temperature in 3rd column.
I have prepared this code but not idea how to move next??
metC = {'linear','cubic','next','pchip','previous','spline','v5cubic','nearest'};
% Read the file data:
S = dir('Output_*.txt');
N = sort({S.name});
nmf = numel(N);
nmr = size(load(N{1},'-ascii'),1);
mat = zeros(nmr,3,nmf);
for k = 1:nmf
mat(:,:,k) = load(N{k},'-ascii');
end
tmp = 0==diff(mat(:,1:2,:),1,3);
assert(all(tmp(:)),'First columns do not match')
% Rearrange:
[VC,NA,IC] = unique(mat(:,1,1));
[VR,NA,IR] = unique(mat(:,2,1));
out = reshape(mat(:,3,:),numel(VR),numel(VC),nmf);
% Detect -9999:
idx = out<(-9998);
Here i have not idea whether i am going right or not??
Interpolate:
vec = 1:nmf;% 24 times
for m = 1:numel(metC)% the length of techniques let say 3 times in this case
metS = metC{m} % the method
for r = 1:numel(VR)% 19 times
for c = 1:numel(VC)% 17 times
idy = squeeze(idx(r,c,:)).'; % removing singleton dimension. Here idy will be position of the without -9999 value
if any(idy) %check for -9999
Xold = vec(~idy); %giving x which are sample points
Yold = squeeze(out(r,c,~idy)).'; % I DO NOT KNOW WHAT THIS LINE DOING?
Xnew = vec(idy); % these are query points which we want to interpolate
out(r,c,idy) = scatteredInterpolant(Xold,Yold,???????????,Xnew,metC,'extrap');
end
end
end
end
My data set look like:
21.500 60.500 295.867
21.500 61.500 295.828
21.500 62.500 295.828
21.500 63.500 295.867
21.500 64.500 296.102
21.500 65.500 296.234
21.500 66.500 296.352
21.500 67.500 296.336
21.500 68.500 296.305
21.500 69.500 298.281
21.500 70.500 301.828
21.500 71.500 302.094
21.500 72.500 299.469
21.500 73.500 301.711
21.500 74.500 -9999.000
21.500 75.500 -9999.000
21.500 76.500 -9999.000
21.500 77.500 -9999.000
21.500 78.500 -9999.000
22.500 60.500 295.477
22.500 61.500 295.484
22.500 62.500 295.516
22.500 63.500 295.547
22.500 64.500 295.852
22.500 65.500 295.859
22.500 66.500 295.852
22.500 67.500 295.711
22.500 68.500 295.969
22.500 69.500 298.562
22.500 70.500 300.828
22.500 71.500 302.352
22.500 72.500 300.570
22.500 73.500 301.383
22.500 74.500 -9999.000
22.500 75.500 -9999.000
22.500 76.500 -9999.000
22.500 77.500 -9999.000
22.500 78.500 -9999.000
23.500 60.500 294.906
23.500 61.500 294.898
23.500 62.500 295.000
23.500 63.500 295.078
23.500 64.500 295.297
23.500 65.500 295.359
23.500 66.500 295.297
23.500 67.500 295.312
23.500 68.500 296.664
23.500 69.500 298.781
23.500 70.500 299.211
23.500 71.500 300.109
23.500 72.500 301.000
23.500 73.500 301.594
23.500 74.500 302.000
23.500 75.500 -9999.000
23.500 76.500 -9999.000
23.500 77.500 -9999.000
23.500 78.500 -9999.000
24.500 60.500 294.578
24.500 61.500 294.516
24.500 62.500 294.734
24.500 63.500 294.789
24.500 64.500 294.844
24.500 65.500 294.562
24.500 66.500 294.734
24.500 67.500 296.367
24.500 68.500 297.438
24.500 69.500 298.531
24.500 70.500 298.453
24.500 71.500 299.195
24.500 72.500 300.062
24.500 73.500 -9999.000
24.500 74.500 -9999.000
24.500 75.500 -9999.000
24.500 76.500 -9999.000
24.500 77.500 -9999.000
24.500 78.500 -9999.000
25.500 60.500 296.258
25.500 61.500 296.391
25.500 62.500 296.672
25.500 63.500 296.398
25.500 64.500 295.773
25.500 65.500 295.812
25.500 66.500 296.609
25.500 67.500 297.977
25.500 68.500 297.109
25.500 69.500 297.828
25.500 70.500 298.430
25.500 71.500 298.836
25.500 72.500 298.703
25.500 73.500 -9999.000
25.500 74.500 -9999.000
25.500 75.500 -9999.000
25.500 76.500 -9999.000
25.500 77.500 -9999.000
25.500 78.500 299.023
26.500 60.500 -9999.000
26.500 61.500 298.266
26.500 62.500 296.773
26.500 63.500 -9999.000
26.500 64.500 -9999.000
26.500 65.500 -9999.000
26.500 66.500 297.250
26.500 67.500 296.188
26.500 68.500 295.938
26.500 69.500 296.906
26.500 70.500 297.828
26.500 71.500 299.312
26.500 72.500 299.359
26.500 73.500 -9999.000
26.500 74.500 -9999.000
26.500 75.500 -9999.000
26.500 76.500 -9999.000
26.500 77.500 298.875
26.500 78.500 296.773
27.500 60.500 -9999.000
27.500 61.500 -9999.000
27.500 62.500 -9999.000
27.500 63.500 -9999.000
27.500 64.500 -9999.000
27.500 65.500 -9999.000
27.500 66.500 -9999.000
27.500 67.500 295.352
27.500 68.500 295.148
27.500 69.500 295.750
27.500 70.500 295.750
27.500 71.500 296.070
27.500 72.500 295.227
27.500 73.500 -9999.000
27.500 74.500 -9999.000
27.500 75.500 -9999.000
27.500 76.500 -9999.000
27.500 77.500 -9999.000
27.500 78.500 296.609
28.500 60.500 -9999.000
28.500 61.500 -9999.000
28.500 62.500 -9999.000
28.500 63.500 -9999.000
28.500 64.500 -9999.000
28.500 65.500 -9999.000
28.500 66.500 -9999.000
28.500 67.500 295.773
28.500 68.500 295.375
28.500 69.500 295.438
28.500 70.500 294.664
28.500 71.500 294.906
28.500 72.500 294.812
28.500 73.500 295.805
28.500 74.500 -9999.000
28.500 75.500 -9999.000
28.500 76.500 -9999.000
28.500 77.500 -9999.000
28.500 78.500 -9999.000
29.500 60.500 -9999.000
29.500 61.500 -9999.000
29.500 62.500 -9999.000
29.500 63.500 -9999.000
29.500 64.500 -9999.000
29.500 65.500 -9999.000
29.500 66.500 -9999.000
29.500 67.500 295.719
29.500 68.500 296.797
29.500 69.500 293.375
29.500 70.500 294.305
29.500 71.500 294.070
29.500 72.500 293.750
29.500 73.500 295.539
29.500 74.500 -9999.000
29.500 75.500 -9999.000
29.500 76.500 -9999.000
29.500 77.500 -9999.000
29.500 78.500 -9999.000
30.500 60.500 -9999.000
30.500 61.500 -9999.000
30.500 62.500 -9999.000
30.500 63.500 -9999.000
30.500 64.500 -9999.000
30.500 65.500 -9999.000
30.500 66.500 -9999.000
30.500 67.500 -9999.000
30.500 68.500 -9999.000
30.500 69.500 -9999.000
30.500 70.500 293.320
30.500 71.500 292.930
30.500 72.500 293.570
30.500 73.500 294.648
30.500 74.500 295.383
30.500 75.500 -9999.000
30.500 76.500 -9999.000
30.500 77.500 -9999.000
30.500 78.500 -9999.000
31.500 60.500 -9999.000
31.500 61.500 -9999.000
31.500 62.500 -9999.000
31.500 63.500 -9999.000
31.500 64.500 -9999.000
31.500 65.500 -9999.000
31.500 66.500 -9999.000
31.500 67.500 -9999.000
31.500 68.500 -9999.000
31.500 69.500 -9999.000
31.500 70.500 293.992
31.500 71.500 293.422
31.500 72.500 294.438
31.500 73.500 294.141
31.500 74.500 -9999.000
31.500 75.500 -9999.000
31.500 76.500 -9999.000
31.500 77.500 -9999.000
31.500 78.500 -9999.000
32.500 60.500 -9999.000
32.500 61.500 -9999.000
32.500 62.500 -9999.000
32.500 63.500 -9999.000
32.500 64.500 -9999.000
32.500 65.500 -9999.000
32.500 66.500 -9999.000
32.500 67.500 -9999.000
32.500 68.500 -9999.000
32.500 69.500 -9999.000
32.500 70.500 -9999.000
32.500 71.500 294.312
32.500 72.500 294.812
32.500 73.500 -9999.000
32.500 74.500 -9999.000
32.500 75.500 -9999.000
32.500 76.500 -9999.000
32.500 77.500 -9999.000
32.500 78.500 -9999.000
33.500 60.500 -9999.000
33.500 61.500 -9999.000
33.500 62.500 -9999.000
33.500 63.500 -9999.000
33.500 64.500 -9999.000
33.500 65.500 -9999.000
33.500 66.500 -9999.000
33.500 67.500 -9999.000
33.500 68.500 -9999.000
33.500 69.500 -9999.000
33.500 70.500 -9999.000
33.500 71.500 -9999.000
33.500 72.500 -9999.000
33.500 73.500 -9999.000
33.500 74.500 -9999.000
33.500 75.500 -9999.000
33.500 76.500 -9999.000
33.500 77.500 -9999.000
33.500 78.500 -9999.000
34.500 60.500 -9999.000
34.500 61.500 -9999.000
34.500 62.500 -9999.000
34.500 63.500 -9999.000
34.500 64.500 -9999.000
34.500 65.500 -9999.000
34.500 66.500 -9999.000
34.500 67.500 -9999.000
34.500 68.500 -9999.000
34.500 69.500 -9999.000
34.500 70.500 -9999.000
34.500 71.500 -9999.000
34.500 72.500 -9999.000
34.500 73.500 -9999.000
34.500 74.500 -9999.000
34.500 75.500 -9999.000
34.500 76.500 -9999.000
34.500 77.500 -9999.000
34.500 78.500 -9999.000
35.500 60.500 -9999.000
35.500 61.500 -9999.000
35.500 62.500 -9999.000
35.500 63.500 -9999.000
35.500 64.500 -9999.000
35.500 65.500 -9999.000
35.500 66.500 -9999.000
35.500 67.500 -9999.000
35.500 68.500 -9999.000
35.500 69.500 -9999.000
35.500 70.500 -9999.000
35.500 71.500 -9999.000
35.500 72.500 -9999.000
35.500 73.500 -9999.000
35.500 74.500 -9999.000
35.500 75.500 -9999.000
35.500 76.500 -9999.000
35.500 77.500 -9999.000
35.500 78.500 -9999.000
36.500 60.500 276.742
36.500 61.500 274.406
36.500 62.500 -9999.000
36.500 63.500 -9999.000
36.500 64.500 -9999.000
36.500 65.500 272.219
36.500 66.500 273.023
36.500 67.500 275.875
36.500 68.500 -9999.000
36.500 69.500 -9999.000
36.500 70.500 -9999.000
36.500 71.500 -9999.000
36.500 72.500 -9999.000
36.500 73.500 -9999.000
36.500 74.500 -9999.000
36.500 75.500 -9999.000
36.500 76.500 -9999.000
36.500 77.500 -9999.000
36.500 78.500 -9999.000
37.500 60.500 277.406
37.500 61.500 277.547
37.500 62.500 276.375
37.500 63.500 275.484
37.500 64.500 276.820
37.500 65.500 275.312
37.500 66.500 274.875
37.500 67.500 275.875
37.500 68.500 -9999.000
37.500 69.500 -9999.000
37.500 70.500 -9999.000
37.500 71.500 -9999.000
37.500 72.500 -9999.000
37.500 73.500 -9999.000
37.500 74.500 -9999.000
37.500 75.500 -9999.000
37.500 76.500 -9999.000
37.500 77.500 -9999.000
37.500 78.500 -9999.000
Here is an example using scatteredInterpolant.
% Load the data
data = load('temperature_data.txt');
% separate the data columns, just to make the code clear
Lat = data(:,1); % Column 1 is Latitude
Lon = data(:,2); % Column 2 is Longitude
Tmp = data(:,3); % Column 3 is Temperature
% Find the "good" data points
good_temp = find(Tmp > -9999);
% create the interpolant object using the good data points
T = scatteredInterpolant(Lat(good_temp),Lon(good_temp),Tmp(good_temp),'linear');
% find the "bad" data points
bad_temp = find(Tmp == -9999);
% use the interpolation object to interpolate temperature values
interp_values = T(Lat(bad_temp),Lon(bad_temp));
% replace the bad values with the interpolated values
Tmp(bad_temp) = interp_values;