My site's Tomcat log has many failure connection (ex 302, 404 status code). such as below. there are too many.
94.191.77.57 - - [16/Jun/2019:05:16:38 +0000] "GET /index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:38 +0000] "GET /phpmyadmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:38 +0000] "GET /phpMyAdmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:39 +0000] "GET /pmd/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:39 +0000] "GET /pma/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:39 +0000] "GET /PMA/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:41 +0000] "GET /PMA2/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:41 +0000] "GET /pmamy/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:41 +0000] "GET /pmamy2/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:42 +0000] "GET /mysql/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:42 +0000] "GET /admin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:42 +0000] "GET /db/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:42 +0000] "GET /dbadmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:45 +0000] "GET /admin/pma/index.php HTTP/1.1" 302
94.191.77.57 - - [16/Jun/2019:05:16:50 +0000] "GET /phpmyadmin2/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:50 +0000] "GET /phpMyAdmin-4.4.0/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:50 +0000] "GET /myadmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:51 +0000] "GET /myadmin2/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:51 +0000] "GET /xampp/phpmyadmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:51 +0000] "GET /phpMyadmin_bak/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:52 +0000] "GET /www/phpMyAdmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:53 +0000] "GET /tools/phpMyAdmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:53 +0000] "GET /phpmyadmin-old/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:53 +0000] "GET /phpMyAdminold/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:53 +0000] "GET /phpMyAdmin.old/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:54 +0000] "GET /pma-old/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:54 +0000] "GET /claroline/phpMyAdmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:54 +0000] "GET /typo3/phpmyadmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:55 +0000] "GET /phpma/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:55 +0000] "GET /phpmyadmin/phpmyadmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:55 +0000] "GET /phpMyAdmin/phpMyAdmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:55 +0000] "GET /phpMyAbmin/index.php HTTP/1.1" 302 -
94.191.77.57 - - [16/Jun/2019:05:16:56 +0000] "GET /phpMyAdmin__/index.php HTTP/1.1" 302-
94.191.77.57 - - [16/Jun/2019:05:16:57 +0000] "GET /phpMyAdmin+++---/index.php HTTP/1.1" 302 -
I think it needs to prepare SQL injecttion attack.
What's your opinion?
You should limit the number of failed requests allowed from any IP address. I'd put the site behind a proxy server such as CloudFlare.
Related
I am debugging http requests to our server and decided to try Dio dart package. After some trials (with no difference in results from standard http packages), I decided to stop using the Dio package.
I though happen to notice extraneous requests from random location (traced back to China telecom). Considering we are only trying to setup the server, and the requests started showing up only after I used Dio in my flutter app - Is DIO snooping on my server?
Seen on Server
X-Forwarded-Protocol: https
X-Real-Ip: 183.136.225.35
Host: 0.0.0.0:5002
Connection: close
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 QIHU 360SE
Accept: */*
Referer: ******
Accept-Encoding: gzip
2022-10-06 15:06:06,768 [DEBUG] root:
X-Forwarded-Protocol: https
X-Real-Ip: 45.79.204.46
Host: 0.0.0.0:5002
Connection: close
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36
Accept: */*
Referer: *****
Accept-Encoding: gzip
Traceroute on IP
4 142 ms 141 ms 153 ms 116.119.68.60
5 140 ms 138 ms 140 ms be6391.rcr21.b015591-1.lon13.atlas.cogentco.com [149.14.224.161]
6 139 ms 139 ms 139 ms be2053.ccr41.lon13.atlas.cogentco.com [130.117.2.65]
7 144 ms 142 ms 142 ms 154.54.61.158
8 191 ms 190 ms 190 ms chinatelecom.demarc.cogentco.com [149.14.81.226]
9 299 ms * 299 ms 202.97.13.18
10 * 316 ms * 202.97.90.30
11 * 317 ms * 202.97.24.141
12 * * * Request timed out.
13 317 ms 308 ms 320 ms 220.191.200.166
14 334 ms 354 ms * 115.233.128.133
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 325 ms 325 ms 333 ms 183.136.225.35
I think this question can be rewritten to "Using Spring Boot and keycloak-spring-boot-starter, what happens after KeycloakSpringBootConfigResolver.resolve()?"
I have a custom keycloak config resolver:
public class CustomKeycloakConfigResolver
extends KeycloakSpringBootConfigResolver {
...
#Override
public KeycloakDeployment resolve(final HttpFacade.Request request) {
LOGGER.debug("-----------------------------------------------");
LOGGER.debug("Resolving Deployment for {}", request.getURI());
...
LOGGER.trace("---------- CREATING KEYCLOAK DEPLOYMENT ---------");
KeycloakDeployment keycloakDeployment =
KeycloakDeploymentBuilder.build(adapterConfig);
LOGGER.trace("---------- /CREATED KEYCLOAK DEPLOYMENT ---------");
return keycloakDeployment;
This pulls the KeycloakDeployment configuration from our database instead of from application.properties. Testing on a local docker swarm cluster, it works like a charm (but this is only on one machine, without SSL enabled, etc).
Pushing out to our QA environment (nginx on one machine with TLS termination, REST service on another, Keycloak on another, database provided by RDS), the generation of a KeycloakDeployment goes off without a hitch. This includes reaching out to .well-known/openid-configuration and resolving all of the Keycloak URLs, and printing both of the final trace() statements.
Almost immediately after (within 10-30ms) creating the KeycloakDeployment and returning it to the keycloak-spring-boot-starter framework, I receive a SocketTimeoutException exception. I can't tell from the exception what the system is trying to do when throws this exception, and I haven't been able to tell from https://github.com/keycloak/keycloak what the workflow is after the deployment is "resolved()".
So - what happens next?
A secured method is accessed
Spring Boot auth hands off to Keycloak auth
Keycloak auth generates a custom KeycloakDeployment
KeycloakDeployment is resolved - reaches out to Keycloak service and obtains the OIDC configuration
KeycloakDeployment is passed back to Keycloak auth framework
... Something happens that throws an immediate socket timeout exception...
Method is never invoked
How do I figure out what's happening in step 6? I find it hard to believe it's an actual socket timeout after only 20ms, and everything I think it should be accessing is up and responsive. But I'm willing to be wrong...
---- Original Question ----
I'm trying to get Keycloak working with a Spring Boot REST service behind an nginx proxy. TLS is terminated at the nginx server. Everything is in docker containers, but on separate machines (no kube, no swarm, etc).
Everything seems good. I can login to the master realm, create a new realm, add users, etc. However when the REST service tries to contact the Keycloak server (through the nginx proxy), I'm getting an SSL timeout error:
2022-03-22 18:59:13.384 INFO 26 --- [nio-8080-exec-3] o.keycloak.adapters.KeycloakDeployment : Loaded URLs from https://hostname/auth/realms/myrealm/.well-known/openid-configuration
javax.net.ssl|WARNING|18|http-nio-8080-exec-3|2022-03-22 18:59:13.409 UTC|SSLSocketImpl.java:1672|handling exception (
"throwable" : {
java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
...
Thing is, this "timeout" happens almost instantaneously after the .well-known/openid-configuration response, so I'm skeptical it's even actually a timeout, unless the threshold is set to like 10ms by default or something.
I cranked up javax.net logging with System.setProperty("javax.net.debug", "ssl:all");, and I can't see anything obvious:
2022-03-22 18:59:13.384 INFO 26 --- [nio-8080-exec-3] o.keycloak.adapters.KeycloakDeployment : Loaded URLs from https://hostname/auth/realms/myrealm/.well-known/openid-configuration
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.385 UTC|SSLSocketOutputRecord.java:331|WRITE: TLSv1.2 application_data, length = 11
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.385 UTC|SSLCipher.java:1770|Plaintext before ENCRYPTION (
0000: 07 00 00 00 03 63 6F 6D 6D 69 74 .....commit
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.388 UTC|SSLSocketOutputRecord.java:346|Raw write (
0000: 17 03 03 00 23 00 00 00 00 00 00 00 80 9C 76 46 ....#.........vF
0010: 44 FA F9 3A A4 9B A1 B2 D8 9B 6A 69 76 C7 1A 3D D..:......jiv..=
0020: 94 C4 40 D2 D8 F2 E4 7E ..#.....
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.388 UTC|SSLSocketInputRecord.java:488|Raw read (
0000: 17 03 03 00 23 ....#
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.388 UTC|SSLSocketInputRecord.java:214|READ: TLSv1.2 application_data, length = 35
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.388 UTC|SSLSocketInputRecord.java:488|Raw read (
0000: 15 13 12 41 88 AD 18 6F B8 5E 25 90 9D BA 23 BF ...A...o.^%...#.
0010: B3 A5 A9 5E 61 FA 77 BD AE A4 C0 57 B2 1D 5B 18 ...^a.w....W..[.
0020: E8 C7 77 ..w
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.388 UTC|SSLSocketInputRecord.java:247|READ: TLSv1.2 application_data, length = 35
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.388 UTC|SSLCipher.java:1672|Plaintext after DECRYPTION (
0000: 07 00 00 01 00 00 00 00 00 00 00 ...........
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.388 UTC|SSLSocketOutputRecord.java:331|WRITE: TLSv1.2 application_data, length = 21
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.389 UTC|SSLCipher.java:1770|Plaintext before ENCRYPTION (
0000: 11 00 00 00 03 53 45 54 20 61 75 74 6F 63 6F 6D .....SET autocom
0010: 6D 69 74 3D 31 mit=1
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.390 UTC|SSLSocketOutputRecord.java:346|Raw write (
0000: 17 03 03 00 2D 00 00 00 00 00 00 00 81 6B 87 27 ....-........k.'
0010: 9C 91 53 E2 F8 70 1C D4 FA F3 4A 79 1B B0 11 05 ..S..p....Jy....
0020: 13 3E 4F 10 A8 E8 43 B3 BB FA 1E 48 82 DF 59 25 .>O...C....H..Y%
0030: CF 9D ..
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.392 UTC|SSLSocketInputRecord.java:488|Raw read (
0000: 17 03 03 00 23 ....#
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.392 UTC|SSLSocketInputRecord.java:214|READ: TLSv1.2 application_data, length = 35
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.392 UTC|SSLSocketInputRecord.java:488|Raw read (
0000: 15 13 12 41 88 AD 18 70 E3 20 7E 21 DA B0 24 28 ...A...p. .!..$(
0010: EF 6D EB BC 5C CE 5D 94 1D BC 04 BB F9 D1 3D 72 .m..\.].......=r
0020: 0C 71 83 .q.
)
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.392 UTC|SSLSocketInputRecord.java:247|READ: TLSv1.2 application_data, length = 35
javax.net.ssl|DEBUG|18|http-nio-8080-exec-3|2022-03-22 18:59:13.393 UTC|SSLCipher.java:1672|Plaintext after DECRYPTION (
0000: 07 00 00 01 00 00 00 02 00 00 00 ...........
)
javax.net.ssl|WARNING|18|http-nio-8080-exec-3|2022-03-22 18:59:13.409 UTC|SSLSocketImpl.java:1672|handling exception (
"throwable" : {
java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
...
So between the well-known configuration call and the final timeout exception:
18:59:13.384
18:59:13.409
all of 25ms passes by - seems hard to believe there would be a legitimate timeout exception thrown, but I can't seem to get any more clue as to what is causing the timeout. The keycloak service definitely IS reachable, and responds quite quickly.
Nothing at all in the Keycloak logs, and not terribly much in the nginx logs:
[--- AUTH ---] [22/Mar/2022:19:19:56 +0000] [200] "POST /auth/realms/myrealm/protocol/openid-connect/token HTTP/1.1" "https://web.mycompany.com/"
[--- AUTH ---] [22/Mar/2022:19:19:57 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:57 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
2022/03/22 19:19:57 [info] 73#73: *659 client #.#.#.# closed keepalive connection
[--- AUTH ---] [22/Mar/2022:19:19:57 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:57 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:58 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:58 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
2022/03/22 19:19:58 [info] 73#73: *670 client #.#.#.# closed keepalive connection
2022/03/22 19:19:58 [info] 73#73: *668 client #.#.#.# closed keepalive connection
[--- AUTH ---] [22/Mar/2022:19:19:58 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:58 +0000] [200] "GET /auth/realms/myrealm/protocol/openid-connect/certs HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:58 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:58 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:59 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:59 +0000] [200] "GET /auth/realms/myrealm/protocol/openid-connect/certs HTTP/1.1" "-"
2022/03/22 19:19:59 [info] 73#73: *677 client #.#.#.# closed keepalive connection
2022/03/22 19:19:59 [info] 73#73: *675 client #.#.#.# closed keepalive connection
[--- AUTH ---] [22/Mar/2022:19:19:59 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:59 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:19:59 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:20:00 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
2022/03/22 19:20:00 [info] 73#73: *664 client #.#.#.# closed keepalive connection
2022/03/22 19:20:00 [info] 73#73: *666 client #.#.#.# closed keepalive connection
2022/03/22 19:20:00 [info] 73#73: *672 client #.#.#.# closed keepalive connection
2022/03/22 19:20:00 [info] 73#73: *682 client #.#.#.# closed keepalive connection
2022/03/22 19:20:00 [info] 73#73: *684 client #.#.#.# closed keepalive connection
2022/03/22 19:20:00 [info] 74#74: *686 client #.#.#.# closed keepalive connection
[--- AUTH ---] [22/Mar/2022:19:20:00 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:20:00 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:20:01 +0000] [200] "GET /auth/realms/myrealm/.well-known/openid-configuration HTTP/1.1" "-"
[--- AUTH ---] [22/Mar/2022:19:20:01 +0000] [200] "GET /auth/realms/myrealm/protocol/openid-connect/certs HTTP/1.1" "-"
[--- REST ---] [22/Mar/2022:19:20:01 +0000] [401] "GET /api/v1/stuff HTTP/1.1" "https://web.mycompany.com/"
Any help as to what's going on?
Other notes:
I've gone through the Setting Up a load balancer or proxy instructions from Keycloak. Specifically, the nginx server has the following:
upstream AUTH {
server #.#.#.#:8080;
server #.#.#.#:8080;
}
...
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_pass http://AUTH;
I think that covers bullet points 1 and 2.
Bullet point 3 is covered by the keycloak server. The standalone-ha.xml file has:
<subsystem xmlns="urn:jboss:domain:undertow:12.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other" statistics-enabled="true">
<buffer-cache name="default"/>
<server name="default-server">
<ajp-listener name="ajp" socket-binding="ajp"/>
<http-listener name="default" socket-binding="http" redirect-socket="https" proxy-address-forwarding="${env.PROXY_ADDRESS_FORWARDING:false}" enable-http2="true"/>
<https-listener name="https" socket-binding="https" ssl-context="applicationSSC" proxy-address-forwarding="${env.PROXY_ADDRESS_FORWARDING:false}" enable-http2="true"/>
<host name="default-host" alias="localhost">
<location name="/" handler="welcome-content"/>
<http-invoker http-authentication-factory="application-http-authentication"/>
</host>
</server>
and the container is launched with the environment variable
PROXY_ADDRESS_FORWARDING=true
I've verified with a false login that Keycloak is seeing the end user's IP address rather than the nginx server:
15:17:07,304 WARN [org.keycloak.events] (default task-14) type=LOGIN_ERROR, realmId=master, clientId=security-admin-console, userId=null, ipAddress=#.#.#.#, error=user_not_found, auth_method=openid-connect, auth_type=code, redirect_uri=https://host.name/auth/admin/master/console/#/realms/MyRealm/login-settings, code_id=..., username=foo, authSessionParentId=..., authSessionTabId=...
Where #.#.#.# is my IP address.
I am trying to parse a log file with millions of records. It contains host name, timestamp, status code etc. After successfully parsing host and status code and url, when I am trying to parse timestamp, I am getting an error. following is my code:
lines=sc.textFile(filepath)
df_log= lines.map(lambda x: Row(header=x)).toDF()
timestamp_pattern= r'\[\d{2}\/\w{3}\/\d{4}\:\d{2}\:\d{2}\:\d{2}\s\S+\d{4}]'
df2=df_log.select(regexp_extract(col('header'),timestamp_pattern,1).alias("timestamp"))
everything is working fine till here. after this when I am trying to df2.show(10). I am getting following error:
Py4JJavaErrorTraceback (most recent call last) <ipython-input-112-5a86d13b2926> in <module>()
----> 1 df2.show(1)
/opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/sql/dataframe.pyc in show(self, n, truncate)
316 """
317 if isinstance(truncate, bool) and truncate:
--> 318 print(self._jdf.showString(n, 20))
319 else:
320 print(self._jdf.showString(n, int(truncate)))
/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in __call__(self, *args) 1131 answer = self.gateway_client.send_command(command) 1132 return_value
= get_return_value(
-> 1133 answer, self.gateway_client, self.target_id, self.name) 1134 1135 for temp_arg in temp_args:
/opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/sql/utils.pyc in deco(*a, **kw)
61 def deco(*a, **kw):
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()
/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
317 raise Py4JJavaError(
318 "An error occurred while calling {0}{1}{2}.\n".
--> 319 format(target_id, ".", name), value)
320 else:
321 raise Py4JError(
Py4JJavaError: An error occurred while calling o965.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 5, ip-20-0-31-210.ec2.internal, executor 2): java.lang.IndexOutOfBoundsException: No group 1 at java.util.regex.Matcher.group(Matcher.java:538) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1430) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1417) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1417) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:797) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:797) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:797) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1645) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1600) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1589) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:623) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1930) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1943) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1956) at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:333) at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38) at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2378) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57) at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2772) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2377) at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2384) at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2120) at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2119) at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2802) at org.apache.spark.sql.Dataset.head(Dataset.scala:2119) at org.apache.spark.sql.Dataset.take(Dataset.scala:2334) at org.apache.spark.sql.Dataset.showString(Dataset.scala:248) at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IndexOutOfBoundsException: No group 1 at java.util.regex.Matcher.group(Matcher.java:538) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231) at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more
Please help me with this.
following is the excerpt of the records in the log file for referance:
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] GET /administrator/ HTTP/1.1 200 4263 - Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] GET /administrator/ HTTP/1.1 200 4263 - Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] GET /administrator/ HTTP/1.1 200 4263 - Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] POST /administrator/index.php HTTP/1.1 200 4494 http://almhuette-raith.at/administrator/ Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
46.72.177.4 - - [12/Dec/2015:18:31:08 +0100] GET /administrator/ HTTP/1.1 200 4263 - Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
46.72.177.4 - - [12/Dec/2015:18:31:08 +0100] POST /administrator/index.php HTTP/1.1 200 4494 http://almhuette-raith.at/administrator/ Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
83.167.113.100 - - [12/Dec/2015:18:31:25 +0100] GET /administrator/ HTTP/1.1 200 4263 - Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
83.167.113.100 - - [12/Dec/2015:18:31:25 +0100] POST /administrator/index.php HTTP/1.1 200 4494 http://almhuette-raith.at/administrator/ Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
95.29.198.15 - - [12/Dec/2015:18:32:10 +0100] GET /administrator/ HTTP/1.1 200 4263 - Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
95.29.198.15 - - [12/Dec/2015:18:32:11 +0100] POST /administrator/index.php HTTP/1.1 200 4494 http://almhuette-raith.at/administrator/ Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
109.184.11.34 - - [12/Dec/2015:18:32:56 +0100] GET /administrator/ HTTP/1.1 200 4263 - Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
109.184.11.34 - - [12/Dec/2015:18:32:56 +0100] POST /administrator/index.php HTTP/1.1 200 4494 http://almhuette-raith.at/administrator/ Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0 -
`
change last line to -
timestamp_pattern= r'\[\d{2}\/\w{3}\/\d{4}\:\d{2}\:\d{2}\:\d{2}\s\S+\d{4}]'
df2=df_log.select(regexp_extract(col('header'),timestamp_pattern,0).alias("timestamp"))
note: I changed the groupId from 1 to 0 since there is no group set in your timestamp_pattern
I tried to download image from my server over www. In UnityPlayer it is all right, but when i run app on xcode it is crash.
I check log on server and there was no request from ios only from webplayer:
109.252.44.138 - - [17/Feb/2018:09:53:18 +0300] "POST /unityConnect.php HTTP/1.0" 200 604 "-" "cocktailparty/27
CFNetwork/893.14.2 Darwin/17.3.0" 452 746 0 125187
109.252.44.138 - - [17/Feb/2018:09:53:39 +0300] "POST /unityConnect.php HTTP/1.0" 200 14 "-" "cocktailparty/27
CFNetwork/893.14.2 Darwin/17.3.0" 563 156 0 2266
109.252.44.138 - - [17/Feb/2018:09:53:39 +0300] "POST /unityConnect.php HTTP/1.0" 200 170 "-" "cocktailparty/27
CFNetwork/893.14.2 Darwin/17.3.0" 421 312 0 1807
109.252.44.138 - - [17/Feb/2018:09:53:43 +0300] "POST /unityConnect.php HTTP/1.0" 200 992 "-" "cocktailparty/27
CFNetwork/893.14.2 Darwin/17.3.0" 430 1134 0 243826
109.252.44.138 - - [17/Feb/2018:09:53:45 +0300] "POST /unityConnect.php HTTP/1.0" 200 604 "-" "cocktailparty/27
CFNetwork/893.14.2 Darwin/17.3.0" 452 746 0 2482
109.252.44.138 - - [17/Feb/2018:09:54:05 +0300] "POST /unityConnect.php HTTP/1.0" 200 604 "-" "cocktailparty/27
CFNetwork/893.14.2 Darwin/17.3.0" 452 746 0 2452
109.252.44.138 - - [17/Feb/2018:09:54:25 +0300] "POST /unityConnect.php HTTP/1.0" 200 13 "-" "UnityPlayer/2017.3.0f3
(UnityWebRequest/1.0, libcurl/7.51.0-DEV)" 595 155 0 1760
109.252.44.138 - - [17/Feb/2018:09:54:25 +0300] "POST /unityConnect.php HTTP/1.0" 200 170 "-" "UnityPlayer/2017.3.0f3
(UnityWebRequest/1.0, libcurl/7.51.0-DEV)" 414 312 0 1034
109.252.44.138 - - [17/Feb/2018:09:54:29 +0300] "POST /unityConnect.php HTTP/1.0" 200 992 "-" "UnityPlayer/2017.3.0f3
(UnityWebRequest/1.0, libcurl/7.51.0-DEV)" 423 1134 0 263959
109.252.44.138 - - [17/Feb/2018:09:54:32 +0300] "POST /unityConnect.php HTTP/1.0" 200 604 "-" "UnityPlayer/2017.3.0f3
(UnityWebRequest/1.0, libcurl/7.51.0-DEV)" 445 746 0 3588
109.252.44.138 - - [17/Feb/2018:09:54:32 +0300] "GET /img/0002.jpg HTTP/1.1" 200 73962 "-" "UnityPlayer/2017.3.0f3 (UnityWebRequest/1.0,
libcurl/7.51.0-DEV)"
Code:
public void GetFullReceiptMySQL(Dictionary <string, string> [] newReader)
{
StartCoroutine(StartLoad(newReader[0]["id"], newReader));
}
IEnumerator StartLoad(string idCocktail, Dictionary <string, string> [] newReader)
{
// testing parameter 0002, image 0002 present on server
string imageUrl = GetFullSourceCocktailImage("0002");
WWW www = new WWW(imageUrl);
Debug.Log(imageUrl);
// Wait for download to complete
Debug.Log ("Done1? " + www.isDone + " progress " + www.progress + " or with error " + www.error) ;
yield return www;
Debug.Log ("Done2? " + www.isDone + " progress " + www.progress + " or with error " + www.error) ;
if (!string.IsNullOrEmpty(www.error))
{
Debug.Log("error " + www.error);
}
else
{
Debug.Log("image ready" + www.error);
RawImage image = GetComponent<KeeperScript>().imageCocktailInReceipt;
Texture2D newTexture = new Texture2D(www.texture.width, www.texture.height);
www.LoadImageIntoTexture(newTexture as Texture2D);
image.GetComponent<RawImage>().texture= newTexture;
MakeAllReceipt(newReader);
}
DestroyImmediate(www.texture);
www.Dispose();
www = null;
Resources.UnloadUnusedAssets();
}
In XCode in logs there was error 404
trying to when change /client/BigBlueButton.html portion of the URL according to - http://docs.bigbluebutton.org/support/faq.html#how-do-i-change-the-client-bigbluebutton-html-portion-of-the-url,
but getting -
404 Not Found
nginx/1.4.6 (Ubuntu)
my /etc/bigbluebutton/nginx/client.nginx:
#location /client/BigBlueButton.html {
# root /home/firstuser/dev/bigbluebutton/bigbluebutton-client;
# index index.html index.htm;
# expires 1m;
#}
# BigBlueButton Flash client.
location /client {
root /home/firstuser/dev/bigbluebutton/bigbluebutton-client;
index index.html index.htm;
}
my /etc/bigbluebutton/nginx/rewrite.nginx:
location /client/BigBlueButton.html {
rewrite ^ /conference permanent;
}
location /conference {
alias /var/www/bigbluebutton/client;
index BigBlueButton.html;
expires 1m;
}
$sudo /etc/init.d/nginx restart
/var/log/nginx/bigbluebutton.access.log
33.126.263.65 - - [16/Aug/2015:12:37:57 -0400] "GET / HTTP/1.1" 200 2852 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:37:57 -0400] "GET /css/bijou.min.css HTTP/1.1" 200 2753 "http://M_IP_Ad/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:37:57 -0400] "GET /css/style.css HTTP/1.1" 200 2918 "http://M_IP_Ad/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:37:57 -0400] "GET /css/font-awesome.min.css HTTP/1.1" 200 20766 "http://M_IP_Ad/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:37:57 -0400] "GET /images/jimtalk-logo.png HTTP/1.1" 200 10251 "http://M_IP_Ad/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:37:57 -0400] "GET /images/bbb-setup-audio.jpg HTTP/1.1" 200 18876 "http://M_IP_Ad/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:37:57 -0400] "GET /images/bbb-viewer-overview.jpg HTTP/1.1" 200 21929 "http://M_IP_Ad/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:37:57 -0400] "GET /images/bbb-presenter-overview.jpg HTTP/1.1" 200 18309 "http://M_IP_Ad/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:37:58 -0400] "GET /fonts/fontawesome-webfont.woff?v=4.1.0 HTTP/1.1" 200 83760 "http://M_IP_Ad/css/font-awesome.min.css" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
M_IP_Ad - - [16/Aug/2015:12:38:00 -0400] "POST /bigbluebutton/api/create?name=Demo+Meeting&meetingID=Demo+Meeting&voiceBridge=72274&attendeePW=ap&moderatorPW=mp&record=false&checksum=a93ab8433532c633ab2467afc0d91e0eb1dc4e88 HTTP/1.1" 200 488 "-" "Java/1.7.0_79"
33.126.263.65 - - [16/Aug/2015:12:38:00 -0400] "GET /demo/demo1.jsp?username=%D7%99%D7%A2%D7%99%D7%A2%D7%99&action=create HTTP/1.1" 200 1003 "http://M_IP_Ad/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:38:01 -0400] "GET /bigbluebutton/api/join?meetingID=Demo+Meeting&fullName=%D7%99%D7%A2%D7%99%D7%A2%D7%99&password=mp&checksum=3438ed39a50723be59798038f86fcba0af30b325 HTTP/1.1" 302 0 "http://M_IP_Ad/demo/demo1.jsp?username=%D7%99%D7%A2%D7%99%D7%A2%D7%99&action=create" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:38:02 -0400] "GET /client/BigBlueButton.html HTTP/1.1" 301 193 "http://M_IP_Ad/demo/demo1.jsp?username=%D7%99%D7%A2%D7%99%D7%A2%D7%99&action=create" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:38:02 -0400] "GET /conference HTTP/1.1" 301 193 "http://M_IP_Ad/demo/demo1.jsp?username=%D7%99%D7%A2%D7%99%D7%A2%D7%99&action=create" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
33.126.263.65 - - [16/Aug/2015:12:38:02 -0400] "GET /conference/ HTTP/1.1" 403 208 "http://M_IP_Ad/demo/demo1.jsp?username=%D7%99%D7%A2%D7%99%D7%A2%D7%99&action=create" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36"
Thank you
The nginx read all the files with nginx extension (*.nginx). I guess the problem is that nginx is not able to handle same locations.