We are using twitter4j for streaming twitter data, but getting "connect timed out" error. But if we use curl with the authorization information and execute from the same system, it works fine and we get the tweets.
Below is the java code snippet:
ArrayList<String> track = new ArrayList<String>();
track.add("#usa");
String[] trackArray = track.toArray(new String[track.size()]);
twitterStream.filter(new FilterQuery(0, null, trackArray));
Below is the stack trace, the google links provided in the error message do not give much information.
Relevant discussions can be found on the Internet at:
http://www.google.co.jp/search?q=944a924a or
http://www.google.co.jp/search?q=24fd66dc
TwitterException{exceptionCode=[944a924a-24fd66dc 944a924a-24fd66b2], statusCode=-1, message=null, code=-1, retryAfter=-1, rateLimitStatus=null, version=3.0.5}
at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:177)
at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:61)
at twitter4j.internal.http.HttpClientWrapper.post(HttpClientWrapper.java:98)
at twitter4j.TwitterStreamImpl.getFilterStream(TwitterStreamImpl.java:304)
at twitter4j.TwitterStreamImpl$7.getStream(TwitterStreamImpl.java:292)
at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:462)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
at java.net.Socket.connect(Socket.java:546)
at sun.net.NetworkClient.doConnect(NetworkClient.java:169)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:289)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:346)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:755)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:858)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:135)
... 5 more
20254 [Twitter Stream consumer-2[Establishing connection]] INFO twitter4j.TwitterStreamImpl - Waiting for 250 milliseconds
Below is the curl command
curl --get 'https://stream.twitter.com/1.1/statuses/filter.json' --data 'track=usa' --header 'Authorization: OAuth oauth_consumer_key="hidden_value", oauth_nonce="hidden_value", oauth_signature="hidden_value", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1388654077", oauth_token="hidden_value", oauth_version="1.0"'
We use a proxy server to connect to internet and are using cntlm with the user credentials along with proxy server details specified in cntlm.
The issue is resolved.
Earlier I was setting the proxy host and port using below code which didn't work:
System.getProperties().put("http.proxyHost", "proxy");
System.getProperties().put("http.proxyPort", "8080");
But it works by setting the proxy using ConfigurationBuilder
cb.setHttpProxyHost("proxy");
cb.setHttpProxyPort(8080);
You can also set proxy host and port by using twitter4j properties in flume-ng command as below
$ flume-ng agent -n TwitAgent -c conf -f /usr/hadoop/FLUME/apache-flume-1.3.1-bin/conf/flume.conf -Dtwitter4j.http.proxyHost=www-proxy.example.com -Dtwitter4j.http.proxyPort=80
Related
Dancer2 is a PSGI web application framework, we can use the "plackup" tool (provided by Plack) for launching the application:
plackup -p 5000 bin/app.psgi
The web application created using Dancer2 script can be viewed on localhost and port 5000.
http://localhost:5000
We can then write a get request as follows:
get '/hello/:name' => sub {
return "Hi there " . route_parameters->get('name');
};
For GET requests to "/hello/...", the code block provided above gets executed.
If the application is down, for GET requests to "/hello/...", the following error is shown:
curl: (7) Failed connect to localhost:5000; Connection refused
How do we integrate Dancer2::Core::HTTP helpers for rendering HTTP status codes for Dancer2 so that the following status code is shown when the app is down? Similarly, there are other error handlers too, how to integrate them as well?
status_500 status_error, status_internal_server_error
My current use case is: I have a frontend application where a user is logged in via Keycloak. I would like to implement some parts of the Ditto HTTP API in this frontend (https://www.eclipse.org/ditto/http-api-doc.html).
For example I want to create policies (https://www.eclipse.org/ditto/basic-policy.html) for authorization. I've read in the documentation that one can use an OpenID Connect compliant provider and the form is : (https://www.eclipse.org/ditto/basic-policy.html#who-can-be-addressed).
There's basic auth example at the bottom of the page, it seems to use the username in this case.
{
"policyId": "my.namespace:policy-a",
"entries": {
"owner": {
"subjects": {
"nginx:ditto": {
"type": "nginx basic auth user"
}
},
...
}
My question is: What exactly would be the sub-claim if I want to use Keycloak? Is it also the username of the user I want to grant rights to? And how would I get this in my frontend where I want to specify the policy for sending it to Ditto afterwards?
UPDATE 1:
I tried to enable keycloak authentication in Ditto like suggested below and as stated here: https://www.eclipse.org/ditto/installation-operating.html#openid-connect
Because I'm running Ditto with Docker Compose, I added the following line as an environment variable in ditto/deployment/docker/docker-compose.yml in line 136: - Dditto.gateway.authentication.oauth.openid-connect-issuers.keycloak=http://localhost:8090/auth/realms/twin
This URL is the same as in the issuer claim of my token which I'm receiving from keycloak.
Now if I try to make for example a post request with Postman to {{basePath}}/things I get the following error:
<html>
<head>
<title>401 Authorization Required</title>
</head>
<body bgcolor="white">
<center>
<h1>401 Authorization Required</h1>
</center>
<hr>
<center>nginx/1.13.12</center>
</body>
</html>
I chose Bearer Token as Auth in Postman and pasted a fresh token. Basic Auth with the default ditto user is still working.
Do I have to specify the new subject/my user in Ditto before?
UPDATE 2:
I managed to turn basic auth in nginx off by commenting out "auth_basic" and "auth_basic_user_file" in nginx.conf!
It seems to be forwarded to Ditto now, because now I get the following error with Postman:
{
"status": 401,
"error": "gateway:jwt.issuer.notsupported",
"message": "The JWT issuer 'localhost:8090/auth/realms/twin' is not supported.",
"description": "Check if your JWT is correct."
}
UPDATE 3:
My configuration in gateway.conf looks now like this:
oauth {
protocol = "http"
openid-connect-issuers = {
keycloak = "localhost:8090/auth/realms/twin"
}
}
I also tried to add these two lines in the docker-compose.yml:
- Dditto.gateway.authentication.oauth.protocol=http
- Dditto.gateway.authentication.oauth.openid-connect-issuers.keycloak=localhost:8090/auth/realms/twin
Unfortunately I still had no luck, same error as above :/ It seems like an user had a similar problem with keycloak before (https://gitter.im/eclipse/ditto?at=5de3ff186a85195b9edcb1a6), but sadly he mentioned no solution.
EDIT: It turns out that I specified these variables in the wrong way, the correct solution is to add them as part of command: java ... more info here
UPDATE 4:
I tried to build Ditto locally instead of using the latest docker images and I think I might be one step further now, it seems like my oauth config is working. I get now:
{
"status": 503,
"error": "gateway:publickey.provider.unavailable",
"message": "The public key provider is not available.",
"description": "If after retry it is still unavailable, please contact the service team."
}
The error message from the log is:
gateway_1 | 2020-11-05 15:33:18,669 WARN [] o.e.d.s.g.s.a.j.DittoPublicKeyProvider - Got Exception from discovery endpoint <http://localhost:8090/auth/realms/twin/.well-known/openid-configuration>.
gateway_1 | akka.stream.StreamTcpException: Tcp command [Connect(localhost:8090,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
gateway_1 | Caused by: java.net.ConnectException: Connection refused
...
gateway_1 | java.util.concurrent.CompletionException: org.eclipse.ditto.services.gateway.security.authentication.jwt.PublicKeyProviderUnavailableException [message='The public key provider is not available.', errorCode=gateway:publickey.provider.unavailable, statusCode=SERVICE_UNAVAILABLE, description='If after retry it is still unavailable, please contact the service team.', href=null, dittoHeaders=ImmutableDittoHeaders [{}]]
...
gateway_1 | Caused by: org.eclipse.ditto.services.gateway.security.authentication.jwt.PublicKeyProviderUnavailableException [message='The public key provider is not available.', errorCode=gateway:publickey.provider.unavailable, statusCode=SERVICE_UNAVAILABLE, description='If after retry it is still unavailable, please contact the service team.', href=null, dittoHeaders=ImmutableDittoHeaders [{}]]
...
gateway_1 | Caused by: akka.stream.StreamTcpException: Tcp command [Connect(localhost:8090,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
gateway_1 | Caused by: java.net.ConnectException: Connection refused
My keyloak is definitely running, I'm able to get tokens. If I'm opening http://localhost:8090/auth/realms/twin/.well-known/openid-configuration which is in the first error message, I'm able to see my openid-configuration from keycloak config.
Edit: It seems that my gateway container cannot reach my keycloak container, will try to figure this out.
FINAL UPDATE:
Unreachable keycloak docker container from the gateway docker container was the issue. I'm now using traefik:
Keycloak container has the following alias: keycloak.localhost
Oauth configuration in the gateway looks like this:
oauth {
protocol = "http"
openid-connect-issuers = {
keycloak = "keycloak.localhost/auth/realms/twin"
}
}
Now the gateway can find the keycloak container via the alias and I can still use the keycloak admin ui from my localhoast: http://keycloak.localhost:8090/auth/admin/
Additional info: Traefic Blog
What exactly would be the sub-claim if I want to use Keycloak?
Keycloak provides you a JWT.
A JWT is an encrypted JSON which contains multiple fields called "claims". You can check how your token looks like by visiting https://jwt.io and pasting your token there. One of those fields is called sub. This is the sub claim.
To enable your keycloak authentication in eclipse ditto you need to add the issuer to the ditto configuration.
An example can be founde here.
The address must match the URL in the issuer claim of your JWT token.
ditto.gateway.authentication {
oauth {
protocol = "http"
openid-connect-issuers = {
some-name = "localhost:8090/auth/realms/twin"
}
}
}
Is it also the username of the user I want to grant rights to?
In eclipse ditto there is not really a concept of "user names". Eclipse ditto authentication is based on authorization subjects. For the basic authentication example you provided, the authorization subject which is generated within ditto is nginx:ditto.
For JWT authentication the authorization subject is generated as a combination of the name for the open id connect issuer which you configured (in my case some-name) and the value of the sub claim. An authorization subject could look like this: some-name:8d078113-3ee5-4dbf-8db1-eb1a6cf0fe81.
And how would I get this in my frontend where I want to specify the policy for sending it to Ditto afterwards?
I'm not sure if I understand the question correctly. If you mean how to authenticate your frontend HTTP requests to eclipse ditto, you need to provide the JWT to eclipse ditto by adding it to the authorization header of your HTTP requests in the following form:
authorization: Bearer yourJWT
If you mean how you would know the sub claim of a JWT, you need to parse the JWT to a JSON object and then read the sub claim out of the payload section.
I'm getting an Exception when running this, but Jenkins actually executes the requested action:
URL url = new URL("https://somehost.com/jenkins/quietDown");
HttpURLConnection c= (HttpURLConnection) url.openConnection();
c.setRequestMethod("POST");
c.setRequestProperty("Authorization", "Basic " + Base64.getEncoder().encodeToString(("user:apiToken").getBytes()));
c.getInputStream().close();
Exception:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://somehost.com/jenkins/
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at build.JenkinsClient.main(JenkinsClient.java:102)
Testing with Jenkins 2.164.3 and Java 8.
Setting this gets rid of the Exception:
connection.setInstanceFollowRedirects(false);
After hours I noticed how the stack trace contained a slightly different URL than the one I was posting to:
https://somehost.com/jenkins/
vs
https://somehost.com/jenkins/quietDown
It seems like Jenkins answers with a redirect (302 Found), which the HttUrlConnections follows by default to read from, which then for some reason caused that exception.
For the longest time, I tried to figure out a way to issue the POST request without calling connection.getInputStream(), but that seems to be the only call which actually triggers the request. If anyone knows a different way to issue a POST request with pure Java, please let me know.
I knew my URL and username:token stuff was correct because I tested with curl (which doesn't complain, even with the follow redirect option turned on):
curl -X POST https://somehost.com/jenkins/quietDown -u admin:token
curl -L -X POST https://somehost.com/jenkins/quietDown -u admin:token
I have the Telit LE920-EUG 4G LTE module. I am trying to execute GET and POST http requests to a remote server. Though the PDP context is activating properly and I have internet access on the SIM that I'm using, I can't seem to be able to connect to a remote server and execute HTTP requests (both POST and GET) from the module.
I have tried two ways, one through direct HTTP commands supported by the module(All commands mentioned in the LE9x0 AT command reference guide), the commands sequence for which is mentioned below, but +CME ERROR: 100 occurs, and it's same for every http command(AT#HHTPQRY, AT#HTTPRCV) that I try to execute.
AT#SGACT=1,1
#SGACT: 31.81.208.1
OK
AT#HTTPCFG=0,"httpbin.org",80,0,,,0,120,1
+CME ERROR: 100
//No configuration details
AT#HTTPCFG?
+CME ERROR: 100
AT#HTTPCFG=?
+CME ERROR: 100
I have also tried the GET and POST commands after socket dialing. The socket connects but they are not receiving any data from the server or posting anything onto the server, the connection closes with a NO CARRIER. The command sequence that I'm using is given below
//Socket Dial
AT#SD=1,0,80,www.m2msupport.net
CONNECT
//GET commands sequence
GET /m2msupport/http_get_test.php HTTP/1.1
Host:www.m2msupport.net
Connection:keep-alive
//Connection closes with No Response
NO CARRIER
//Socket info shows the bytes sent
at#si=1
#SI: 1,86,0,0,0
OK
I want to download the result of a Express.js REST API which is very slow to process (~10 minutes). I tried few timeout options with wget but it gives up after few minutes while I ask it to wait around ~60 000 years.
wget "http://localhost:5000/slowstuff" --http-user=user --http-password=password --read-timeout=1808080878708 --tries=1
--2015-02-26 11:14:21-- http://localhost:5000/slowstuff
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:5000... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="Authorization Required"
Reusing existing connection to [localhost]:5000.
HTTP request sent, awaiting response... No data received.
Giving up.
EDIT:
The problem doesn't come from the wget timeout value. With a timeout set to 4 seconds, the error is different: Read error (Connection timed out) in headers. And I have exactly the same problem with curl.
I think the problem comes from my API. It looks like a timeout of 2 minutes is set by default in NodeJS.
Now, I need to find how to change this value.
This
--http-password=password--read-timeout=1808080878708
is missing a blank. Use
--http-password=password --read-timeout=1808080878708