InfluxDB http calls sending credentials (username & password) in URL as query params - rest

For a sample project for Weather service I needed to store time series data. This is the first time I am using any time-series database. I did some reading on those and their comparison and found that InfluxDB is open-source and is one of the best, so decided to use that.
For my PoC I installed it locally on my machine and connecting it from my application. However, when I see the logs for various queries run against InfluxDB, found that it makes http calls to InfluxDB and it passes the username and password in the Query params in URL. This certainly seems like bad practice to pass credentials as as plain text in the URL while making the http call. Can someone comment why is it designed like this and is it supposed to be like this in real world scenario as well?
Logs:
2019-07-19 12:01:00.304 INFO 69709 --- [pool-1-thread-1] okhttp3.OkHttpClient : --> POST http://127.0.0.1:8086/write?u=admin&p=admin&db=weatherdata&rp=defaultPolicy&precision=n&consistency=one (78-byte body)
2019-07-19 13:48:28.461 INFO 69709 --- [nio-8080-exec-9] okhttp3.OkHttpClient : --> GET http://127.0.0.1:8086/query?u=admin&p=admin&db=weatherdata&q=Select+*+from+weather
2019-07-19 13:48:28.530 INFO 69709 --- [nio-8080-exec-9] okhttp3.OkHttpClient : <-- 200 OK http://127.0.0.1:8086/query?u=admin&p=admin&db=weatherdata&q=Select+*+from+weather (68ms, unknown-length body)

InfluxDB supports HTTP Basic Auth where username and password are passed via HTTP auth headers instead of the URL. I suspect you just need to configure your client to do that instead of using the URL parameters. Credentials are still in plaintext, but I think if you set up HTTPS, Basic Auth is secure-ish.
In general I don't think the Influx Devs expect InfluxDB to be a standalone, public-facing service. Instead, they expect that you're going to use InfluxDB as a backend, and then use something like Chronograf (which is their own visualization tool) or Grafana as a front end. So if they're going to spend time on more sophisticated authentication protocols, they're going to do it on the front end side.
The expectation would be that the front end and back end run on the same network, and communications between them can be secured via network segmentation.

Related

How do I resolve RESTEASY002186 so my Wildfly 26 web application can use SSE over https?

I have a web application running on Wildfly 26 that uses SSE broadcasting and works correctly with http. However, when I switch to using an https endpoint, I get Wildfly log entries of:
WARN [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-1)
RESTEASY002186: Failed to set servlet request into asynchronous mode,
server sent events may not work
This happens with each registration attempt of the https endpoint but I never see this when registering with the http endpoint.
Testing with curl against the http endpoint results in curl waiting for events to show up (and keeps printing them out as it receives them) until I quit. Using curl to test the https endpoint, I will see the same headers I got from the http endpoint, namely:
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked
Content-Type: text/event-stream
But after printing out my registration successful event, curl seems to believe the stream is closed and exits -- giving me my command prompt back.
My #GET MediaType.SERVER_SENT_EVENTS registration endpoint will create an OutboundSseEvent and send it to the SseEventSink to acknowledge successful registration to my SseBroadcaster instance (this is the event curl sees and prints before exiting). I then log a registration successful message before exiting the method. All of this appears to work correctly for both http and https but the stream doesn't stay open once the request endpoint completes because of the failure to run asynchronously as outlined above.
I have not found information on the causes and/or workaround solutions for my RESTEASY002186 problem. I posted a question on this issue last week using the Wildfly Google Group (https://groups.google.com/g/wildfly/c/SO2eHdvMEko) but thought I would try a wider audience since this doesn't seem to be a commonly experienced condition. I don't see any indications during initialization that WildFly will be unable to use asynchronous mode, it just complains when it tries and fails... Any help would be greatly appreciated!
Edit 6/6/2022
The code is running on an isolated network so I can't just cut/paste the code here, but I gutted the resources file to a bare minimum -- just leaving enough for the client to be able to register. The problem remains unchanged. The code is now essentially:
#Path("sse")
public class SseResources {
#GET
#Produces(MediaType.SERVER_SENT_EVENTS)
public void listen(#Context Sse sse, #Context SseEventSink sseEventSink) {
SseRegComplete regComplete = new SseRegComplete("sse-server");
OutboundSseEvent event = sse.newEventBuilder().
name(regComplete.getType().toString()).
id(regComplete.getEventId()).
mediaType(MediaType.APPLICATION_JSON_TYPE).
data(SseRegComplete.class, regComplete).
comment("Event Stream Registration Completed Successfully").
build();
sseEventSink.send(event);
}
}
Before the above simplified code, I had declared the resource as #ApplicationScoped, had Sse injected into it, and kept a reference to the SseBroadcaster so I could use it whenever an event would come in. I was catching the events to broadcast by using an #Observes method (which I also got rid of). I was calling register(sseEventSink) on the SseBroadcaster in the listen method so I could later call broadcast(outboundEvent) whenever I had updates to publish. I got rid of all that just to see if I could get the stream to stay open but to no avail. I still get the RESTEASY002186 message and curl still exits after printing out the regComplete event sent to it in the code above.
Edit 6/7/2022
Yesterday I was able to get my code working in a new vanilla Wildfly 26 install using an https endpoint URL by following these configuration instructions. Something I hadn't mentioned in the original post is that I am trying to add SSE functionality to an already existing app. It is several years old and we actually moved to Wildfly 26 about 6 months ago because of the log4j vulnerability in the earlier version of Wildfly we were using. I suspect that the problem is related to either our Wildfly configuration (perhaps because old settings were brought over that shouldn't have been) or some 3rd party dependency that is preventing Wildfly from using asynchronous mode.
We are using Shiro for authentication and authorization against an LDAP server -- perhaps Shiro has some hooks into the Wildfly runtime that are causing issues? After initial login, we use a session cookie in all subsequent calls. That is a difference from my test server but I don't think it is relevant because the call definitely passed authentication before executing the registration code. The only other thing that comes to mind right now is our web app ships with LogBack and tells Wildfly not to use the default logging framework.
I plan to start today by comparing the two standalone.xml files to see if anything jumps out at me as being fundamentally different. Is there anything else I should be checking for differences (I think there is a domain.xml file somewhere...)?
Edit 6/14/2022
This definitely has something to do with Shiro being in the loop. When I edit the web.xml file to have Shiro's filter-mapping url-pattern to not include the SSE endpoint, everything works as expected.

Postman : socket hang up

I just started using Postman. I had this error "Error: socket hang up" when I was executing a collection runner. I've read a few post regarding socket hang up and it mention about sending a request and there's no response from the server side and probably timeout. How do I extend the length of time of the request in Postman Collection Runner?
For me it was because my application was switched to https and my postman requests still had http in them. Changing postman to https fixed it.
Socket hang up, error is port related error. I am sharing my experience. When you use same port for connecting database, which port is already in use for other service, then "Socket Hang up" error comes out.
eg:- port 6455 is dedicated port for some other service or connection. You cannot use same port (6455) for making a database connection on same server.
Sometimes, this error rises when a client waits for a response for a very long time. This can be resolved using the 202 (Accepted) Http code. This basically means that you will tell the server to start the job you want it to do, and then, every some-time-period check if it has finished the job.
If you are the one who wrote the server, this is relatively easy to implement. If not, check the documentation of the server you're using.
Postman was giving "Could not get response" "Error: socket hang up".
I solved this problem by adding the Content-Length http header to my request
Are you using nodemon, or some other file-watcher? In my case, I was generating some local files, uploading them, then sending the URL back to my user. Unfortunately nodemon would see the "changes" to the project, and trigger a restart before a response was sent. I ignored the build directories from my file-watcher and solved this issue.
Here is the Nodemon readme on ignoring files: https://github.com/remy/nodemon#ignoring-files
I have just faced the same problem and I fixed it by close my VPN. So I guess that's a network agent problem. You can check if you have some network proxy is on.
this happaned when client wait for response for long time
try to sync your API requests from postman
then make login post and your are done
I defined Authenticate method to generate a token and mentioned its return type as nullable string as:
public string? Authenticate(string username, string password)
{
if(!users.Any(u => u.Key==username && u.Value == password))
{
return null;
}
var tokenHandler = new JwtSecurityTokenHandler();
var tokenKey = Encoding.ASCII.GetBytes(key);
var tokenDescriptor = new SecurityTokenDescriptor()
{
Subject = new ClaimsIdentity(new Claim[]
{
new Claim(ClaimTypes.Name, username)
}),
Expires = DateTime.UtcNow.AddHours(1),
SigningCredentials = new SigningCredentials(new
SymmetricSecurityKey(tokenKey),
SecurityAlgorithms.HmacSha256Signature)
};
var token = tokenHandler.CreateToken(tokenDescriptor);
return tokenHandler.WriteToken(token);
}
Changing nullable string to simply string fixed "Socket Hang Up" issue for me!
If Postman doesn't get response within a specified time it will throw the error "socket hang up".
I was doing something like below to achieve 60 minutes of delay between each scenario in a collection:
get https://postman-echo.com/delay/10
pre request script :-
setTimeout(function(){}, [50000]);
I reduced time duration to 30 seconds:
setTimeout(function(){}, [20000]);
After that I stopped getting this error.
I solved this problem with disconnection my vpn. you should check if there is vpn connected.
What helped for me was replacing 'localhost' in the url to http://127.0.0.1 or whatever other address your local machine has assigned localhost to.
Socket hang up error could be due to the wrong URL of the API you are trying to access in the postman. please check the URL once carefully.
It's possible there are 2 things, happening at the same time.
The url contains a port which is not commonly used AND
you are using a VPN or proxy that does not support that port.
I had this problem. My server port was 45860 and I was using pSiphon anti-filter VPN. In that condition my Postman reported "connection hang-up" only when server's reply was an error with status codes bigger than 0. (It was fine when some text was returning from server with no error code.)
When I changed my web service port to 8080 on my server, WOW, it worked! even though pSiphon VPN was connected.
Following on Abhay's answer: double check the scheme. A server that is secured may disconnect if you call an https endpoint with http.
This happened to me while debugging an ASP.NET Core API running on localhost using the local cert. Took me a while to figure out since it was inside a Postman environment and also it was a Monday.
In my case, adding in the header the "Content-length" parameter did the job.
My environment is
Mac:
[Terminal command: sw_vers]
ProductName: macOS
ProductVersion: 12.0.1. (Monterey)
BuildVersion: 21A559
mysql:
[Terminal command: mysql --version]
Ver 8.0.27 for macos11.6 on x86_64 (Homebrew)
Apache:
[Terminal command: httpd -v]
Server version: Apache/2.4.48 (Unix)
Server built: Oct 1 2021 20:08:18.
*Laravel
[Terminal command: php artisan --version]
Laravel Framework 8.76.2
Postman
Version 9.1.5 (9.1.5)
socket hang up error can also occur due to backend API handling logic.
For example - I was trying to create an Nginx config file and restart the service by using the incoming API request body. This resulted in temporary disconnection of the Nginx service while handling the API request and resulted in socket hang up.
If you have tried all the steps mentioned in other comments, and still face the issue. I suggest you check the API handler code thoroughly.
I handled the above-mentioned example by calling the Nginx reset method with delay and a separate API to check the status of the prev reset request.
For me it was giving Socket Hung Up error only while running Collection Runner not with single request.
Adding a small delay (100-300ms) in the collection Runner solved issue for me.
In my case, I had to provide --ssl-client-key and --ssl-client-cert files to overcome these errors.
Great error, it is so general that for everyone something different helps.
In my case I was not able to fix it and what is really funny is fact that I am expecting to get multipart file on one endpoint. When I prepare request in postman I get "Error: socket hang up". If I change for other endpoint(even not existing) is exactly that same error. But when I call any endpoint without body that request works and after that all subsequent attempts works perfectly.
In my case this is purely postman issue. Any request using curl is never giving that error.
For me the issue was related to the mismatch of the http versions on the client and server.
Client was assuming http v2 while server (spring boot/ tomcat) in the case was http v1
When on the server I configured server to v2, the issue got resolved in a go.
In spring boot you can configure the http v2 as below:-
server.http2.enabled=true
Note - Also the scenario was related to using client auth mechanism (i.e. MTLS)
Without client auth/ MTLS it worked without issues but for client auth the version setting in spring boot was the important rescue point
"socket hang up" is proxy related issue. when we run same collection with the help of newman on jenkins then all test are passed.
change the proxy setting
https://docs.cloudfoundry.org/cf-cli/http-proxy.html
I had the same issue: "Error: socket hang up" when sending a request to store a file and backend logs mentioned a timeout as you described. In my case I was using mongoDB and the real problem was my collection’s array capacity was full. When I cleared the documents in that collection the error was dismissed. Hope this will help someone who faces a similar scenario.
"Socket Hung Up" can be on-premise issue some time's, because, of bottle neck in %temp% folder, try to free up the "temp" folder and give a try
I fixed this issue by disabling Postman token header. Screenshot:
I face the same issue in when calling a SOAP API with POSTMAN
by adding the following data in the header my issue was fixed
Key:Content-Length
Value:<calculated when request is sent>
In my case, I was incorrectly using a port reserved for https version of my api.
For example, I was supposed to use https://localhost:6211, but I was using http://localhost:6211.
It is port related error. I was trying to hit the API with an invalid port.
if it helps to anybody... In my case, i just forgot to use json parser (const jsonParser = express.json();) to have access to json type of objects sending to the server from the client. Be careful, don't waste your time =)
This happened to me while I was learning ASP.NET Web API.
In my case it was because the SSL certificate verification.
I was using VS Code so I oversee about SSL certificate verification and it came with https protocol.
I solved this with testing my endpoints with http protocol.
Another approach can be just disabling the SSL certificate Verification on Postman Settings.
This error was coming for me since the request url is not correct --> here you can see my url does not contains : after http
The url I was using was : http//locahost:9090/someApi
Solution
adding a colon new url is http://localhost:9090/someApi
the socket error was not coming
This is just my case may be your case is totally different as mentioned in the other answers :)

How to send PHP app logs directly to ELK service?

According to the documentation there are two ways to send log information to the SwisscomDev ELK service.
Standard way via STDOUT: Every output to stdout is sent to Logstash
Directly send to Logstash
Asking about way 2. How is is this achieved, especially how is the input expected?
We're using Monolog in our PHP buildpack based application and using its stdout_handler is working fine.
I was trying the GelfHandler (connection refused), SyslogUdPHandler (no error, but no result), both configured to use VCAPServices logstashHost and logstashPort as API endpoint / host to send logs to.
Binding works, env variables are set, but I have no idea how to send SwisscomDev ELK service Logstash API endpoint compatible log information from our application.
Logstash is configured with a tcp input, which is reachable via logstashHost:logstashPort. The tcp input is configured with its default codec, which is the line codec (source code; not the plain codec as stated in the documentation).
The payload of the log event should be encoded in JSON so that the fields are automatically recognized by Elasticsearch. If this is the case, the whole log event is forwarded without further processing to Elasticsearch.
If the payload is not JSON, the whole log line will end up in the field message.
For your use case with Monolog, I suggest you to use the SocketHandler (pointing it to logstashHost:logstashPort) in combination with the LogstashFormatter which will take care of the JSON encoding with the log events being line delimited.

WSO2 Carbon 404 Error Redirection for Webapp Deployment?

We are using WSO2 Carbon 4.2.0 through the WSO2 Application Server (AS) package. In replacing an older, highly customized Carbon installation (provided by a company that no longer supports the product, has abandoned it and refuses to work on it, and left us no details on how/what they modified in Carbon), we have deployed a couple web applications in the webapps container as they were deployed before in the older instance. We have changed our WebContextRoot in the carbon.xml from the default "/" to a sub-URL of ex: "/stuff", as is also detailed in the self-answered SO question here. However the answer given there is not detailed in what the OP actually encountered when he modified his WSO2 instance.
In testing the above configuration we noticed that if a user were to go to a non-existent web address on the server, depending on the format of the URL they are either:
redirected to a blank page;
receive a "500 Internal server error" (I suspect this is the embedded Tomcat?);
get sent to the Carbon login page (which we definitely do not want to happen for security reasons); or
get an XML document stating:
<faultString> The service cannot be found for the endpoint reference (EPR) /stuff/services/nonexistantservicename </faultString>
At least in the case of missing content we wish the user to be sent to a standardized 404 error page, or at the least be sent an HTTP 404 error by the server. For services the XML error is palatable, we can deal with that.
The only option for us right now to circumvent this issue is to place a proxy in front of the WSO2 instance, which would be another layer to manage and tune, and possibly degrade performance. Please know that I am not a programmer but just an admin with DevOps experience. I would not know how to handle this with e.g. a Java solution or re-coding parts of WSO2. Customizing the core product would also hamper future upgrades of WSO2, a scenario we are trying to dig ourselves out of now as detailed above. Is there no internal WSO2 mechanism to handle non-existent content? Can we not redirect any errors to a standard canned response page?

Perl SOAP::WSDL accessing HTTPS Unathorized error

I'm trying to generate a Perl library to connect to a WebService. This webservice is in an HTTPS server and my user has access to it.
I've executed wsdl2perl.pl several times, with different options, and it always fails with the message: Unauthorized at /usr/lib/perl5/site_perl/5.8.8/SOAP/WSDL/Expat/Base.pm line 73.
The thing is, when I don't give my user/pass as arguments, it doesn't even asks for them.
I've read [SOAP::WSDL::Manual::Cookbook] (http://search.cpan.org/~mkutter/SOAP-WSDL-2.00.10/lib/SOAP/WSDL/Manual/Cookbook.pod) and done what it says about HTTPS: Crypt::SSLeay is instaleld, and both SOAP::WSDL::Transport::HTTP and SOAP::Transport::HTTP are modified.
Can you give any hint about what may be going wrong?
Can you freely access the WSDL file from your web browser?
Can someone else in your network access it without any problems?
Maybe the web server hosting the WSDL file requires Basic or some other kind of Authentication...
If not necessary ,I don't recommend you to use perl as a web service client .As you know ,perl is a open-source language,although it do support soap protocol,but its support do not seem very standard.At first,its document is not very clear.And also ,its support sometimes is limited.At last,bug always exists here and there.
So ,if you have to use wsdl2perl,you can use komodo to step into the code to find out what happened.This is just what I used to do when using perl as a web service client.You know ,in the back of https is SSL,so ,if your SSL is based on certificate-authorized,you have to set up your cert path and the list of trusted server cert.You'd better use linux-based firefox to have a test.As I know ,you can set up firefox's cert path and firefox's trusted cert list.If firefox can communicated with your web service server succefully,then,it's time to debug your perl client.
To debug situations with Perl and SOAP, interpose a web proxy so you can see exactly what data is being passed and what response comes back from the server. You were getting a 401 Not authorized, I expect, but there may be more detail in the server response.
Both Fiddler http://docs.telerik.com/fiddler and Charles proxy https://www.charlesproxy.com/ can do this.
The error message you quote seems to be from this line :
die $response->message() if $response->code() ne '200';
and in HTTP world, Unauthorized is clearly error code 401, which means your website asks for a username and password (most probably, some website may "hijack" this error code to cater for other conditions like a filter on the source IP).
Do you have them?
If so, you can
after wdsl2perl has run, find in the created files where set_proxy() is called and change the URL in there to include the username and password like that : ...->set_proxy('http://USERNAME:PASSWORD#www.example.com/...')
or your in code, after instantiating the SOAP::WSDL object, call service(SERVICENAME) on it (for each service you have defined in your WSDL file), which gives you a new object, on which you call transport() to access the underlying transport object on which you can call proxy() with the URL as formatted above (yes it is proxy() here and set_proxy() above); or you call credentials() instead of proxy() and you pass 4 strings:
'HOSTNAME:PORT'
the realm, as given by the webserver but I think you can put anything
the username
the password