Challenge in data from REST API using Azure Data Factory - access issue - rest

We are trying to reach to an API hosted in our company network using rest connector in ADF (SHIR is used). Linked service connection is successful but dataset is unable to read the data and copy activity is as well failing with below error. Please suggest your thoughts in resolving the same.
Failure happened on 'Source' side. ErrorCode=UserErrorFailToReadFromRestResource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred while sending the request.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=Unable to connect to the remote server,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ,Source=System,'

This error is mostly seen due to firewall issues. You might want to verify your network firewall setting to allow the API request to be read.
Also, verify if your API call is working as expected using other API testing tools. If the issue persists you can raise a support ticket for engineers to investigate more on the issue.

If you are able to preview data in your source , then check your sink connection as this issue can occur when the Sink in the copy activity is behind a firewall, I was getting the same issue and I tried copying to a container without a firewall and it worked. Its weird that the error is related to Source and the issue is with Sink.

Related

The listener for Azure function was unable to start - Microsoft.Azure.EventHubs.Processor Encountered error

I am getting the below error whilst running my Python Azure Function on the local machine in VSCode.
For clarification the message is:
The listener for function 'Functions.IoT_Data-Handler' was unable to
start. Microsoft.Azure.EventHubs.Processor: Encountered error while
fetching the list of EventHub PartitionIds. System.Private.CoreLib: A
connection attempt failed because the connected party did not properly
respond after a period of time, or established connection failed
because connected host has failed to respond.
This error has never occurred before in the time I have started using VSCode for Azure functions (since last September). The only thing that has changed recently is that I now deploy this function within an Azure Function premium resource, but really that should not matter in the dev environment.
For information, this function is hooked up to an Azure IoT-Hub endpoint and is simply reading and processing the uplink data before saving it to an Azure SQL database.
Can anyone offer any advice?
Check if my below findings help to fix your issue:
As #PeterBons said, check the connection string given correctly in the local.settings.json:
Whatever the Event Hub Endpoint/IoT Hub Endpoint Connection String given in the file local.settings.json, that property name should be mapped in the function.json file.
Try replacing the IoT Hub Connection String without the consumer group name as mentioned in this GitHub Issue #5512
I found similar issues in the SO 1 & 2 which will be helpful to fix your issue.

Kafka Connect: Error detection when worker fails

I'am submitting a connector to kafka. The connector created is sftp connector. Now when the password is wrong the connector sends back success response when the connector fails. The password is wrong response is not given at that time. This is a single scenario there could be mutliple scenarios like this. Now when I use the <host>/connectors/<connector-name>/status, I get the error saying failed to establish connection. But this endpoint has a little delay. If I'am immediately trying after creating the connector, I may not get any response(404).
What is the proper way of handling this using the status api call.Is there any delay that needs to be used before firing this API. Or can it be handled while submitting the connector to API?
When you create the connector, it naturally needs to load the JAR(s) responsible for the tasks, then distribute the tasks to actually start the connector code (which is responsible for connecting to the SFTP server with the connection details).
Therefore, the delay is natural, and there's no way to know your connection details are incorrect unless you try to use them before launching the connector.

AWS AmazonS3Client request returns error "The remote certificate is invalid according to the validation procedure"

We have an application using AWS SDK AmazonS3Client to communicate with the S3 service to get files downloaded. With thousands of instances running fine, however, we got a few sites getting the following exception error message:
The remote certificate is invalid according to the validation procedure.
The versions of the AmazonSDK.S3.dll and AmazonS3.Core.dll we're using are 3.3.102.18 and 3.3.103.1 respectively. These had been running for over a year without problems until recently.
Has anyone else experienced the similar issue? What could be the root cause of the problem? How do we resolve it?
Thanks!
This has been determined as an issue with proxy server loaded with incorrect certificate from the service provider. So, not really the application issue.

Drools controller URL responding with error code 405

We are exploring Drools BPM for one of our new project. I am basically from .Net background and doesn't have much knowledge with java techs.
Our expectation is that we should able to create rules and workflows using Drools so that it can be exposed as REST API's and then the rules should be executed from .net/angular client applications using this REST.
I have configured the KIE server using WildFly10 server. It seems that all my configurations are working fine but when i try to connect to controller ("http://localhost:8080/kie-wb/rest/controller") it gives the below error message.
Error Message :
2018-01-22 12:30:09,263 INFO [org.kie.server.controller.websocket.client.WebSocketKieServerControllerImpl] (KieServer-ControllerConnect) Kie Server points to non Web Socket controller 'http://localhost:8080/kie-wb/rest/controller', using default REST mechanism
2018-01-22 12:30:09,263 WARN [org.kie.server.common.KeyStoreHelperUtil] (KieServer-ControllerConnect) Unable to load key store. Using password from configuration
2018-01-22 12:30:09,274 WARN [org.kie.server.services.impl.controller.DefaultRestControllerImpl] (KieServer-ControllerConnect) Exception encountered while syncing with controller at http://localhost:8080/kie-wb/rest/controller/server/wildfly-kieserver error Error while sending PUT request to http://localhost:8080/kie-wb/rest/controller/server/wildfly-kieserver response code 405
The below URL's are working fine
http://localhost:8080/kie-wb
http://localhost:8080/kie-server/services/rest/server/
http://localhost:8080/kie-wb/rest/controller/management/servers
I was able to get both workbench and kie server to work on the same web and/or app server following various examples that are proliferated on the internet. However, I encountered this particular error when setting up for a more realistic production ready environment using separate app servers on the same host running on different ports. The issue is, the REST version of the url is only supported when both workbench and kie server are running on the same app or web server. When on separate app or web servers, it must use web sockets (see docs for property org.kie.workbench.controller):
http://docs.jboss.org/drools/release/7.8.0.Final/drools-docs/html_single/#_wb.systemproperties
Only Web Socket protocol is supported for connecting with a remote Kie Server Controller. When specifying this proporty, the Workbench will automatically disable all the features related to running the embbeded controller.
After reading this, the 405 makes sense (method not allowed). I changed the URL from
http://ACTUAL_HOST_NAME:8080/kie-drools-wb/rest/controller
to
ws://ACTUAL_HOST_NAME:8080/kie-drools-wb/websocket/controller
and that resolved the 405.
Also, I was originally using 'localhost' for the actual host name, but it still would not connect, so I changed it to the actual host name. Finally, I was still seeing a connection problem and then found out that both servers need
-Dorg.kie.workbench.controller.user=kieserver
-Dorg.kie.workbench.controller.pwd=kieserver1!
-Dorg.kie.server.user=kieserver
-Dorg.kie.server.pwd=kieserver1!
to be set on both servers (See https://groups.google.com/d/msg/drools-setup/Bpt-r_bIa2U/g0H1J77sBgAJ and https://groups.google.com/d/msg/drools-setup/Aru_F98nL9g/Z6vUCm6hBQAJ).
Edit:
After fiddling with it some more and read the documentation a little bit more, the referenced documentation was referring to workbench connecting to an external controller, instead of configuring the controller within workbench.
After much torment, my problem was solved by adding role "kieserver=kie-server,rest-all" and user "kieserver" to Business Central Workbench.
Without this data, I consistently received 405.

Web Deploy not working. Timeout on client. Schannel 1203 on server

I'm trying to deploy an application via web deploy to a windows 2012 r2 server.
First tried installing iis, web deploy using the msi. After getting an unreachable error I saw somewhere that I needed wmsvc to be installed and installed it via powershell command Install-WindowsFeature Web-Mgmt-Service.
I stopped getting the unreachable error and started getting this error ->
Error: Could not complete the request to remote agent URL 'http://xx.x.xx.xx:8172/MSDEPLOYAGENTSERVICE'.
Error: The operation has timed out
On the server's event viewer under system I get the following error ->
A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 10. The Windows SChannel error state is 1203.
Any idea of what could be going on? Did I miss something on the installation?
After this I also tried installing everything but without iis and I get the same error.
So, this is really not a PoSH issue.
Maybe you should consider movin gthis question to another forum.
Anyway, you have either real server config issues (this can sometimes mean wipe and rebuild - very ugly option - but if it's a virtual machine and you have a snapshot, just roll back), or an app causing this.
Typically for Schannel type of error, it has to do with application or service in machine not able to complete any SSL connection sort of connection.
The error state parameter of 1203 means client error connecting to server, ie invalid ClientHello from the client
See if you can collect more data by...
https://support.microsoft.com/en-us/help/260729/how-to-enable-schannel-event-logging-in-iis
I realized this is an old question, but in case somebody ran into the same issue, here are my solution.
I had to follow this guidelines : https://learn.microsoft.com/en-us/aspnet/web-forms/overview/deployment/configuring-server-environments-for-web-deployment/configuring-a-web-server-for-web-deploy-publishing-web-deploy-handler
Copy the resulting publishsetting file. In the Publish/Profile page import it.
In the Publish/Connection page, change the server to the server address with the 8172 port. Add the password. Change the Destination URL to the site to go to after the publish is completed.
Then after Validate connection was completed, go ahead and publish it!