How do I connect to Hive on EMR through Eclipse? - eclipse

I am trying to connect to Hive on EMR through Eclipse, but I get an error.
Exception in thread "main" java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:8158: java.net.ConnectException: Connection refused
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:215)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.readypulse.sparkanalytics.HiveQLConnector.<init>(HiveQLConnector.java:31)
at com.readypulse.sparkanalytics.HiveQLConnector.main(HiveQLConnector.java:83)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

You would need a SSH Tunnel to poke a hole to EMR master. And then connect via JDBC.
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
private static String hiveConnectionString = "jdbc:hive2://localhost:8158";
//Need port forwarding
SparkPortForwarding.portForwardForSpark();
Class c = Class.forName(driverName);
Connection connection = DriverManager.getConnection(hiveConnectionString,
"user", "pwd");
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery(sql);
You can do port forwarding via shell as:
ssh -i ~/mykey.pem -N -L 8158:<maserhost>:10000 hadoop#<masterhost>
Or You can use the code using the library JSch
public static void portForwardForSpark() {
try {
if(session != null && session.isConnected()) {
return;
}
JSch jsch = new JSch();
jsch.addIdentity(PATH_TO_SSH_KEY_PEM);
String host = REMOTE_HOST;
session = jsch.getSession(USER, host, 22);
// username and password will be given via UserInfo interface.
UserInfo ui = new MyUserInfo();
session.setUserInfo(ui);
session.connect();
int assingedPort = session.setPortForwardingL(LPORT, RHOST, RPORT);
System.out.println("Port forwarding done for the post : " + assingedPort);
} catch (Exception e) {
System.out.println(e);
}
}

Related

HttpClient socket exception

I am trying to call some external API hosted on azure from my web Api. The API is working fine on my local machine but when I deploy it IIS on server it starts throwing System.Net.Sockets.SocketException (10060).A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. Although I have increased the request timeout to 5 minutes but connection is silently stopping after 21 seconds and throwing aforementioned exception.
Here is my code:
var telemetries = new TelemetryResponse();
var client = httpClientFactory.CreateClient("Lynx");
client.Timeout = TimeSpan.FromMinutes(5);
var httpResponseMessage = await client.GetAsync("vehicletelemetries/All?key=iLJIbAVXOnpKz5xyF0zV44yepu5OVfmZFhkHM7x");
if (httpResponseMessage.IsSuccessStatusCode)
{
string content = await httpResponseMessage.Content.ReadAsStringAsync();
telemetries = JsonConvert.DeserializeObject<TelemetryResponse>(content);
}
Exception I am getting is:
System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
You got an error code:
WSAETIMEDOUT
10060
Connection timed out.
A connection attempt failed because the connected party did not properly respond after a period of time, or the established connection failed because the connected host has failed to respond.
https://learn.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2#WSAETIMEDOUT
Check your network connection, ip adress, port and maybe the connection got blocked from a firewall.
Hi guys the culprit was the proxy and we have to configure our HttpClient with proxy while creating it/registering in the DI container. I have registered the HttpClient in DI like this.
var proxySettings = new ProxySetting();
Configuration.Bind(nameof(ProxySetting), proxySettings);
services.AddHttpClient("Lynx", client =>
{
client.BaseAddress = new Uri(Configuration.GetSection("LynxUrl").Value);
}).ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler { Proxy = new MyProxy(proxySettings)});
And my proxy code is
public class MyProxy : IWebProxy
{
private readonly ProxySetting _proxySetting;
public MyProxy(ProxySetting proxySetting)
{
_proxySetting = proxySetting;
}
public ICredentials Credentials
{
//get { return new NetworkCredential("username", "password"); }
get { return new NetworkCredential(_proxySetting.UserName, _proxySetting.Password,_proxySetting.Domain); }
set { }
}
public Uri GetProxy(Uri destination)
{
return new Uri(_proxySetting.ProxyUrl);
}
public bool IsBypassed(Uri host)
{
return false;
}
}
It has resolved my problem completely.

MailKit - smtp.Connect throw exception in C# .Net

Exception while connecting to the server: A connection attempt failed because the connected party did not properly respond after a period of time, or an established connection failed because the connected host has failed to respond.
1.) able to ping the server
2.) telnet smtp.office365.com 587 respond as expected.
public ConnectSmtpServer()
{
using MailKit.Net.Smtp;
string _SmtpServer = "smtp.office365.com";
int _portNumber = 587;
Console.WriteLine("Welcome!");
using (SmtpClient smtp = new SmtpClient())
{
try
{
smtp.Timeout = 30 * 1000;
smtp.Connect(_SmtpServer, _portNumber, SecureSocketOptions.Auto);
Console.WriteLine("Test smtp connection using MailKit.Net !");
smtp.Connect(_SmtpServer, _portNumber, SecureSocketOptions.Auto);
Console.WriteLine();
}
}
}

Not able to connet to ejabberd server through smack

I am setting up a chat service using ejabberd and building an XMPP client for android devices using smack.
Here are some important details.
server OS: ubuntu 18.04
server hosted as localhost (jid format: alice#localhost).
server system IP : 192.168.4.162
Client:
Smack 4.3.1
Using external phone through USB debugging : Nokia 3.1 Plus.
Here is my code
Here are some of the configurations I tried.
private class MyLoginTask extends AsyncTask<String, String, String> {
#Override
protected String doInBackground(String... params) {
// Create a connection to the jabber.org server.
InetAddress addr = null;
try{
addr = InetAddress.getByName("192.168.4.162");
}catch(UnknownHostException e){
e.printStackTrace();
}
XMPPTCPConnectionConfiguration config = null;
DomainBareJid serviceName = null;
try{
serviceName = JidCreate.domainBareFrom("localhost");
System.out.println("serviceName: "+serviceName);
}catch(XmppStringprepException e){
e.printStackTrace();
}
HostnameVerifier verifier = new HostnameVerifier() {
#Override
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
// ConnectionConfiguration connConfig = new ConnectionConfiguration("talk.google.com", PORT, SERVICE);
try{
config = XMPPTCPConnectionConfiguration.builder()
.setUsernameAndPassword("alice", "9009")
.setHost("192.168.4.162")
.setSecurityMode(ConnectionConfiguration.SecurityMode.disabled)
.setXmppDomain("localhost")
.setHostnameVerifier(verifier)
.setHostAddress(addr)
.setPort(5222)
.build();
}catch(XmppStringprepException e){
e.printStackTrace();
}
AbstractXMPPConnection conn1 = new XMPPTCPConnection(config);
try {
System.out.println("Connecting......."); AndroidUsingLinkProperties.setup(getApplicationContext());
conn1.connect().login();
if(conn1.isConnected()) {
Log.w("app", "conn done");
}
conn1.login();
if(conn1.isAuthenticated()) {
Log.w("app", "Auth done");
}
}
catch (Exception e) {
Log.w("app", e.toString());
}
return "";
}
#Override
protected void onPostExecute(String result) {
}
}
The configurations i tried above the result is:
org.jivesoftware.smack.SmackException$ConnectionException: The following addresses failed: '192.168.4.162:5222' failed because: /192.168.4.162 exception: java.net.ConnectException: failed to connect to /192.168.4.162 (port 5222) from /192.168.4.182 (port 39568) after 30000ms: isConnected failed: EHOSTUNREACH (No route to host)
however I am successfully able to build a connection to my server through any other client(psi, gajim, My web app(BOSH connection)).
following may help if networking is the issue:
$ nmap localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2019-08-28 18:00 IST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000056s latency).
Not shown: 992 closed ports
PORT STATE SERVICE
111/tcp open rpcbind
139/tcp open netbios-ssn
445/tcp open microsoft-ds
631/tcp open ipp
5222/tcp open xmpp-client
5269/tcp open xmpp-server
5280/tcp open xmpp-bosh
8600/tcp open asterix
Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds
I have been there and I found out the accidentally my phone and my ejabberd server were on different network. They should be on the same network as in this case. Make sure you are on same network and this error should go away.

RMI and 2 machines - two way communication

I've got problem with RMI comunication between 2 machines (win 7 and win xp VM). The exception with I have problem is:
java.rmi.ConnectException: Connection refused to host: 169.254.161.21; nested exception is:
java.net.ConnectException: Connection timed out: connect
It's really weired because during connection I use address 192.168.1.4 (server), but exception somehow show sth different. I disabled firewall on both side. Ping working to both side. I tried telnet to server and use server port:
telnet 192.168.1.4 1099 and it's working... I can't figure out where the problem is.
If I run this on host side (eg server side) everything works fine.
How is it look from SERVER:
public class Server
{
public static void main(String args[])
{
InputStreamReader is = new InputStreamReader(System.in);
BufferedReader br = new BufferedReader(is);
String portNum, registryURL;
try{
System.out.println("Enter the RMIregistry port number:");
portNum = (br.readLine()).trim();
int RMIPortNum = Integer.parseInt(portNum);
startRegistry(RMIPortNum); // Registry registry = LocateRegistry.createRegistry(RMIPortNum);
ServerSide_Impl exportedObj = new ServerSide_Impl();
registryURL = "rmi://localhost:" + portNum + "/callback";
//registryURL = "rmi://192.168.1.4:" + portNum + "/callback";
Naming.rebind(registryURL, exportedObj);
System.out.println("Callback Server ready.");
}// end try
catch (Exception re) {
System.out.println(
"Exception in HelloServer.main: " + re);
} // end catch
} // end main
//This method starts a RMI registry on the local host, if
//it does not already exists at the specified port number.
private static void startRegistry(int RMIPortNum) throws RemoteException
{
try
{
Registry registry = LocateRegistry.getRegistry(RMIPortNum);
registry.list( );
// This call will throw an exception
// if the registry does not already exist
}
catch (RemoteException e)
{
Registry registry = LocateRegistry.createRegistry(RMIPortNum);
}
} // end startRegistry
} // end class
Client side is look like:
try
{
this.serverAd = serverAddress.getText();
String path = System.getProperty("user.dir");
String pathAfter = path.replace("\\", "/");
String pathFile = "file:/"+pathAfter + "/wideopen.policy";
System.setProperty("java.security.policy", pathFile);
System.setSecurityManager(new RMISecurityManager());
this.hostName = hostNameTextField.getText();
this.portNum = hostPortNumberTextField.getText();
RMIPort = Integer.parseInt(this.portNum);
this.time = Integer.parseInt(timeTextField.getText());
//this.registryURL = "rmi://localhost:" + this.portNum + "/callback";
String registryURLString = "rmi://"+this.serverAd+":" + this.portNum + "/callback";
this.registryURL = registryURLString;
ConsoleTextField.append("\n"+ this.registryURL + "\n");
// find the remote object and cast it to an
// interface object
obj = (ServerSide_Interface)Naming.lookup(this.registryURL);
boolean test = obj.Connect();
if(test)
{
callbackObj = new ClientSide_Impl();
// register for callback
obj.registerForCallback(callbackObj);
isConnected = true;
ConsoleTextField.append("Nawiązano połaczenie z serwerem\n");
TableModel modelTemp = obj.Server_GenerateStartValues();
myDataTable.setModel(modelTemp);
myDataTable.setEnabled(true);
}
else ConsoleTextField.append("Brak połączenia z serwerem\n");
}
catch (Exception ex ){
ConsoleTextField.append(ex + "\n");
System.out.println(ex);
}
This connection is working fine if I run client on host side. If I use VM and try connect between 2 different machines, I can;t figure out what did I do bad
There is something wrong with your etc/hosts file or your DNS setup. It is providing the wrong IP address to the server as the server's external IP address, so RMI embeds the wrong address into the stub, so the client attempts to connect to the wrong IP address.
If you can't fix that, you can set the system property java.rmi.server.hostname to the correct IP address at the server JVM, before exporting any RMI objects (including Registries). That tells RMI to embed that address in the stub.

java.sql.SQLException: Network error IOException: Connection timed out: connect

I have installed SQL Server 2012 Express in my laptop and tried to run a sample java program as follows:
import java.sql.*;
public class Ex2 {
Connection con;
Statement st;
ResultSet rs;
public Ex2(String query){
System.out.println("IN Const1");
try{
System.out.println("IN try");
Class.forName("net.sourceforge.jtds.jdbc.Driver");
String dbName = "HOME";
String user = "sa";
String password = "JUL#2012";
//the name of my SQL SERVER 2005 instance
String SqlServerInstance = "SQLEXPRESS";
String url = "jdbc:jtds:sqlserver://localhost:1433";
url = url + "/" + dbName;
url = url + ";" + SqlServerInstance;
con = DriverManager.getConnection(url, user, password);
System.out.println("nach DriverManager");
st = con.createStatement();
st.executeQuery(query);
rs = st.getResultSet();
while ( rs.next() ){
System.out.println(rs.getString(0));
System.out.println(rs.getString(1));
System.out.println(rs.getString(2));
System.out.println(rs.getString(3));
}
}
catch(Exception e){
e.printStackTrace();
System.out.println("Exception: " + e.getMessage());
}
}
public static void main(String[] argv){
System.out.println("IN MAIN");
Ex2 test = new Ex2("SELECT * FROM HOME.dbo.employee;");
}
}*
and getting the following error log in the console:
java.sql.SQLException: Network error IOException: Connection timed out: connect
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:410)
at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)
at java.sql.DriverManager.getConnection(DriverManager.java:525)
at java.sql.DriverManager.getConnection(DriverManager.java:171)
at Ex2.<init>(Ex2.java:22)
at Ex2.main(Ex2.java:43)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:516)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:307)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:257)
at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:311)
Exception: Network error IOException: Connection timed out: connect
... 6 more
When I checked the sql server configuration manager the services SQL Server and SQL Server Browser are running.
I have been googling for the last two days and struggle to fix this issue and couldn't. So please help me.