HBase Connection Instance - scala

Question

I have this code below:
DStream.map {
_.message()
}.foreachRDD { rdd =>
rdd.foreachPartition{iter =>
val conf = HBaseUtils.configureHBase("iemployee")
val connection = ConnectionFactory.createConnection(conf)
val table = connection.getTable(TableName.valueOf("""iemployee"""))
iter.foreach{elem =>
/* loop through the records in the partition and push them out to the DB */
}
}
Can someone please tell me if the connection object created val connection = ConnectionFactory.createConnection(conf) here is the same connection object used in each partition (since I never close it) or will a new connection object be created for every partition?

Answer 1

New connection instance for each partition..
Please see the below code & documentation of Connection Factory. it was also mentioned that its callers responsibility to close the connection.
/**
* Create a new Connection instance using the passed <code>conf</code> instance. Connection
* encapsulates all housekeeping for a connection to the cluster. All tables and interfaces
* created from returned connection share zookeeper connection, meta cache, and connections
* to region servers and masters.
* <br>
* The caller is responsible for calling {#link Connection#close()} on the returned
* connection instance.
*
* Typical usage:
* <pre>
* Connection connection = ConnectionFactory.createConnection(conf);
* Table table = connection.getTable(TableName.valueOf("table1"));
* try {
* table.get(...);
* ...
* } finally {
* table.close();
* connection.close();
* }
* </pre>
*
* #param conf configuration
* #param user the user the connection is for
* #param pool the thread pool to use for batch operations
* #return Connection object for <code>conf</code>
*/
public static Connection createConnection(Configuration conf, ExecutorService pool, User user)
throws IOException {
if (user == null) {
UserProvider provider = UserProvider.instantiate(conf);
user = provider.getCurrent();
}
String className = conf.get(ClusterConnection.HBASE_CLIENT_CONNECTION_IMPL,
ConnectionImplementation.class.getName());
Class<?> clazz;
try {
clazz = Class.forName(className);
} catch (ClassNotFoundException e) {
throw new IOException(e);
}
try {
// Default HCM#HCI is not accessible; make it so before invoking.
Constructor<?> constructor =
clazz.getDeclaredConstructor(Configuration.class,
ExecutorService.class, User.class);
constructor.setAccessible(true);
return (Connection) constructor.newInstance(conf, pool, user);
} catch (Exception e) {
throw new IOException(e);
}
}
}
Hope this helps!!