I have a small sqoop import project where i have to import from mysql to hdfs. It is a maven project, while i am running in my local system it works fine. Now when i am trying to run in my ubuntu server by running a jar file, it is executing but not showing any output. It is not a web based application, normal application where i am running hadoop deamons and running this project. Tell me how to run this type of projects in servers and run.
package org.ezytruk.sqoop;
import java.io.FileInputStream;
import java.util.Properties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.sqoop.tool.ImportTool;
import com.cloudera.sqoop.SqoopOptions;
import com.cloudera.sqoop.SqoopOptions.FileLayout;
public class SqoopImport {
public static void importMySQLToHDFS() throws Exception {
try {
String driver = "com.mysql.jdbc.Driver";
Class.forName(driver).newInstance();
Configuration config = new Configuration();
config.addResource(new Path("/usr/local/hadoop/etc/hadoop/conf/core-site.xml"));
config.addResource(new Path("/usr/local/hadoop/etc/hadoop/conf/hdfs-site.xml"));
Properties properties = new Properties();
properties.load(new FileInputStream("/home/*****/SqoopImportHDFS/resources/sqoopImport.properties"));
//org.apache.sqoop.SqoopOptions sqoopOptions = new org.apache.sqoop.SqoopOptions();
SqoopOptions sqoopOptions = new SqoopOptions();
sqoopOptions.setDriverClassName(driver);
sqoopOptions.setHadoopHome("/usr/local/hadoop");
sqoopOptions.setConnectString(properties.getProperty("db_mysql_connection_string"));
sqoopOptions.setTableName(properties.getProperty("db_mysql_table_name"));
sqoopOptions.setUsername(properties.getProperty("db_mysql_username"));
sqoopOptions.setPassword(properties.getProperty("db_mysql_password"));
sqoopOptions.setNumMappers(1);
//sqoopOptions.setWarehouseDir(properties.getProperty("db_hdfs_warehouse_directory"));
sqoopOptions.setTargetDir(properties.getProperty("db_hdfs_output_directory"));
sqoopOptions.setFileLayout(FileLayout.TextFile);
new ImportTool().run(sqoopOptions);
} catch (Exception e) {
}
}
public static void main(String[] args) throws Exception {
SqoopImport mySqlImport = new SqoopImport();
mySqlImport.importMySQLToHDFS();
}
}
Related
When I try to connect to facebook.com through PhantomJS I receive this error. I could not find a solution.
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriverService;
import org.openqa.selenium.remote.DesiredCapabilities;
public class SeleniumDemo {
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
DesiredCapabilities caps = new DesiredCapabilities();
caps.setJavascriptEnabled(true);
caps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY, "E:/ProjectTrials/phantomjs-2.1.1-windows/phantomjs-2.1.1-windows/bin/phantomjs.exe");
System.out.println("c1");
WebDriver driver = new PhantomJSDriver(caps);
System.out.println("c2");
driver.get("https://www.facebook.com");
System.out.println("c3");
System.out.println("Page Title: " + driver.getTitle());
}
}
I am not able to compile my DriverClass at the job.waitforcompletion(boolean) clause.It gives me a NoClassFoundException.If I catch the exception ,the run method throws the error that its expecting a int value.I am using MapReduce New API.Could anyone suggest what is the issue :
import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class Dist_Driver extends Configured implements Tool {
public int run(String args[]) throws IOException, InterruptedException {
// Configuration phase
// Configuration conf=new Configuration();
Job job = new Job(new Configuration());
job.setJarByClass(Dist_Driver.class);
// Mapper Reducer InputFormat
job.setInputFormatClass(FileInputFormat.class);
// Mapper and Reducer Class
job.setMapperClass(Dist_Mapper.class);
job.setReducerClass(DistCache_Reducer.class);
job.setOutputFormatClass(TextOutputFormat.class);
job.setInputFormatClass(KeyValueTextInputFormat.class);
// set FileInputOutput
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
// setting number of reduce tasks and submit it
job.setNumReduceTasks(2);
// Lets check if the file exist
File f1 = new File("/home/hdfs/trials_mapreduce_progams/emp_id");
if (f1.exists())
System.out.println("The Files Exists");
else
System.out.println("The File doesnot exist");
URI path1;
try {
path1 = new URI(
"/home/hdfs/trials_mapreduce_progams/emp_lookup.txt");
DistributedCache.addCacheFile(path1, job.getConfiguration());
} catch (URISyntaxException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (job.waitForCompletion(true))
return 0;
else
return 1;
}
public static void main(String[] args) throws Exception {
int exitcode = ToolRunner.run(new Dist_Driver(), args);
System.exit(exitcode);
}
}
Just add the ClassNotFoundException to the run method signature
public int run(String args[]) throws IOException,
InterruptedException,
ClassNotFoundException {
The reason you get an error when you try and try/catch it is because if there is a ClassNotFoundException thrown during execution, there will be no return value, and the method has to return something.
If you really want to catch it, just return 1 in the catch clause, which is the error exit code
I am trying to embed an OSGI bundle that uses in my java application.
Below is my only java file in the bundle (I am doing it simple now):
package it.eng.test.ds.consumer;
public class Consumer {
public void activate(){
System.out.println("I'm here, in the activate ds automatic method ;)");
}
public void deactivate()
{
System.out.println("Bye Bye!");
}
}
Below is consumer.xml:
<?xml version="1.0" encoding="UTF-8"?>
<scr:component xmlns:scr="http://www.osgi.org/xmlns/scr/v1.1.0" name="it.eng.test.ds.consumer"
activate="activate"
deactivate="deactivate"
modified="modified"
enabled="true"
immediate="true">
<implementation class="it.eng.test.ds.consumer.Consumer"/>
</scr:component>
Below is MANIFEST.MF:
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Ds-bundle
Bundle-SymbolicName: it.eng.test.ds.consumer
Bundle-Version: 1.0.0.qualifier
Bundle-RequiredExecutionEnvironment: JavaSE-1.7
-Component: OSGI-INF/consumer.xml
Next, I embedded equinox in my application, started the required bundles (org.eclipse.equinox.ds and its dependencies), and finally started my bundle which is called it.eng.test.ds.consumer_1.0.0.201401071018.jar.
Below is my application code:
import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.ServiceLoader;
import org.osgi.framework.Bundle;
import org.osgi.framework.BundleContext;
import org.osgi.framework.BundleException;
import org.osgi.framework.launch.Framework;
import org.osgi.framework.launch.FrameworkFactory;
public class MainEmbedder {
public static void main(String[] args)
{
FrameworkFactory frameworkFactory = ServiceLoader.load(FrameworkFactory.class).iterator().next();
Map<String, String> config = new HashMap<String, String>();
//TODO: add some config properties
Framework framework = frameworkFactory.newFramework(config);
try {
framework.start();
} catch (BundleException e) {
e.printStackTrace();
}
BundleContext context = framework.getBundleContext();
List<Bundle> installedBundles = new LinkedList<Bundle>();
try {
installedBundles.add(context.installBundle("file:C:\\Users\\student\\Desktop\\DS-Plugins\\org.eclipse.osgi.services_3.3.100.v20120522-1822.jar"));
installedBundles.add(context.installBundle("file:C:\\Users\\student\\Desktop\\DS-Plugins\\org.eclipse.equinox.util_1.0.200.v20100503.jar"));
installedBundles.add(context.installBundle(
"file:C:\\Users\\student\\Desktop\\DS-Plugins\\org.eclipse.equinox.ds_1.4.1.v20120926-201320.jar"));
installedBundles.add(context.installBundle(
"file:C:\\Users\\student\\Desktop\\DS-Plugins\\plugins\\it.eng.test.ds.consumer_1.0.0.201401071018.jar"));
} catch (BundleException e) {
e.printStackTrace();
}
for (Bundle bundle : installedBundles) {
try {
bundle.start();
} catch (BundleException e) {
e.printStackTrace();
}
}System.out.println("after starting bundles");
}
}
When I run my app, I can see the message "after starting bundles", but I do not see the message "I'm here, in the activate ds automatic method ;)". This means that my bundle did not enter the activate method. Why is that? How can I solve this problem? Thanks.
Update:
I tried to start the bundle with Felix framework, using the below code:
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.felix.framework.Felix;
import org.apache.felix.framework.util.FelixConstants;
import org.osgi.framework.Bundle;
import org.osgi.framework.BundleContext;
import org.osgi.framework.BundleException;
import org.osgi.framework.Constants;
import org.osgi.framework.ServiceRegistration;
public class HostApplication
{
private HostActivator m_activator = null;
private Felix m_felix = null;
public HostApplication()
{
// Create a configuration property map.
Map config = new HashMap();
config.put(Constants.FRAMEWORK_STORAGE_CLEAN, Constants.FRAMEWORK_STORAGE_CLEAN_ONFIRSTINIT);
// Create host activator;
m_activator = new HostActivator();
List list = new ArrayList();
list.add(m_activator);
config.put(FelixConstants.SYSTEMBUNDLE_ACTIVATORS_PROP, list);
try
{
// Now create an instance of the framework with
// our configuration properties.
m_felix = new Felix(config);
// Now start Felix instance.
m_felix.start();
}
catch (Exception ex)
{
System.err.println("Could not create framework: " + ex);
ex.printStackTrace();
}
// Register the application's context as an OSGi service!
BundleContext bundleContext1 = m_felix.getBundleContext();
System.out.println("6");
try {
Bundle dsBundle = bundleContext1.installBundle("file:C:\\Users\\student\\Desktop\\DS-Plugins\\org.apache.felix.scr-1.8.0.jar");
dsBundle.start();
if(dsBundle.getState()== dsBundle.ACTIVE)
System.out.println("DS Bundle is Active!");
Bundle consumerBundle = bundleContext1.installBundle("file:C:\\Users\\student\\Desktop\\DS-Plugins\\plugins\\it.eng.test.ds.consumer_1.0.0.201401071018.jar");
consumerBundle.start();
if(consumerBundle.getState()== consumerBundle.ACTIVE)
System.out.println("Consumer Bundle is Active!");
System.out.println("done!");
} catch (BundleException e) {
e.printStackTrace();
System.out.println("Not done!");
}
shutdownApplication();
}
public Bundle[] getInstalledBundles()
{
// Use the system bundle activator to gain external
// access to the set of installed bundles.
return m_activator.getBundles();
}
public void shutdownApplication()
{
// Shut down the felix framework when stopping the
// host application.
try {
m_felix.stop();
} catch (BundleException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
try {
m_felix.waitForStop(0);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Now I get the following error:
ERROR: it.eng.test.ds.consumer (2): Component descriptor entry 'OSGI-INF/consumer.xml' not found.
I guess there is something wrong with my bundle, but it is very simple and just as I descried it above. Any ideas?
I've replayed your setup, and I've got it to work.
(A github repo is here, if all else fails)
Check if OSGI-INF/ is actually in your bundle, did you add it to build.properties (if you use PDE)
regards, Frank
I am not sure what your exact intention is but embedding osgi bundles into a standard java application is maybe not the best thing to do. I would recommend to convert your application to osgi (if possible and it's not too big). If you cannot do to that consider to emulate the bundle with the ds component, e.g. just create the component instance manually and set it's dependencies on foot. But don't mix both worlds.
So I'm using Cassandra in my project and I have to make a connection between Eclipse and the data base. I tried to use a JDBC-compliant driver that I found on code.google.com but I'm getting this exception:
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
This is my code:
package cassandrasampledriver;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import org.apache.cassandra.cql.jdbc.DriverResolverException;
import org.apache.cassandra.cql.jdbc.InvalidUrlException;
public class CassandraDriver
{
public static void main(String[] args) {
Connection con = null;
String KS = "cassandrademocql";
try
{
Class.forName("org.apache.cassandra.cql.jdbc.CassandraDriver");
con = DriverManager.getConnection("jdbc:cassandra://localhost:9160/" + KS);
Statement stmt = con.createStatement();
String query = "DROP KEYSPACE cassandrademocql;";
ResultSet result = stmt.executeQuery(query);
}
catch (ClassNotFoundException e) {
e.printStackTrace();
}
catch (SQLException e) {
e.printStackTrace();
}
}
}
Thanks in advance :)
I ran your code using this cql driver (Version 1.1.2 compatible with cassandra 1.2.
on JDK6 without an Error. Maybe you are missing references in your classpath? The only problem with the code is that you are assigning a variable to a method call that wont return anything ResultSet result = stmt.executeQuery(query); and so a java.sql.SQLNonTransientException exception will be thrown.
You said you are new to Cassandra, just some friendly advice, I would do some research on the available APIs for Cassandra before making a choice.
These are the JARs I used
apache-cassandra-1.2.0-rc1-SNAPSHOT.jar
apache-cassandra-clientutil-1.2.0-rc1-SNAPSHOT.jar
apache-cassandra-thrift-1.2.0-rc1-SNAPSHOT.jar
cassandra-jdbc-1.1.2.jar libthrift-0.9.0.jar log4j-1.2.16.jar
log4j-over-slf4j-1.7.2.jar selenium-server-standalone-2.21.0.jar
slf4j-api-1.7.2.jar slf4j-simple-1.7.2.jar
Full code that I ran:
package cassandrasampledriver;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class CassandraDriver {
public static void main(String[] args) {
Connection con = null;
String KS = "cassandrademocql";
try
{
Class.forName("org.apache.cassandra.cql.jdbc.CassandraDriver");
con = DriverManager.getConnection("jdbc:cassandra://localhost:9160/" + KS);
Statement stmt = con.createStatement();
String query = "DROP KEYSPACE cassandrademocql;";
// Because you are droping the KS this will not return anything
// So the result set will be null and a java.sql.SQLNonTransientException exception will be thrown every time.
ResultSet result = stmt.executeQuery(query);
}
catch (Exception ex) {
ex.printStackTrace();
}
}
}
Hi
I have a java class which is working fine in windows but not in Mac OSX snow leopard. I am using Eclipse on both operating systems. On Mac OSX its throwing file not found exception.
Basically I am trying to read a file using BufferedReader and FileReader and I put my file in \resources\
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class ReadFileContents {
/**
* #param args
*/
public static void main(String[] args) {
BufferedReader br = null;
String line = "";
try {
br = new BufferedReader(new FileReader("resources\\abc"));
while((line = br.readLine())!= null)
{
System.out.println("Read ::: "+line+" From File.");
}
} catch (FileNotFoundException fne) {
fne.printStackTrace();
}catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
On Mac it is giving
java.io.FileNotFoundException: resources\abc (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at java.io.FileInputStream.<init>(FileInputStream.java:66)
at java.io.FileReader.<init>(FileReader.java:41)
at ReadFileContents.main(ReadFileContents.java:18)
Do I need any special configuration in my eclipse to get this working...
Mac uses forward slashes: "resources/abc". (This will actually work on Windows as well. Only the command line interpreter requires backslashes.)