No manager for connect string - postgresql

As soon as my EMR-Cluster was ready to be run.
I started facing some issues when listing databases and importing sqoop
Apparently, sqoop has been installed normally and it is working normally when I type "sqoop help" in Linux terminal.
using sqoop help
as you can see, the command could be recognized normally.
However, if I try out the sqoop import command, this one cannot be and it faces an error:
sqoop import \
--connect jdbc:postgres://sportsdb.cxri########.us-east-2.rds.amazonaws.com/SportsDB \
--username postgres \
--password mypassword \
--table addresses --target-dir s3://sqoop-table-from-rds-to-s3/sqoop-table/ -m 1 --fields-terminated-by '\t' --lines-terminated-by ','
sqoop import
The same goes to the second one, which is "sqoop list-databases" as shown below:
sqoop list-databases \
--connect jdbc:postgres://sportsdb.cxri########.us-east-2.rds.amazonaws.com \
--username postgres \
--password mypassword
sqoop list-databases
they don't really works and anything happens ;/
I also downloaded jar and put into /usr/lib/sqoop/lib/ where is the jar files on sqoop
To do it I run these two follow commands:
1) wget -O postgresql-jdbc.jar https://jdbc.postgresql.org/download/postgresql-42.3.1.jar
2) sudo mv postgresql-jdbc.jar /usr/lib/sqoop/lib/
Jar file added to sqoop/lib
Someone else has a suggestion about what can be done in order to fix this issue?

The issue could be solved after following a tip received about a typo.
Then, I just changed the word postgres for postgresql as follows:
sqoop list-databases \
--connect jdbc:postgresql://sportsdb.cxri########.us-east-2.rds.amazonaws.com \
--username postgres \
--password mypassword
That is it. The issue was fixed just adjusting something pretty simple

Related

mongoexport works on local but not on remote server

When I run this code to export my local database, it works well and a file is successfully generated:
mongoexport \
--host="localhost:27017" \
-d springDb \
-c posts \
--out social.json
But when I try to access my remote server with the similar command here:
mongoexport \
--host="mongodb+srv://clustxxx.xxx.mongodb.net/xxxx" \
-d socialDeveloper \
-c posts \
--out social.json
I get this error:
error parsing command line options: error parsing URI from mongodb:///clustxxx.xxx.mongodb.net/xxxx:27017/?replicaSet=mongodb+srv:: error parsing uri: must have at least 1 host
What am I doing wrong please?

Syntax error when running Query: "syntax error at or near "\""

I've generated a PostgreSQL script that I want to use to restore a database. When I go to my backup server to try to restore, I get the error: syntax error at or near "\".
It's getting stuck on the following characters \.
These appear like this:
COPY admin.roles (role_id, role_name, is_role_auto) from stdin;
\.
What's wrong with this statement? Is there config I missed? I'm on PostgreSQL 11.4 on Windows, the backup was taken with pg_dump, and I restore it using pgAdmin.
You cannot use pgAdmin to restore a "plain format" dump taken with pg_dump. It doesn't understand the psql syntax where COPY and its data are interleaved.
You will have to use psql to restore the dump:
psql -h server -p 5432 -U user -d database -f dumpfile.sql
It's really hard to know the specific error without seeing your backup and restore commands in their entirety, but if it helps, here is the boilerplate I use when I want to copy a table from production to a backup server:
$BIN/pg_dump -h production_server -p 5432 \
--dbname=postgres \
--superuser=postgres \
--verbose \
--format=c \
--compress=9 \
--table=admin.roles > backup.sql
$BIN/pg_restore \
--host=backup_server \
--port=5432 \
--username=postgres \
--dbname=postgres \
--clean \
--format=custom \
backup.sql
The format=c (or --format=custom) makes the content completely unreadable, but on a plus side it also avoids any weird errors with delimiters and the like, and it also perfectly copies complex data structures like arrays and BLOBs.

Sqoop command with --query in Scala using Process("<sqoop command>").! not working

I am trying to run a Sqoop command having --query option on Scala shell using Process("<sqoop command>").! but it is showing an error described below.
If I use --table instead of --query, it works like a charm.
But for my requirement, I need to use --query.
The error it gives is :
ERROR tool.BaseSqoopTool: Error parsing arguments for import:
ERROR tool.BaseSqoopTool: Unrecognized argument: *
ERROR tool.BaseSqoopTool: Unrecognized argument: from
etc.
Thanks in advance
The below code works properly in Spark-Scala shell.
Process("""sqoop import <queuename> --connect <connection-string-db2> --
username <username> --password <password> --table "<database.table>" --
target-dir <target directory> --delete-target-dir --fields-terminated-by
'\\001' --lines-terminated-by '\\n' --null-string '\\\\N' --null-non-
string '\\\\N' --hive-drop-import-delims -m 1""").!
But when I change --table by --query (and use whatever basic query in it e.g. "SELECT * FROM DATABASE.TABLE"), it doesn't work... Below is the command which doesn't work:
Process("""sqoop import <queuename> --connect <connection-string-db2> --
username <username> --password <password> --query "SELECT * FROM
DATABASE.TABLE AND \$CONDITIONS" --target-dir <target directory> --
delete-target-dir --fields-terminated-by '\\001' --lines-terminated-by
'\\n' --null-string '\\\\N' --null-non-string '\\\\N' --hive-drop-
import-delims -m 1""").!
I think you need to escape the double quotes after --query .

Cloudera-Sqoop import with/without --hive-import

I'm trying to do an activity whereas i'll import a table from MSSQL then export to MSSQL again in another database for the sake of testing sqoop1. So far, my imports are successful. My concern is regarding the export, if i import a table without --hive-import option, i'll be able to export it successfully. But if i include --hive-import option, sqoop wont be able to export it and prompts an error:
17/04/02 23:08:20 ERROR sqoop.Sqoop: Got exception running Sqoop:
org.kitesdk.data.DatasetIOException: Unable to load descriptor
file:hdfs://quickstart.cloudera:8020/user/hive/warehouse/customer/.metadata/descriptor.properties
for dataset:customer org.kitesdk.data.DatasetIOException: Unable to
load descriptor
file:hdfs://quickstart.cloudera:8020/user/hive/warehouse/customer/.metadata/descriptor.properties
for dataset:customer
As per checking, there's a difference in the metadata with --hive-imports. Imports with --hive-import parameter only does not have the required metadata:
Supplier/.metadata/descriptor.properties
My question is, is it possible to import a table in sqoop with --as-parquetfile and --hive-import option then be able to export it also?
here's my sample import and export code for referrence:
sqoop export \
--connect "jdbc:sqlserver://192.168.1.23;database=SqoopDB;schema=dbo;" \
--username sa \
--password Password1 \
--export-dir /user/hive/warehouse/customer \
--table customer
sqoop import \
--connect "jdbc:sqlserver://192.168.1.23;database=SourceDB;schema=dbo" \
--username sa \
--password Password1 \
--table Customer \
--as-parquetfile \
--hive-import \
--hive-overwrite \
-m 1

Sqoop: how switch off Prepared Statements?

I use Sqoop 1.4.5-cdh5.4.2 and Postgresql.
If Sqoop connects directly to the database - all right.
But need use Sqoop over pgbouncer, and I have problem with this.
In pgbouncer you can not do prepared statements transaction mode.
... connect command:
sqoop import \
--connect "$db_name" \
--username "$db_user" \
--password "$db_pass" \
--direct \
--hive-import \
--hive-table "$hive_schema.$t" \
--hive-overwrite \
--num-mappers 10 \
--fetch-size 100000 \
--split-by "object_id" \
--target-dir "/user/$hive_schema/$t" \
--table "$t"
... and error:
org.postgresql.util.PSQLException: ERROR: prepared statement "S_3" already exists
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2270)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1998)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Connection.executeTransactionCommand(AbstractJdbc2Connection.java:791)
at org.postgresql.jdbc2.AbstractJdbc2Connection.commit(AbstractJdbc2Connection.java:815)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:315)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
at org.apache.sqoop.hive.TableDefWriter.getCreateTableStmt(TableDefWriter.java:126)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:188)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:514)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Add prepareThreshold=0 to the connection string
SQOOP don't work with pgbouncer and transaction pool :(