When I use sqoop, it tells me:
'--table: command not found'
The code is as follows:
sqoop export --connect jdbc:mysql://hadoop01:3306/result \
--username root \
--password 000000 \
--table stats_event \
--export-dit hdfs://hadoop01:8020/user/hive/warehouse/eclog.db/stats_event \
--input-fields-terminated-by '\t' \
--update-node allowsert \
--update-key paltform_dimension_id,date_dimension_id,event_dimension_id \
- m 1;
Typo:
--export-dit → --export-dir
it's typo mistake:
--export-dir and also
--update-node allowser --> --update-node allowinsert
Related
I have dumped a PostgreSQL database using:
pg_dump --file "dump_file.sql" \
--host "localhost" \
--port "5432" \
--username "username" \
--verbose \
--format=c \
--blobs \
--encoding "UTF8" \
--schema "public" \
"dbname"
Then when I try to restore it:
pg_restore --file "dump_file.sql" \
--clean \
--create \
--no-privileges \
--no-owner \
--format=c \
-U "username" \
-d "newdatabasename"
it says:
pg_restore: error: options -d/--dbname and -f/--file cannot be used together.
And if I remove -d "newdatabasename" it says:
pg_restore: error: could not read from input file: end of file
Question
How could I restore this database under a new name?
After some readings, I thought --no-privileges --no-owner would have fixed it but it's not the case...
With pg_restore, the dump file is not the argument to --file. It is not an option argument at all.
pg_restore --clean \
--create \
--no-privileges \
--no-owner \
--format=c \
-U "username" \
-d "newdatabasename" \
"dump_file.sql"
From Add protocol-mapper to keycloak using kcadm.sh
Has anyone figured this out yet? I tried it the way Oscar suggested and it still does not work.
The lines that are not commented work perfectly.
The lines that are commented do not work. I get an error that says "./clientmapper.sh: 59 (or whatever line number that I have uncommented): -s: not found"
sudo docker exec $keycontainer /opt/jboss/keycloak/bin/kcadm.sh create \
clients/$cid/protocol-mappers/models \
-r myrealm \
-s name=roles \
-s protocol=openid-connect \
-s protocolMapper=oidc-usermodel-attribute-mapper
#-s 'config."id.token.claim"=true' \
#-s claim.name=roles \
#-s jsonType.label=String \
#-s multivalued=true \
#-s userinfo.token.claim=true \
#-s access.token.claim=true
I made this work by formatting as Oscar suggested and using -i after the docker exec command. It works perfectly now.
sudo docker exec -i $keycontainer /opt/jboss/keycloak/bin/kcadm.sh create \
clients/$cid/protocol-mappers/models \
-r testrealm \
-s name=testmap \
-s protocol=openid-connect \
-s protocolMapper=oidc-usermodel-realm-role-mapper \
-s 'config."id.token.claim"=true' \
-s 'config."claim.name"=testmap' \
-s 'config."jsonType.label"=String' \
-s 'config."multivalued"=true' \
-s 'config."userinfo.token.claim"=true' \
-s 'config."access.token.claim"=true'
I am trying to export the table from hdfs to postgres
Below is the query which I used for export:
sqoop export --connect jdbc:postgresql:hostname:5432/postgresDB --username user --password password --input-fields-terminated-by '\001' --fields-terminated-by ',' --table customer --export-dir /hdfs/location/customer --input-null-string '\\N' --input-null-non-string '\\N' --direct --update-key customer_id
The sqoop query completes with success message. Please see the screenshot below:
But when I query the table, I am not finding any data.
Any help is appreciated. Thanks in advance.
sqoop export --connect jdbc:postgresql:hostname:5432/postgresDB \
--username user \
--password password \
--input-fields-terminated-by '\001' \
--fields-terminated-by ',' \
--table customer \
--export-dir /hdfs/location/customer \
--input-null-string '\\N' --input-null-non-string '\\N' \
--direct \
--update-mode allowinsert
This worked ..
For me, this worked after I added the schema name for my table name.
-- -- schema my_schema
I use Sqoop 1.4.5-cdh5.4.2 and Postgresql.
If Sqoop connects directly to the database - all right.
But need use Sqoop over pgbouncer, and I have problem with this.
In pgbouncer you can not do prepared statements transaction mode.
... connect command:
sqoop import \
--connect "$db_name" \
--username "$db_user" \
--password "$db_pass" \
--direct \
--hive-import \
--hive-table "$hive_schema.$t" \
--hive-overwrite \
--num-mappers 10 \
--fetch-size 100000 \
--split-by "object_id" \
--target-dir "/user/$hive_schema/$t" \
--table "$t"
... and error:
org.postgresql.util.PSQLException: ERROR: prepared statement "S_3" already exists
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2270)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1998)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Connection.executeTransactionCommand(AbstractJdbc2Connection.java:791)
at org.postgresql.jdbc2.AbstractJdbc2Connection.commit(AbstractJdbc2Connection.java:815)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:315)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
at org.apache.sqoop.hive.TableDefWriter.getCreateTableStmt(TableDefWriter.java:126)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:188)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:514)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Add prepareThreshold=0 to the connection string
SQOOP don't work with pgbouncer and transaction pool :(
I'm seeing INFO messages when I run my tests and I thought that I had gotten rid of them by setting the client_min_messages PGOPTION. Here's my command:
PGOPTIONS='--client-min-messages=warning' \
psql -h localhost \
-p 5432 \
-d my_db \
-U my_user \
--no-align \
--field-separator '|' \
--pset footer \
--quiet \
-v AUTOCOMMIT=off \
-X \
-v VERBOSITY=terse \
-v ON_ERROR_STOP=1 \
--pset pager=off \
-f tests/test.sql \
-o "$test_results"
Can someone advise me on how to turn off the INFO messages?
This works for me: Postgres 9.1.4 on Debian GNU Linux with bash:
env PGOPTIONS='-c client_min_messages=WARNING' psql ...
(Still works for Postgres 12 on Ubuntu 18.04 LTS with bash.)
It's also what the manual suggests. In most shells, setting environment variables also works without an explicit leading env. See maxschlepzig's comment.
Note, however, that there is no message level INFO for client_min_messages.
That's only applicable to log_min_messages and log_min_error_statement.