I am trying to run a copy command in pyspark. as below. How to get rid of this error?
spark.write.format("snowflake") \
.options(**config.sfparams) \
.option("query", "copy into people_data from (select $1:Company_ID::varchar as Company_ID from #company_stage/pitchbook/"+config.todays_date+"/Company/")\
.load()
Error:
py4j.protocol.Py4JJavaError: An error occurred while calling o49.load.
: net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error:
syntax error line 1 at position 15 unexpected 'copy'.
Copy doesnt work in pyspark. created a external table to read data directly from s3 and enabled autorefresh=true
Related
I have created an table and then I'm trying to import a csv file from my local machine. I wrote on the console in the following way.
copy schema_name.newtble (col_1, col_2, col_3, ID) FROM ‘D:\directory\subderictory\filename.csv’ DELIMITER ‘,’ CSV HEADER;
But I got the error
SQL Error [42601]: ERROR: syntax error at or near "‘D"
Please help me to solve it out. It seems like I have done some silly mistake.
I am trying to load a .sql dump file using pgAdmin. However getting the following error :
ERROR: syntax error at or near "\"
LINE 2757: \.
^
SQL state: 42601
Character: 68298
It looks like this in code :
Even if I delete it, it gets the same error in next \.
If I delete all, gives another error.
I am new to pyspark and I did some initial tutorials. When I am trying to load a CSV file on my local host in the Spark framework using Jupyter Notebook, the below mentioned error pops. My java version is 8.0
from pyspark.sql import SparkSession
spark = SparkSession \
.builder \
.appName('sql based spark data analysis') \
.config('spark.some.config.option', 'some-value') \
.getOrCreate()
df = spark.read.csv('C:/Users/sitaram/Downloads/creditcardfraud/creditcard.csv')
My error is as follows:
Py4JJavaError: An error occurred while calling o55.csv.
: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException:
java.lang.RuntimeException: Error while running command to get file
permissions : java.io.IOException: (null) entry in command string: null ls -F C:\tmp\hive
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:65
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Error
while running command to get file permissions : java.io.IOException: (null) entry in command string: null ls -F C:\tmp\hive
Please try C://Users//sitaram//Downloads//creditcardfraud//creditcard.csv
I used the command:
COPY studentapp_deg_course_cat(degree_code, specialization, category_level1, category_level2, category_level3, min_credit, max_credit, primary)
FROM '/home/abhishek/Downloads/courses.csv'
USING DELIMITERS ';'
and i am getting the following error:
ERROR: syntax error at or near "COPY" LINE 1: COPY studentapp_deg_course_cat(degree_code, specialization,...
^
********** Error **********
ERROR: syntax error at or near "COPY" SQL state: 42601 Character: 1
I would like to know the error in my code
Valid COPY syntax for version 9.1 on Linux should be:
COPY studentapp_deg_course_cat(degree_code, specialization, category_level1,
category_level2, category_level3, min_credit, max_credit, "primary")
FROM '/home/abhishek/Downloads/courses.csv'
WITH (DELIMITER ';')
WITH, not USING.
DELIMITER instead of DELIMITERS.
And do not use primary as column name. It's a reserved word.
This line(518):
COPY wp_commentmeta (meta_id, comment_id, meta_key, meta_value) FROM stdin;
\.
is giving this error:
[ERROR ] 518.0: syntax error, unexpected character
What is this?
I have done backup before with this database, and now I'm just trying to restore all the tables back to the database.
The error:
ERROR: syntax error at or near "\"
LINE 1: ...a (meta_id, comment_id, meta_key, meta_value) FROM stdin; \.
^
********** Error **********
ERROR: syntax error at or near "\"
SQL state: 42601
Character: 77
points to the \ in the \. as being the issue.
Are you sure you require the \.?
Per the documentation:
End of data can be represented by a single line containing just
backslash-period (.). An end-of-data marker is not necessary when
reading from a file, since the end of file serves perfectly well; it
is needed only when copying data to or from client applications using
pre-3.0 client protocol.
Try removing your \. from the line and see if your copy works as expected.