Yesterday I installed PostgreSQL 14.5 on a Windows 10 laptop.
I then ran an old script to load images into a table.
The script uses the pg_read_binary_file function.
Some of the images are .jpg files and some are .png files.
Of the 34 files, only 5 were successfully processed (1 .jpg and 4 .png). The other 29 failed with the following error:
[Exception, Error code 0, SQLState XX000] ERROR: could not open file "file absolute path" for reading: Invalid argument
For instance, the following statement executes without errors
select pg_read_binary_file('C:\Users\Jorge\OneDrive\Documents\000\020-logos\adalid.png') as adalid_png;
... and the following statement fails
select pg_read_binary_file('C:\Users\Jorge\OneDrive\Documents\000\020-logos\oper.png') as oper_png;
... with the following error message
[Exception, Error code 0, SQLState XX000] ERROR: could not open file "C:/Users/Jorge/OneDrive/Documents/000/020-logos/oper.png" for reading: Invalid argument
So far, I have not been able to identify any difference in the files that could be the cause of the error. Also, I'm pretty sure the script works on earlier releases of version 14. Unfortunately I have not been able to find a website to download any of those earlier releases to test it again.
Has anyone else found this problem, and its solution?
I think the issue is somehow caused by OneDrive. This laptop is new. When I logged in with my Microsoft account, the OneDrive directory was automatically created and updated. Apparently this operation only updates the directory entries, leaving the contents of the files in the cloud until they are opened. When I zipped the directory that contains all my images, a message from OneDrive appeared saying that in that moment it will restore some files. After that, all the commands in my scripts work.
My theory is that pg_read_binary_file gets the file entry from the directory, so it doesn't give the "No such file or directory" message; but then fails reading the contents, giving the "Invalid argument" message instead.
The unanswered question would be: why does 7-Zip make OneDrive restore the files but pg_read_binary_file does not?
UPDATE
After more testing, and reading Save disk space with OneDrive Files On-Demand for Windows, now I am sure that pg_read_binary_file could fail and send the message "Invalid argument" when the OneDrive file is not a locally available file. In Windows File Explorer such file has a blue cloud icon next to it.
Related
I'm trying to create parquet files for several days locally. The first time I run the code, everything works fine. The second time it fails to delete a file. The third time it fails to delete another file. It's totally random which file can not be deleted.
The reason I need this to work is because I want to create parquet files everyday for the last seven days. So the parquet files that are already there should be overwritten with the updated data.
I use Project SDK 1.8, Scala version 2.11.8 and Spark version 2.0.2.
After running that line of code the second time:
newDF.repartition(1).write.mode(SaveMode.Overwrite).parquet(
OutputFilePath + "/day=" + DateOfData)
this error occurs:
WARN FileUtil:
Failed to delete file or dir [C:\Users\...\day=2018-07-15\._SUCCESS.crc]:
it still exists.
Exception in thread "main" java.io.IOException:
Unable to clear output directory file:/C:/Users/.../day=2018-07-15
prior to writing to it
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:91)
After the third time:
WARN FileUtil: Failed to delete file or dir
[C:\Users\day=2018-07-20\part-r-00000-8d1a2bde-c39a-47b2-81bb-decdef8ea2f9.snappy.parquet]: it still exists.
Exception in thread "main" java.io.IOException: Unable to clear output directory file:/C:/Users/day=2018-07-20 prior to writing to it
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:91)
As you see it's another file than the second time running the code.
And so on.. After deleting the files manually all parquet files can be created.
Does somebody know that issue and how to fix it?
Edit: It's always a crc-file that can't be deleted.
Thanks for your answers. :)
The solution is not to write in the Users directory. There seems to be a permission problem. So I created a new folder in the C: directory and it works perfect.
this problem occurs when you open the destination directory in windows. You just need to close the directory.
Perhaps another Windows process has a lock on the file so it can't be deleted.
I copy an old excel file (excel 97) from one location to another location using PowerShell. It worked until last Sunday and it got error:
"error in method invocation failed (system.IO.file into) ..does not contain a method named "saved""
error is access to the path denied
Do you know what I need to do with this error?
Make sure the account you are using to run this code still holds the permission to the destination path.
Note:Post the code here with error messages
I have a script that reads data, process it, and prints an output. The script is not running in the directory where it is saved. I tried to change directory via OS.chdir, but still I get the error "file not found". I placed the scripts in the phone storage by drag and drop from my PC.
When I execute the following script:
copy (
select agk_p_id Promoter_agk, multiplication_lr_agk_p_k4, agk_lr_rvd, status_agk_p_k4
from patient_agk_p_expr
where status_agk_p_k4='Preferentially')
to 'g:\boom.csv'
With CSV HEADER;
It works just beautifully, and creates the boom.csv file on my g drive.
I get:
Query returned successfully: 8486 rows affected, 631 ms execution time.
I should note that my 'g' drive is an external harddrive that is connected to my computer.
And my cygwin refers to my g harddrive like this:
blumr04#SRB524YBZ1 /cygdrive/g/
$ pwd
/cygdrive/g
Now, my computer has also access to a server harddrive of my organization.
On my windows explorer it refers to as (Z:)
My cygwin refers to the 'Z' drive accordingly (just the same it does to my C: drive):
blumr04#SRB524YBZ1 /cygdrive/z/
$ pwd
/cygdrive/z
But I have troubles when it comes to having postgres recognizing this harddrive - when I attempt to run the following script in order to save my table to the Z harddrive :
copy (
select agk_p_id Promoter_agk, multiplication_lr_agk_p_k4, agk_lr_rvd, status_agk_p_k4
from patient_agk_p_expr
where status_agk_p_k4='Preferentially')
to 'z:\boom.csv'
With CSV HEADER;
I get the following error message:
ERROR: could not open file "z:\boom.csv" for writing: No such file or directory
********** Error **********
ERROR: could not open file "z:\boom.csv" for writing: No such file or directory
SQL state: 58P01
Does anyone know how can I save (copy to) my files when it comes to a harddrive that is not physically connected to my computer, but rather is a server harddrive?
- Is there a command / script in postgres that would be able to show me which harddrives are accessible to postgres? It looks like for some reason the Z harddrive is not accessible for postgres read/write, at least not in the way that I attempt it, while G,J,K, and other harddrive which are external HD - are accessible. I would be glad to know if I could expand postgres accessibility somehow..
Thanks!
#Mike Sherrill 'Cat Recall'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
My Z drive is referred to in Windows Explorer also as:
(\shares.nyumc.org\research)(Z:),
therefore I tried also the following:
copy (
select agk_p_id Promoter_agk, multiplication_lr_agk_p_k4, agk_lr_rvd, status_agk_p_k4
from patient_agk_p_expr
where status_agk_p_k4='Preferentially')
to '\\shares.nyumc.org\research\boom.csv'
With CSV HEADER;
This scripts gives me the following error, indeed all about permissions:
ERROR: could not open file "\\shares.nyumc.org\research\boom.csv" for writing: Permission denied
********** Error **********
ERROR: could not open file "\\shares.nyumc.org\research\boom.csv" for writing: Permission denied
SQL state: 42501
So it looks like the right path is:
\\shares.nyumc.org\research\
And that in this case (Z:) is merely an alias name(?!) as the error message is this time NOT about "No such file or directory", but rather about permissions.
Is there a way I could facilitate the necessary permission to Postgres so it could write to the server drive?
The most common problem with running COPY tablename to filename is dealing with path and permissions from the point of view of the PostgreSQL server.
Files named in a COPY command are read or written directly by the
server, not by the client application. Therefore, they must reside on
or be accessible to the database server machine, not the client. They
must be accessible to and readable or writable by the PostgreSQL user
(the user ID the server runs as), not the client. Source
If you try to write to a file that the PostgreSQL server can't "see", you'll get "No such file or directory". If you try to write to a file in a directory for which the PostgreSQL server lacks permissions, you'll get "Permission denied".
So odds are good that the PostgreSQL user (the user ID the server runs as) lacks permissions on "z".
Hallo everyone once again,
I did various searches but couldn't gind a suitable/applicable answer to the simple problem below:
On pgAdminIII (Windows 7 64-bit) I am running the following command using SQL editor:
COPY public.Raw20120113 FROM 'D:\my\path\to\Raw CSV Data\13_01_2012.csv';
I tried many different variations for the path name and verified the path, but I keep getting:
ERROR: could not open file "D:\my\path\to\Raw CSV Data\13_01_2012.csv" for reading: No such file or directory
Any suggestions why this happens?
Thank you all in advance
Petros
UPDATE!!
After some tests I came to the following conclusion: The reason I am getting this error is that the path includes some Greek characters. So, while Windows uses codepage 1253, the console is using 727 and this whole thing is causing the confusion. So, some questions arise, you may answer them if you like or prompt me to other questions?
1) How can I permanently change the codepageof the console?
2) How can I define the codepage is SQL editor?
Thank you again, and sorry if the place to post the question was inappropriate!
Try DIR "D:\my\path\to\Raw CSV Data\13_01_2012.csv" from command line and see if it works - just to ensure that you got the directory, file name, extension etc correct.
The problem is that COPY command runs on server so it takes the path to the file from the server's scope.
To use local file to import you need to use \COPY command. This takes local path to the file into account and loads it correctly.