I'm importing a folder (not a file) from Google Storage within Google Data Prep. I need to get the file name for all of the files from within the storage as a column within the finished dataset.
Related
in a scheduler action i need to add files to FAL (to sys_file) which are already in the storage, uploaded via ftp. The normal way
storage->addFile(....)
which copies the file from a temporary folder to the file storage and adds it to the sys_file table does not work, because the file is already in the fileadmin directory. If i try i get this error message:
[ERROR] Cannot add a file that is already part of this storage.
How is it possible to add a file to sys_file which is already in fileadmin?
Thanks!
AFAIK addFile() is to be used for files uploaded from a local disk. If the files are already available on the remote server, you should go for addUploadedFile() instead.
I am trying to create an GCS object (File) with GCS create plugin of Data Fusion.
but it is creating a folder instead.
How I can have a file created instead of a folder ??
It seems that the description of the plugin leads to a misunderstanding. Cloud Storage doesn't work like a conventional filesystem, so you cannot "strictly" create empty files. The gsutil command doesn't have an equivalent to a touch command (on Linux) and all "basic" operation in this product is limited to the cp command (upload and download files).
Therefore, since there is no file when you specify the storage url, it's expected that a folder will be created instead of a file.
Based on this, I would like to suggest you two workarounds:
If you are using this plugin to create a file as a ‘flag’, you can continue using the plugin since the created folder also serves as a flag (to trigger a Cloud Function, for example)
If you need to create a file, you can create files with the GCS plugin located in ‘Sink’ plugins group to write records to one or more files in a directory on Google Cloud Storage. Files can be written in various formats such as csv, avro, parquet, and json.
Moodle stores references to all files uploaded to it in its database, the files themselves get hashed and stored on disk but you cannot work out what the files are by looking at them on the disk.
Is there somewhere in the Moodle interface that lets you browse all the files that have been uploaded as an admin?
Is this available as a plugin? Looking for some kind of global file store browser.
I am using Oracle SQL Developer on MacOS and I am trying to save zip file into Blob field. I know how to load any other file type by clicking "Load" button and then selecting the file.
The problem is that when I select the zip file and click on Open, it does not select that file, but shows the me the content of zip file and then I can select only one file from that zip. This is not what I want, because I want to upload the whole zip.
Is there any setting in Oracle SQL Developer or any other way?
I do not have such problem on same table when using PLSQL Developer on Windows machine.
I have found out that when I have a zip file in a folder, then Oracle Sql Developer automatically shows also folders for this file, even if does not exist in reality on disk. The name of the folder and the file is the same including extension. Different is only the icon. If you select archive then loading to blob works ok. This is the behaviour on MacOS. I did not test it on other systems. On bottom picture is the example of folder containing only 2 zip files and how it looks when you try selecting file from that folder.
Hi I want to know how to copy files to HDFS from source file system(Local File system),if source file already copied to HDFS,then how to eliminate or ignore that file to copy again in HDFS using Talend.
Thanks
Venkat
To copy files from local file system to the HDFS, you need to use tHDFSPut components if you have Talend for big data. If you use Talend for data integration you can easily use tSystem component with the right command.
To avoid duplicated files, you need to create a table in a RDBMS and keep track of all copied files. Each time the job start copying file, it should check if it already exists in the table.