I am trying to rename the S3 files which basically
copy to target + delete source
But in my case I am able to copy target but not able to delete source properly .
All directory structure remains same without any file ..
also it creates temp files in the main directory .
Do I have to explicitly delete it after renaming ?
Here is my code which renames the files
I have subfolders insode the folder.
val file = fs.globStatus(new Path(outputFileURL + "/*/*"))
for (urlStatus <- file) {
val DataPartitionName = urlStatus.getPath.toString.split("=")(1).split("\\/")(0).toString
val finalFileName = finalPrefix + DataPartitionName + "." + intFileCounter + "." + fileVersion + currentTime + fileExtention
val dest = new Path(mainFileURL + "/" + finalFileName)
fs.rename(urlStatus.getPath, dest)
intFileCounter += 1
}
If you check apache hadoop rename documentation, it says
The core operation of rename()—moving one entry in the filesystem to another ..
So its just moving the files and not renaming . More detail on the link above.
So I guess you will have to explicitly delete the folder name after renaming is complete
Instead of renaming the files only, you could rename the folder as well. That would not require you to explicitly delete the folder name.
Related
I am trying to write a script that takes a file from a specified location and copies it to the destination folder.
I currently have have the following with moves the file, but i need it to copy instead:
NT_JOB CCD1222
RUN THURSDAY
DELAYSUB 11:00
AGENT atl_stsprd01
USER connectd
ARGS source-file.txt +
W1222 +
CD.PRDCL.LNX +
"\\path\to\source\folder\" +
W99999_UNIX.properties +
/path/to/destination/folder/ +
destination-file.txt +
binary
EXITCODE 0 SUCCESS
CMDNAME \data\scripts\filetransfer.bat
RESOURCE ADD(1, DRES)
ENDJOB
What do i need to take the above and make it copy the file over to the destination folder instead!?
I have a directory which again contains subdirectories, which are built has part of other recipe and moved to DEPLOY_DIR_IMAGE using deploy bb class. So now I want to copy it to main image boot partition.
If it was a single file then appending required filename to IMAGE_EFI_BOOT_FILES variable, then yocto copies it to /boot. But same doesn't work for directories containing subdirectories please provide style to include even the subdirectories. Thank you
PS: I have tried appending IMAGE_EFI_BOOT_FILES += "parent_dir/*" didnt work.
It is obvious that IMAGE_EFI_BOOT_FILES is acting like the well known IMAGE_BOOT_FILES and other variables that are responsible for having the files necessary to be shipped in the boot partition. And that needs files and not directories.
So, if you do not need to specify all the files by hand, but instead you want to pass the directory, I suggest you use a python method to collect the files for you and append them to the variable.
See the following example I developed and tested:
def get_files(d, dir):
import os
dir_path = dir
if not os.path.exists(os.path.dirname(dir)):
dir_path = d.getVar(dir)
return ' '.join(f for f in os.listdir(d.getVar(dir)) if os.path.isfile(f))
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
The method will test if the argument is a real path then it will directly check for files, if not it will assume that it is a bitbake variable and it will get its content, so if DEPLOY_DIR_IMAGE is, for example, /home/user/dir, passing DEPLOY_DIR_IMAGE or /home/usr/dir will give the same result.
IMPORTANT
It is obvious also that IMAGE_EFI_BOOT_FILES is used in a .conf file such as local.conf or a custom machine configuration file. So adding that python function in .conf file will not work. I suggest creating a class for it and inherit it globally in your .conf file:
meta-custom/classes/utils.bbclass
local.conf:
INHERIT += "utils"
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
Try this and let me know in the comments.
EDIT
I have just realized that bitbake already imports os within python expressions expansions, so you can do it in one line without any need for a separate python function:
PATH = "/path/to/directory/" or
PATH = "A variable containing the path"
IMAGE_EFI_BOOT_FILES += "${#' '.join('%s' % f for f in os.listdir('${PATH}') if os.path.isfile(f))}"
Note: I am looking for Yocto built-in which can achieve solution for above mentioned , would like to share other way to resolve the functionality for community's benefit.
Add following in bb file if you are using one or refer to talel-belhadjsalem answer to use utils.bbclass.
def add_directory_bootfs(d, dirname, bootfs_dir):
file_list = list()
boot_files_list = list()
deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE')
for (dirpath, dirnames, filenames) in os.walk(os.path.join(deploy_dir_image, dirname)):
file_list += [os.path.join(dirpath, file) for file in filenames]
for file in file_list:
file_rel_path = os.path.relpath(file, os.path.join(deploy_dir_image, dirname))
boot_file_entry = os.path.join(dirname, file_rel_path) + ';' + os.path.join(bootfs_dir, dirname, file_rel_path)
boot_files_list.append(boot_file_entry)
return ' '.join(boot_files_list)
IMAGE_EFI_BOOT_FILES += "${#add_directory_bootfs(d, 'relative_path_to_dir_in_deploy_dir_image', 'EFI/BOOT')}"
I have a script that writes Hive table content into a CSV file in HDFS.
The target folder name is given in a JSON paramater file. When I launch the script I notice that the folder that I already created is deleted automatically and then an error is thrown saying that the target file does not exist. This my script:
sigma.cache // sigma is the df that contains the hive table. Tested OK
sigma.repartition(1).write.mode(SaveMode.Overwrite).format("csv").option("header", true).option("delimiter", "|").save(Parametre_vigiliste.cible)
val conf = new Configuration()
val fs = FileSystem.get(conf)
//Parametre_vigiliste.cible is the variable inide the JSON file that contains the target folder name
val file = fs.globStatus(new Path(Parametre_vigiliste.cible + "/part*"))(0).getPath().getName();
fs.rename(new Path(Parametre_vigiliste.cible + "/" + file), new Path(Parametre_vigiliste.cible + "/" + "FIC_PER_DATALAKE_.txt"));
sigma.unpersist()
ERROR THROWN:
exception caught: java.lang.UnsupportedOperationException: CSV data
source does not support null data type.
Can this code delete a folder for a certain reason? Thank you.
So as Prateek suggested I tried sigma.printSchema and I discovered some null columns. I rectified that and it worked.
I'm trying to develop an Word-AddIn where I can upload a file, or where I could make changes to the file itself. I can do that with any File on the Local NTFS Windows Filesystem. But when a File is located to any "OneDrive"-Folder it won't work, as the file-Document path starts with:
https://companyname-my.sharepoint.com/personal/name_whatever/Documents
according to the return values of:
var doc = Globals.ThisAddIn.Application.ActiveDocument;
MessageBox.Show("Document Name : " + doc.Name);
MessageBox.Show("Document Full Name : " + doc.FullName);
MessageBox.Show("Document Path : " + doc.Path);
rather than expected to be:
C:\Users\myname\OneDrive-My_Company_Name\Documents
So basically the local folder is not callable. So does anybody has already faced this kind of issue?
You can use the Save or SaveAs methods of the Document to save the document on the local folder.
I use universal sbt-native-packager to generate a zip file distribution.
sbt universal:packageBin
The generated zip file, once extracted, contains everything inside a main directory named as my zip file:
unzip my-project-0.0.1.zip
my-project-0.0.1/bin
my-project-0.0.1/lib
my-project-0.0.1/conf
...
How can I create a zip that has no root folder, so that when extracted it will have a structure like that?
bin
lib
conf
Thanks
I'm not confident enough with sbt and scala to submit a pull request.
bash scripting has to be excluded right now, so my current (and ugly) solution is this one:
packageBin in Universal := {
val originalFileName = (packageBin in Universal).value
val (base, ext) = originalFileName.baseAndExt
val newFileName = file(originalFileName.getParent) / (base + "_dist." + ext)
val extractedFiles = IO.unzip(originalFileName,file(originalFileName.getParent))
val mappings: Set[(File, String)] = extractedFiles.map( f => (f, f.getAbsolutePath.substring(originalFileName.getParent.size + base.size + 2)))
val binFiles = mappings.filter{ case (file, path) => path.startsWith("bin/")}
for (f <- binFiles) f._1.setExecutable(true)
ZipHelper.zip(mappings,newFileName)
IO.move(newFileName, originalFileName)
IO.delete(file(originalFileName.getParent + "/" + originalFileName.base))
originalFileName
}
The solution proposed on github seems to be way nicer than mine even tough it doesn't work for me:
https://github.com/sbt/sbt-native-packager/issues/276
Unfortunately it looks like the name of that top-level directory is fixed to be the same as the name of the distributable zip (check out line 24 of the ZipHelper source on GitHub).
So unless you feel like making it configurable and submitting a pull request, it might just be easier to modify the resulting zip on the command line (assuming some flavour of UNIX):
unzip my-project-0.0.1.zip && cd my-project-0.0.1 && zip -r ../new.zip * && cd -
This will create new.zip alongside the existing zipfile - you could then mv it over the top if you like.