I am deleting output folders before running my spark job .
Sometime it deletes it but sometime it delete all the files inside it but top level folder still remains .
I have sub folders kind of structure .
This is how i am deleting the folders .
def DeleteDescrFolder(fs: org.apache.hadoop.fs.FileSystem, descrFileURL: String) = {
val bDescr = fs.exists(new Path(descrFileURL))
if (true.equals(bDescr)) {
val outputFile = fs.globStatus(new Path(descrFileURL))
for (DeleteFilePath <- outputFile) {
fs.delete(DeleteFilePath.getPath)
}
println("Descr File is delete from " + descrFileURL)
} else {
println(descrFileURL + "Decsr Does not Exist")
}
}
How can i remove folder name also ?
You are deleting the files inside the folder specified. Try the following code which will delete the folder as well
def DeleteDescrFolder(fs: org.apache.hadoop.fs.FileSystem, descrFileURL: String) = {
if (fs.exists(new Path(descrFileURL))) {
try{
fs.delete(new Path(descrFileURL),true)
println("Descr folder is deleted " + descrFileURL)
}catch{case e: Exception =>
print("exeption "+e)
}
} else {
println(descrFileURL + "Decsr Does not Exist")
}
}
Related
I have following piece of code. I'm trying to check out two files from Perforce and put them in a changelist. But run_add does not check the files out. The only thing I see in Perforce is a empty changelist with no files in it.
""" Checks out files from workspace using P4"""
files = ['analyse-location.cfg', 'CMakeLists.txt']
p4 = P4()
# Connect and disconnect
if (p4.connected()):
p4.disconnect()
p4.port = portp4
p4.user = usernameP4
p4.password = passwordP4
p4.client = clientP4
try:
p4.connect()
if p4.connected():
change = p4.fetch_change()
change['Description'] = "Auto"
change['Files'] = []
changeList = p4.save_change(change)[0].split()[1]
for items in files:
abs_path = script_dir + "\\" + items
p4.run_add("-c", changeList, items)
print("Adding file "+ abs_path + " to "+ changeList)
# Done! Disconnect!
p4.disconnect()
except P4Exception:
print("Something went wrong in P4 connection. The errors are: ")
for e in p4.errors:
print(e)
p4.disconnect()
However, when I have instead p4.run("edit", items) it puts the files in the default changelist.It really gets on my nervs. I don't know I am doing that is wrong. The changes list created as well. I use python 3.7 32 bits on Windows
Your script discards the output of the run_add call. Try changing this:
for items in files:
abs_path = script_dir + "\\" + items
p4.run_add("-c", changeList, items)
print("Adding file "+ abs_path + " to "+ changeList)
to:
for items in files:
abs_path = script_dir + "\\" + items
output = p4.run_add("-c", changeList, items)
print("Adding file "+ abs_path + " to "+ changeList)
if output:
print(output)
if p4.errors:
print(p4.errors)
if p4.warnings:
print(p4.warnings)
That will show you the results of the p4 add commands that you're running. Based on the fact that a p4 edit opens the files, I expect you'll find a message like this:
C:\Perforce\test>p4 add foo
//stream/main/foo - can't add existing file
The p4 add and p4 edit commands are not synonymous; one is for adding a new file, one is for editing an existing file. If your script is editing existing files, it should be calling run_edit, not run_add.
I changed my question to following and it worked.
p4.port = portp4
p4.user = usernameP4
p4.password = passwordP4
p4.client = clientP4
try:
p4.connect()
if p4.connected():
change = p4.fetch_change()
change['Description'] = "Auto"
change['Files'] = []
changeList = p4.save_change(change)[0].split()[1]
for items in files:
abs_path = script_dir + "\\" + items
output = p4.run_edit("-c", changeList, items)
print("Adding file "+ abs_path + " to "+ changeList)
if output:
print(output)
if p4.errors:
print(p4.errors)
if p4.warnings:
print(p4.warnings)
p4.disconnect()
except P4Exception:
print("Something went wrong in P4 connection. The errors are: ")
for e in p4.errors:
print(e)
p4.disconnect()
Thanks to #Sam Stafford for his hint. Now it works just like a charm. The key was to change p4.run_add("-c", changelist, items) to p4.run_edit("-c", changelist, items)
i would like merging from local file from /opt/one.txt with file at my hdfs hdfs://localhost:54310/dummy/two.txt.
contains at one.txt : f,g,h
contains at two.txt : 2424244r
my code :
val cfg = new Configuration()
cfg.addResource(new Path("/usr/local/hadoop/etc/hadoop/core-site.xml"))
cfg.addResource(new Path("/usr/local/hadoop/etc/hadoop/hdfs-site.xml"))
cfg.addResource(new Path("/usr/local/hadoop/etc/hadoop/mapred-site.xml"))
try
{
val srcPath = "/opt/one.txt"
val dstPath = "/dumCBF/two.txt"
val srcFS = FileSystem.get(URI.create(srcPath), cfg)
val dstFS = FileSystem.get(URI.create(dstPath), cfg)
FileUtil.copyMerge(srcFS,
new Path(srcPath),
dstFS,
new Path(dstPath),
true,
cfg,
null)
println("end proses")
}
catch
{
case m:Exception => m.printStackTrace()
case k:Throwable => k.printStackTrace()
}
i was following tutorial from : http://deploymentzone.com/2015/01/30/spark-and-merged-csv-files/
and it's not working at all, error would below this :
java.io.FileNotFoundException: File does not exist: /opt/one.txt
i dont why, sounds of error like that? BTW the file one.txt is exist
and then, im add some code to check exist file :
if(new File(srcPath).exists()) println("file is exist")
any idea or references? thanks!
EDIT 1,2 : typo extensions
Why is it when I don't include the file I get this
[IOException: can not replace a non-empty directory: Path(./public/upload)]
request.body.file("resourceFile").map { k =>
val t = new java.io.File(s"./public/upload/${k.filename}")
k.ref.moveTo(t, true)
println("Ok File Upload" + k.filename)
How do you stop this from happening?
Ta
I don't understand why it is happening.
You could add an ugly if statement to prevent the error :
request.body.file("resourceFile").map { k =>
if (!k.filename.isEmpty) {
val t = new java.io.File(s"./public/upload/${k.filename}")
k.ref.moveTo(t, true)
println("Ok File Upload" + k.filename)
}
I'm creating my first gradle plugin. I'm trying to copy a file from the distribution jar into a directory I've created at the project. Although the file exists inside the jar, I can't copy it to the directory.
This is my task code:
import org.gradle.api.DefaultTask;
import org.gradle.api.tasks.TaskAction;
class InitTask extends DefaultTask {
File baseDir;
private void copyEnvironment(File environments) {
String resource = getClass().getResource("/environments/development.properties").getFile();
File input = new File(resource);
File output = new File(environments, "development.properties");
try {
copyFile(input, output);
}
catch (IOException e) {
e.printStackTrace();
}
}
void copyFile(File sourceFile, File destFile) {
destFile << sourceFile.text
}
#TaskAction
void createDirectories() {
logger.info "Creating directory."
File environments = new File(baseDir, "environments");
File scripts = new File(baseDir, "scripts");
File drivers = new File(baseDir, "drivers");
[environments, scripts, drivers].each {
it.mkdirs();
}
copyEnvironment(environments);
logger.info "Directory created at '${baseDir.absolutePath}'."
}
}
And this is the error I'm getting:
:init
java.io.FileNotFoundException: file:/path-to-jar/MyJar.jar!/environments/development.properties (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at groovy.util.CharsetToolkit.<init>(CharsetToolkit.java:69)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.newReader(DefaultGroovyMethods.java:15706)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.getText(DefaultGroovyMethods.java:14754)
at org.codehaus.groovy.runtime.dgm$352.doMethodInvoke(Unknown Source)
at org.codehaus.groovy.reflection.GeneratedMetaMethod$Proxy.doMethodInvoke(GeneratedMetaMethod.java:70)
at groovy.lang.MetaClassImpl$GetBeanMethodMetaProperty.getProperty(MetaClassImpl.java:3465)
at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:61)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:227)
at br.com.smartcoders.migration.tasks.InitTask.copyFile(InitTask.groovy:29)
Just to emphasize, the development.properties is inside the environments directory inside the MyJar.jar
getClass().getResource() returns a URL. To access that URL, you'll have to read it directly (e.g. with url.text) rather than first converting it to a String/File. Or you can use getClass().getResourceAsStream().text, which is probably more accurate. In both cases you can optionally specify the file encoding.
Kotlin DSL answer!
For cases like this it is good to have extensions:
fun Any.getResource(filename: String): File? {
val input = this::class.java.classLoader.getResourceAsStream(filename) ?: return null
val tempFile = File.createTempFile(
filename.substringBeforeLast('.'),
"." + filename.substringAfterLast('.')
)
tempFile.deleteOnExit()
tempFile.writer().use { output ->
input.bufferedReader().use { input ->
output.write(input.readText())
}
}
return tempFile
}
I've added a new exception to my plugin:
class UnzipException extends RuntimeException {
String message
String defaultMessage
String fileName
}
. . .
else {
throw new UnzipException(
message:"grailsant.unzipexception.badfile",
defaultMessage: "Invalid zip file: ${zipFile}",
fileName: zipFile)
}
...
And in the plugin's messages.properties I have:
grailsant.unzipexception.badfile=Invalid zip file: {0}
Two questions:
How do I get {0} filled in with fileName?
Can an application override the grailsant.unzipexception.badfile message?
(1) It seems like this has to be done by app:
try {
. . .
} catch (org.grails.plugins.grailsant.UnzipException e) {
flash.message = e.message
flash.args = [e.fileName]
flash.defaultMessage = e.defaultMessage
}
(2) Yep, if the message.properties in the app has the same key as the plugin, the app's values will be used.