I am trying to get a file path from conf, this is my conf file:
uploadedFilePath.conf
file.path="public/img/"
This is how I am getting path from conf file in my code:
val conf = ConfigFactory.load()
var path : String = conf.getString("file.path")
I am getting exception on second line
09:58:11.527 108649 [application-akka.actor.default-dispatcher-10]
PlayDefaultUpstreamHandler ERROR - Cannot invoke the action
com.typesafe.config.ConfigException$WrongType: system properties: path
has type OBJECT rather than STRING
at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:159)
~[com.typesafe.config-1.3.0.jar:na]
at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
~[com.typesafe.config-1.3.0.jar:na]
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
~[com.typesafe.config-1.3.0.jar:na]
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
~[com.typesafe.config-1.3.0.jar:na]
at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
~[com.typesafe.config-1.3.0.jar:na]
I am do not know what I am doing wrong.
remove the quotes
file.path=public/img/
Related
I have a JAR file with below structure:
example.jar
|
+-org
| +-springframework
| +-boot
| +-loader.jar
+-BOOT-INF
+-classes
| +- kafka
| truststore.jks ==> I want to get the path here
+-lib
+-dependency1.jar
How can I get the configuration file path, only path (string) of 'kafka/truststore.jks' file ?
Because I am applying the SSL for KafkaProducer, I using below code and it work fine on local:
#Value("classpath:kafka/truststore.jks")
private org.springframework.core.io.Resource sslTruststoreResource;
...
String sslTruststoreLocation = sslTruststoreResource.getFile().getAbsolutePath(); // ==\> ***it throw FileNotFoundException here on deployed Server, local env run fine !***
Map\<String, Object\> config = Maps.newHashMap();
config.put("ssl.truststore.location", sslTruststoreLocation);
but when I deploy on Server, it throw FileNotFoundException :(
After many days to research, I found that the sslTruststoreResource.getFile() will be fail for JAR file case as mentioned at here
The sslTruststoreResource.getInputStream() or sslTruststoreResource.getFilename() are ok for JAR file but they are not path I need for kafka configuration.
In my project, the 'truststore.jks' file is located as below:
src
-- java
-- resources
. -- kafka
-- truststore.jks
So, is there any solution for my issue ? Thank you.
I tried to use ClassPathResource, ResourcePatternResolver but they not working
After many ways I still could not get the path from JKS file then I copy it to another path out of jar file where my code can refer to its path
final String FILE_NAME = env.getProperty("kafka.metadata.ssl.truststore.location");
String sslTruststoreLocation = "*-imf-kafka-client.truststore.jks";
try {
InputStream is = getClass().getClassLoader().getResourceAsStream(FILE_NAME);
// Get the destination path where contains JKS file
final String HOME_DIR = System.getProperty("user.home");
final Path destPath = Paths.get(HOME_DIR, "tmp");
if (!Files.isDirectory(destPath)) {
Files.createDirectories(destPath);
}
// Copy JKS file to destination path
sslTruststoreLocation = destPath.toFile().getAbsolutePath() + "/" + FILE_NAME;
File uploadedFile = new File(sslTruststoreLocation);
if(!uploadedFile.exists()) {
uploadedFile.getParentFile().mkdirs();
uploadedFile.createNewFile();
FileCopyUtils.copy(is, new FileOutputStream(sslTruststoreLocation));
log.debug("Copied {} file from resources dir to {} done !", FILE_NAME, sslTruststoreLocation);
}
config.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, sslTruststoreLocation);
} catch (IOException e) {
final String message = "The " + sslTruststoreLocation + " file not found to construct a KafkaProducer";
log.error(message);
}
Looks like this is a known issue in Kafka.
Spring Boot propose a workaround similar to yours.
I am trying to get the directory name from hdfs location using spark. I am getting the whole path to the directory instead of just the directory name.
val fs = FileSystem.get(sc.hadoopConfiguration)
val ls = fs.listStatus(new Path("/user/rev/raw_data"))
ls.foreach(x => println(x.getPath))
This gives me
hdfs://localhost/user/rev/raw_data/191622-140
hdfs://localhost/user/rev/raw_data/201025-001
hdfs://localhost/user/rev/raw_data/201025-002
hdfs://localhost/user/rev/raw_data/2065-5
hdfs://localhost/user/rev/raw_data/223575-002
How can I just get the output as below (i.e. just the directory name)
191622-140
201025-001
201025-002
2065-5
223575-002
As you work with Path objects when using status.getPath, you can simply use the getName function on Path objects:
FileSystem
.get(sc.hadoopConfiguration)
.listStatus(new Path("/user/rev/raw_data"))
.filterNot(_.isFile)
.foreach(status => println(status.getPath.getName))
which would print:
191622-140
201025-001
201025-002
2065-5
223575-002
I have Loaded a DataFrame into HDFS as text format using below code. finalDataFrame is the DataFrame
finalDataFrame.repartition(1).rdd.saveAsTextFile(targetFile)
After executing the above code I found that a directory created with the file name I provided and under the directory a file created but not in text format. The file name is like part-00000.
I have resolved this in HDFS using below code.
val hadoopConfig = new Configuration()
val hdfs = FileSystem.get(hadoopConfig)
FileUtil.copyMerge(hdfs, new Path(srcPath), hdfs, new Path(dstPath), true, hadoopConfig, null)
Now I can get the text file in the mentioned path with given file name.
But when I am trying to do the same operation in S3 it is showing some exception
FileUtil.copyMerge(hdfs, new Path(srcPath), hdfs, new Path(dstPath), true, hadoopConfig, null)
java.lang.IllegalArgumentException: Wrong FS:
s3a://globalhadoop/data, expected:
hdfs://*********.aws.*****.com:8050
It seems that S3 path is not supporting over here. Can anyone please assist how to do this.
I have solved the problem using below code.
def createOutputTextFile(srcPath: String, dstPath: String, s3BucketPath: String): Unit = {
var fileSystem: FileSystem = null
var conf: Configuration = null
if (srcPath.toLowerCase().contains("s3a") || srcPath.toLowerCase().contains("s3n")) {
conf = sc.hadoopConfiguration
fileSystem = FileSystem.get(new URI(s3BucketPath), conf)
} else {
conf = new Configuration()
fileSystem = FileSystem.get(conf)
}
FileUtil.copyMerge(fileSystem, new Path(srcPath), fileSystem, new Path(dstPath), true, conf, null)
}
I have written the code for filesystem of S3 and HDFS and both are working fine.
You are passing in the hdfs filesystem as the destination FS in FileUtil.copyMerge. You need to get the real FS of the destination, which you can do by calling Path.getFileSystem(Configuration) on the destination path you have created.
I am trying to read the following config file using typesafe config
common = {
jdbcDriver = "com.mysql.jdbc.Driver"
slickDriver = "slick.driver.MySQLDriver"
port = 3306
db = "foo"
user = "bar"
password = "baz"
}
source = ${common} {server = "remoteserver"}
target = ${common} {server = "localserver"}
When I try to read my config using this code
val conf = ConfigFactory.parseFile(new File("src/main/resources/application.conf"))
val username = conf.getString("source.user")
I get an error
com.typesafe.config.ConfigException$NotResolved: source.user has not been resolved, you need to call Config#resolve(), see API docs for Config#resolve()
I don't get any error if I put everything inside "source" or "target" tags. I get errors only when I try to use "common"
I solved it myself.
ConfigFactory.parseFile(new File("src/main/resources/application.conf")).resolve()
I solved it.
Config confSwitchEnv = ConfigFactory.load("env.conf");
the env.conf is in the resources dir.
reference: https://nicedoc.io/lightbend/config
I am trying to annotate a corpus using Syntaxnet. I added the following lines in the end of the /models/syntaxnet/syntaxnet/models/parsey_mcparseface/context.pbtxt file:
input {
name: 'input_file'
record_format: 'english-text'
Part {
file_pattern: '/home/melvyn/text.txt'
}
}
output {
name: 'output_file'
record_format: 'english-text'
Part {
file_pattern: '/home/melvyn/text-tagged.txt'
}
}
When i run the command:
./demo.sh --input=input_file --output=output_file
I am getting:
./demo.sh: line 31: bazel-bin/syntaxnet/parser_eval: No such file or directory
./demo.sh: line 43: bazel-bin/syntaxnet/parser_eval: No such file or directory
./demo.sh: line 55: bazel-bin/syntaxnet/conll2tree: No such file or directory
According to the answer given ## here ## I changed my demo.sh file and now I get some errors which say:
[libprotobuf ERROR external/tf/google/protobuf/src/google/protobuf/text_format.cc:291] Error parsing text-format syntaxnet.TaskSpec: 200:8: Message type "syntaxnet.TaskOutput" has no field named "Part".
E external/tf/tensorflow/core/framework/op_segment.cc:53] Create kernel failed: Invalid argument: Could not parse task context at syntaxnet/models/parsey_mcparseface/context.pbtxt
E external/tf/tensorflow/core/common_runtime/executor.cc:333] Executor failed to create kernel. Invalid argument: Could not parse task context at syntaxnet/models/parsey_mcparseface/context.pbtxt
[[Node: DocumentSource = DocumentSourcebatch_size=32, corpus_name="stdin-conll", task_context="syntaxnet/models/parsey_mcparseface/context.pbtxt", _device="/job:localhost/replica:0/task:0/cpu:0"]]
What could be a possible solution?
Though it's not certain but I think you are not running the shell script from the root directory. Please try running it as per the instructions mentioned here
I hope it helps.