Below is my project structure. I'm trying to read my properties file pgmproperties.properties in my program below but getting error File Not found. Please tell me where should i place my file. I'm using SBT build tool.
My code
def main(args: Array[String]) {
// This is used to read Properties files and get the values
val is: InputStream = ClassLoader.getSystemResourceAsStream("pgmproperties.properties")
val prop: Properties = new Properties()
if (is != null) {
prop.load(is)
}
else {
throw new FileNotFoundException("Properties file cannot be loaded")
}
val sndrmailid = prop.getProperty("SndrMailId") //Sender Mail id
val psswd = prop.getProperty("SndrMailPsswd") //Sender gmail Mailbox id password
val tomailid = prop.getProperty("RcvrMailId") // Receiver email id.
println( sndrmailid )
}
Error getting
The right place where to put resources is: src/main/resources (as said in the official SBT documentation).
Try to move your file there and it should work :)
Related
I am trying to read parquet files from directory at S3
val bucketKey = "s3a://foo/direcoty_to_retrieve/"
val conf: Configuration = new Configuration()
conf.setBoolean(AvroReadSupport.AVRO_COMPATIBILITY, true)
val inputFile = HadoopInputFile.fromPath(new Path(bucketKey), conf)
val reader: ParquetReader[GenericRecord] = AvroParquetReader.builder[GenericRecord](inputFile).withConf(conf).build()
however I am getting
Exception in thread "main" java.io.FileNotFoundException: No such file or directory: s3a://foo/direcoty_to_retrieve
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
at org.apache.parquet.hadoop.util.HadoopInputFile.fromPath(HadoopInputFile.java:39)
EDIT:
When I use the AvroParquetReader.builder with filePath e.g :
val reader: ParquetReader[GenericRecord] = AvroParquetReader.builder[GenericRecord](new Path(bucketKey)).withConf(conf).build()
it works, however this option is deprecated and I rather not use it.
on local directory it works. my env variables for AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY are set correctly. what can be the problem ?
I faced the same issue with reading parquet file from Amazon S3 storage via Alpakka Avro parquet lib (which depends on parquet-hadoop-1.10.1.jar). After some debugging I found out the issue in ParquetReader.build method.
public ParquetReader<T> build() throws IOException {
ParquetReadOptions options = optionsBuilder.build();
if (path != null) {
FileSystem fs = path.getFileSystem(conf);
FileStatus stat = fs.getFileStatus(path);
if (stat.isFile()) {
return new ParquetReader<>(
Collections.singletonList((InputFile) HadoopInputFile.fromStatus(stat, conf)),
options,
getReadSupport());
} else {
List<InputFile> files = new ArrayList<>();
for (FileStatus fileStatus : fs.listStatus(path, HiddenFileFilter.INSTANCE)) {
files.add(HadoopInputFile.fromStatus(fileStatus, conf));
}
return new ParquetReader<T>(files, options, getReadSupport());
}
} else {
return new ParquetReader<>(Collections.singletonList(file), options, getReadSupport());
}
}
When HadoopInputFile is used as input, builder path property set to null and reader initiated in else block. As parquet file represented as a directory in filesystem this leads to java.io.FileNotFoundException.
Solution for now will be to use deprecated method:
AvroParquetReader.builder[GenericRecord](new Path(bucketKey)).withConf(conf).build()
In the file system of Hadoop I have Excel file.
I have task to copy that file from Hadoop to remote SFTP server in my Scala/Spark application.
I have formed the opinion that directly it will not work. If my fears are correct, I need to make next steps:
1) Remove excel file from Hadoop to local directory. For example I can make it with Scala DSL:
import scala.sys.process._
s"hdfs dfs -copyToLocal /hadoop_path/file_name.xlsx /local_path/" !
2) From local directory send file to remote SFTP server. What kind of library you can recommend for this task?
Is my reasoning correct? What the best way to solve my problem?
As mentioned in the comment spark-sftp is good choice
if not you can try below sample code from apache-commons-ftp libraries.. which will list all remote files.. similarly you can delete the files as well.. untested pls try it.
Option1:
import java.io.IOException
import org.apache.commons.net.ftp.FTPClient
//remove if not needed
import scala.collection.JavaConversions._
object MyFTPClass {
def main(args: Array[String]): Unit = {
// Create an instance of FTPClient
val ftp: FTPClient = new FTPClient()
try {
// Establish a connection with the FTP URL
ftp.connect("ftp.test.com")
// Enter user details : user name and password
val isSuccess: Boolean = ftp.login("user", "password")
if (isSuccess) {
// empty array is returned
val filesFTP: Array[String] = ftp.listNames()
var count: Int = 1
// Iterate on the returned list to obtain name of each file
for (file <- filesFTP) {
println("File " + count + " :" + file) { count += 1; count - 1 }
}
}
// Fetch the list of names of the files. In case of no files an
// Fetch the list of names of the files. In case of no files an
ftp.logout()
} catch {
case e: IOException => e.printStackTrace()
} finally try ftp.disconnect()
catch {
case e: IOException => e.printStackTrace()
}
}
}
Option 2:
There is something called jsch library you can see this question and example snippet from SO
Well, finally I found the way to solve the task. I decided to use jsch library.
build.sbt:
libraryDependencies += "com.jcraft" % "jsch" % "0.1.55"
.scala:
import scala.sys.process._
import com.jcraft.jsch._
// Copy Excel file from Hadoop file system to local directory with Scala DSL.
s"hdfs dfs -copyToLocal /hadoop_path/excel.xlsx /local_path/" !
val jsch = new JSch()
val session = jsch.getSession("XXX", "XXX.XXX.XXX.XXX") // Set your username and host
session.setPassword("XXX") // Set your password
val config = new java.util.Properties()
config.put("StrictHostKeyChecking", "no")
session.setConfig(config)
session.connect()
val channelSftp = session.openChannel("sftp").asInstanceOf[ChannelSftp]
channelSftp.connect()
channelSftp.put("excel.xlsx", "sftp_path/") // set your path in remote sftp server
channelSftp.disconnect()
session.disconnect()
The following code works as expected, for each iteration the next value from the valueFeed is popped and written to the output.csv file
class TestSimulation extends Simulation {
val valueFeed = csv("input.csv")
val writer = {
val fos = new java.io.FileOutputStream("output.csv")
new java.io.PrintWriter(fos, true)
}
val scn = scenario("Test Sim")
.repeat(2) {
feed(valueFeed)
.exec(session => {
writer.println(session("value").as[String])
session
})
}
setUp(scn.inject(constantUsersPerSec(1) during (10 seconds)))
}
When feed creation is inlined in the feed method the behaviour is still exactly the same
class TestSimulation extends Simulation {
val writer = {
val fos = new java.io.FileOutputStream("output.csv")
new java.io.PrintWriter(fos, true)
}
val scn = scenario("Test Sim")
.repeat(2) {
feed(csv("input.csv"))
.exec(session => {
writer.println(session("value").as[String])
session
})
}
setUp(scn.inject(constantUsersPerSec(1) during (10 seconds)))
}
Since the feed creation is not extracted I would not expect each iteration to be using the same feed but creating it's own feed instance.
Why then is it the behaviour implies the same feed is being used and the first value from the input file not always written to the output?
Example input file (data truncated, tested with more lines to prevent empty feeder exception):
value
1
2
3
4
5
Because csv(...) is in fact FeederBuilder which is called once to produce the feeder to be used within the scenario.
The gatling DSL defines builders - these are executed only once at startup, so even when you inline you get a feeder shared between all users as the same (and only) builder is used to create all the users.
if you want to have each user have its own copy of the data, you can't use the .feed method, but you can get all the records and use other looping constructs to iterate through them
val records = csv("foo.csv").records
foreach(records, "record") {
exec(flattenMapIntoAttributes("${record}"))
}
When I run WURFL demo app for scala:
object Demo {
def main(args: Array[String]) {
// Create WURFL passing a GeneralWURFLEngine object with a wurfl xml
val wurflWrapper = new Wurfl(new GeneralWURFLEngine("classpath:/resources/wurfl.zip"))
// Set cache provider
wurflWrapper.setCacheProvider(new LRUMapCacheProvider)
// Set Performance/Accuracy Mode
wurflWrapper.setTargetAccuracy
// Set Capability Filter
wurflWrapper.setFilter(
"can_assign_phone_number",
"marketing_name",
"brand_name",
"model_name",
"is_smarttv",
"is_wireless_device",
"device_os",
"device_os_version",
"is_tablet",
"ux_full_desktop",
"pointing_method",
"preferred_markup",
"resolution_height",
"resolution_width",
"xhtml_support_level")
// User-Agent here
var userAgent = ""
// Defining headers
var headers = Map("Accept-Datetime"->"Thu, 31 May 2007 20:35:00 GMT")
headers += ("Content-Type"-> "application/x-www-form-urlencoded")
var device = wurflWrapper.deviceForHeaders(userAgent, headers)
val matchType = device.matchType
if (matchType == MatchType.conclusive)
{
println("Match Type is conclusive")
}
val wireless = device.capability("is_wireless_device")
println("Is wireless: " + wireless)
}
}
I get this exception:
[main] ERROR net.sourceforge.wurfl.core.GeneralWURFLEngine - cannot initialize: java.lang.NullPointerException: in is null
java.lang.NullPointerException: in is null
at java.util.zip.ZipInputStream.<init>(ZipInputStream.java:101)
at java.util.zip.ZipInputStream.<init>(ZipInputStream.java:80)
at net.sourceforge.wurfl.core.resource.FileLoader.fromZipFile(FileLoader.java:248)
at net.sourceforge.wurfl.core.resource.FileLoader.openInputStream(FileLoader.java:230)
at net.sourceforge.wurfl.core.resource.FileLoader.getStream(FileLoader.java:288)
at net.sourceforge.wurfl.core.resource.XMLResource.getData(XMLResource.java:163)
at net.sourceforge.wurfl.core.resource.DefaultWURFLModel.init(DefaultWURFLModel.java:115)
at net.sourceforge.wurfl.core.resource.DefaultWURFLModel.<init>(DefaultWURFLModel.java:107)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.init(GeneralWURFLEngine.java:340)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.initIfNeeded(GeneralWURFLEngine.java:319)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.getDeviceForRequest(GeneralWURFLEngine.java:451)
at com.scientiamobile.wurfl.Wurfl.deviceForHeaders(Wurfl.scala:77)
at com.Demo$.main(Demo.scala:49)
at com.Demo.main(Demo.scala)
Exception in thread "main" net.sourceforge.wurfl.core.exc.WURFLRuntimeException: WURFL unexpected exception
at net.sourceforge.wurfl.core.GeneralWURFLEngine.initIfNeeded(GeneralWURFLEngine.java:322)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.getDeviceForRequest(GeneralWURFLEngine.java:451)
at com.scientiamobile.wurfl.Wurfl.deviceForHeaders(Wurfl.scala:77)
at com.Demo$.main(Demo.scala:49)
at com.Demo.main(Demo.scala)
Caused by: java.lang.NullPointerException: in is null
at java.util.zip.ZipInputStream.<init>(ZipInputStream.java:101)
at java.util.zip.ZipInputStream.<init>(ZipInputStream.java:80)
at net.sourceforge.wurfl.core.resource.FileLoader.fromZipFile(FileLoader.java:248)
at net.sourceforge.wurfl.core.resource.FileLoader.openInputStream(FileLoader.java:230)
at net.sourceforge.wurfl.core.resource.FileLoader.getStream(FileLoader.java:288)
at net.sourceforge.wurfl.core.resource.XMLResource.getData(XMLResource.java:163)
at net.sourceforge.wurfl.core.resource.DefaultWURFLModel.init(DefaultWURFLModel.java:115)
at net.sourceforge.wurfl.core.resource.DefaultWURFLModel.<init>(DefaultWURFLModel.java:107)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.init(GeneralWURFLEngine.java:340)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.initIfNeeded(GeneralWURFLEngine.java:319)
... 4 more
The "wurfl.zip" is well located under "resources".
I also tried adding it to main Scala classes path, but still not luck.
From a code perspective
val wurflWrapper = new Wurfl(new GeneralWURFLEngine("classpath:/resources/wurfl.zip"))
is a proper way to initialize your WURFL engine.
You may want to provide information about how you're running the demo, if you are running it inside an IDE (IDEA, Eclipse or Netbeans), or using command line, or other ways. It can also be useful to tell whether you're using Maven or not.
In case you are running it using command line, please provide a sample of how you launch the Scala app and how you set the classpath.
Assuming a scenario where you are compiling with maven and executing the project directly into the target dir using -cp classes, execution will result in your classpath error because resource files are not included in the classes directory.
Make sure that wurfl-scala-example-.jar is included your classpath.
If you are using the Demo project inside IntelliJ IDEA, please make sure that the resource directory is marked as "resource", otherwise IDEA run tool will not include the wurfl.zip file as a resource.
Hope this helps.
I've got an XML file that I need to read from the classpath in order to load some test data for my project with DBUnit when running a custom runTask in SBT.
The XML file is located in /src/main/resources and is copied properly to the /target/scala_2.8.1/classes during the build, but I get a MalformedURLException when trying to access it.
The weird thing is, I can access the file when this data loading functionality was part of my Scala specs unit tests.
Any ideas?
In my case the problem was that I used getClass.getResourceAsStream() in early initialiser. Had to specify the class explicitly with Class.forName() to solve it: Class.forName(<class name>).getResourceAsStream("/data.xml")
If the error is saying that the URL is malformed, it's probably true.
Here's a code I use to grab file from resource during test:
def copyFileFromResource(source: String, dest: File) {
val in = getClass.getResourceAsStream(source)
val reader = new java.io.BufferedReader(new java.io.InputStreamReader(in))
val out = new java.io.PrintWriter(new java.io.FileWriter(dest))
var line: String = null
line = reader.readLine
while (line != null) {
out.println(line)
line = reader.readLine
}
in.close
out.flush
}