I'm trying to read text file located in resources directory using Scala version 2.12.3.
However I'm getting file not found error.
my project in eclipse
my scala code:
package main.scala
import scala.io.Source
import scala.io.Codec
object Application {
def main(args: Array[String]) {
try {
val source = Source.fromFile("sample.txt")(Codec.UTF8)
for (line <- source.getLines) {
println(line.toUpperCase)
}
source.close
} catch {
case e: Throwable => e.printStackTrace()
}
}
}
I also tried using
val source = Source.fromFile("sample.txt")(Codec.UTF8)
but got the same error.
If you want to read file from src/main/resources directory you should use Source.fromResource method, so try this:
Source.fromResource("sample.txt")(Codec.UTF8)
Update
In your case you have to use either Source.fromFile("src/main/resources/sample.txt") or
Source.fromFile("sample.txt") if you put your file in root project directory
Related
I have following scala code:
val file = new FileReader("myfile.txt")
try {
// do operations on file
} finally {
file.close() // close the file
}
How do I handle FileNotFoundException thrown when I read the file? If I put that line inside try block, I am not able to access the file variable inside finally.
For scala 2.13:
you can just use Using to acquire some resource and release it automatically without error handling if it's an AutoClosable:
import java.io.FileReader
import scala.util.Using
val newStyle: Try[String] = Using(new FileReader("myfile.txt")) {
reader: FileReader =>
// do something with reader
"something"
}
newStyle
// will be
// Failure(java.io.FileNotFoundException: myfile.txt (No such file or directory))
// if file is not found or Success with some value it will not fall
scala 2.12:
You can wrap your reader creation by scala.util.Try and if it will fall on creation you will get Failure with FileNotFoundException inside.
import java.io.FileReader
import scala.util.Try
val oldStyle: Try[String] = Try{
val file = new FileReader("myfile.txt")
try {
// do operations on file
"something"
} finally {
file.close() // close the file
}
}
oldStyle
// will be
// Failure(java.io.FileNotFoundException: myfile.txt (No such file or directory))
// or Success with your result of file reading inside
I recommend not to use try ... catch blocks in scala code. It's not type safety for some cases and can lead to non-obvious results but for release some resource in old scala versions there is the only way to do it - using try-finally.
I have a file src/main/scala/foo.scala which needs to be inside package bar. Ideally the file should be inside src/main/scala/bar/foo.scala.
// src/main/scala/foo.scala
package bar
// ...
How can I auto-fix this issue throughout my project such that the folder structure matches the package structure?
Is there any SBT plugin etc that can help me fix this issue?
As far as I am aware there are not such tools, though AFAIR IntelliJ can warn about package-directory mismatch.
Best I can think if is custom scalafix (https://scalacenter.github.io/scalafix/) rule - scalafix/scalameta would be used to check file's actual package, translate it to an expected directory and if they differ, move file.
I suggest scalafix/scalameta because there are corner cases like:
you are allowed to write your packages like:
package a
package b
package c
and it almost like package a.b.c except that it automatically imports everything from a and b
you can have package object in your file and then if you have
package a.b
package object c
this file should be in a/b/c directory
so I would prefer to check if file didn't fall under any of those using some existing tooling.
If you are certain that you don't have such cases (I wouldn't without checking) you could:
match the first line with regexp (^package (.*))
translate a.b.c into a/b/c (matched.split('.').map(_.trim).mkString(File.separator))
compare generated location to an actual location ( I suggest resolving absolute file locations)
move file if necessary
If there is a possibility of having more complex case than that, I could replace first step by querying scalafix/scalameta utilities.
Here is an sbt plugin providing packageStructureToDirectoryStructure task that reads package statements from source files, creates corresponding directories, and then moves files to them
import sbt._
import sbt.Keys._
import better.files._
object PackagesToDirectories extends AutoPlugin {
object autoImport {
val packageStructureToDirectoryStructure = taskKey[Unit]("Make directory structure match package structure")
}
import autoImport._
override def trigger = allRequirements
override lazy val projectSettings = Seq(
packageStructureToDirectoryStructure := {
val log = streams.value.log
log.info(s"Refactoring directory structure to match package structure...")
val sourceFiles = (Compile / sources).value
val sourceBase = (Compile / scalaSource).value
def packageStructure(lines: Traversable[String]): String = {
val packageObjectRegex = """package object\s(.+)\s\{""".r
val packageNestingRegex = """package\s(.+)\s\{""".r
val packageRegex = """package\s(.+)""".r
lines
.collect {
case packageObjectRegex(name) => name
case packageNestingRegex(name) => name
case packageRegex(name) => name
}
.flatMap(_.split('.'))
.mkString("/")
}
sourceFiles.foreach { sourceFile =>
val packagePath = packageStructure(sourceFile.toScala.lines)
val destination = file"$sourceBase/$packagePath"
destination.createDirectoryIfNotExists(createParents = true)
val result = sourceFile.toScala.moveToDirectory(destination)
log.info(s"$sourceFile moved to $result")
}
}
)
}
WARNING: Make sure to backup the project before running it.
I have a folder structure like below:
- main
-- java
-- resources
-- scalaresources
--- commandFiles
and in that folders I have my files that I have to read.
Here is the code:
def readData(runtype: String, snmphost: String, comstring: String, specificType: String): Unit = {
val realOrInvFile = "/commandFiles/snmpcmds." +runtype.trim // these files are under commandFiles folder, which I have to read.
try {
if (specificType.equalsIgnoreCase("Cisco")) {
val specificDeviceFile: String = "/commandFiles/snmpcmds."+runtype.trim+ ".cisco"
val realOrInvCmdsList = scala.io.Source.fromFile(realOrInvFile).getLines().toList.filterNot(line => line.startsWith("#")).map{
//some code
}
val specificCmdsList = scala.io.Source.fromFile(specificDeviceFile).getLines().toList.filterNot(line => line.startsWith("#")).map{
//some code
}
}
} catch {
case e: Exception => e.printStackTrace
}
}
}
Resources in Scala work exactly as they do in Java.
It is best to follow the Java best practices and put all resources in src/main/resources and src/test/resources.
Example folder structure:
testing_styles/
├── build.sbt
├── src
│ └── main
│ ├── resources
│ │ └── readme.txt
Scala 2.12.x && 2.13.x reading a resource
To read resources the object Source provides the method fromResource.
import scala.io.Source
val readmeText : Iterator[String] = Source.fromResource("readme.txt").getLines
reading resources prior 2.12 (still my favourite due to jar compatibility)
To read resources you can use getClass.getResource and getClass.getResourceAsStream .
val stream: InputStream = getClass.getResourceAsStream("/readme.txt")
val lines: Iterator[String] = scala.io.Source.fromInputStream( stream ).getLines
nicer error feedback (2.12.x && 2.13.x)
To avoid undebuggable Java NPEs, consider:
import scala.util.Try
import scala.io.Source
import java.io.FileNotFoundException
object Example {
def readResourceWithNiceError(resourcePath: String): Try[Iterator[String]] =
Try(Source.fromResource(resourcePath).getLines)
.recover(throw new FileNotFoundException(resourcePath))
}
good to know
Keep in mind that getResourceAsStream also works fine when the resources are part of a jar, getResource, which returns a URL which is often used to create a file can lead to problems there.
in Production
In production code I suggest to make sure that the source is closed again.
For Scala >= 2.12, use Source.fromResource:
scala.io.Source.fromResource("located_in_resouces.any")
One-liner solution for Scala >= 2.12
val source_html = Source.fromResource("file.html").mkString
Important taken from comments (thanks to #anentropic): with Source.fromResource you do not put the initial forward slash.
import scala.io.Source
object Demo {
def main(args: Array[String]): Unit = {
val ipfileStream = getClass.getResourceAsStream("/folder/a-words.txt")
val readlines = Source.fromInputStream(ipfileStream).getLines
readlines.foreach(readlines => println(readlines))
}
}
The required file can be accessed as below from resource folder in scala
val file = scala.io.Source.fromFile(s"src/main/resources/app.config").getLines().mkString
For Scala 2.11, if getLines doesn't do exactly what you want you can also copy the a file out of the jar to the local file system.
Here's a snippit that reads a binary google .p12 format API key from /resources, writes it to /tmp, and then uses the file path string as an input to a spark-google-spreadsheets write.
In the world of sbt-native-packager and sbt-assembly, copying to local is also useful with scalatest binary file tests. Just pop them out of resources to local, run the tests, and then delete.
import java.io.{File, FileOutputStream}
import java.nio.file.{Files, Paths}
def resourceToLocal(resourcePath: String) = {
val outPath = "/tmp/" + resourcePath
if (!Files.exists(Paths.get(outPath))) {
val resourceFileStream = getClass.getResourceAsStream(s"/${resourcePath}")
val fos = new FileOutputStream(outPath)
fos.write(
Stream.continually(resourceFileStream.read).takeWhile(-1 !=).map(_.toByte).toArray
)
fos.close()
}
outPath
}
val filePathFromResourcesDirectory = "google-docs-key.p12"
val serviceAccountId = "[something]#drive-integration-[something].iam.gserviceaccount.com"
val googleSheetId = "1nC8Y3a8cvtXhhrpZCNAsP4MBHRm5Uee4xX-rCW3CW_4"
val tabName = "Favorite Cities"
import spark.implicits
val df = Seq(("Brooklyn", "New York"),
("New York City", "New York"),
("San Francisco", "California")).
toDF("City", "State")
df.write.
format("com.github.potix2.spark.google.spreadsheets").
option("serviceAccountId", serviceAccountId).
option("credentialPath", resourceToLocal(filePathFromResourcesDirectory)).
save(s"${googleSheetId}/${tabName}")
The "resources" folder must be under the source root. if using intellj check for the blue folder in the project folders on the left side. eg AppName/src/main/scala or Project/scala/../main/ etc.
If using val stream: InputStream = getClass.getResourceAsStream("/readme.txt") don't forget the "/" (forward slash), given readme.txt is the file inside resources
I have a spark/scala project named as Omega
I have a conf file inside Omega/conf/omega.config
I use API's from typesafe to load the config file from conf/omega.config.
It was working fine and I was able to read the respective value for each key
Now today, For the first time I added some more key-value pairs in my omega.config file and tried to retrieve them from my scala code. It throws
Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'job_name'
This issue started happening after adding new value for the key job_name in my omega.config file.
Also I am not able to read the newly added -key-values, I am still able to read all old values using config. getString method
I am building my spark/scala application using maven.
Omega.config
input_path="/user/cloudera/data
user_name="surender"
job_name="SAMPLE"
I am Not able to access the recently added key "job_name" alone
package com.pack1
import com.pack2.ApplicationUtil
object OmegaMain {
val config_loc = "conf/omega.config"
def main(args: Array[String]): Unit = {
val config = ApplicationUtil.loadConfig(config_loc)
val jobName = ApplicationUtil.getFromConfig(config,"job_name")
}
}
package com.pack2
import com.typesafe.config.{Config, ConfigFactory}
object ApplicationUtil {
def loadConfig(filePath:String):Config={
val config = ConfigFactory.parseFile(new File(filePath))
config
}
def getFromConfig(config:Config,jobName:String):String={
config.getString(jobName)
}
}
Could some one help me what went wrong?
You can try something like:
def loadConfig(filename: String, syntax: ConfigSyntax): Config = {
val in: InputStream = getClass.getResourceAsStream(filename)
if (in == null) return null
val file: File = File.createTempFile(String.valueOf(in.hashCode()), ".conf")
file.deleteOnExit()
val out: FileOutputStream = new FileOutputStream(file)
val buffer: Array[Byte] = new Array(1024)
var bytesRead: Int = in.read(buffer)
while (bytesRead != -1) { out.write(buffer, 0, bytesRead); bytesRead = in.read(buffer) }
out.close()
val conf: Config = ConfigFactory.parseFile(file, ConfigParseOptions.defaults().setSyntax(syntax).setAllowMissing(false).setOriginDescription("Merged with " + filename))
conf
}
filename is some file path in the classpath. If you want to update this method to taking some external file into account, change update the 4th with val file: File = new File("absolute Path of he file")
I am guessing the file isn't on the classpath after you build with Maven.
Since you are using Maven to build a jar, you need your omega.config to be in the classpath. This means that you either have to put it into src/main/resources by default or explicitly tell Maven to add conf to the default resources classpath.
Using Scala 2.10.2------------
I am a Java engineer, and started to learn Scala from yesterday, but I got stuck now, this simply code does not work for me, but when I use java to write it, it works fine:
package lesson4
import scala.io.Source
import scala.reflect.io.File
object Test {
def main(args: Array[String]): Unit = {
var filePath = Source.getClass().getResource("/lesson4/test.txt")
var file = Source.fromFile(filePath.getFile())
var lines = file.getLines
lines.foreach(println)
}
}
The file is in the right path:
But the code just does not work:
Exception in thread "main" java.io.FileNotFoundException: /Users/wenjiezhang/Desktop/source_files/git_hub%20workspace/Learning%20Scala/ScalaLearning/bin/lesson4/test.txt (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at scala.io.Source$.fromFile(Source.scala:90)
at scala.io.Source$.fromFile(Source.scala:75)
at scala.io.Source$.fromFile(Source.scala:53)
at lesson4.Test$.main(Test.scala:20)
at lesson4.Test.main(Test.scala)
As you can see from the error log, the path you're providing is being converted to a URL (for example SPACE becomes %20). You should use the fromURL method Source.fromURL(Source.getClass().getResource("/lesson4/test.txt"))
scala.io.Source docs
You have spaces in your path converted to encoded from the URL.