get GCS file metadata using scala - scala

I want to get the time creation of files in GCS, I used the code below :
println(Files
.getFileAttributeView(Paths.get("gs://datalake-dev/mu/tpu/file.0450138"), classOf[BasicFileAttributeView])
.readAttributes.creationTime)
The problem is that the Paths.get function replace // with / so I will get gs:/datalake-dev/mu/tpu/file.0450138 instead of gs://datalake-dev/mu/tpu/file.0450138.
Anyone can help me with this ?
Thanks a lot !

I solved the problem by adding the following java code and then calling the java function in scala.
import com.google.cloud.storage.*;
import java.sql.Timestamp;
public class ExtractDate {
public static String getTime(String fileName){
String bucketName = "bucket-data";
String blobName = "doc/files/"+fileName;
// Instantiates a client
Storage storage_client = StorageOptions.getDefaultInstance().getService();
Bucket bucket = storage_client.get(bucketName);
//val storage_client = Storage.
BlobId blobId = BlobId.of(bucketName, blobName);
Blob blob = storage_client.get(blobId);
Timestamp tmp = new Timestamp(bucket.get(blobName).getCreateTime());
System.out.print(bucket.get(blobName).getContent());
// return the year of the file date creation
return tmp.toString().substring(0,4);
}
}

You can use the file_get_contents method to read the contents of the path. From the documentation on Reading and Writing Files
Read objects contents using PHP to fetch an object's custom metadata from Google Cloud Storage.An App Engine PHP 5 app must use the Cloud Storage stream wrapper to write files at runtime. However, if an app needs to read files, and these files are static, you can optionally read static files uploaded with your app using PHP filesystem functions such as file_get_contents.
$fileContents = file_get_contents($filePath);
where the path specified must be a path relative to the script accessing them.
You must upload the file or files in an application subdirectory when you deploy your app to App Engine, and must configure the app.yaml file so your app can access those files. For complete details, see PHP 5 Application Configuration with app.yaml.
In the app.yaml configuration, notice that if you use a static file or directory handler (static_files or static_dir) you must specify application_readable set to true or your app won't be able to read the files. However, if the files are served by a script handler, this isn't necessary, because these files are readable by script handlers by default.

Related

Changing working directory in Scala [duplicate]

How can I change the current working directory from within a Java program? Everything I've been able to find about the issue claims that you simply can't do it, but I can't believe that that's really the case.
I have a piece of code that opens a file using a hard-coded relative file path from the directory it's normally started in, and I just want to be able to use that code from within a different Java program without having to start it from within a particular directory. It seems like you should just be able to call System.setProperty( "user.dir", "/path/to/dir" ), but as far as I can figure out, calling that line just silently fails and does nothing.
I would understand if Java didn't allow you to do this, if it weren't for the fact that it allows you to get the current working directory, and even allows you to open files using relative file paths....
There is no reliable way to do this in pure Java. Setting the user.dir property via System.setProperty() or java -Duser.dir=... does seem to affect subsequent creations of Files, but not e.g. FileOutputStreams.
The File(String parent, String child) constructor can help if you build up your directory path separately from your file path, allowing easier swapping.
An alternative is to set up a script to run Java from a different directory, or use JNI native code as suggested below.
The relevant OpenJDK bug was closed in 2008 as "will not fix".
If you run your legacy program with ProcessBuilder, you will be able to specify its working directory.
There is a way to do this using the system property "user.dir". The key part to understand is that getAbsoluteFile() must be called (as shown below) or else relative paths will be resolved against the default "user.dir" value.
import java.io.*;
public class FileUtils
{
public static boolean setCurrentDirectory(String directory_name)
{
boolean result = false; // Boolean indicating whether directory was set
File directory; // Desired current working directory
directory = new File(directory_name).getAbsoluteFile();
if (directory.exists() || directory.mkdirs())
{
result = (System.setProperty("user.dir", directory.getAbsolutePath()) != null);
}
return result;
}
public static PrintWriter openOutputFile(String file_name)
{
PrintWriter output = null; // File to open for writing
try
{
output = new PrintWriter(new File(file_name).getAbsoluteFile());
}
catch (Exception exception) {}
return output;
}
public static void main(String[] args) throws Exception
{
FileUtils.openOutputFile("DefaultDirectoryFile.txt");
FileUtils.setCurrentDirectory("NewCurrentDirectory");
FileUtils.openOutputFile("CurrentDirectoryFile.txt");
}
}
It is possible to change the PWD, using JNA/JNI to make calls to libc. The JRuby guys have a handy java library for making POSIX calls called jnr-posix. Here's the maven info
As mentioned you can't change the CWD of the JVM but if you were to launch another process using Runtime.exec() you can use the overloaded method that lets you specify the working directory. This is not really for running your Java program in another directory but for many cases when one needs to launch another program like a Perl script for example, you can specify the working directory of that script while leaving the working dir of the JVM unchanged.
See Runtime.exec javadocs
Specifically,
public Process exec(String[] cmdarray,String[] envp, File dir) throws IOException
where dir is the working directory to run the subprocess in
If I understand correctly, a Java program starts with a copy of the current environment variables. Any changes via System.setProperty(String, String) are modifying the copy, not the original environment variables. Not that this provides a thorough reason as to why Sun chose this behavior, but perhaps it sheds a little light...
The working directory is a operating system feature (set when the process starts).
Why don't you just pass your own System property (-Dsomeprop=/my/path) and use that in your code as the parent of your File:
File f = new File ( System.getProperty("someprop"), myFilename)
The smarter/easier thing to do here is to just change your code so that instead of opening the file assuming that it exists in the current working directory (I assume you are doing something like new File("blah.txt"), just build the path to the file yourself.
Let the user pass in the base directory, read it from a config file, fall back to user.dir if the other properties can't be found, etc. But it's a whole lot easier to improve the logic in your program than it is to change how environment variables work.
I have tried to invoke
String oldDir = System.setProperty("user.dir", currdir.getAbsolutePath());
It seems to work. But
File myFile = new File("localpath.ext");
InputStream openit = new FileInputStream(myFile);
throws a FileNotFoundException though
myFile.getAbsolutePath()
shows the correct path.
I have read this. I think the problem is:
Java knows the current directory with the new setting.
But the file handling is done by the operation system. It does not know the new set current directory, unfortunately.
The solution may be:
File myFile = new File(System.getPropety("user.dir"), "localpath.ext");
It creates a file Object as absolute one with the current directory which is known by the JVM. But that code should be existing in a used class, it needs changing of reused codes.
~~~~JcHartmut
You can use
new File("relative/path").getAbsoluteFile()
after
System.setProperty("user.dir", "/some/directory")
System.setProperty("user.dir", "C:/OtherProject");
File file = new File("data/data.csv").getAbsoluteFile();
System.out.println(file.getPath());
Will print
C:\OtherProject\data\data.csv
You can change the process's actual working directory using JNI or JNA.
With JNI, you can use native functions to set the directory. The POSIX method is chdir(). On Windows, you can use SetCurrentDirectory().
With JNA, you can wrap the native functions in Java binders.
For Windows:
private static interface MyKernel32 extends Library {
public MyKernel32 INSTANCE = (MyKernel32) Native.loadLibrary("Kernel32", MyKernel32.class);
/** BOOL SetCurrentDirectory( LPCTSTR lpPathName ); */
int SetCurrentDirectoryW(char[] pathName);
}
For POSIX systems:
private interface MyCLibrary extends Library {
MyCLibrary INSTANCE = (MyCLibrary) Native.loadLibrary("c", MyCLibrary.class);
/** int chdir(const char *path); */
int chdir( String path );
}
The other possible answer to this question may depend on the reason you are opening the file. Is this a property file or a file that has some configuration related to your application?
If this is the case you may consider trying to load the file through the classpath loader, this way you can load any file Java has access to.
If you run your commands in a shell you can write something like "java -cp" and add any directories you want separated by ":" if java doesnt find something in one directory it will go try and find them in the other directories, that is what I do.
Use FileSystemView
private FileSystemView fileSystemView;
fileSystemView = FileSystemView.getFileSystemView();
currentDirectory = new File(".");
//listing currentDirectory
File[] filesAndDirs = fileSystemView.getFiles(currentDirectory, false);
fileList = new ArrayList<File>();
dirList = new ArrayList<File>();
for (File file : filesAndDirs) {
if (file.isDirectory())
dirList.add(file);
else
fileList.add(file);
}
Collections.sort(dirList);
if (!fileSystemView.isFileSystemRoot(currentDirectory))
dirList.add(0, new File(".."));
Collections.sort(fileList);
//change
currentDirectory = fileSystemView.getParentDirectory(currentDirectory);

HTML5 File API in Firefox Addon SDK

Is there a way to access Html5 file api in Fire Fox addon sdk in the content script?
This is needed in order to store user added words and their meanings. The data can grow large and so local storage isn't an option.
window.requestFileSystem3 = window.requestFileSystem || window.webkitRequestFileSystem;
gives me the error TypeError: window.requestFileSystem3 is not a function.
I am asking this because i am porting this code from a Google Chrome Extension which allows accessing the file api in a content script.
Additional Questions
1) If HTML5 File API is not allowed then should i use file module?
2) Does the file module allow access to any file on the file system as opposed to the Html5 file api which only access to a sandboxed access to file system?
3) Assuming i have to use file module what would be the best location to store my files ( like the user profile directory or extension directory ) and how would i get this path in code.
I apologize for so many sub questions inside this questions. Google wasn't very helpful regarding this topic.
Any sample code would be very helpful.
Firefox doesn't support writing files via File API yet and even when this will be added it will probably be accessible to web pages only and not extensions. In other words: yes, if you absolutely need to write to files then you should use low-level APIs. You want to store your data in the user profile directory (there is no extension directory, your extension is usually installed as a single packed file). Something like this should work to write a file:
var file = require("sdk/io/file");
var profilePath = require("sdk/system").pathFor("ProfD");
var filePath = file.join(profilePath, "foo.txt");
var writer = file.open(filePath, "w");
writer.writeAsync("foo!", function(error)
{
if (error)
console.log("Error: " + error);
else
console.log("Success!");
});
For reference: sdk/io/file, sdk/system
You could use TextReader.read() or file.read() to read the file. Unfortunately, Add-on SDK doesn't seem to support asynchronous file reading so the read will block the Firefox UI. The only alternative would be importing NetUtil and FileUtils via chrome authority, something like this:
var {components, Cu} = require("chrome");
var {NetUtil} = Cu.import("resource://gre/modules/NetUtil.jsm", null);
var {FileUtils} = Cu.import("resource://gre/modules/FileUtils.jsm", null);
NetUtil.asyncFetch(new FileUtils.File(filePath), function(stream, result)
{
if (components.isSuccessCode(result))
{
var data = NetUtil.readInputStreamToString(stream, stream.available());
console.log("Success: " + data);
}
else
console.log("Error: " + result);
});

How to serve uploaded files in Play!2 using Scala?

I'm trying to allow users to upload photos to the server and then view them. Uploading happens as described in this guide. Here is the code:
def upload = Action(parse.multipartFormData) { request =>
request.body.file("picture").map { picture =>
import java.io.File
val filename = picture.filename
val contentType = picture.contentType
picture.ref.moveTo(new File("/tmp/picture"))
Ok("File uploaded")
}.getOrElse {
Redirect(routes.Application.index).flashing(
"error" -> "Missing file"
)
}
}
It is unclear to me how to serve the uploaded images back to users that want to see them. Right now I am hosting the server on my own machine, so the code snippet from the guide writes the files to my D: drive, which isn't (and shouldn't be) available from the Internet. As far as I can see there are 2 options:
Store the photos under the /public folder in my project (the one that is dedicated to assets). See here: http://www.playframework.org/documentation/2.0/Assets
Write my own controller that servs images form custom locations from my drive.
For 1, I'm not sure if that is the purpose of assets.
For 2, I have no idea how to write such a controller.
The simple example is
def index = Action {
Ok.sendFile(new java.io.File("/tmp/fileToServe.pdf"))
}
there is "Serving files" section at https://www.playframework.com/documentation/2.4.x/ScalaStream#Serving-files which explains how to serve files
2.0.3 will feature an external Assets controller which might be (mis)used for this. Writing such a controller is no magic though, you have predefined folder where all your uploads are saved, and that's where you read them from. In the database you save the (unique) file name.
A different approach would be to save the uploaded files in the database. We do this with GridFS in MongoDB. A custom controller serves them back to the user. This way your data is stored in one central place, which also makes backups and recoveries simpler.
You can add a new route like this:
GET /myFiles/*file controllers.Assets.at(path="/tmp", file)

iPhone - Connection String and DB File

I am developing a iPhone app using Monotouch. I need to access a Sqlite DB. In my soultion, I have a contracts, data access, business access and UI project. I have two questions:
Where should I keep my DB file? Originally, I put it in the data access project. When I compile my business access project it copies the DB file to the output, but when I compile my UI project it does not (UI has a reference to business access which has a ref to data access). I moved it to the UI project, but it feels wrong to keep it there.
Where should I keep the connection string to the DB? Is there a concept of config files?
Here is what we do:
We ship a copy of the DB in the application. It is included as Content, Always Copy in the project.
On the user's machine, it is stored in the special directory Environment.SpecialFolder.Personal.
When the app is started, we check to see if the database exists on the user's system and, if not, copy it there.
The connection string is just "Data Source=" + sDatabasePath.
Here is a sample of the code that we use for this (I hacked in the connection stuff since we use a homebuilt class for managing the DB, but you should get the idea):
const string DATABASE_FILE_NAME = "MyDB.db3";
bool fSuccess = false;
DbConnection conn = new DbConnection ();
string sApplicationDir = Path.Combine (Environment.GetFolderPath (Environment.SpecialFolder.Personal), "MyApplicationSubDir");
if (!Directory.Exists (sApplicationDir)) {
Directory.CreateDirectory (sApplicationDir);
}
// Generate the directory to the database file
string sDatabaseDir = Path.Combine (sApplicationDir, "Database");
m_sDatabaseDir = sDatabaseDir;
if (!Directory.Exists (sDatabaseDir)) {
Directory.CreateDirectory (sDatabaseDir);
}
// Generate the path to the database file
string sDatabasePath = Path.Combine (sDatabaseDir, DATABASE_FILE_NAME);
m_sDatabaseFile = sDatabasePath;
// If the file does not not exist
if (!File.Exists (sDatabasePath)) {
// Copy the base implementation
File.Copy (Path.Combine (Path.Combine (Environment.CurrentDirectory, "Database"), DATABASE_FILE_NAME), sDatabasePath);
}
// Initialize the DB
conn.ConnectionString = "Data Source=" + sDatabasePath;
out of interest, have you looked at sqlite-net ? http://code.google.com/p/sqlite-net/
Makes your DB handling a lot easier.

Mirth: How to get source file directory from file reader channel

I have a file reader channel picking up an xml document. By default, a file reader channel populates the 'originalFilename' in the channel map, which ony gives me the name of the file, not the full path. Is there any way to get the full path, withouth having to hard code something?
You can get any of the Source reader properties like this:
var sourceFolder = Packages.com.mirth.connect.server.controllers.ChannelController.getInstance().getDeployedChannelById(channelId).getSourceConnector().getProperties().getProperty('host');
I put it up in the Mirth forums with a list of the other properties you can access
http://www.mirthcorp.com/community/forums/showthread.php?t=2210
You could put the directory in a channel deploy script:
globalChannelMap.put("pickupDirectory", "/Mirth/inbox");
then use that map in both your source connector:
${pickupDirectory}
and in another channel script:
function getFileLastModified(fileName) {
var directory = globalChannelMap.get("pickupDirectory").toString();
var fullPath = directory + "/" + fileName;
var file = Packages.java.io.File(fullPath);
var formatter = new Packages.java.text.SimpleDateFormat("yyyyMMddhhmmss");
formatter.setTimeZone(Packages.java.util.TimeZone.getTimeZone("UTC"));
return formatter.format(file.lastModified());
};
Unfortunately, there is no variable or method for retrieving the file's full path. Of course, you probably already know the path, since you would have had to provide it in the Directory field. I experimented with using the preprocessor to store the path in a channel variable, but the Directory field is unable to reference variables. Thus, you're stuck having to hard code the full path everywhere you need it.