User will upload EDUPUB/Zip file from UI. We want to implement a REST api extension module to take EDUPUB/Zip file and ingest into a MarkLogic database. Does MarkLogic API api support this? Any suggestions?
I implemented the code below for extracting and uploading a EDUPUB/Zip file
xquery version "1.0-ml";
declare namespace zip="xdmp:zip";
declare function local:epubupload ($filepath as xs:string)
{
let $get_document :=xdmp:document-get($filepath)
let $get_uri := fn:document-uri($get_document)
let $get_document_uri := fn:concat($get_uri, "/")
let $get_collection := fn:tokenize($get_uri, "\\")[last()]
let $epub_extract := xdmp:zip-manifest($get_document)
for $each_file in $epub_extract/zip:part/text()
let $document_data := xdmp:zip-get($get_document, $each_file)
let $full_document_uri := fn:concat($get_document_uri, $each_file)
return xdmp:document-insert($full_document_uri, $document_data, (), $get_collection)
};
local:epubupload("c:\data\sample.epub")
But for the REST api what is the parameter? And how to get whole file from user system?
If you are creating your own REST extension, then you can use the following pattern on the zip payload:
1 Iterate over the zip file using xdmp:zip-manifest
2 For each entry, use xdmp:zip-get to extract the file
3 Save it into MarkLogic via xdmp:document-insert
Depending on how you posted the content, xdmp:base64-decode may be part of your code to actually get to your zip file.
Related
What is the best way to download all files in a remote directory using C# and FTP and save them to a local directory?
Thanks.
downloading all files in a specific folder seems to be an easy task. However, there are some issues which has to be solved. To name a few:
How to get list of files (System.Net.FtpWebRequest gives you unparsed list and directory list format is not standardized in any RFC)
What if remote directory has both files and subdirectories. Do we have to dive into the subdirs and download it's content?
What if some of the remote files already exist on the local computer? Should they be overwritten? Skipped? Should we overwrite older files only?
What if the local file is not writable? Should the whole transfer fail? Should we skip the file and continue to the next?
How to handle files on a remote disk which are unreadable because we don’t have sufficient access rights?
How are the symlinks, hard links and junction points handled? Links can easily be used to create an infinite recursive directory tree structure. Consider folder A with subfolder B which in fact is not the real folder but the *nix hard link pointing back to folder A. The naive approach will end in an application which never ends (at least if nobody manage to pull the plug).
Decent third party FTP component should have a method for handling those issues. Following code uses our Rebex FTP for .NET.
using (Ftp client = new Ftp())
{
// connect and login to the FTP site
client.Connect("mirror.aarnet.edu.au");
client.Login("anonymous", "my#password");
// download all files
client.GetFiles(
"/pub/fedora/linux/development/i386/os/EFI/*",
"c:\\temp\\download",
FtpBatchTransferOptions.Recursive,
FtpActionOnExistingFiles.OverwriteAll
);
client.Disconnect();
}
The code is taken from my blogpost available at blog.rebex.net. The blogpost also references a sample which shows how ask the user how to handle each problem (e.g. Overwrite/Overwrite older/Skip/Skip all).
Using C# FtpWebRequest and FtpWebReponse, you can use the following recursion (make sure the folder strings terminate in '\'):
public void GetAllDirectoriesAndFiles(string getFolder, string putFolder)
{
List<string> dirIitems = DirectoryListing(getFolder);
foreach (var item in dirIitems)
{
if ( item.Contains('.') )
{
GetFile(getFolder + item, putFolder + item);
}
else
{
var subDirPut = new DirectoryInfo(putFolder + "\\" + item);
subDirPut.Create();
GetAllDirectoriesAndFiles(getFolder + item + "\\", subDirPut.FullName + "\\");
}
}
}
The "item.Contains('.')" is a bit primitive, but has worked for my purposes. Post a comment if you need an example of the methods:
GetFile(string getFileAndPath, string putFileAndPath)
or
DirectoryListing(getFolder)
For FTP protocol you can use FtpWebRequest class from .NET framework. Though it does not have any explicit support for recursive file operations (including downloads). You have to implement the recursion yourself:
List the remote directory
Iterate the entries, downloading files and recursing into subdirectories (listing them again, etc.)
Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the FtpWebRequest. The FtpWebRequest unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.
Your options are:
Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name". If that succeeds, it's a file, if that fails, it's a directory. But that can become a performance problem, when you have a large number of entries.
You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)
void DownloadFtpDirectory(
string url, NetworkCredential credentials, string localPath)
{
FtpWebRequest listRequest = (FtpWebRequest)WebRequest.Create(url);
listRequest.UsePassive = true;
listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
listRequest.Credentials = credentials;
List<string> lines = new List<string>();
using (WebResponse listResponse = listRequest.GetResponse())
using (Stream listStream = listResponse.GetResponseStream())
using (StreamReader listReader = new StreamReader(listStream))
{
while (!listReader.EndOfStream)
{
lines.Add(listReader.ReadLine());
}
}
foreach (string line in lines)
{
string[] tokens =
line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
string name = tokens[8];
string permissions = tokens[0];
string localFilePath = Path.Combine(localPath, name);
string fileUrl = url + name;
if (permissions[0] == 'd')
{
Directory.CreateDirectory(localFilePath);
DownloadFtpDirectory(fileUrl + "/", credentials, localFilePath);
}
else
{
var downloadRequest = (FtpWebRequest)WebRequest.Create(fileUrl);
downloadRequest.UsePassive = true;
downloadRequest.UseBinary = true;
downloadRequest.Method = WebRequestMethods.Ftp.DownloadFile;
downloadRequest.Credentials = credentials;
var response = downloadRequest.GetResponse();
using (Stream ftpStream = response.GetResponseStream())
using (Stream fileStream = File.Create(localFilePath))
{
ftpStream.CopyTo(fileStream);
}
}
}
}
The url must be like:
ftp://example.com/ or
ftp://example.com/path/
Or use 3rd party library that supports recursive downloads.
For example with WinSCP .NET assembly you can download whole directory with a single call to Session.GetFiles:
// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Ftp,
HostName = "example.com",
UserName = "user",
Password = "mypassword",
};
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
// Download files
session.GetFiles("/home/user/*", #"d:\download\").Check();
}
Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.
(I'm the author of WinSCP)
You could use System.Net.WebClient.DownloadFile(), which supports FTP. MSDN Details here
You can use FTPClient from laedit.net. It's under Apache license and easy to use.
It use FtpWebRequest :
first you need to use WebRequestMethods.Ftp.ListDirectoryDetails to get the detail of all the list of the folder
for each files you need to use WebRequestMethods.Ftp.DownloadFile to download it to a local folder
I want to get the time creation of files in GCS, I used the code below :
println(Files
.getFileAttributeView(Paths.get("gs://datalake-dev/mu/tpu/file.0450138"), classOf[BasicFileAttributeView])
.readAttributes.creationTime)
The problem is that the Paths.get function replace // with / so I will get gs:/datalake-dev/mu/tpu/file.0450138 instead of gs://datalake-dev/mu/tpu/file.0450138.
Anyone can help me with this ?
Thanks a lot !
I solved the problem by adding the following java code and then calling the java function in scala.
import com.google.cloud.storage.*;
import java.sql.Timestamp;
public class ExtractDate {
public static String getTime(String fileName){
String bucketName = "bucket-data";
String blobName = "doc/files/"+fileName;
// Instantiates a client
Storage storage_client = StorageOptions.getDefaultInstance().getService();
Bucket bucket = storage_client.get(bucketName);
//val storage_client = Storage.
BlobId blobId = BlobId.of(bucketName, blobName);
Blob blob = storage_client.get(blobId);
Timestamp tmp = new Timestamp(bucket.get(blobName).getCreateTime());
System.out.print(bucket.get(blobName).getContent());
// return the year of the file date creation
return tmp.toString().substring(0,4);
}
}
You can use the file_get_contents method to read the contents of the path. From the documentation on Reading and Writing Files
Read objects contents using PHP to fetch an object's custom metadata from Google Cloud Storage.An App Engine PHP 5 app must use the Cloud Storage stream wrapper to write files at runtime. However, if an app needs to read files, and these files are static, you can optionally read static files uploaded with your app using PHP filesystem functions such as file_get_contents.
$fileContents = file_get_contents($filePath);
where the path specified must be a path relative to the script accessing them.
You must upload the file or files in an application subdirectory when you deploy your app to App Engine, and must configure the app.yaml file so your app can access those files. For complete details, see PHP 5 Application Configuration with app.yaml.
In the app.yaml configuration, notice that if you use a static file or directory handler (static_files or static_dir) you must specify application_readable set to true or your app won't be able to read the files. However, if the files are served by a script handler, this isn't necessary, because these files are readable by script handlers by default.
I tried to change client page size in Azure server
it's default is 50 and I want to make it bigger
so i use Microsoft tutorial in this link
https://learn.microsoft.com/en-us/azure/app-service-mobile/app-service-mobile-ios-how-to-use-client-library#querying
var client : MSClient?
let client = MSClient(applicationURLString: "AppUrl")
let table = client.tableWithName("TodoItem")
let query = table.query()
let pullSettings = MSPullSettings(pageSize: 3000)
but when I write
table.pullWithQuery(query, queryId:nil, settings: pullSettings) { (error) in
if let err = error {
print("ERROR ", err)
}
}
there are error "Value of type 'MSTable?' has no member 'pullWithQuery'"
what is the problem ?
is the function name changed ?
Two problems:
The documentation has not been updated for current versions of Swift
(an update request has been filed). The correct function name in modern Swift is pull rather than pullWithQuery.
The pullWithQuery function is on MSSyncTable, not MSTable. Pull is part of the offline sync system. The MSTable analog is read.
More details:
The SDK itself defines the function as MSSyncTable.pullWithQuery, but one of the features of Swift 3.0 is that it renames Objective C methods when it projects them into Swift to remove redundant arguments from the name, so verbWithX(X) becomes just verb(with:x) and pullWithQuery (MSQuery) becomes pull(with:MSQuery).
For more information on Swift 3 changes please see https://swift.org/blog/swift-3-0-released/ . I believe this particular change is SE-0005: Better Translation of Objective-C APIs Into Swift
If you download the Swift quickstart from your Azure Portal then you’ll get the correct modern pattern there:
self.table!.pull(with: self.table?.query(), queryId: "AllRecords")
or with your arguments:
self.table!.pull(with: self.table?.query(), queryId: nil, settings: pullSettings)
I am trying to generate external modules rather than a type definition file. I believe I need to do the following:
Change the extension of the file to .ts instead of .d.ts.
Generate one file per module.
Add the key word "Export" in front of each interface and enum.
I was easily able to change the extension of the file by changing the "output extension" setting in the tt file.
I cannot figure out how to split the modules into separate files.
I cannot figure out how to add the Export key word to each interface.
TypeLITE doesn't support generating multiple files. This feature has been requested by several users, but I am not aware of a simple way to generate multiple files from the single tt file.
export keyword can't be added without changing source code of the library (TsGenerator.cs). This is very specific requirement, so I probably won't add it to the library.
TypeLite is a good project but lacking in Documentation and examples, it's open source so anyone can contribute and make it better.
As for creating a file per class i solved it using the code below.
private static void GenerateTypeScriptContracts(string assemblyFile, string outputPath)
{
// Clean TS Folder
System.IO.DirectoryInfo di = new DirectoryInfo(outputPath);
foreach (FileInfo file in di.GetFiles())
{
file.Delete();
}
// --
var assembly = Assembly.LoadFrom(assemblyFile);
// If you want a subset of classes from this assembly, filter them here
var models = assembly.GetTypes();
foreach (var model in models)
{
var generator = new TypeScriptFluent()
.WithConvertor<Guid>(c => "string")
.WithMemberFormatter((identifier) => Char.ToLower(identifier.Name[0]) + identifier.Name.Substring(1));
generator.ModelBuilder.Add(model);
// Generate TS interface definitions
var tsClassDefinitions = generator.Generate(TsGeneratorOutput.Properties | TsGeneratorOutput.Fields);
File.WriteAllText(Path.Combine(outputPath, "I" + model.FullName.Replace("ProjectName.DtoModels.", "") + ".ts"), tsClassDefinitions);
}
}
Is there a way to access Html5 file api in Fire Fox addon sdk in the content script?
This is needed in order to store user added words and their meanings. The data can grow large and so local storage isn't an option.
window.requestFileSystem3 = window.requestFileSystem || window.webkitRequestFileSystem;
gives me the error TypeError: window.requestFileSystem3 is not a function.
I am asking this because i am porting this code from a Google Chrome Extension which allows accessing the file api in a content script.
Additional Questions
1) If HTML5 File API is not allowed then should i use file module?
2) Does the file module allow access to any file on the file system as opposed to the Html5 file api which only access to a sandboxed access to file system?
3) Assuming i have to use file module what would be the best location to store my files ( like the user profile directory or extension directory ) and how would i get this path in code.
I apologize for so many sub questions inside this questions. Google wasn't very helpful regarding this topic.
Any sample code would be very helpful.
Firefox doesn't support writing files via File API yet and even when this will be added it will probably be accessible to web pages only and not extensions. In other words: yes, if you absolutely need to write to files then you should use low-level APIs. You want to store your data in the user profile directory (there is no extension directory, your extension is usually installed as a single packed file). Something like this should work to write a file:
var file = require("sdk/io/file");
var profilePath = require("sdk/system").pathFor("ProfD");
var filePath = file.join(profilePath, "foo.txt");
var writer = file.open(filePath, "w");
writer.writeAsync("foo!", function(error)
{
if (error)
console.log("Error: " + error);
else
console.log("Success!");
});
For reference: sdk/io/file, sdk/system
You could use TextReader.read() or file.read() to read the file. Unfortunately, Add-on SDK doesn't seem to support asynchronous file reading so the read will block the Firefox UI. The only alternative would be importing NetUtil and FileUtils via chrome authority, something like this:
var {components, Cu} = require("chrome");
var {NetUtil} = Cu.import("resource://gre/modules/NetUtil.jsm", null);
var {FileUtils} = Cu.import("resource://gre/modules/FileUtils.jsm", null);
NetUtil.asyncFetch(new FileUtils.File(filePath), function(stream, result)
{
if (components.isSuccessCode(result))
{
var data = NetUtil.readInputStreamToString(stream, stream.available());
console.log("Success: " + data);
}
else
console.log("Error: " + result);
});