I'm using Papa.unparse() to convert a JSON object to csv then downloading the file. The method fails with:
"allocation size overflow papaparse.min.js:6:1580"
This happens in firefox when there's > than 500,000 items to unparse in the JSON array.
The Papa.parse() method allows you to stream data from a file. Is there any similar approach you can take for Papa.unparse()?
You don't need PapaParse to do this.
The allocation size overflow problem is from converting your 500,000 rows into 500,000 strings and concatenating them together to form one massive string to create the CSV file with. JavaScript creates a new String when you concatenate one or more together, and eventually you run out of memory and crash.
The solution is to use TextEncoder to encode your strings into utf-8 (or whatever you need) ArrayBuffers, push each one into a giant array, and then use that array to create your file.
Here's some rough code for how you might do that:
var textEncoder = new TextEncoder("utf-8");
var headers = ["header1","header2","header3"];
var row1 = ["column1-1","column1-2","column1-3"];
var row2 = ["column2-1","column-2-2","column2-3"];
var data = [headers,row1,row2];
var arrayBuffers = [];
var csvString = '';
var encodedString = null;
var file = null;
for(var x=0;x<data.length;x++){
csvString = data[x].join(",").concat('\r\n');
encodedString = textEncoder.encode(csvString);
arrayBuffers.push(encodedString);
}
file = new File(arrayBuffers,"yourCsvFile.csv",{ type: "text/csv" });
I use text-encoding for a TextEncoder polyfill, and presumably if you're trying to Papa.unparse you've got FileAPI already.
i 'm .net developer. i want to Zip all files and make a one zip file with this technique.
ZipFile multipleFilesAsZipFile = new ZipFile();
Response.AddHeader("Content-Disposition", "attachment; filename=" + DateTime.Now.ToString("ddMMyyyy_HHmmss") + ".zip");
Response.ContentType = "application/zip";
for (int i = 0; i < filename.Length; i++)
{
string filePath = Server.MapPath("~/PostFiles/" + filename[i]);
multipleFilesAsZipFile.AddFile(filePath, string.Empty);
}
multipleFilesAsZipFile.Save(Response.OutputStream);
how ever for making this Zip i use third party library Ionic.
all files are ziped successfully but not extracted to client desktop. is there problem with my code. or this library that i'm using has been expired.
Is there free full version .net compatible library to zip all files.
Use SharpZipLib:
Nuget Package
Install-Package SharpZipLib
OR Download here
http://www.icsharpcode.net/OpenSource/SharpZipLib/
Snippet from examples:
private void CompressFolder(string path, ZipOutputStream zipStream, int folderOffset) {
string[] files = Directory.GetFiles(path);
foreach (string filename in files) {
FileInfo fi = new FileInfo(filename);
string entryName = filename.Substring(folderOffset); // Makes the name in zip based on the folder
entryName = ZipEntry.CleanName(entryName); // Removes drive from name and fixes slash direction
ZipEntry newEntry = new ZipEntry(entryName);
newEntry.DateTime = fi.LastWriteTime; // Note the zip format stores 2 second granularity
// Specifying the AESKeySize triggers AES encryption. Allowable values are 0 (off), 128 or 256.
// A password on the ZipOutputStream is required if using AES.
// newEntry.AESKeySize = 256;
// To permit the zip to be unpacked by built-in extractor in WinXP and Server2003, WinZip 8, Java, and other older code,
// you need to do one of the following: Specify UseZip64.Off, or set the Size.
// If the file may be bigger than 4GB, or you do not need WinXP built-in compatibility, you do not need either,
// but the zip will be in Zip64 format which not all utilities can understand.
// zipStream.UseZip64 = UseZip64.Off;
newEntry.Size = fi.Length;
zipStream.PutNextEntry(newEntry);
// Zip the file in buffered chunks
// the "using" will close the stream even if an exception occurs
byte[ ] buffer = new byte[4096];
using (FileStream streamReader = File.OpenRead(filename)) {
StreamUtils.Copy(streamReader, zipStream, buffer);
}
zipStream.CloseEntry();
}
string[ ] folders = Directory.GetDirectories(path);
foreach (string folder in folders) {
CompressFolder(folder, zipStream, folderOffset);
}
}
Taken from : https://github.com/icsharpcode/SharpZipLib/wiki/Zip-Samples
Works awesome!
I have an app that is storing images in a Windows Azure Block Blob. I'm adding meta data to each blob that gets uploaded. The metadata may include some special characters. For instance, the registered trademark symbol (®). How do I add this value to meta data in Windows Azure?
Currently, when I try, I get a 400 (Bad Request) error anytime I try to upload a file that uses a special character like this.
Thank you!
You might use HttpUtility to encode/decode the string:
blob.Metadata["Description"] = HttpUtility.HtmlEncode(model.Description);
Description = HttpUtility.HtmlDecode(blob.Metadata["Description"]);
http://lvbernal.blogspot.com/2013/02/metadatos-de-azure-vs-caracteres.html
The supported characters in the blob metadata must be ASCII characters. To work around this you can either escape the string ( percent encode), base64 encode etc.
joe
HttpUtility.HtmlEncode may not work; if Unicode characters are in your string (i.e. ’), it will fail. So far, I have found Uri.EscapeDataString does handle this edge case and others. However, there are a number of characters that get encoded unnecessarily, such as space (' '=chr(32)=%20).
I mapped the illegal ascii characters metadata will not accept and built this to restore the characters:
static List<string> illegals = new List<string> { "%1", "%2", "%3", "%4", "%5", "%6", "%7", "%8", "%A", "%B", "%C", "%D", "%E", "%F", "%10", "%11", "%12", "%13", "%14", "%15", "%16", "%17", "%18", "%19", "%1A", "%1B", "%1C", "%1D", "%1E", "%1F", "%7F", "%80", "%81", "%82", "%83", "%84", "%85", "%86", "%87", "%88", "%89", "%8A", "%8B", "%8C", "%8D", "%8E", "%8F", "%90", "%91", "%92", "%93", "%94", "%95", "%96", "%97", "%98", "%99", "%9A", "%9B", "%9C", "%9D", "%9E", "%9F", "%A0", "%A1", "%A2", "%A3", "%A4", "%A5", "%A6", "%A7", "%A8", "%A9", "%AA", "%AB", "%AC", "%AD", "%AE", "%AF", "%B0", "%B1", "%B2", "%B3", "%B4", "%B5", "%B6", "%B7", "%B8", "%B9", "%BA", "%BB", "%BC", "%BD", "%BE", "%BF", "%C0", "%C1", "%C2", "%C3", "%C4", "%C5", "%C6", "%C7", "%C8", "%C9", "%CA", "%CB", "%CC", "%CD", "%CE", "%CF", "%D0", "%D1", "%D2", "%D3", "%D4", "%D5", "%D6", "%D7", "%D8", "%D9", "%DA", "%DB", "%DC", "%DD", "%DE", "%DF", "%E0", "%E1", "%E2", "%E3", "%E4", "%E5", "%E6", "%E7", "%E8", "%E9", "%EA", "%EB", "%EC", "%ED", "%EE", "%EF", "%F0", "%F1", "%F2", "%F3", "%F4", "%F5", "%F6", "%F7", "%F8", "%F9", "%FA", "%FB", "%FC", "%FD", "%FE" };
private static string MetaDataEscape(string value)
{
//CDC%20Guideline%20for%20Prescribing%20Opioids%20Module%206%3A%20%0Ahttps%3A%2F%2Fwww.cdc.gov%2Fdrugoverdose%2Ftraining%2Fdosing%2F
var x = HttpUtility.HtmlEncode(value);
var sz = value.Trim();
sz = Uri.EscapeDataString(sz);
for (int i = 1; i < 255; i++)
{
var hex = "%" + i.ToString("X");
if (!illegals.Contains(hex))
{
sz = sz.Replace(hex, Uri.UnescapeDataString(hex));
}
}
return sz;
}
The result is:
Before ==> "1080x1080 Facebook Images"
Uri.EscapeDataString =>
"1080x1080%20Facebook%20Images"
After => "1080x1080 Facebook
Images"
I am sure there is a more efficient way, but the hit seems negligible for my needs.
I have built a Server that you can upload files to and download, using Eclipse, servlet and jsp, it's all very new to me. (more info).
Currently the upload system works with the file's name. I want to programmatically assign each file a random key. And with that key the user can download the file. That means saving the data in a config file or something like : test.txt(file) fdjrke432(filekey). And when the user inputs the filekey the servlet will pass the file for download.
I have tried using a random string generator and renameTo(), for this. But it doesn't work the first time, only when I upload the same file again does it work. And this system is flawed, the user will receive the file "fdjrke432" instead of test.txt, their content is the same but you can see the problem.
Any thoughts, suggestions or solutions for my problem?
Well Sebek, I'm glad you asked!! This is quite an interesting one, there is no MAGIC way to do this. The answer is indeed to rename the file you uploaded. But I suggest adding the random string before the name of the file; like : fdjrke432test.txt.
Try this:
filekey= RenameRandom();
File renamedUploadFile = new File(uploadFolder + File.separator+ filekey+ fileName);
item.write(renamedUploadFile);
//remember to give the user the filekey
with
public String RenameRandom()
{
final int LENGTH = 8;
StringBuffer sb = new StringBuffer();
for (int x = 0; x < LENGTH; x++)
{
sb.append((char)((int)(Math.random()*26)+97));
}
System.out.println(sb.toString());
return sb.toString();
}
To delete or download the file from the server you will need to locate it, the user will input the key, you just need to search the upload folder for a file that begins with that key:
filekey= request.getParameter("filekey");
File f = new File(getServletContext().getRealPath("") + File.separator+"data");
File[] matchingFiles = f.listFiles(new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.startsWith(filekey);
}
});
String newfilename = matchingFiles[0].getName();
// now delete or download newfilename