I have the following seetings in my web.config file.
<httpRuntime executionTimeout="90000" maxRequestLength="204800" />
But I am not able to upload anyfile which are greater than 50MB. Wht could be the reason. The web browser keeps on waiting for the upload to finish but anyfile lesser than 50MB gets uploaded without any issues. Are there any other pleases I need to check?
Place this in your web.config
<system.web>
<httpRuntime executionTimeout="360" maxRequestLength="100000" />
That enables a 360 second timeout and 100,000 Kb of upload data at a time.
If that doesn't work, run this command on your IIS server. (replace [IISWebsitename])
C:\Windows\System32\inetsrv>appcmd set config "[IISWebsitename]" -section:requestFiltering -requestLimits.maxAllowedContentLength:100000000 -commitpath:apphost
That enables 100,000,000 bytes of upload data at a time.
Perhaps 50MB is the maximum allowed filesize upload on the webserver? If this is the case usually an error message is returned.
Can you find out what the allowed maximum uploaded filesize on the webserver is?
P.S.: Information about the system would also be nice.
(Windows/Linux/Mac, IIS/Apache ...)
Related
I have FilePulse correctly configured, so that when I create a file inside the reading folder, it reads it and ingests it in the topic.
Now I need to do continuous reading of each of the files in that folder, since they are continually being updated.
I have to change any property of properties file?
My filePulseTxtFile.properties:
name=connect-file-pulse-txt
connector.class=io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector
topic=lineas-fichero
tasks.max=1
File types
fs.scan.filters=io.streamthoughts.kafka.connect.filepulse.scanner.local.filter.RegexFileListFilter
file.filter.regex.pattern=.*\\.log$
task.reader.class=io.streamthoughts.kafka.connect.filepulse.reader.RowFileInputReader
File scanning
fs.cleanup.policy.class=io.streamthoughts.kafka.connect.filepulse.clean.LogCleanupPolicy
fs.scanner.class=io.streamthoughts.kafka.connect.filepulse.scanner.local.LocalFSDirectoryWalker
fs.scan.directory.path=/home/ec2-user/parser/scanDirKafka
fs.scan.interval.ms=10000
Internal Reporting
internal.kafka.reporter.bootstrap.servers=localhost:9092
internal.kafka.reporter.id=connect-file-pulse-txt
internal.kafka.reporter.topic=connect-file-pulse-status
Track file by name
offset.strategy=name
Thanks a lot!
Continious reading is only supported by the RowFileInputReader that you can configure with the read.max.wait.ms property - The maximum time to wait in milliseconds for more bytes after hitting end of file.
For example, if you configure that property to 10000 then the reader will wait 10 seconds for new lines to be added to the file before considering it completed.
Also, you should note that as long as there are task processing files, then new files that are added to the source directory will not be selected. But, you can configure the allow.tasks.reconfiguration.after.timeout.ms to force all tasks to be restarted after a given period so that new files will be scheduled.
Finally, you must take care to correctly set the max.tasks property so that all files can be processed in parallel (a task can only process one file at a time).
I'm uploading files, in chunks, to SharePoint Online through the REST API.
I am doing this via the StartUpload, ContinueUpload and FinishUpload methods and passing them a chunk as a byte[].
I have used this as the example to work from: Using chunked upload/StartUpload with sharepoint REST api
Documentation is here: https://msdn.microsoft.com/library/office/microsoft.sharepoint.client.file.startupload.aspx
(I have not included the code as I don't think it's relevant, please correct me if I'm mistaken)
This works as long as the total file size is larger than the chunk size.
For example, if the chunk size is 1MB, but the total file size is 4MB, then the method will work.
If the chunk size is 4MB, but the total file size is 1MB, then I end up with empty or corrupt files once uploaded.
This is because the initial call to StartUpload contains the entire file in one chunk, so FinishUpload never gets called to close the file.
If I call FinishUpload with an empty byte[0] then I get:
Error 500 - Internal Server Error: The upload was incomplete. Try to save again.
Under normal circumstances I would simply check if the total file size was smaller than the chunk size and instead add the file directly using /Files/add(...).
Unfortunately I am being passed the file chunks in a stream and I don't know the total file size beforehand. I also can't save all chunks before processing, I have to pass each chunk straight to SharePoint as I am given them.
So how do I use FinishUpload to close a file upload when I have no more bytes to upload?
For that matter you could consider the following modifications:
1) in case if total file size is less then chunk size, then the file
is getting uploaded via a single request. For example:
var fi = new FileInfo(fileName);
if(fi.Length <= chunkSize)
{
this.UploadFile(address, fileName);
return;
}
where WebClient.UploadFile Method is used for uploading a file via SharePoint RPC
2) otherwise file is getting uploaded via chunk session
Refer SharePointClient.cs for a complete example.
The pagesize of the file I read is 32768. When i set the JET_paramDatabasePageSize to 32768,JetInit returns -1213.Then,i set the JET_paramRecovery to "Off",JetInit succeeds.But,when I use JetAttachDatabase,it returns -550.
Here is my code:
err=JetSetSystemParameter(&instance,sesid,JET_paramDatabasePageSize ,32768 ,NULL);
err=JetCreateInstance(&instance,NULL);
err=JetSetSystemParameter(&instance,sesid,JET_paramRecovery,0,"Off");
err=JetInit(&instance);
err=JetBeginSession(instance,&sesid,NULL,NULL);
err=JetAttachDatabase(sesid,buffer, JET_bitDbReadOnly );
err=JetOpenDatabase ( sesid, buffer, NULL, &dbid, JET_bitDbReadOnly );
What's wrong with it？I am running a Windows 7 32bit.
The page size is global to the process (NOT just the instance) and is persisted in the log files and the database, so changing the page size can be annoyingly tricky.
Is there information in the database that you're trying to access? Or did you just experience this during development?
If you saw this during development, then the easiest thing to do is to blow everything away (del .edb edb) [Assuming that you kept the prefix as "edb"].
Also, are you sure that the database is 32k pages? You can confirm with esentutl.exe -mh <database-name>.
It will be trickier to recover the data if you do care about, and you switched the page size. (I don't know off the top of my head, and I'd have to try a few things out...)
-martin
We send bitmaps from our Silverlight client to the server to be saved, using a WCF-RIA defined service operation. Our DomainService class looks a little like this:
[EnableClientAccess()]
public class CBitmapSavingService : DomainService
{
public void SaveBitmap(string bitmapGuid, byte[] pngBytes)
{
// Save PNG on server file system
}
}
Works fine, until we get a large bitmap. Then we get a 'DomainOperationException' exception.
I suspect that we are surpassing a size limit for either the parameter or the message.
Can I reconfigure my service such that larger bitmaps can be sent from the client with WCF-RIA-Services?
I made the following change to my web.config file:
<httpRuntime requestValidationMode="2.0" maxRequestLength="6225920"/>
and it worked.
(why 6225920? Size of 2048*760 bitmap before compression, I gotta choose something)
I found the answer on another site: http://forums.silverlight.net/forums/p/186772/440463.aspx
This is only intended as a short term fix for us, because we don't really want such large bitmaps on the server. I plan to make a client side change, such that the picture type (PNG or JPEG) and quality will be changed to create an image within a defined maximum size.
I want to get file size I'm doing this:
my $filename=$query->param("upload_file");
my $filesize = (-s $filename);
print "Size: $filesize ";`
Yet it is not working. Note that I did not upload the file. I want to check its size before uploading it. So to limit it to max of 1 MB.
You can't know the size of something before uploading. But you can check the Content-Length request header sent by the browser, if there is one. Then, you can decide whether or not you want to believe it. Note that the Content-Length will be the length of the entire request stream, including other form fields, and not just the file upload itself. But it's sufficient to get you a ballpark figure for conformant clients.
Since you seem to be running under plain CGI, you should be able to get the request body length in $ENV{CONTENT_LENGTH}.
Also want to sanity check against possibly already having post max set (from perldoc CGI):
$CGI::POST_MAX
If set to a non-negative integer, this variable puts a ceiling on the size of
POSTings, in bytes. If CGI.pm detects a POST that is greater than the ceiling,
it will immediately exit with an error message. This value will affect both
ordinary POSTs and multipart POSTs, meaning that it limits the maximum size of
file uploads as well. You should set this to a reasonably high value, such as
1 megabyte.
The uploaded file is stashed in a tmp location on the server when the form is submitted, check the file size there.
Supply the value for $field.
my $upload_filehandle = $query->upload($field);
my $tmpfilename = $query->tmpFileName($upload_filehandle);
my $file_size = (-s $tmpfilename);
This has nothing to do with Perl.
You are trying to read the filesize of a file on the user's computer using commands that read files on your server, what you want can't be done using Perl.
This is something that has to be done in the browser, and looking briefly at these questions it's either very hard or impossible.
Your best bet is to allow the user to start the upload and abort if the file is too big.
If you want to check before you process the request, you might be better off checking on the web page that triggers the request. I don't think the web browser can do it on it's own, but if you don't mind Flash, there are many Flash upload tools that can check things like size (as well as file types) and prevent uploading.
A good one to start with is the YUI Uploader. Lots more here: What is the best multiple file JavaScript / Flash file uploader?
Obviously you would want to check on the server side too, but by the time the user has started sending the request to the server, you are already using up your CPU cycles and bandwidth.
Thanks everyone for your replies; I just found out why $filesize = (-s $filename); was not working before, it is due that I was checking file size while sending Ajax request and not while re submitting the page.That's why I was having size to be zero. I fixed that to submit the page and it worked. Thanks.
Just read this post but while checking the content-length is a good approximate pre-check you could also save the file to temporary folder and then perform any kind of check on it. If it doesn't meet your criteria just delete and don't send it to it's final destination.
Look at the perl documentation for file stats -X - perldoc.perl.org and stat-perldoc.perl.org. Also, you can look at this upload script which is doing the similar thing what you are trying to do.