companion: 2022-09-22T23:31:07.088Z [error] a62da431-f9ce-4fae-b18d-dc59189a53ea root.error PayloadTooLargeError: request entity too large
at readStream (/usr/local/share/.config/yarn/global/node_modules/raw-body/index.js:155:17)
at getRawBody (/usr/local/share/.config/yarn/global/node_modules/raw-body/index.js:108:12)
at read (/usr/local/share/.config/yarn/global/node_modules/body-parser/lib/read.js:77:3)
at jsonParser (/usr/local/share/.config/yarn/global/node_modules/body-parser/lib/types/json.js:135:5)
at Layer.handle [as handle_request] (/usr/local/share/.config/yarn/global/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/local/share/.config/yarn/global/node_modules/express/lib/router/index.js:317:13)
at /usr/local/share/.config/yarn/global/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/usr/local/share/.config/yarn/global/node_modules/express/lib/router/index.js:335:12)
at next (/usr/local/share/.config/yarn/global/node_modules/express/lib/router/index.js:275:10)
at middleware (/usr/local/share/.config/yarn/global/node_modules/express-prom-bundle/src/index.js:174:5)
::ffff:172.29.0.1 - - [22/Sep/2022:23:31:07 +0000] "POST /s3/multipart/FqHx7wOxKS8ASbAWYK7ZtEfpWFOT2h9KIX2uHTPm2EZ.k1INl8vxfdpH7KBXhLTii1WL7GeDLzLcAKOW0vmxKhfCrcUCRMgHGdxEd5Nwxr._omBrtqOQFuY.Fl9nX.Vy/complete?key=videos%2Fuploads%2Ff86367432cef879b-4d84eb44-thewoods_weddingfilm_1.mp4 HTTP/1.1" 413 211 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
It will always give out a 413 on the last request with about 188k of payload describing all the parts.
I've tried anything including:
var bodyParser = require("body-parser");
app.use(bodyParser.json({ limit: "50mb" }));
app.use(bodyParser.urlencoded({limit: "50mb", extended: false}));
but it has no effect. I've been spending months on this problem, read every article, complaint, issue on the internet regarding this and still, no one knows why it's happening and no one knows how to resolve it.
Can anyone help?
This is a known issue with the S3 plugin. It is fixed in the latest version of Uppy, but Companion is still on an older version. You can use the S3 Multipart plugin directly, which is what Companion uses under the hood.
const Uppy = require('#uppy/core')
const AwsS3Multipart = require('#uppy/aws-s3-multipart')
const uppy = new Uppy()
uppy.use(AwsS3Multipart, {
companionUrl: 'https://companion.uppy.io',
companionCookiesRule: 'same-origin',
limit: 5,
getUploadParameters (file) {
return {
method: 'post',
endpoint: 'https://companion.uppy.io/s3/multipart',
fields: {
filename: file.name,
size: file.size,
contentType: file.type
}
}
}
})
uppy.on('upload-success', (file, data) => {
console.log('Upload successful', file, data)
})
uppy.on('upload-error', (file, error) => {
console.log('Upload error', file, error)
})
uppy.addFile({
name: 'test.txt',
type: 'text/plain',
data: new Blob(['hello world'], { type: 'text/plain' })
})
uppy.upload()
The S3 Multipart plugin is a bit more complicated to use than the S3 plugin, but it is more flexible. It allows you to upload files larger than 5GB, and it allows you to upload files in parallel. It also allows you to upload files to S3-compatible services like Minio. The S3 plugin is a bit simpler to use, but it has some limitations. It doesn’t allow you to upload files larger than 5GB, and it doesn’t allow you to upload files in parallel. It also doesn’t allow you to upload files to S3-compatible services like Minio.
I wanted to extract the real tiktok video link but my code seems to not be working. I want to get the
https://www.tiktok.com/#lilymaycreative/video/6911015584570395906?sender_device=pc&sender_web_id=6894321561748211206&is_from_webapp=v
from the shorten link which is
https://vm.tiktok.com/ZSTjLwCK/
var dio =
Dio(BaseOptions(connectTimeout: 10000, receiveTimeout: 10000, headers: {
'User-Agent':
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
}));
try {
Response response = await dio.get('https://vm.tiktok.com/ZSTjLwCK/');
_document = parse(response.data);
if (_document != null) print(_document);
print(jsonData);
} catch (e) {
print(e);
}
I can tell you a way that can help solve your problem.
If you press F12(developer tools) in your example site.
You detect video tag in html like below and you may access the video link.
I have a script the parse a specific webpage.
When I set headless false, the puppeteer doesn't load page
await page.goto('https://www.google.com', {
waitUntil: 'load',
// Remove the timeout
timeout: 0
});
I tried with a lot of configurations, like:
const args = [
'--no-sandbox',
'--enable-logging',
' --v=1',
'--disable-gpu',
'--disable-extension',
'--disable-setuid-sandbox',
'--disable-infobars',
'--window-position=0,0',
'--ignore-certifcate-errors',
'--ignore-certifcate-errors-spki-list',
'--user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3312.0 Safari/537.36"'
];
const options = {
args,
headless: false, // default is true
userDataDir: "./user_data",
defaultViewport: null,
devtools: true,
ignoreHTTPSErrors: true,
};
But the script stops to await page.goto until timeout.
As I understand you're parsing not the google.com page.
The first thing that should be considered it's the waitUntil: 'load'. What it done, it considers navigation to be finished when the load event is fired.
The load event is fired when the whole webpage (HTML) has loaded fully, including all dependent resources such as JavaScript files, CSS files, and images.
There is a big chance that this event is not firing in your case for a reasonable timeout, so I would suggest not to rely on this waitUntil but use another wait like the presence of some selector, for example
await page.goto('https://www.google.com');
await page.waitForSelector('[name="q"]');
I've got a next.js react app running on a custom Express server with custom routes. I'm working on this project by myself, but I'm hoping I might have a collaborator at some point, and so my main goal is really just to clean things up and make everything more legible.
As such, I've been trying move as much of the Express routing logic as possible to the built in Next.js api routes. I'm also trying to replace all the fetch calls I have with axios requests, since they look less verbose.
// current code
const data = await fetch("/api/endpoint", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ foo: "bar" })
}).then(x => x.json());
// what I'd like
const data = await axios.post( "/api/endpoint", { foo: "bar" });
The problem I've been having is that the dynamic next.js api routes stall as soon as there's JSON data in the body. I'm not even getting an error, the request just gets stuck as "pending" and the await promise never resolved.
I get responses from these calls, but I can't pass in the data I need:
// obviously no data passed
const data = await axios.post( "/api/endpoint");
// req.body = {"{ foo: 'bar' }":""}, which is weird
const data = await axios.post( "/api/endpoint", JSON.stringify({ foo: "bar" }));
// req.body = "{ foo: 'bar' }" if headers omitted from fetch, so I could just JSON.parse here, but I'm trying to get away from fetch and possible parse errors
const data = await fetch("/api/endpoint", {
method: "POST",
// headers: { "Content-Type": "application/json" },
body: JSON.stringify({ foo: "bar" })
}).then(x => x.json());
If I try to call axios.post("api/auth/token", {token: "foo"}), the request just gets stuck as pending and is never resolved.
The Chrome Network panel gives me the following info for the stalled request:
General
Request URL: http://localhost:3000/api/auth/token
Referrer Policy: no-referrer-when-downgrade
Request Headers
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,es;q=0.8
Connection: keep-alive
Content-Length: 26
Content-Type: application/json;charset=UTF-8
Cookie: token=xxxxxxxxxxxxxxxxxxxx; session=xxxxxxxxxxxxxxxxxxxxxxx
Host: localhost:3000
Origin: http://localhost:3000
Referer: http://localhost:3000/dumbtest
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36
Request Payload
{token: "foo"}
I've tried looking into what might be causing this, and everything seems to point towards there being an issue with preflight requests, but, since those are related to CORS policies, I don't understand why I'd be encountering those. I'm making a request from http://localhost:3000 to http://localhost:3000/api/auth/token.
Even so, I did try to add cors middleware as shown in the next.js example, but that didn't make a difference. As far as I can tell, the request never even hits the server - I've got a console.log call as the first line in the handler, but it's never triggered by these requests.
Is there something obvious I'm missing? This feels like it should be a simple switch to make, but I've spent the last day and a half trying to figure this out, but I keep reaching the same point with every solution I try - staring at a gray pending request in my Network tab and a console reporting no errors or anything.
After a few more hours searching, I found my answer here
Turns out that since I was using a bodyParser middleware in my express server, I had to disable the Next body parsing by adding this at the top of my file:
export const config = {
api: {
bodyParser: false,
},
}
I am making a POST request to server to upload an image and sending formdata using axios in react-native. i am getting "Network Error". i also try fetch but nothing work.using react native image picker libeary for select image.in postman api working fine
formData.append('title', Title);
formData.append('class_id', selectClass._id)
formData.append('subject_id', checkSelected)
formData.append('teacher_id', userId)
formData.append('description', lecture);
formData.append('type', 'image');
var arr=[];
arr.push(imageSource)
arr.map((file,index)=>{
formData.append('file',{
uri:file.path,
type:file.type,
name:file.name
})
})
axios({
method: 'post',
url: URL + 'admin/assignment/create',
data: data,
headers: {
"content-type": "multipart/form-data",
'x-auth-token': token,
},
})
.then(function (response) {
//handle success
console.log('axios assigment post',response);
})
.catch(function (response) {
//handle error
console.log('axios assigment post',response);
});
Project keeps flipper java file under app > source > debug in react native > 0.62. There is an issue with Flipper Network that causes the problem in your case. If you remove the debug folder, you will not be able to debug Android with Flipper, so the best solution is upgrading Flipper version in android > gradle.properties to 0.46.0 that fixes the problem.
You can change it with this line
FLIPPER_VERSION=0.46.0
react-nativeandroid
The issue that I was facing which is close to what you are mentioning is that I was getting NetworkError when using image-picker and trying to upload the file using axios. It was working perfectly in iOS but not working in android.
This is how I solved the issue.
There are two independent issues at action here. Let’s say we get imageUri from image-picker, then we would use these following lines of code to upload from the frontend.
const formData = new FormData();
formData.append('image', {
uri : imageUri,
type: "image",
name: imageUri.split("/").pop()
});
The first issue is with the imageUri itself. If let’s say photo path is /user/.../path/to/file.jpg. Then file picker in android would give imageUri value as file:/user/.../path/to/file.jpg whereas file picker in iOS would give imageUri value as file:///user/.../path/to/file.jpg.
The solution for the first issue is to use file:// instead of file: in the formData in android.
The second issue is that we are not using proper mime-type. It is working fine on iOS but not on Android. What makes this worse is that the file-picker package gives the type of the file as “image” and it does not give proper mime-type.
The solution is to use proper mime-type in the formData in the field type. Ex: mime-type for .jpg file would be image/jpeg and for .png file would be image/png. We do not have to do this manually. Instead, you can use a very famous npm package called mime.
The final working solution is:
import mime from "mime";
const newImageUri = "file:///" + imageUri.split("file:/").join("");
const formData = new FormData();
formData.append('image', {
uri : newImageUri,
type: mime.getType(newImageUri),
name: newImageUri.split("/").pop()
});
I hope this helps to solve your problem :)
REACT NATIVE SOLUTION
If you are using Axios or Fetch in React Native and you got Network Error when uploading the file or data.
Try to commenting below line from the /android/app/src/main/java/com/{your_project}/MainApplication.java
its located around the 40-50 line
initializeFlipper(this, getReactNativeHost().getReactInstanceManager())
https://github.com/facebook/react-native/issues/28551
I faced the same issue. The following steps worked for me.
update FLIPPER_VERSION=0.52.0 latest
for formData code as below:
let formData = new FormData();
let file = {
uri: brand.uri,
type: 'multipart/form-data',
name: brand.uri
};
formdata.append('logo', file);
The type must be 'multipart/form-data' as the post header.
"react-native": "0.62.1",
"react": "16.11.0",
"axios": "^0.19.2",
weird solution i have to delete debug folder
in android ->app->source->debug
and restart the app again
its solve my problem. i think it's cache problem.
I had this problem and solve it via commenting the 43 line in
android/src/debug/.../.../ReactNativeFlipper.java
// builder.addNetworkInterceptor(new FlipperOkhttpInterceptor(networkFlipperPlugin));
could you test it?
in my case, the solution was to change to
const headers = {
accept: 'application/json',
'content-type': 'multipart/form-data',
};
change this line: form_data.append('file', data);
To form_data.append('file', JSON.stringify(data));
from https://github.com/react-native-image-picker/react-native-image-picker/issues/798
You need to add this uesCleartextTraffic="true" to the AndroidManifest.xml file found inside the dir android/app/src/main/AndroidManifest.xml
<application ... android:usesCleartextTraffic="true"> Then, Because of issue with Flipper Network.
I commented initializeFlipper(this, getReactNativeHost().getReactInstanceManager())
in this file /android/app/src/main/java/com/{your_project}/MainApplication.java
Also, commenting out line number 43 in this file android/app/src/debug/java/com/**/ReactNativeFlipper.java
line43: builder.addNetworkInterceptor(new FlipperOkhttpInterceptor(networkFlipperPlugin));
If using expo and expo-image-picker, then the problem is only with the image type and nothing else.
In the latest updates, they removed the bug related to path (as other answers mention to change the beginning of the path which was correct for the older versions).
Now to remove the problem, we need to change the type only and is mentioned by other answers to use mime which works fine;
import mime from 'mime'
const data = new FormData();
data.append('image', {
uri: image.uri,
name: image.uri.split('/').pop() // getting the text after the last slash which is the name of the image
type: mime.getType(image.uri) // image.type returns 'image' but mime.getType(image.uri) returns 'image/jpeg' or whatever is the type
})
In my case, after debbuging for a while, the issue was in nginx.
The image was "too big".
I Had to add annotations to the Kubernetes ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 20m
....
It was a bit tricky to debug since the request never got through the load balancer (nginx) to the Api service.
The "Network error" message didn't help a lot either.
I am using Expo SDK 42 (react-native v0.63). And I was using the expo-document-picker library to pick the documents & upload to server.
This is the code I am using to open the picker & get the metadata about the file.
const result = await DocumentPicker.getDocumentAsync({
type: 'image/*',
copyToCacheDirectory: false,
multiple: false,
});
if (result.type === 'success') {
const name_split = result.name.split('.');
const ext = name_split[name_split.length - 1];
// result.uri = 'file://' + result.uri;
result.type = helper.get_mime_type(ext);
delete result.size;
}
(You can write your function to get the mime type from the file extension or use some library from npm)
And this is the code I am using to upload the file to server:
const formdata = new FormData();
formdata.append('custom_param', 'value');
formdata.append('file', result); // 'result' is from previous code snippet
const headers = {
accept: 'application/json',
'content-type': 'multipart/form-data',
};
const opts = {
method: 'POST',
url: 'your backend endpoint',
headers: headers,
data: formdata,
};
return await axios.request(axiosopts);
The above code is the working code. I want to explain what I did wrong initially that was causing the Network Error in axios.
I had set copyToCacheDirectory to true initially and the uri I was getting in the result was in the format /data/path/to/file.jpeg. And I also tried appending file:// to beginning of the uri but it didn't work.
I then set copyToCacheDirectory to false and the uri I was getting in the result was in the format content://path/to/file.jpeg. And this didn't cause any Network Error in axios.
I have faced this error as well. I found out that I got this error because the file path is wrong and axios couldn't find the file. It is a weird error message when the uri is wrong though but that's what actually has happened. So double checking the uri of the file would fix the issue. Mainly consider file://.
I faced the same issue.
After capturing the photo wait for 10sec then start uploading. It worked for me.
Also got the same issue. I spent almost 4 days to find reason.
So in my case it was Content-Type: multipart/form-data. I forgot
indicate it. In Android it should be indicated explicitly...
After two days of searching for a solution, I found that the problem was in my rn-fetch-blob library, Changed it to in package.json dependencies
"rn-fetch-blob": "^0.12.0",
fix my Netowk issue and app crash on uploading. I am using
react-native-image-crop-picker
always send image and file in formdata in post api through axios in react js and react native
Blockquote
REACT NATIVE SOLUTION USING AXIOS
I face the same issue after upgrading react native cli project.
I'm using
"react-native": "0.70.6",
"react": "18.1.0",
"axios": "^1.1.3"
AND FLIPPER_VERSION=0.125.0
The below code solves my issue
const imageData = image;
const form = new FormData();
form.append("ProfileImage", {
type: imageData.mime,
uri: imageData.path,
name: imageData.path.split("/").pop(),
});
axios({
method: "put",
url: `${URL}/api/UploadPhoto`,
data: formData,
headers: {
"Content-Type": "multipart/form-data",
"cache-control": "no-cache",
},
processData: false,
contentType: false,
mimeType: "multipart/form-data",
});
For me, I didn't comment on any line from the /android/app/src/main/java/com/{your_project}/MainApplication.java
initializeFlipper(this, getReactNativeHost().getReactInstanceManager())
Also not changed FLIPPER_VERSION