I have created a review request on ReviewBoard using the api, but I am now unable to add a diff file to this review request.
I am following the ReviewBoard instructions to sent a POST request to ReviewBoard API --> https://www.reviewboard.org/docs/manual/2.0/webapi/2.0/resources/diff-list/
My request header looks like this
{Content-Type: multipart/form-data; boundary=------- 1511347017267}{Accept: application/xml}{Authorization: token 4937d...sometoken...584b23}
My request body looks like this
------- 1511345733192
Content-Disposition: form-data; name="basedir"
/trunk_projectABC/
------- 1511345733192
Content-Disposition: form-data; name="path"; filename="build.diff"
Content-Type: text/xml
Index: utility/build.bat
===================================================================
--- utility/build.bat (revision 67210)
+++ utility/build.bat (working copy)
## -10,8 +10,8 ##
echo off
set thpcra=false
set patchBuild=false
-set help=true
-set clientBuild=true
+set help=false
+set clientBuild=false
set runJavaTestCases=false
set buildpath="%BUILD_PATH%"
set jdkpath="%JDK_17%"
------- 1511345733192 --
I found another question here
review board diff not uploading
and made sure that the repo URL + basedir + relative path is set right but I still get BAD REQUEST response from the server.
NOTE: I generated the diff using tortoiseSVN tool
I am using HttpURLConnection conn = (HttpURLConnection) url.openConnection();
Any inputs on why I am getting BAD REQUEST from the server ?
Answering my own question.
I was using PrintWriter to write the request body on the connection output stream. Probably using a stream introduced any extra characters or somehow malformed the request body, but it was the reason why I was getting bad request.
I am now using MultipartRequestEntity to create the request body. Here is the code snippet
PostMethod method = new PostMethod(apiURL);
File diff = new File("D:/path_to_diff/build.diff");
partsList.add(new FilePart("path", diff.getName(), diff));
partsList.add(new StringPart("basedir", "/trunk_projectABC/"));
Part[] parts = partsList.toArray(new Part[partsList.size()]);
RequestEntity entity = new MultipartRequestEntity(parts, method.getParams());
method.setRequestEntity(entity);
This works and uploads a diff file on the review request
Related
We seem to be hitting a limit when trying to add a large binary file to Azure DevOps using the REST APIs. The same file check-in works fine using the old SOAP APIs and also works using the TFVC CLI (tf.exe). But we have a use case where we need to occasionally check in large files programatically from machines that don't have VS installed. We're trying to migrate our application from the old SOAP APIs to the REST API because we're moving to .NET Core where the SOAP API is not supported.
A POST to the /_apis/tfvc/changesets (Create Changeset) API with large (> about 19 MB) files results in:
HTTP 400: Bad Request
This issue has been reported a couple of times on the Azure DevOps .NET Samples github repo, but that isn't the right forum for this question so it hasn't been answered there.
https://github.com/microsoft/azure-devops-dotnet-samples/issues/176
https://github.com/microsoft/azure-devops-dotnet-samples/issues/104
How do we create large files in TFVC using the REST API?
Update 2020-01-13
Here's a sample console app we've used to demonstrate the problem:
using Microsoft.TeamFoundation.SourceControl.WebApi;
using Microsoft.VisualStudio.Services.Common;
using Microsoft.VisualStudio.Services.WebApi;
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApp1
{
internal class Program
{
internal static async Task Main(string[] args)
{
var orgUrl = new Uri(args[0]);
string serverPath = args[1];
string localPath = args[2];
string contentType = args[3];
string pat = args[4];
var changes = new List<TfvcChange>()
{
new TfvcChange()
{
ChangeType = VersionControlChangeType.Add,
Item = new TfvcItem()
{
Path = serverPath,
ContentMetadata = new FileContentMetadata()
{
Encoding = Encoding.UTF8.WindowsCodePage,
ContentType = contentType,
}
},
NewContent = new ItemContent()
{
Content = Convert.ToBase64String(File.ReadAllBytes(localPath)),
ContentType = ItemContentType.Base64Encoded
}
}
};
var changeset = new TfvcChangeset()
{
Changes = changes,
Comment = $"Added {serverPath} from {localPath}"
};
var connection = new VssConnection(orgUrl, new VssBasicCredential(string.Empty, pat));
var tfvcClient = connection.GetClient<TfvcHttpClient>();
await tfvcClient.CreateChangesetAsync(changeset);
}
}
}
Running this console app with a ~50MB zip file against an Azure DevOps TFVC repository results in a VssServiceResponseException (with inner ArgumentException) with a message of: The maximum request size of 26214400 bytes was exceeded.
Update 2020-01-14
Added display of file size and base64 content length before sending TFVC request in my sample code.
Corrected my observations about the file size limit.
Corrected the error code returned by Azure DevOps (400, not 413).
Originally I stated that the limit was around 13 MB. This was based on the failures I saw with files > 20MB, success with files < about 10 MB, and the limits described in the linked github issues.
I have run more tests of my own and have narrowed in on about 19,200 KB as the actual limit. This seems correct based on the error message indicating a 26,214,400 byte limit. A 19 MB base-64 encoded file would expand to about 26 MB.
I also noticed in my most recent tests, when monitoring with Fiddler, that Azure DevOps returns a 400 status code (not 413). My notes had indicated a 413 Request Entity Too Large was observed at some point in the past. Perhaps this with with an older version of Azure DevOps Server? In any case, the error we see now is 400 Bad Request.
Here is what Fiddler shows for the request headers:
POST /<...REMOVED...>/_apis/tfvc/changesets HTTP/1.1
Host: dev.azure.com
Accept: application/json; api-version=5.1
User-Agent: VSServices/16.153.29226.1 (NetStandard; Microsoft Windows 10.0.18363)
X-VSS-E2EID: 6444f0b5-57e0-45da-bd86-a4c62d8a1794
Accept-Language: en-US
X-TFS-FedAuthRedirect: Suppress
X-TFS-Session: 9f8e8272-db48-4e93-b9b0-717937244aff
Expect: 100-continue
Authorization: Basic <...REMOVED...>
Accept-Encoding: gzip
Content-Type: application/json; charset=utf-8; api-version=5.1
Content-Length: 26324162
And the raw response:
HTTP/1.1 400 Bad Request
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 206
Content-Type: application/json; charset=utf-8
Expires: -1
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
X-TFS-ProcessId: 7611d69f-e722-4108-8050-e55a61b1cbb4
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: 15e046e5-3788-4fdb-896a-7d0482121ddd
X-TFS-Session: 9f8e8272-db48-4e93-b9b0-717937244aff
X-VSS-E2EID: 6444f0b5-57e0-45da-bd86-a4c62d8a1794
X-VSS-UserData: <...REMOVED...>
X-FRAME-OPTIONS: SAMEORIGIN
Request-Context: appId=cid-v1:e3d45cd2-3b08-46bc-b297-cda72fdc1dc1
Access-Control-Expose-Headers: Request-Context
X-Content-Type-Options: nosniff
X-MSEdge-Ref: Ref A: 7E9E95F5497946AC87D75EF3AAD06676 Ref B: CHGEDGE1521 Ref C: 2020-01-14T14:01:59Z
Date: Tue, 14 Jan 2020 14:02:00 GMT
{"$id":"1","innerException":null,"message":"The maximum request size of 26214400 bytes was exceeded.","typeName":"System.ArgumentException, mscorlib","typeKey":"ArgumentException","errorCode":0,"eventId":0}
Until now, the most useful solution is to allocated a sufficiently large buffer via modifying the configuration in IIS.
Go machine -> open IIS manager:
(1) Select the site of your collection
(2) Double click “Configuration Editor”
(3) Insert system.webServer/serverRuntime into Section
(4) Expand the uploadReadAheadSize value based on your scenario.
Then click Apply to apply above changes.
I know, it would be very costly for large request bodies. But, as I know, this is the often method we used.
I opened a support ticket with Microsoft. The support engineer indicated that this was a known limitation of the Azure DevOps service. He suggested that I create a feature request. I've done that. If this limitation is problematic for you, please upvote the feature request.
https://developercommunity.visualstudio.com/idea/1130401/allow-creating-large-tfvc-changesets-via-the-api.html
I'm unable to send both, JSON body data and a video/image file together in one request on JMeter. It gives a 401 unauthorized, even though I have added the Authorization header in the HTTP Header Manager. The same request with headers works fine on Postman.
I've already referred to this guide here with no luck.
https://www.blazemeter.com/blog/testing-advanced-rest-api-file-uploads-jmeter/
Content-Type: multipart/form-data
{
"email": "xyz#gmail.com",
"password": "xyz",
"file": ${__FileToString("/Downloads/SampleVideo_1280x720_1mb.mp4")}
}
SAMPLER RESULT:
Thread Name: Thread Group 1-1
Sample Start: 2019-09-30 14:26:25 IST
Load time: 2922
Connect Time: 838
Latency: 2922
Size in bytes: 202
Sent bytes:1056397
Headers size in bytes: 202
Body size in bytes: 0
Sample Count: 1
Error Count: 1
Data type ("text"|"bin"|""):
Response code: 401
Response message: Unauthorized
HTTPSampleResult fields:
ContentType:
DataEncoding: null
REQUEST:
POST data:
--o3F8APyqP080W3wk0N_-96jzl11Bfsa
Content-Disposition: form-data; name="file"; filename="SampleVideo_1280x720_1mb.mp4"
Content-Type: video/mp4
Content-Transfer-Encoding: binary
<actual file content, not shown here>
--o3F8APyqP080W3wk0N_-96jzl11Bfsa--
[no cookies]
JMeter should be doing what real browser (or other application using your API endpoint) is doing.
If the browser (or the application) sends 2 requests - JMeter must be sending 2 requests as well. If the browser (or the application) sends 1 request - JMeter must be sending 1 request. In case of 1 request my expectation is that you should be sending application/json as the value of the Content-Type header like it's done in the guide you're referring.
--boundary
Content-Type: application/json; charset=UTF-8
JSON Metadata
--boundary
Content-Type: file MIME type
File content
--boundary--
The answer is quite simple: just record the request using JMeter's HTTP(S) Test Script Recorder and JMeter will come up with proper configuration of the HTTP Request sampler(s), HTTP Header Manager, etc. All you will need to do is to implement parameterization and correlation so the script could be replayed successfully.
One important bit: during the recording you need to have SampleVideo_1280x720_1mb.mp4 file in JMeter's "bin" folder as modern browsers don't return the full path hence JMeter will be able to properly capture the file upload request only if the file lives in its "bin" folder. More information: Recording File Uploads with JMeter
To post json body and attachment you can add your "email" and "password" in Parameters tab of JMeter HTTPS request and file in Files Upload tab.
Is it possible to send a gzipped soap request?
I added an HTTP Header Manager with the following headers:
Content-Type: application/soap+xml; charset=Utf-8
Content-Encoding: gzip
I added a Beanshell PreProcessor as a child of the request which needs to be encoded, and I defined the following script:
import org.apache.commons.io.IOUtils;
import java.util.zip.GZIPOutputStream;
// This only works for the HTTP Request, not Soap Request.
// String bodyString = sampler.getArguments().getArgument(0).getValue();
String bodyString = ctx.getCurrentSampler().getXmlData();
byte [] requestBody = bodyString.getBytes();
ByteArrayOutputStream out = new ByteArrayOutputStream(requestBody.length);
GZIPOutputStream gzip = new GZIPOutputStream(out);
gzip.write(requestBody);
gzip.close();
// This only works for the HTTP Request, not Soap Request.
// sampler.getArguments().getArgument(0).setValue(out.toString(0));
ctx.getCurrentSampler().setXmlData(???);
My problem is the last line, how can I set xmlData?
Jmeter version 3.1
Replace SOAP/XML-RPC Sampler which is deprecated with the HTTP Request sampler
Upgrade to JMeter 5.0 (or whatever is the latest version available at JMeter Downloads page)
Switch from Beanshell to Groovy
Use sampler.getArguments().getArgument(0).setValue(out.toString(0)); in order to generate the request body.
After setting up my API to upload files, I realised that there is a special case where you want to upload a picture (jpg), you defined the binary support at the API, but you get the following error:
The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method.
Consult the service documentation for details.
The Canonical String for this request should have been
'PUT /test/vi-dummy-bucket/testImg2.jpg
content-type:application/x-www-form-urlencoded
host:qhweyos7z2.execute-api.us-west-1.amazonaws.com
x-amz-date:20170808T154441Z
x-amz-security-token: // security token string no quotes
content-type;host;x-amz-date;x-amz-security-token 5fa90f0 ...'
The String-to-Sign should have been
'AWS4-HMAC-SHA256\n20170808T154441Z
20170808/us-west-1/execute-api/aws4_request
f7a38fa ...'
The strange thing is that uploading simple text files works with the exact same api call, then only thing I have to change is
Content-Type 'text/plain'
and write a text in the raw portion of the request.
Not sure if this is a Content-Type issue or a Request Body Issue, if I leave everything in the working state (text/plain & text in the body) and just change the body to binary and set the image, I get the above error.
My API gateway is in us-west-1 region
My S3 bucket is in us-east-1 region
And the request I am using is:
PUT /test/vi-dummy-bucket/testImg2.jpg HTTP/1.1
Host: qhwe7z2.execute-api.us-west-1.amazonaws.com
Content-Type: application/x-www-form-urlencoded
X-Amz-Security-Token: FQoDYX ...
X-Amz-Date: 20170808T154441Z
Authorization: AWS4-HMAC-SHA256
Credential=ASIAJICO6JFTJWN7A/20170808/us-west-1/execute-
api/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-
security-token,
Signature=6a792 ... Cache-Control: no-cache
Postman-Token: e9d1f730-f50b-7e27-70cc-c15a138d8cc6
(Binary Image)
This is another version of the request (same error):
PUT /test/vi-dummy-bucket/testImg2.jpg HTTP/1.1
Content-Type: image/jpeg
x-amz-security-token: FQoDY ...
x-amz-date: 20170808T190134Z
Authorization: AWS4-HMAC-SHA256
Credential=ASIAIZSP5YKVLJ3GVVQA/20170808/us-west-1/execute-
api/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-
security-token,
Signature=b2324 ...
Host: qhos7z2.execute-api.us-west-1.amazonaws.com
Connection: close
User-Agent: Paw/3.1.2 (Macintosh; OS X/10.12.6) GCDHTTPRequest
Content-Length: 823236
--- UPDATE ---
After implementing the sigV4 sigining manually using the generated SDK, the signature is no longer an issue.
The only problem left, is that the generated SDK only accepts a string as the "body", so I have to convert the file to a binary string. Then it passes correctly and a file is created in S3, but the size is now double and its not viewable, as if the binary string wasn't converted back to the binary file. So frustrating...
BTW, I've already tried PASSTHROUGH and CONVERT_TO_BINARY.
Updated: It looks like this may be related to a known error in Postman. For reference here is a related SO question: AWS Signature Error using Postman to access the AWS API Gateway when posting a binary
and here is the bug report for Postman: https://github.com/postmanlabs/postman-app-support/issues/3232
Does the request work if you use an alternate rest client and/or a command line utility like curl or httpie?
If you configured the binary support you should probably set the Content-Type to match the binary content you're sending.
From what you've posted you're sending the binary content with Content-Type application/x-www-form-urlencoded but if the body is actually a binary jpeg file I'd expect that you should be sending Content-Type image/jpeg
So lets say I have a resource at /things.
If client does GET /things/myid I'll serve the default response (json).
If client passes an Accept: csv header I will serve the response as csv.
But what if I want to enable the client to tell the server to include a Content-disposition: attachment header in the response (to force a download)?