I'm using a Prometheus datasource from Grafana, and I'm sometimes getting 200 OK, sometimes getting 405 Method not allowed when looking at graphs, or inserting new graphs.
It is very strange that it only appears sometimes, for random graphs, sometimes only for some graphs in a single dashboard.
The datasource is set up to proxy requests through the backend.
Both Grafana and Prometheus are running in Kubernetes as StatefulSets in Google cloud.
I'm accessing Grafana at localhost:3000 through an SSH tunnel to the pod in Kubernetes, and Grafana is accessing Prometheus at http://prometheus:9090/.
I've tried changing the method from GET to POST in the datasource setup, but then I get 405 on every request.
The raw headers in the request for http://localhost:3000/api/datasources/proxy/1/api/v1/query_range?query=kafka_topic_highwater{topic="test"}&start=1541499015&end=1541499930&step=15 is
Host: localhost:3000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://localhost:3000/d/tDB6XEaiz/kafka-realtime-timeseries?orgId=1
X-Grafana-Org-Id: 1
DNT: 1
Connection: keep-alive
Cookie: grafana_user=admin; grafana_remember=asdf8a620; grafana_sess=<secret>
And the response is:
HTTP/1.1 405 Method Not Allowed
Cache-Control: no-cache
Content-Length: 19
Content-Type: text/plain; charset=utf-8
Date: Tue, 06 Nov 2018 10:25:22 GMT
Expires: -1
Pragma: no-cache
X-Content-Type-Options: nosniff
Any ideas what might be causing this?
The problem was that I had two Prometheus instances running in the same cluster, with the same service name, so that requests were distributed across them... One of them replied with 405 because it was set up to forward metrics directly to StackDriver..
Related
I have an application (React SPA) that calls a bunch of servers on different subdomains of the application domain, i.e.:
the web app sits at foo.bar.com,
and talks to api.foo.bar.com and media.foo.bar.com.
When accessing api.foo.bar.com, I get an error from the browser (be it Edge, Chrome, or Firefox) telling me that the origin (foo.bar.com) is different from the value of the Access-Control-Allow-Origin response header. However, by inspection of the response, they are the same:
(I unfortunately have to obfuscate the address.)
Those apps are hosted on Kubernetes; the ingress is NGINX, and it's is not providing CORS (cors-enabled annotation is false). Both applications (api and media) are Express apps, and both have the same CORS configuration allowing the specific origin.
I'm wondering if this has something to do with the redirect - the call to the media... endpoint returns a redirect (302) whose Location is a api... address.
Other than that, I have no clue what could be wrong. Something is, for sure, because all browsers agree that my request should be blocked (on account of the origin).
In all cases, I've checked the address multiple times for typos, ending forward-slashes, etc. I've called OPTIONS on those endpoints with cURL and Postman, using all headers or just a few. They always answer the correct address.
Additional information, as requested:
Preflight request:
OPTIONS /media/1.0.0/rtsp/hls?feedUrl=https%3A%2F%2Flive.monuv.com.br%2Fa1%2F14298.stream%2Fstr27%2Fchunklist.m3u8%3Fm_hash%3DkhV_hCnKG3nhaNCFaYZxBnoMz-99idQVHiQh80ADW78%253D HTTP/2
Host: media.aiXXXXXXXXXXXXXX.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: GET
Access-Control-Request-Headers: feedurl
Referer: https://aiXXXXXXXXXXXXXXXX.com/
Origin: https://aiXXXXXXXXXXXXXXXX.com
DNT: 1
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
Pragma: no-cache
Cache-Control: no-cache
TE: trailers
Preflight response:
HTTP/2 204 No Content
date: Fri, 08 Oct 2021 13:33:10 GMT
x-powered-by: Express
access-control-allow-origin: https://aiXXXXXXXXXXXXXXXXXX.com
vary: Origin
access-control-allow-credentials: true
access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
access-control-allow-headers: Content-Type, feedUrl
strict-transport-security: max-age=15724800; includeSubDomains
X-Firefox-Spdy: h2
Request
The preflight passes, and the browsers starts a "flight" request:
GET /media/1.0.0/rtsp/hls?feedUrl=https%3A%2F%2Flive.monuv.com.br%2Fa1%2F14298.stream%2Fstr27%2Fchunklist.m3u8%3Fm_hash%3DkhV_hCnKG3nhaNCFaYZxBnoMz-99idQVHiQh80ADW78%253D HTTP/2
Host: media.aiXXXXXXXXXXXXXXXXXXXXX.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
feedUrl: https://live.monuv.com.br/a1/14298.stream/str27/chunklist.m3u8?m_hash=khV_hCnKG3nhaNCFaYZxBnoMz-99idQVHiQh80ADW78%3D
Origin: https://aiXXXXXXXXXXXXXXXX.com
DNT: 1
Connection: keep-alive
Referer: https://aiXXXXXXXXXXXXXXXXX.com/
Cookie: ory_kratos_session=MTYzMzYzODY1OHxEdi1CQkFFQ180SUFBUkFCRUFBQVJfLUNBQUVHYzNSeWFXNW5EQThBRFhObGMzTnBiXXXXXXXXXXXXYVc1bkRDSUFJSHBtUWxsaWFsVlJhWGRTVGxSMmIzZHRkbTFqYm5CUlRWVkdkelpPWkRoWnXXXTyqwgK-0Pe0qtZHjNhfU-YoASjg3istMZi672swQ==
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
Pragma: no-cache
Cache-Control: no-cache
TE: trailers
Response
HTTP/2 302 Found
date: Fri, 08 Oct 2021 13:33:10 GMT
content-type: text/plain; charset=utf-8
content-length: 129
location: https://api.aiXXXXXXXXXXXXXXXXXX.com/media/1.0.0/hls/streams/19dd149d-f551-4093-b2aa-e5558388d545/hls.m3u8
x-powered-by: Express
access-control-allow-origin: https://aiXXXXXXXXXXXXXXXX.com
vary: Origin, Accept
access-control-allow-credentials: true
strict-transport-security: max-age=15724800; includeSubDomains
X-Firefox-Spdy: h2
At this response, the browser fails saying that the origin don't match the access-control-allow-origin.
(the first image was from Edge, since the log was more clear; this log is from Firefox)
Problem
The error message—I'm using dummy URLs and origins below—from the browser can be a bit confusing:
Access to XMLHttpRequest at 'https://api.example.com/' (redirected from 'https://media.example.com/') from origin 'https://example.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header has a value 'https://example.com' that is not equal to the supplied origin.
The key here is that, as sideshowbarker hinted at in his comment, because your first preflighted request to https://media.example.com/ responds with a cross-origin redirect to https://api.example.com/, the browser performs another whole CORS access-control check for that resource. However, because the redirect resulting from the first preflighted request happens to be cross-origin, the browser sets the origin of the second preflight request (which the error message refers to as the "supplied origin"), not as https://example.com, but as the null origin!
Here's a rundown of what is likely happening:
Because https://api.example.com likely doesn't (and shouldn't!) allow the null, the second access-control check fails and you get that annoying CORS error.
Solution
Resist the temptation to allow the null origin on https://api.example.com/, as doing so has serious security ramifications: it amount to voiding the protection that the Same-Origin Policy provides.
Instead, you should get rid of that redirect from https://media.example.com/ to https://api.example.com/ and make your frontend request the https://api.example.com/ resource directly.
Alternatively, if you cannot completely get rid of the redirect but you can change its destination, make it a same-origin redirect (from somewhere https://media.example.org to elsewhere on https://media.example.org).
My Traefik config for WordPress contains the following docker-labels:
- "traefik.backend=wordpress"
- "traefik.docker.network=web"
- "traefik.frontend.rule=Host:MyHostName.net;PathPrefix:/blog"
- "traefik.enable=true"
- "traefik.port=80"
Now requesting the url "https://MyHostName/blog" seems to reach the service which seems to return a redirect to "https://MyHostName/wp-admin...".
I cannot use subdomains.
How can I solve this?
UPDATE 0
First thing to do should be adding the Filter "PathPrefixStrip:/blog" to remove the "/blog" prefix when forwarding the request to the service. Correct?
But how do I modify (for example) a redirect request to add the prefix "/blog" to the redirect URL?
UPDATE 1
At https://github.com/containous/traefik/issues/985 my question is "discussed" and a solution seems to be merged (https://github.com/containous/traefik/pull/1442).
In short: Stripped prefixes will be added as the respective header (X-Forwarded-Prefix).
I will check that and write down the results here.
Additional resources:
Routing paths with Traefik
Is there an equivalent to ReverseProxyPass for Apache in Traefik?
UPDATE 2
Now I created a request looking like this:
https://MYHOSTNAME/blog
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: de,en-US;q=0.7,en;q=0.3
Connection: keep-alive
Cookie: ocuvhr6ala6i=d2cd9020839889a752b4375a63dedad0; oc_sessionPassphrase=qJu13Q%2FlAoSsv5b0qC18Re%2BcrcML6o32c2XuDJEGViIMI4uERIf%2Bs77DvFbMSkEBkZs%2Bn%2FfnUjdB9APvk4zq2qlj6AiDXX2CGYf31MPVci8HkgcsXFcpL7cRLBbRGRWS; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true
Host: MYHOSTNAME
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
The "PathPrefixStrip" seems to work in the direction CLIENT>>SERVICE. This is what my traefik log contains:
traefik | time="2018-04-04T18:12:54Z" level=debug msg="vulcand/oxy/roundrobin/rr: competed ServeHttp on request" Request="
{
"Method":"GET",
"URL":{
"Scheme":"",
"Opaque":"",
"User":null,
"Host":"",
"Path":"/",
"RawPath":"",
"ForceQuery":false,
"RawQuery":"",
"Fragment":""
},
"Proto":"HTTP/2.0",
"ProtoMajor":2,
"ProtoMinor":0,
"Header":{
"Accept":[
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
],
"Accept-Encoding":[
"gzip, deflate, br"
],
"Accept-Language":[
"de,en-US;q=0.7,en;q=0.3"
],
"Cookie":[
"ocuvhr6ala6i=d2cd9020839889a752b4375a63dedad0; oc_sessionPassphrase=qJu13Q%2FlAoSsv5b0qC18Re%2BcrcML6o32c2XuDJEGViIMI4uERIf%2Bs77DvFbMSkEBkZs%2Bn%2FfnUjdB9APvk4zq2qlj6AiDXX2CGYf31MPVci8HkgcsXFcpL7cRLBbRGRWS; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true"
],
"Upgrade-Insecure-Requests":[
"1"
],
"User-Agent":[
"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
],
"X-Forwarded-Prefix":[
"/blog"
]
},
"ContentLength":0,
"TransferEncoding":null,
"Host":"MYHOSTNAME",
"Form":null,
"PostForm":null,
"MultipartForm":null,
"Trailer":null,
"RemoteAddr":"81.128.35.176:33468",
"RequestURI":"/",
"TLS":null
}
"
But the redirection answer looks as follows in my browser:
HTTP/2.0 302 Found
cache-control: no-cache, must-revalidate, max-age=0
content-length: 0
content-type: text/html; charset=UTF-8
date: Wed, 04 Apr 2018 18:44:18 GMT
expires: Wed, 11 Jan 1984 05:00:00 GMT
location: https://MYHOSTNAME/wp-admin/install.php
server: Apache/2.4.25 (Debian)
X-Firefox-Spdy: h2
x-powered-by: PHP/7.2.2
So the redirect-response does not contain any information about the stripped path prefix "/blog".
UPDATE 3
At the end it looks like a problem of the served software inside the container that does not handle the header.
Additional resources:
Wordpress & Nginx with Docker: Static files not loaded
Any ideas?
Since v2.0, Traefik doesn't support PathPrefixStrip anymore, you need to use a middleware as specified in this article : https://doc.traefik.io/traefik/migration/v1-to-v2/#strip-and-rewrite-path-prefixes 😊
Maybe you should add all possible values in your PathPrefixStrip: / blog rule eg.
PathPrefixStrip: /blog,/wp-admin,/abc,/xyz
In many cases, it works for standard routes. The biggest problem is when your backend service does not listen to requests in the root / but in some sub-dir /something/index.html and that sub-dir takes resources from the root /.
I have three websites hosted (example1.com, example2.com, example3.com) on a server. There is a page (test.php) on example1.com with just code below inside it:
<?php
header('Location:http://example2.com/a.php');
?>
When I browse test.php it goes to http://example1.com/a.php . it doesn't understand it is another domain url, it tried to find the page on itself.
but when I put http://google.com instead of example2.com/a.php it works correct. I really get confused.
What is the problem ? Should I set some configuration on the server?
( I am administrator of the hosting server ).
Ps. The server is behind a pound server.
Edited:
Here's the Firebug Net output for example1.com/test.php
Response Headers:
HTTP/1.1 302 Found
Date: Tue, 09 Oct 2012 09:03:34 GMT
Server: Apache/2.2.16 (Debian)
Location: http://example1.com/a.php
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 21
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
Request Headers:
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Connection keep-alive
Cookie mycookie
Host example1.com
User-Agent Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/20100101 Firefox/14.0.1
the problem is solved. it was because of pound server configuration. 'RewriteLocation' entry in pound server configuration must be set to 2 to this server doesn't change the redirect location.
anyway, thank you for answering.
I have two possible flows:
ConsoleClient -(1)-> ApplicationServer
or
SilverlightClient -(2)-> WebServer -(3)-> ApplicationServer
Fiddler successfully captures the HTTP traffic on the (1) and the (2), but not on the (3). Here is a sample capture on (1):
POST /WcfDemo/ws HTTP/1.1
Content-Type: application/soap+xml; charset=utf-8
Host: il-mark-lt
Content-Length: 521
Expect: 100-continue
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing"><s:Header><a:Action s:mustUnderstand="1">http://tempuri.org/IWcfDemoService/Add</a:Action><a:MessageID>urn:uuid:d7fde351-12fd-4872-bc26-52ff97f126e9</a:MessageID><a:ReplyTo><a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address></a:ReplyTo><a:To s:mustUnderstand="1">http://il-mark-lt/WcfDemo/ws</a:To></s:Header><s:Body><Add xmlns="http://tempuri.org/"><x>4</x><y>5</y></Add></s:Body></s:Envelope>
HTTP/1.1 200 OK
Content-Length: 399
Content-Type: application/soap+xml; charset=utf-8
Server: Microsoft-HTTPAPI/2.0
Date: Sat, 17 Sep 2011 20:57:16 GMT
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing"><s:Header><a:Action s:mustUnderstand="1">http://tempuri.org/IWcfDemoService/AddResponse</a:Action><a:RelatesTo>urn:uuid:d7fde351-12fd-4872-bc26-52ff97f126e9</a:RelatesTo></s:Header><s:Body><AddResponse xmlns="http://tempuri.org/"><AddResult>9</AddResult></AddResponse></s:Body></s:Envelope>
And here is an example of (2):
POST /WcfDemoService.svc/ws HTTP/1.1
Host: localhost:56970
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.2) Gecko/20100101 Firefox/6.0.2
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8,application/json
Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.6,he-IL;q=0.5,he;q=0.4,ru-RU;q=0.3,ru;q=0.1
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Referer: http://localhost:56970/ClientBin/SilverlightClient.xap
Content-Length: 581
Content-Type: application/soap+xml; charset=utf-8
<s:Envelope xmlns:a="http://www.w3.org/2005/08/addressing" xmlns:s="http://www.w3.org/2003/05/soap-envelope"><s:Header><a:Action s:mustUnderstand="1">http://tempuri.org/IWcfDemoService2/Add</a:Action><a:MessageID>urn:uuid:e8420d3e-f568-49ce-bfc7-5631d5bf3fd0</a:MessageID><a:ReplyTo><a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address></a:ReplyTo><a:To s:mustUnderstand="1">http://localhost:56970/WcfDemoService.svc/ws</a:To></s:Header><s:Body><Add xmlns="http://tempuri.org/"><x>11</x><y>22</y><serverChannelKind>ws</serverChannelKind></Add></s:Body></s:Envelope>
HTTP/1.1 200 OK
Server: ASP.NET Development Server/10.0.0.0
Date: Sat, 17 Sep 2011 20:59:23 GMT
X-AspNet-Version: 4.0.30319
Content-Length: 401
Cache-Control: private
Content-Type: application/soap+xml; charset=utf-8
Connection: Close
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing"><s:Header><a:Action s:mustUnderstand="1">http://tempuri.org/IWcfDemoService2/AddResponse</a:Action><a:RelatesTo>urn:uuid:e8420d3e-f568-49ce-bfc7-5631d5bf3fd0</a:RelatesTo></s:Header><s:Body><AddResponse xmlns="http://tempuri.org/"><AddResult>33</AddResult></AddResponse></s:Body></s:Envelope>
Now, I am absolutely sure the (3) does get through. So, it all boils down to some misconfiguration on the WebServer, but I cannot nail it. The Web server is just a trivial ASP.NET application hosted within IIS. It even has the following lines in the web.config:
<system.net>
<defaultProxy>
<proxy bypassonlocal="false" usesystemdefault="true" />
</defaultProxy>
</system.net>
Still, this does not work.
To further strengthen my suspicion on the web server configuration, I have checked the SilverlightClient --> ApplicationServer flow and it is captured just fine.
I am using the Asp.Net development server.
Edit
Running procmon reveals that the following suspicious registry key is consulted (amongst others):
HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\ProxyBypass
And on my machine it was set to 1. I have changed it to 0 and seems like it solved my issue. The only problem is that when I change it back to 1 Fiddler continues to capture the problematic leg! Very interesting.
Anyway, I am satisfied, for now.
You are calling "localhost" right?
Fiddler is not able to capture the local traffic if you are using "localhost" as hostname.
Solutions:
Use servername (e.g. myserver)
Use ip4.fiddler (e.g. http://ipv4.fiddler:8787)
Not sure if these are causing it ... but,
A few things to check:
In IIS7 the appPool has a loadUserProfile setting. It causes the session to load a user profile which means it can get system proxy settings.
Check the code making the request from the webServer - even if you configure to use the system proxy and bypass onLocal (which only applies to names without dots in it), code making the request can still explicitly set to use or not to use a proxy.
Far fetched but you may want to play with the account the appPool runs as - local account with profile vs. Network Service.
Hope that helps - these network things have a lot of variables between two points :)
I'm trying to implement file upload functionality in the iPhone app. Server code is tested and works when files are uploaded from the desktop browser, so I moved to implementing the Objective-C client code. I'm assembling HTTP requests body manually, and despite that it looks correct, it is rejected by the server (server handler unable to extract the parts from multipart content). In desperation I've simplified the form to having only one parameter, but it still does not work.
I've captured the network traffic and I could see that Wireshark could not parse my multipart content as well (have a look at screenshots: Firefox request, iPhone request). I'm pasting it below in hope that you could see the errors I can't see.
Thanks in advance.
Firefox:
POST /cubepaint/actions/gallery/post HTTP/1.1
Host: [...]
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Authorization: Basic [...]
Content-Type: multipart/form-data; boundary=---------------------------20072377098235644401115438165
Content-Length: 180
-----------------------------20072377098235644401115438165
Content-Disposition: form-data; name="deviceId"
12345
-----------------------------20072377098235644401115438165--
HTTP/1.1 200 OK
Date: Sat, 17 Oct 2009 22:09:21 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_python/3.2.10 Python/2.4.4 mod_ssl/2.2.3 OpenSSL/0.9.8c
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
iPhone:
POST /cubepaint/actions/gallery/post HTTP/1.1
Host: [...]
User-Agent: Copenhagen/1.0 CFNetwork/459 Darwin/9.8.0
Content-Type: multipart/form-data; boundary=----------0E7B16E6-CD3D-4213-9B42-07DA30822C74
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Authorization: Basic [...]
Content-Length: 187
Connection: keep-alive
----------0E7B16E6-CD3D-4213-9B42-07DA30822C74
Content-Disposition: form-data; name="deviceId"
00000000-0000-1000-8000-0016CBCC0B61
----------0E7B16E6-CD3D-4213-9B42-07DA30822C74--
HTTP/1.1 200 OK
Date: Sat, 17 Oct 2009 22:04:07 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_python/3.2.10 Python/2.4.4 mod_ssl/2.2.3 OpenSSL/0.9.8c
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Your iPhone version indicates keep-alive but doesn't specify a length. Not sure that's enough to cause trouble.
Also, is it possible your server is checking for user-agent strings it recognizes (say, for backward-compatibility mode)?
I'd also compare the two in a text editor that shows CR/LF characters to make sure you're getting proper line endings.
Another thing you could try is create a simple web-page that does a multipart POST and run it from the iPhone browser (instead of the Mac one) then check the headers that go across the wire. Or you could snag a toolkit like ASIHTTPRequest and see what kind of output it generates for multi-part posts (or just use the toolkit instead of trying to write your own).
Good luck
Solved by reading RFC 2046 (MIME specification): boundary between parts of multipart message should contain two leading '-'s, and last boundary should additionally contain two trailing '-'s. The boundary in the request header and request body in the Firefox request differ:
---------------------------20072377098235644401115438165
and
-----------------------------20072377098235644401115438165
The last boundary looks like this:
-----------------------------20072377098235644401115438165--
You really could not see this with the eye when there are so many leading '-'s in the original boundary.