When ever the mirc radio bot announces the song playing and the ammount of listeners a stray html and body tag show up as seen below.
The code for the radio bot is as follows
#announcer on
ctcp *:*:*:{
if ($1 == SRstats) {
set %sctat.chan $chan
sockclose scstat
sockopen scstat 149.202.90.221 8132
}
}
on *:SOCKOPEN:scstat:{
sockwrite -n $sockname GET /7.html HTTP/1.0
sockwrite -n $sockname User-Agent: Mozilla
sockwrite -n $sockname $crlf
}
on *:sockread:scstat:{
if ($sockerr > 0) return
:nextread
sockread -f %scasttemp
if ($sockbr == 0) return
if (%scasttemp == $null) %scasttemp = empty
set %scasttemp $remove(%scasttemp,<html><head><meta http-equiv="Pragma" content="no-cache"></head><body>,</body></html>)
if ((HTTP/1.* !iswm %scasttemp) && (content-type* !iswm %scasttemp) && (%scasttemp != empty)) {
set %scstat.song.temp $gettok(%scasttemp,7-,44)
set %scstat.bitrate $gettok(%scasttemp,6,44)
set %scstat.listeners $gettok(%scasttemp,1,44)
set %scstat.maxlist $gettok(%scasttemp,4,44)
set %scstat.peak $gettok(%scasttemp,3,44)
if ($gettok(%scasttemp,2,44) == 1) set %scstat.livedj connected
else set %scstat.livedj not connected
; changing some of the html codes back to regular characters
set %scast.song $replace(%scast.song,&,$chr(38),',$chr(39))
}
goto nextread
}
on *:sockclose:scstat:{
if (( %scstat.song.temp == %scstat.song ) || ( %scstat.song.temp == Line Recording )) { goto scstat.end }
else {
set %scstat.song %scstat.song.temp
set %song.msg 6,0 $+ %dj_nick is playing 6 : 12 %scstat.song $+ . 0,1 Tune into Radio-Airwaves, type !radiohelp/4 %scstat.listeners $+ --listeners are tuned in.
; set %chans $chan(0)
; while %chans {
/scid -a msg #Radio-Airwaves-Lounge %song.msg
; dec %chans
; }
:scstat.end
}
}
on *:TEXT:!playing:#: msg $chan %song.msg
#announcer end
I thought the first fix should be changing the , between the body tags to html number code but that just displayed the number code and not the actually comma. I also maybe there were mismatched tags/stray tags so I check for that. I didn't find any. I've yet to see the cause of the stray tags appearing when the accouner is on. Any help would be greatly appericated.
The line your trying to extract the information from, <html><body> at the start and end of the retrieved text.
You can solve this by setting %scasttemp using several techniques.
Strip the Html tags using $nohtml a like script. - Recomended
Staticly Substring from $right(%text, -12)
Dynamically find the first occurrence after body**>** and substring the rest of the text.
Use Regular Expressions
And many more..
$nohtml
alias nohtml { var %x,%y = $regsub($1-,/(<[^>]+>)/g,$null,%x) | return %x }
Additionally, when handling sockread i would use Tokenize to handle $1.. identifier instead of tokens.
if (!$sockbr || !%scasttemp) {
return
}
tokenize 32 $nohtml(%scasttemp)
;;; Comment out the below line if you still want to use the old variable, otherwise you should change the rest of the code.
;;;set %scasttemp $1-
;;; Identify the data we wish to extract the information, else return.
if ($numtok($1-, 44) < 7) {
return
}
;;; Rest of the code here..
Sending to server header request suggesting to close the connection after the information received is a good practice.
sockwrite -n $sockname Connection: close
Also adding the sockclose after all information received is a good convention instead of letting the sockets to hang around. (If Connection: close was not requested)
goto nextread
sockclose $sockname
Related
We are using Magento 2 with Varnish cache
We only get Varnish Cache HIT on very few /catalogsearch/result/ pages, and we really can not figure out why we don't get cache HIT on all /catalogsearch/result/ pages.
Please help us in the right direction :-)
Ex.
we always get HIT on this url
https://www.babygear.dk/catalogsearch/result/?q=bog
We always get MISS on a lot of other search queries
https://www.babygear.dk/catalogsearch/result/?q=black
https://www.babygear.dk/catalogsearch/result/?q=box
https://www.babygear.dk/catalogsearch/result/?q=box
Here is our varnish.vlc
# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 5
vcl 4.0;
import std;
# The minimal Varnish version is 5.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'
backend default {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 600s;
}
acl purge {
"127.0.0.1";
}
sub vcl_recv {
if (req.method == "PURGE") {
if (client.ip !~ purge) {
return (synth(405, "Method not allowed"));
}
# To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
# has been added to the response in your backend server config. This is used, for example, by the
# capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
}
if (req.http.X-Magento-Tags-Pattern) {
ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
}
if (req.http.X-Pool) {
ban("obj.http.X-Pool ~ " + req.http.X-Pool);
}
return (synth(200, "Purged"));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
# Bypass shopping cart, checkout
if (req.url ~ "/checkout") {
return (pass);
}
# Bypass health check requests
if (req.url ~ "/pub/health_check.php") {
return (pass);
}
# Set initial grace period usage status
set req.http.grace = "none";
# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");
# collect all cookies
std.collect(req.http.Cookie);
# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
# No point in compressing these
unset req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
set req.http.Accept-Encoding = "deflate";
} else {
# unknown algorithm
unset req.http.Accept-Encoding;
}
}
# Remove all marketing get parameters to minimize the cache objects
if (req.url ~ "(\?|&)(gclid|ff|fp|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
set req.url = regsuball(req.url, "(gclid|ff|fp|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
set req.url = regsub(req.url, "[?|&]+$", "");
}
# Static files caching
if (req.url ~ "^/(pub/)?(media|static)/") {
# Static files should not be cached by default
#return (pass);
# But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
unset req.http.Https;
unset req.http.X-Forwarded-Proto;
unset req.http.Cookie;
}
return (hash);
}
sub vcl_hash {
if (req.http.cookie ~ "X-Magento-Vary=") {
hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}
# For multi site configurations to not cache each other's content
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
hash_data(req.http.X-Forwarded-Proto);
}
if (req.url ~ "/graphql") {
call process_graphql_headers;
}
}
sub process_graphql_headers {
if (req.http.Store) {
hash_data(req.http.Store);
}
if (req.http.Content-Currency) {
hash_data(req.http.Content-Currency);
}
}
sub vcl_backend_response {
set beresp.grace = 3d;
if (beresp.http.content-type ~ "text") {
set beresp.do_esi = true;
}
if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
set beresp.do_gzip = true;
}
if (beresp.http.X-Magento-Debug) {
set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}
# cache only successfully responses
if (beresp.status != 200) {
set beresp.ttl = 0s;
set beresp.uncacheable = true;
return (deliver);
} elsif (beresp.http.Cache-Control ~ "private") {
set beresp.uncacheable = true;
set beresp.ttl = 44400s;
return (deliver);
}
# validate if we need to cache it and prevent from setting cookie
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
unset beresp.http.set-cookie;
}
# If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
if (beresp.ttl <= 0s ||
beresp.http.Surrogate-control ~ "no-store" ||
(!beresp.http.Surrogate-Control &&
beresp.http.Cache-Control ~ "no-cache|no-store") ||
beresp.http.Vary == "*") {
# Mark as Hit-For-Pass for the next 2 minutes
set beresp.ttl = 120s;
set beresp.uncacheable = true;
}
return (deliver);
}
sub vcl_deliver {
if (resp.http.X-Magento-Debug) {
if (resp.http.x-varnish ~ " ") {
set resp.http.X-Magento-Cache-Debug = "HIT";
set resp.http.Grace = req.http.grace;
} else {
set resp.http.X-Magento-Cache-Debug = "MISS";
}
} #else {
# unset resp.http.Age;
# }
# Not letting browser to cache non-static files.
if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
set resp.http.Pragma = "no-cache";
set resp.http.Expires = "-1";
set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
}
unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;
}
sub vcl_hit {
if (obj.ttl >= 0s) {
# Hit within TTL period
return (deliver);
}
if (std.healthy(req.backend_hint)) {
if (obj.ttl + 3000000s > 0s) {
# Hit after TTL expiration, but within grace period
set req.http.grace = "normal (healthy server)";
return (deliver);
} else {
# Hit after TTL and grace expiration
return (miss);
}
} else {
# server is not healthy, retrieve from cache
set req.http.grace = "unlimited (unhealthy server)";
return (deliver);
}
}
It's a bit tough to judge what's really going on, because there's both a Varnish cache in front of Magento and Cloudflare as the CDN.
A no-cache/no-store Cache-Control value
What I am seeing in general for your searches, is the following Cache-Control value:
cache-control: no-store, no-cache, must-revalidate, max-age=0
Based on this value, Varnish will decide not to cache. In the response headers of must of your search results you will see that Age: 0 is set. This means that Varnish doesn't hold the value in cache.
A cached search result that shouldn't be cacheable
However, weirdly enough https://www.babygear.dk/catalogsearch/result/?q=bog does have a an Age header with a value greater than zero:
age: 38948
This means it's been in cache for 38948 seconds. But really, this shouldn't be happening, because the page is not supposed to be cacheable.
Making search results cacheable:
Please make sure you have a Cache-Control header that allows caching, if you want to cache search results.
Example:
Cache-Control: public, s-maxage=3600
Debugging using Varnishlog
If you really want to know what happens in behind the scenes in Varnish, you can perform some debugging using the varnishlog binary.
You could run the following command to get debug output:
varnishlog -g request -q "ReqUrl eq '/catalogsearch/result/\?q=bog'"
This will print some very verbose logs on how Varnish treats the URL that causes the hit. You can add this output to your question, and I can try to examine what's going on.
FYI: I wrote a very detailed blog post about varnishlog a couple of years ago. Please go to https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/ to have a look, and to learn.
What's Cloudflare doing?
All that being said, I have no clue what the impact of Cloudflare is on the cacheability of the website. The varnishlog output will give us some insight, but if those results diverge from reality, Cloudflare is probably getting in our way.
Keep this in mind while debugging.
Inside vcl_recv Add
# Bypass search requests
if (req.url ~ "/catalogsearch") {
return (pass);
}
check .htaccess file for Cache-Control setting.
I've searched through the various questions and tried different permutations of curl commands but yet to find one the works so posting the question as I'm likely missing something obvious and cannot see it for looking.
I'm running a curl command to try and upload a file for parsing.
S-MBP:project SG$ curl -i -X POST -F "data=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" 'http://localhost:3030/upload/'
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Access-Control-Allow-Headers: accept, content-type
Access-Control-Allow-Methods: POST
Access-Control-Allow-Origin: *
Date: Sat, 25 Jul 2020 14:56:05 GMT
Content-Length: 19
Content-Type: text/plain; charset=utf-8
http: no such file
Some of the permutations tried based on previous answers:
curl -i -X POST -F "filename=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" http://localhost:3030/upload/
curl -i -X POST -F "filename=#mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" http://localhost:3030/upload
curl -i -X POST -F filename=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz http://localhost:3030/upload/
curl -i -X POST -F "filename=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" "http://localhost:3030/upload/"
Interestingly, if I pass in the name of a file that doesn't exist I get the same error but if I change the name of the directory to one that doesn't exist the error is a curl(26) making me think that the command couldn't care less about the file at the moment. I am running this on a mac if that's of nay relevance, I saw a post that implied there may be an issue with brew curl.
The form that I'm trying to target is part of the docker image https://hub.docker.com/repository/docker/simagix/maobi
The form with some values omitted
<form action="/upload" id="maobi" class="dropzone dz-clickable dz-started">
<div class="dz-message">
Drop files here or click to upload.<br>
</div>
<div class="dz-filename"><span data-dz-name="">mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz</span></div></div></form>
and then the script I see in the page that I believe is used to upload and then parse the document generating the output.
<script>
Dropzone.options.maobi = {
timeout: 300000,
init: function() {
this.on("success", function(file, responseText) {
blob = new Blob([responseText], { type: 'text/html' }),
anchor = document.createElement('a');
filename = file.upload.filename;
if ((n = filename.indexOf("mdiag-")) == 0 ) {
n = filename.lastIndexOf(".json")
filename = filename.substring(0, n) + ".html";
} else if ((n = filename.lastIndexOf(".json")) > 0 ) {
//...
//...
} else if ((n = filename.indexOf(".log")) > 0 && (n = filename.lastIndexOf(".gz")) > 0) {
filename = filename.substring(0, n) + ".html";
} else if ((n = filename.lastIndexOf(".bson")) > 0 ) {
filename = filename.substring(0, n) + "-cluster.html";
}
anchor.download = filename;
anchor.href = (window.webkitURL || window.URL).createObjectURL(blob);
anchor.dataset.downloadurl = ['text/html', anchor.download, anchor.href].join(':');
anchor.click();
});
this.on("error", function(file, responseText) {
alert(responseText);
});
}
};
</script>
It seems to me that you are not passing the file in the right form field.
From looking at the Dropzone.js documentation, it seems the right field name is file (since that's the default for the paramName configuration), not data or filename. But to be entirely sure, it'd be best to look at the network request in your browser's devtools and see what POST field name is used there for passing the file.
curl -i -X POST -F "file=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" 'http://localhost:3030/upload/'
I am trying to setup token based authentication on Fastly CDN with Varnish VCL and using this sample VCL snippet to generate and validate JWT tokens here -
sub vcl_recv {
#FASTLY recv
if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") {
return(pass);
}
// Generate synth
if(req.url ~ "generate") {
error 901;
}
// Validate token
if(req.url ~ "validate") {
// Ensure token exists and parse into regex
if (req.http.X-JWT !~ "^([a-zA-Z0-9\-_]+)?\.([a-zA-Z0-9\-_]+)?\.([a-zA-Z0-9\-_]+)?$") {
// Forbidden
error 403 "Forbidden";
}
// Extract token header, payload and signature
set req.http.X-JWT-Header = re.group.1;
set req.http.X-JWT-Payload = re.group.2;
set req.http.X-JWT-Signature = digest.base64url_nopad_decode(re.group.3);
set req.http.X-JWT-Valid-Signature = digest.hmac_sha256("SupSecretStr",
req.http.X-JWT-Header "." req.http.X-JWT-Payload);
// Validate signature
if(digest.secure_is_equal(req.http.X-JWT-Signature, req.http.X-JWT-Valid-Signature)) {
// Decode payload
set req.http.X-JWT-Payload = digest.base64url_nopad_decode(req.http.X-JWT-Payload);
set req.http.X-JWT-Expires = regsub(req.http.X-JWT-Payload, {"^.*?"exp"\s*?:\s*?([0-9]+).*?$"}, "\1");
// Validate expiration
if (time.is_after(now, std.integer2time(std.atoi(req.http.X-JWT-Expires)))) {
// Unauthorized
synthetic {"{"sign":""} req.http.X-JWT-Signature {"","header":""} req.http.X-JWT-Header {"","payload":""} req.http.X-JWT-Payload {"","valid": ""} req.http.X-JWT-Valid-Signature {""}"};
return(deliver);
}
// OK
synthetic {"{"header2":""} req.http.X-JWT-Header {"","payload":""} req.http.X-JWT-Payload {"","sign":""} req.http.X-JWT-Signature {"","valid": ""} req.http.X-JWT-Valid-Signature {""}"};
return(deliver);
} else {
// Forbidden
synthetic {"{"header3":""} req.http.X-JWT-Header {"","payload":""} req.http.X-JWT-Payload {"","sign":""} req.http.X-JWT-Signature {"","valid": ""} req.http.X-JWT-Valid-Signature {""}"};
return(deliver);
}
}
return(lookup);
}
sub vcl_error {
#FASTLY error
// Generate JWT token
if (obj.status == 901) {
set obj.status = 200;
set obj.response = "OK";
set obj.http.Content-Type = "application/json";
set obj.http.X-UUID = randomstr(8, "0123456789abcdef") "-" randomstr(4, "0123456789abcdef") "-4" randomstr(3, "0123456789abcdef") "-" randomstr(1, "89ab") randomstr(3, "0123456789abcdef") "-" randomstr(12, "0123456789abcdef");
set obj.http.X-JWT-Issued = now.sec;
set obj.http.X-JWT-Expires = strftime({"%s"}, time.add(now, 3600s));
set obj.http.X-JWT-Header = digest.base64url_nopad({"{"alg":"HS256","typ":"JWT""}{"}"});
set obj.http.X-JWT-Payload = digest.base64url_nopad({"{"sub":""} obj.http.X-UUID {"","exp":"} obj.http.X-JWT-Expires {","iat":"} obj.http.X-JWT-Issued {","iss":"Fastly""}{"}"});
set obj.http.X-JWT-Signature = digest.base64url_nopad(digest.hmac_sha256("SupSecretStr", obj.http.X-JWT-Header "." obj.http.X-JWT-Payload));
set obj.http.X-JWT = obj.http.X-JWT-Header "." obj.http.X-JWT-Payload "." obj.http.X-JWT-Signature;
unset obj.http.X-UUID;
unset obj.http.X-JWT-Issued;
unset obj.http.X-JWT-Expires;
unset obj.http.X-JWT-Header;
unset obj.http.X-JWT-payload;
unset obj.http.X-JWT-Signature;
synthetic {"{"payload":""} obj.http.X-JWT-Payload {"","header":""} obj.http.X-JWT-Header {"","sign":""} obj.http.X-JWT-Signatre {"","token": ""} obj.http.X-JWT {""}"};
return(deliver);
}
// Valid token
if (obj.status == 902) {
set obj.status = 200;
set obj.response = "OK";
set obj.http.Content-Type = "application/json";
synthetic {"{ "token": ""} req.http.X-JWT {"" }"};
return(deliver);
}
}
Now, when I am trying to compile this it returns -
Syntax error: Unterminated _short-string_
at: (input Line 106 Pos 197)
synthetic {"{"sign":""} req.http.X-JWT-Signature {"","header":""} req.http.X-JWT-Header {"","payload":""} req.http.X-JWT-Payload {"","valid": ""} req.http.X-JWT-Valid-Signature {""}"};
Looks like i am not somehow escaping the values correctly here during the synthetic block.
The only reason I am trying to do add this synthetic block in the vcl_recv subroutine is because I want to test how the digest is generating the JWT token and validating it and with that I wanted to create similar JWT tokens on server side in Node.Js so i was trying to output the different intermediate parts of the token for debugging.
I am not quite familiar with the Varnish syntax and semantics but still I looked for help finding any docs regarding this schedule subroutine but found none so far.
So, can anyone help out in how to fix this and have the vcl_recv, vcl_error interpolate different intermediate values in the json response.
I tried to use some of the Node.Js based base64 url decode libraries to decode the returned token parts and was able to decode the header and payload part but the signature part I am not able to generate from the Node.Js. So, can anyone suggest what is the equivalent of base64url_nopad() in node.js or any javascript libraries ?
For the hmac_256 encryption part we are trying to use the crypto library and creating an hmac like crypto.createHmac('sha256', 'SupSecretStr').update().digest('hex'); but all base64 encode url libraries in js i think return padded urls which is why the base64 encoded part of this hmac 256 digest doesn't match the one generated with varnish
My syntax coloring tool is telling me pretty much the same thing as the error message: you screwed up your quotes :-)
Your last block {""}"}; is opening quotes ({"), immediately closing them ("}), then you are opening simple quotes " and the newline arrives before you close them.
To fix, just put a space between after the final quote of the json: {"" }"};
I'm developping a streaming application with tcl.
I have a vlc sever that broadcast a flow in http mode. what I'm trying to do is to develop a client who will try to connect to server with a particular ip adress and port number, and then try to save the flow in a file.
the code that i'm using is simple:
set server localhost
set sockChan [socket $server 1234]
set line [read $sockChan 1000]
puts " vidéo: $line"
close $sockChan
the problem when i try to test my script, I see that I realise the connection, but I can't read the flow!
the 'puts' doesn't show anything in the console...
Have you any ideas!
thank you..
If you're just wanting to save the contents of a URL to a file, the standard http package has a -channel option which lets you dump directly. For example:
package require http
set f [open video.dump w]
fconfigure $f -translation binary
set tok [http::geturl "http://server:port/url" -channel $f]
close $f
if {[http::ncode $tok] != 200} {
# failed somehow...
} else {
# succeeded
}
http::cleanup $tok
Edit: Doing it asynchronously (requires the event loop going, e.g. via vwait forever):
package require http
set f [open video.dump w]
fconfigure $f -translation binary
proc done {f tok} {
close $f
if {[http::ncode $tok] != 200} {
# failed somehow...
} else {
# succeeded
}
http::cleanup $tok
}
http::geturl "http://server:port/url" -channel $f -command "done $f"
# Your code runs here straight away...
Note that the code's recognizably similar, but now in a slightly different order! If you've got Tcl 8.5 — if not, why not? — then you can use a lambda application instead to make the apparent order of the code even more similar:
package require http
set f [open video.dump w]
fconfigure $f -translation binary
http::geturl "http://server:port/url" -channel $f -command [list apply {{f tok} {
close $f
if {[http::ncode $tok] != 200} {
# failed somehow...
} else {
# succeeded
}
http::cleanup $tok
}} $f]
# Your code runs here straight away...
Since you are working with HTTP, I would suggest looking at libcurl bindings for TCL.
Below is partial code to an experimental http server app I'm building from scratch from a PHP CLI script (Why? Because I have too much time on my hands). The example below more closely matches PHP's manual page on this function. The problem I'm getting is when connecting to this server app via a browser (Firefox or IE8 from two separate systems tested so far), the browser sends an empty request payload to the server and aborts roughly every 1 in 6 page loads.
The server console displays the "Connected with [client info]" each time. However, about 1 in 6 connections will result in a "Client request is empty" error. No error is given telling the header/body response write to the socket failed. The browser will generally continue to read what I give it, but this isn't usable as I can't fulfill the client's intended request without knowing what it is.
<?php
$s_socket_uri = 'tcp://localhost:80';
// establish the server on the above socket
$s_socket = stream_socket_server($s_socket_uri, $errno, $errstr, 30) OR
trigger_error("Failed to create socket: $s_socket_uri, Err($errno) $errstr", E_USER_ERROR);
$s_name = stream_socket_get_name($s_socket, false) OR
trigger_error("Server established, yet has no name. Fail!", E_USER_ERROR);
if (!$s_socket || !$s_name) {return false;}
/*
Wait for connections, handle one client request at a time
Though to not clog up the tubes, maybe a process fork is
needed to handle each connection?
*/
while($conn = stream_socket_accept($s_socket, 60, $peer)) {
stream_set_blocking($conn, 0);
// Get the client's request headers, and all POSTed values if any
echo "Connected with $peer. Request info...\n";
$client_request = stream_get_contents($conn);
if (!$client_request) {
trigger_error("Client request is empty!");
}
echo $client_request."\n\n"; // just for debugging
/*
<Insert request handling and logging code here>
*/
// Build headers to send to client
$send_headers = "HTTP/1.0 200 OK\n"
."Server: mine\n"
."Content-Type: text/html\n"
."\n";
// Build the page for client view
$send_body = "<h1>hello world</h1>";
// Make sure the communication is still active
if ((int) fwrite($conn, $send_headers . $send_body) < 1) {
trigger_error("Write to socket failed!");
}
// Response headers and body sent, time to end this connection
stream_socket_shutdown($conn, STREAM_SHUT_WR);
}
?>
Any solution to bring down the number of unintended aborts down to 0, or any method to get more stable communication going? Is this solvable on my server's end, or just typical browser behavior?
I tested your code and it seems I got better results reading the socket with fread(). You also forgot the main loop(while(1), while(true) or for(;;).
Modifications to your code:
stream_socket_accept with #stream_socket_accept [sometimes you get warnings because "the connected party did not properly respond", which is, of course, the timeout of stream_socket_accept()]
Added the big while(1) { } loop
Changed the reading from the socket from $client_request = stream_get_contents($conn);
to while( !preg_match('/\r?\n\r?\n/', $client_request) ) { $client_request .= fread($conn, 1024); }
Check the source code below (I used 8080 port because I already had an Apache listening on 80):
<?php
$s_socket_uri = 'tcp://localhost:8080';
$s_socket = stream_socket_server($s_socket_uri, $errno, $errstr, 30) OR
trigger_error("Failed to create socket: $s_socket_uri, Err($errno) $errstr", E_USER_ERROR);
$s_name = stream_socket_get_name($s_socket, false) OR
trigger_error("Server established, yet has no name. Fail!", E_USER_ERROR);
if (!$s_socket || !$s_name) {return false;}
while(1)
{
while($conn = #stream_socket_accept($s_socket, 60, $peer))
{
stream_set_blocking($conn, 0);
echo "Connected with $peer. Request info...\n";
// $client_request = stream_get_contents($conn);
$client_request = "";
// Read until double \r
while( !preg_match('/\r?\n\r?\n/', $client_request) )
{
$client_request .= fread($conn, 1024);
}
if (!$client_request)
{
trigger_error("Client request is empty!");
}
echo $client_request."\n\n";
$headers = "HTTP/1.0 200 OK\n"
."Server: mine\n"
."Content-Type: text/html\n"
."\n";
$body = "<h1>hello world</h1><br><br>".$client_request;
if ((int) fwrite($conn, $headers . $body) < 1) {
trigger_error("Write to socket failed!");
}
stream_socket_shutdown($conn, STREAM_SHUT_WR);
}
}
Add sleep(1) after stream_set_blocking