Foswiki plugin on Perl/FCGI failed to use File:Find at the 5th attempt - perl

I am writing a REST plugin for Foswiki using Perl and I am facing an reliability issue when using File::Find. I have tried my best to write a minimal reproducible example. The plugin uses File::Find to traverse directories and print the filenames in the HTTP response. The REST request is working properly 4 times, but stop to work the 5th time. The HTTP status remain “HTTP/1.1 200 OK” but no file is reported by File::Find anymore.
The webserver is nginx and is configured to use FastCGI. It appear to run 4 working threads managed by foswiki-fcgi-pm:
> ps aux
www-data 16957 0.0 7.7 83412 78332 ? Ss 16:52 0:00 foswiki-fcgi-pm
www-data 16960 0.0 7.5 83960 76740 ? S 16:52 0:00 foswiki-fcgi
www-data 16961 0.0 7.6 84004 76828 ? S 16:52 0:00 foswiki-fcgi
www-data 16962 0.0 7.6 83956 76844 ? S 16:52 0:00 foswiki-fcgi
www-data 16963 0.0 7.5 83960 76740 ? S 16:52 0:00 foswiki-fcgi
Firstly, the plugin initialization simply register the REST handler:
sub initPlugin {
my ( $topic, $web, $user, $installWeb ) = #_;
# check for Plugins.pm versions
if ( $Foswiki::Plugins::VERSION < 2.3 ) {
Foswiki::Func::writeWarning( 'Version mismatch between ',
__PACKAGE__, ' and Plugins.pm' );
return 0;
}
Foswiki::Func::registerRESTHandler(
'restbug', \&RestBug,
authenticate => 0, # Set to 0 if handler should be useable by WikiGuest
validate => 0, # Set to 0 to disable StrikeOne CSRF protection
http_allow => 'GET,POST', # Set to 'GET,POST' to allow use HTTP GET and POST
description => 'Debug'
);
# Plugin correctly initialized
return 1;
}
Secondly, the REST handler is implemented as follow, printing all the files it can possibly find:
sub RestBug {
my ($session, $subject, $verb, $response) = #_;
my #Directories = ("/var/www/foswiki/tools");
sub findfilestest
{
$response->print("FILE $_\n");
}
find({ wanted => \&findfilestest }, #Directories );
}
When I test the REST service with a HTTP request, the first 4 times I get the following HTTP response, which seems quite satisfying:
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Tue, 22 Nov 2022 09:23:10 GMT
Content-Length: 541
Connection: keep-alive
Set-Cookie: SFOSWIKISID=385db599c5d66bb19591e1eef7f1a854; path=/; secure; HttpOnly
FILE .
FILE foswiki.freebsd.init-script
FILE bulk_copy.pl
FILE dependencies
FILE mod_perl_startup.pl
FILE geturl.pl
FILE extender.pl
FILE extension_installer
FILE configure
FILE lighttpd.pl
FILE foswiki.freebsd.etc-defaults
FILE save-pending-checkins
FILE babelify
FILE upgrade_emails.pl
FILE tick_foswiki.pl
FILE foswiki.defaults
FILE rewriteshebang.pl
FILE fix_file_permissions.sh
FILE foswiki.init-script
FILE convertTopicSettings.pl
FILE mailnotify
FILE html2tml.pl
FILE tml2html.pl
FILE systemd
FILE foswiki.service
The following attempts give this unexpected response:
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Tue, 22 Nov 2022 09:24:56 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: SFOSWIKISID=724b2c4b1ddfbebd25d0dc2a0f182142; path=/; secure; HttpOnly
Note that if I restart Foswiki with the command systemctl restart foswiki, the REST service work again 4 more times.
How to make this REST service work more than 4 times in a row?

Related

Does openstack4j support set header when upload image?

I have some problem about OpenStack swift object storage. I want to set an expiration for objects. I use openstack4j. My upload object code
public void add(String objectName, InputStream imageStream) {
OSClientV3 clientV3 = OSFactory.clientFromToken(swiftOS.getToken());
Map<String, String> metaData = new HashMap<>();
metaData.put("X-Delete-After", "120");
ObjectPutOptions objectPutOptions = ObjectPutOptions.create();
objectPutOptions.metadata(metaData);
clientV3.objectStorage().objects().put(container, objectName,
Payloads.create(imageStream), objectPutOptions);
}
But it doesn't work.
Then I tried to use swift command line.
swift stat test test-19b8e4d4-1085-490d-b866-97f0ada0d46c
What I get is
Account: AUTH_01d73f1e49ed4dfd9535c831eed4ccf9
Container: test
Object: test-19b8e4d4-1085-490d-b866-97f0ada0d46c
Content Type: application/octet-stream
Content Length: 2414
Last Modified: Wed, 20 Dec 2017 07:58:56 GMT
ETag: 1cb55838010ed189c0698b6b5cade3ed
Meta X-Delete-After: 120
X-Openstack-Request-Id: tx4f1f57ef08e34d9296bfd-005a3a184b
X-Timestamp: 1513756735.97761
X-Trans-Id: tx4f1f57ef08e34d9296bfd-005a3a184b
Accept-Ranges: bytes
When I upload an object by
swift upload test test.jpg -H "X-Delete-After: 120"
and then
swift stat test test.jpg
What I get is
Account: AUTH_01d73f1e49ed4dfd9535c831eed4ccf9
Container: test
Object: test.jpg
Content Type: application/octet-stream
Content Length: 1688
Last Modified: Wed, 20 Dec 2017 08:03:20 GMT
ETag: 8a2d75ff8db40610a52a492abac09d3b
Meta Mtime: 1513755398.217256
X-Delete-At: 1513757119
Accept-Ranges: bytes
X-Timestamp: 1513756999.02865
X-Trans-Id: txc016e1aff901450aa934b-005a3a194c
X-Openstack-Request-Id: txc016e1aff901450aa934b-005a3a194c
It is like openstack document said.
The X-Delete-After header takes an integer number of seconds.
The proxy server that receives the request will convert this
header into an X-Delete-At header using its current time plus
the value given.
But why openstack4j doesn't work?
Looks like we can use
objectPutOptions.getOptions().put("X-Delete-After", "120");
As the getOptions call returns the headers map straight up.
You should use header instead of metadata:
objectPutOptions.header("X-Delete-After", "120");

Perl Fastcgi - HTTP::Message content must be bytes at

I am getting the following errors:
[Fri Feb 19 08:55:41 2016] [error] [client 172.16.30.100] FastCGI:
server "/home/admin/soap/server.fcgi" stderr: HTTP::Message content
must be bytes at
/usr/local/lib/perl5/site_perl/5.8.9/SOAP/Transport/HTTP.pm line 317
[Fri Feb 19 08:55:41 2016] [error] [client 172.16.30.100] FastCGI:
incomplete headers (0 bytes) received from server
"/home/admin/soap/server.fcgi" [Fri Feb 19 08:55:41 2016] [warn]
FastCGI: server "/home/admin/soap/server.fcgi" (pid 45466) terminated
by calling exit with status '255'
Here is line 317 of HTTP.pm I appreciate any input on a fix.
$self->response(HTTP::Response->new(
$code => undef,
HTTP::Headers->new(
'SOAPServer' => $self->product_tokens,
$compressed ? (
'Content-Encoding' => $COMPRESS) : (),
'Content-Type' => join(
'; ',
'text/xml',
!$SOAP::Constants::DO_NOT_USE_CHARSET && $encoding ? 'charset=' . lc($encoding) : ()
),
'Content-Length' => SOAP::Utils::bytelength $response),
$response,
));
$self->response->headers->header('Content-Type' => 'Multipart/Related; type="text/xml"; start="<main_envelope>"; boundary="'.$is_multipart.'"') if $is_multipart; }
Error message is pretty much telling you all about it: content for HTTP::Response object that you provide with $response variable must be bytes - i.e. a buffer without any characters outside 0-255 range. Use Encode::encode to transform it from Perl internal representation to whatever encoding required for your protocol.
By the way, this has nothing to do with fastcgi. The first error cause the script to die without emitting a response, resulting in the second error message.

How to format the HTTP response

I have written a socket program in C. I used this program as a chat server/client using TCP. I tried to change the chat server to use it as a HTTP server by changing the port to 80. I referred to the HTTP request/response format in http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Example_session , and made my program to reply with sample response. I tried the url
http://127.0.0.1/
in browser. My program read the request and replied with response. At first, I used google-chrome. Chrome didn't load the page correctly until i added the correct data length in Content-Length header. After setting content length header, chrome loaded the page correctly. But, firefox doesn't load the page. Firefox doesn't showed any errors, but still loading the page like it is still waiting for some data. Only When i stop the server or close the socket, complete page is loaded. I tried to follow rfc2616 https://www.rfc-editor.org/rfc/rfc2616 , and made the response exactly , but the still the results are same.
Request:
GET / HTTP/1.1\r\nHost: 127.0.0.1:8080\r\nUser-Agent: Mozilla/5.0
(X11; Ubuntu; Linux x86_64; rv:33.0) Gecko/20100101
Firefox/33.0\r\nAccept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8\r\nAccept-Language:
en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection:
keep-alive\r\n\r\n
For the above request, my program write to the socket with following response & content.
Response:
HTTP/1.1 200 OK\r\nCache-Control : no-cache, private\r\nContent-Length
: 107\r\nDate : Mon, 24 Nov 2014 10:21:21 GMT\r\n\r\n
Content:
<html><head><title></title></head><body>TIME : 1416824843 <br>DATE: Mon Nov 24 15:57:23 2014 </body></html>
This response is loading in Chrome, but not in firefox. Chrome is loading the page instantly whereas firefox is waiting for data. Note that the data length 107 is specified in the header. I donot have any addons enabled in firefox. My firefox version is in the request. Chrome version: Version 38.0.2125.111 (64-bit).
Code:
void *socket_read(void *args)
{
int socket,*s,length;
char buf[1024];
s=(int *)args;
socket=*s;
while(1){
buf[0]='\0';
length=read(socket,buf,1024);
if(length==0)
break;
else if(length==-1){
perror("Read");
return;
}
else{
printf("Request: %s\n",buf);
send_response(socket);
}
}
printf("End of read thread [%d]\n",socket);
}
int start_accept(int port)
{
int socket,csocket;
pthread_t thread;
struct sockaddr_in client;
socklen_t addrlen=sizeof(client);
pthread_attr_t attr;
socket=create_socket(port);
while(1){
if((csocket=accept(socket,(struct sockaddr *)&client,&addrlen))==-1)
{
perror("Accept");
break;
}
pthread_attr_init(&attr);
if(0!=pthread_create(&thread,&attr,socket_read,&csocket))
{
perror("Read thread");
return;
}
usleep(10000);
}
}
void send_response(int socket)
{
char buf1[1024];
int content_length;
char buf2[1024]="<html><head><title></title></head><body>TIME : 1416824843 <br>DATE: Mon Nov 24 15:57:23 2014 </body></html>";
content_length=strlen(buf2);
sprintf(buf1,"HTTP/1.1 200 OK\r\nCache-Control : no-cache, private\r\nContent-Length : %d\r\nDate : Mon, 24 Nov 2014 12:03:43 GMT\r\n\r\n",content_length);
printf("Written: %d \n",write(socket,buf1,strlen(buf1)));
fflush(stdout);
printf("Written: %d \n",write(socket,buf2,content_length));
fflush(stdout);
}
I have found the problem.
The Response is incorrect. There should not be any spaces between the header field name and colon(':'). Found this in http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 .
My correct response is
HTTP/1.1 200 OK\r\nCache-Control: no-cache, private\r\nContent-Length: 107\r\nDate: Mon, 24 Nov 2014 10:21:21 GMT\r\n\r\n
I had put a space between 'Content-Length' and ':' . That's the reason Firefox ignored the content length header and reading the socket. Chrome accepted the header fields with spaces, so the problem didn't occurred in chrome.
After removing the space, program works fine.
It actually loads the page. If you add content-type header you will see the HTML page (Content-Type: text/html; charset=UTF-8\r\n)
Anyway, both in Chrome and in Firefox you will see the connections never stops because the server doesn't close the socket. If you closed csocket, you would see the HTML page in both browsers but as you said it should be a persistent connection.

500 internal server error on certain page after a few hours

I am getting a 500 Internal Server Error on a certain page of my site after a few hours of being up. I restart uWSGI instance with uwsgi --ini /home/metheuser/webapps/ers_portal/ers_portal_uwsgi.ini and it works again for a few hours.
The rest of the site seems to be working. When I navigate to my_table, I am directed to the login page. But, I get the 500 error on my table page on login. I followed the instructions here to set up my nginx and uwsgi configs.
That is, I have ers_portal_nginx.conf located i my app folder that is symlinked to /etc/nginx/conf.d/. I start my uWSGI "instance" (not sure what exactly to call it) in a Screen instance as mentioned above, with the .ini file located in my app folder
My ers_portal_nginx.conf:
server {
listen 80;
server_name www.mydomain.com;
location / { try_files $uri #app; }
location #app {
include uwsgi_params;
uwsgi_pass unix:/home/metheuser/webapps/ers_portal/run_web_uwsgi.sock;
}
}
My ers_portal_uwsgi.ini:
[uwsgi]
#user info
uid = metheuser
gid = ers_group
#application's base folder
base = /home/metheuser/webapps/ers_portal
#python module to import
app = run_web
module = %(app)
home = %(base)/ers_portal_venv
pythonpath = %(base)
#socket file's location
socket = /home/metheuser/webapps/ers_portal/%n.sock
#permissions for the socket file
chmod-socket = 666
#uwsgi varible only, does not relate to your flask application
callable = app
#location of log files
logto = /home/metheuser/webapps/ers_portal/logs/%n.log
Relevant parts of my views.py
data_modification_time = None
data = None
def reload_data():
global data_modification_time, data, sites, column_names
filename = '/home/metheuser/webapps/ers_portal/app/static/' + ec.dd_filename
mtime = os.stat(filename).st_mtime
if data_modification_time != mtime:
data_modification_time = mtime
with open(filename) as f:
data = pickle.load(f)
return data
#a bunch of authentication stuff...
#app.route('/')
#app.route('/index')
def index():
return render_template("index.html",
title = 'Main',)
#app.route('/login', methods = ['GET', 'POST'])
def login():
login stuff...
#app.route('/my_table')
#login_required
def my_table():
print 'trying to access data table...'
data = reload_data()
return render_template("my_table.html",
title = "Rundata Viewer",
sts = sites,
cn = column_names,
data = data) # dictionary of data
I installed nginx via yum as described here (yesterday)
I am using uWSGI installed in my venv via pip
I am on CentOS 6
My uwsgi log shows:
Wed Jun 11 17:20:01 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET /whm-server-status (127.0.0.1)
IOError: write error
[pid: 9586|app: 0|req: 135/135] 127.0.0.1 () {24 vars in 292 bytes} [Wed Jun 11 17:20:01 2014] GET /whm-server-status => generated 0 bytes in 3 msecs (HTTP/1.0 404) 2 headers in 0 bytes (0 switches on core 0)
When its working, the print statement in the views "my_table" route prints into the log file. But not once it stops working.
Any ideas?

Mojolicious::Lite with Template Toolkit

I'm trying to get Template Toolkit to work as the default renderer in Mojolicious::Lite. What I have:
use strict;
use warnings;
use Mojolicious::Lite;
use Mojolicious::Plugin::TtRenderer;
plugin tt_renderer => { template_options => { INCLUDE_PATH => './tmpl', DEBUG => 1 } };
get '/' => sub {
my $self = shift;
$self->render( 'index' );
};
app->renderer->default_handler( 'tt' );
app->start;
When I try to hit the test server, I get:
[Fri Oct 12 14:02:02 2012] [info] Listening at "http://*:3000".
Server available at http://127.0.0.1:3000.
[Fri Oct 12 14:02:08 2012] [debug] Your secret passphrase needs to be changed!!!
[Fri Oct 12 14:02:08 2012] [debug] GET / (Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:16.0) Gecko/20100101 Firefox/16.0).
[Fri Oct 12 14:02:08 2012] [debug] Routing to a callback.
[Fri Oct 12 14:02:08 2012] [debug] Nothing has been rendered, expecting delayed response.
This happens regardless of what I pass as parameters to 'render'. I can't seem to figure out how to get any useful debugging information out of this; but I haven't used Mojo before.
I've confirmed by sprinkling in some warn statements that my get handler is being called.
After looking at the source of Mojolicious::Plugin::TtRenderer::Engine, I figured it out. The plugin ignores the INCLUDE_PATH option passed to Template Toolkit, and instead gets the path from $app->renderer_paths. So updating my code to include:
app->renderer->default_handler( 'tt' );
app->renderer->paths( [ './tmpl' ] );
makes it work.