Unicodedecode Error when pushing to CouchDB: 'utf8' codec can't decode byte 0xe9 - encoding

I am using couchdb and couchapp on windows.
I'm working on an ongoing project of a professor https://github.com/Hypertopic/Tire-a-part. I'm currently trying to set up the app on my computer.
When I do:
couchapp push http://127.0.0.1:5984/tire-a-part
I get an error:
Traceback (most recent call last):
File "couchapp\dispatch.pyc", line 48, in dispatch
File "couchapp\dispatch.pyc", line 92, in _dispatch
File "couchapp\commands.pyc", line 79, in push
File "couchapp\localdoc.pyc", line 123, in push
File "couchapp\client.pyc", line 294, in save_doc
File "json\__init__.pyc", line 231, in dumps
File "json\encoder.pyc", line 201, in encode
File "json\encoder.pyc", line 264, in iterencode
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 1:
invalid continuation byte
My professor and my friends all have mac and don't have this problem.
After a few hours on the net trying to search for similar problems I understand that it is an encoding error but i don't understand what is not correctly encoded and what should I do.
Thanks
Edit: I have discovered the debug option of couchapp. It gives much more detail but i still don't really understand as it is my first time with couchapp and couchdb. This is the last part of the debug as i don't think the begining is important:
2018-04-14 12:42:16 [DEBUG] push spec/samples/scopus.bib
2018-04-14 12:42:16 [DEBUG] push spec/spec_helper.rb
2018-04-14 12:42:16 [DEBUG] Resource uri: http://127.0.0.1:5984/tire-a-part
2018-04-14 12:42:16 [DEBUG] Request: GET _design/Tire-a-part
2018-04-14 12:42:16 [DEBUG] Headers: {'Accept': 'application/json', 'User-
Agent': 'couchapp/0.7.5'}
2018-04-14 12:42:16 [DEBUG] Params: {}
2018-04-14 12:42:16 [DEBUG] Start to perform request: GET 127.0.0.1:5984
/tire-a-part/_design/Tire-a-part
2018-04-14 12:42:16 [DEBUG] Send headers: ['GET /tire-a-part/_design/Tire-a-
part HTTP/1.1\r\n', 'Host: 127.0.0.1:5984\r\n', 'User-Agent:
restkit/3.0.4\r\n', 'Accept-Encoding: identity\r\n', 'Accept:
application/json\r\n']
2018-04-14 12:42:16 [DEBUG] Start to parse response
2018-04-14 12:42:16 [DEBUG] Got response: 404 Object Not Found
2018-04-14 12:42:16 [DEBUG] headers: [MultiDict([('X-CouchDB-Body-
Time','0'),('X-Couch-Request-ID', '5ab9eee6cb'), ('Server', 'CouchDB/2.1.1
(Erlang OTP/18)'), ('Date', 'Sat, 14 Apr 2018 10:42:16 GMT'), ('Content-
Type','application/json'), ('Content-Length', '41'), ('Cache-Control',
'must-revalidate')])]
2018-04-14 12:42:16 [DEBUG] return response class
2018-04-14 12:42:16 [DEBUG] release connection
2018-04-14 12:42:16 [DEBUG] C:\Users\jules\Desktop\LO10 projet\Tire-a-
part\vendor don't exist
2018-04-14 12:42:16 [CRITICAL] 'utf8' codec can't decode byte 0xe9 in
position
1: invalid continuation byte
I compared this with what my friend got on mac and it is the exact same except for the [CRITICAL] line. after the 'vendordon't exist' couchapp put _design/Tire-a-part

It really seem it was an encoding error of 'é'. There were 2 files with 'é' in their name. After changing it to 'e' the push command work. The app don't work but that's a story for another day....

I don't have the answer, but I tried something: I started a Python3.5 command line, and declared a variable byte='\xe9' and then printed the variable with print(byte). As can be seen below, the 0xe9 byte looks like to be the é character:
$ python3.5
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> byte='\xe9'
>>> print(byte)
é
>>>
I'm not sure why Windows has problem with the é character but macOS works fine.
On Linux shell command-line, when I put the é character in a file and take a hex-dump of the file, I see that the é character is actually equal to c3 a9, notice that the 0a is the new-line or line-feed:
$ echo 'é' > file
$ cat file
é
$ hd file
00000000 c3 a9 0a |...|
00000003
Therefore, I think the problem is that the é character is encoded with one byte of 0xe9 rather than two bytes of 0xc3 0xa9.
I played around with Go to see where the e9 comes from and I notice that the Unicode for é is actually \u00e9 and it is corresponding the two bytes of \xc3\xa9 i.e. 0xc3 and 0xa9, as shown below. Therefore, on your Windows machine, somehow the Unicode is mixed up with the hexadecimal bytes.

Related

How to correctly display messages in console?

I use mojolicious application. When I do logging all if fine, except when I run application under morbo I see text like:
$app->log->info('тест лога');
[Sat Oct 6 15:22:43 2018] [info] �е�� лога
here is some problem with utf8.
What should I do that messages will be correctly displayed?
My terminal have support for utf8. I run Linux Mint v19.3
Here is how messages looks which come from script:
Test terminal:
Don't use the following when using Mojo::Log to write to STDERR:
binmode STDERR, ':encoding(UTF-8)';
Mojo::Log explicitly encodes everything it logs using UTF-8, even when writing to STDERR.
sub append {
my ($self, $msg) = #_;
return unless my $handle = $self->handle;
flock $handle, LOCK_EX;
$handle->print(encode('UTF-8', $msg)) or croak "Can't write to log: $!";
flock $handle, LOCK_UN;
}
When using STDERR as the log output (the default), this conflicts with the well-established practice of adding an encoding layer to STD*. In such a circumstance, double-encoding occurs.
One must therefore avoid doing
binmode STDERR, ':encoding(UTF-8)';
Note that this is done as part of
use open ':std', ':encoding(UTF-8)';
Try to run following code sample in your system. The test confirmed with correct UTF-8 output in terminal
#!/usr/bin/env perl
use Mojolicious::Lite -signatures;
get '/' => sub ($c) {
$c->render(text => 'Hello World!');
};
app->log->info('тест лога');
app->start;
Run as morbo test.pl produces following output
:u99257852:~/work/perl/mojo$ morbo ./test.pl
Web application available at http://127.0.0.1:3000
[2020-10-31 13:33:57.42056] [83940] [info] тест лога
[2020-10-31 13:35:16.72465] [83940] [debug] [hga9Tgyy] GET "/"
[2020-10-31 13:35:16.72528] [83940] [debug] [hga9Tgyy] Routing to a callback
[2020-10-31 13:35:16.72574] [83940] [debug] [hga9Tgyy] 200 OK (0.001078s, 927.644/s)
(uiserver):u99257852:~/work/perl/mojo$
Tested locally with nc localhost 3000
(uiserver):u99257852:~$ nc localhost 3000
GET / HTTP/1.1
HTTP/1.1 200 OK
Date: Sat, 31 Oct 2020 17:35:16 GMT
Server: Mojolicious (Perl)
Content-Type: text/html;charset=UTF-8
Content-Length: 12
Hello World!
Output of uname -a
(uiserver):u99257852:~/work/perl/mojo$ uname -a
Linux infongwp-us19 4.4.223-icpu-044 #2 SMP Sun May 10 11:26:44 UTC 2020 x86_64 GNU/Linux
(uiserver):u99257852:~/work/perl/mojo$
Bash is user's shell configured with ~/.bashrc with following settings to support UTF-8
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8:

Why does WWW::Mechanize fail with "X-Died: Illegal field name 'X-Meta-Twitter:title'"?

Why does WWW::Mechanize have blank content after getting the following URL? Using a browser or curl there is a full HTML page retrieved.
use WWW::Mechanize;
$mech = new WWW::Mechanize;
$mech->get("http://www.belizejudiciary.org/web/judgements2/");
print $mech->content # prints nothing
Here is the dump of the response:
HTTP/1.1 200 OK
Connection: close
Date: Fri, 10 Feb 2017 00:51:47 GMT
Server: Apache/2.4
Content-Type: text/html; charset=UTF-8
Client-Aborted: die
Client-Date: Fri, 10 Feb 2017 00:51:48 GMT
Client-Peer: 98.129.229.64:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
Link: <http://www.belizejudiciary.org/web/wp-json/>; rel="https://api.w.org/"
Link: <http://www.belizejudiciary.org/web/?p=468>; rel=shortlink
Set-Cookie: X-Mapping-hepadkon=FAB86566672CEB74D66B2818CA030616; path=/
X-Died: Illegal field name 'X-Meta-Twitter:title' at /usr/local/lib/perl5/site_perl/5.16.3/sun4-solaris/HTML/HeadParser.pm line 207.
X-Pingback: http://www.belizejudiciary.org/web/xmlrpc.php
I have version 3.70 of HTML::Parser installed.
Your dump shows that there was an error parsing the response:
X-Died: Illegal field name 'X-Meta-Twitter:title' at /usr/local/lib/perl5/site_perl/5.16.3/sun4-solaris/HTML/HeadParser.pm line 207.
This is caused by a bug in HTML::HeadParser:
<meta> tags can have name attributes with colons in them, and this is perfectly valid. But HTML::HeadParser then tries to register these as X-Meta-<name> headers using HTTP::Headers. Newer versions of HTTP::Headers (since 6.05) have stricter checks for headers, and will refuse them if they contain colons.
This was fixed in version 3.71 of the HTML-Parser distribution, so you should upgrade.

Getting error in perl Can't find string terminator "EOF" anywhere before EOF at /var/www/sandeep/testeof.cgi line 2

iam using perl to print some data but it is giving me error as Can't find string terminator "EOF" anywhere before EOF at
The code is:
#!/usr/local/bin/perl -w
print <<EOF;
hello
EOF
ERROR:
[Mon May 23 11:32:12 2016] [error] [client 192.168.10.117] Directory index forbidden by Options directive: /var/www/, referer: http://192.168.10.100/
[Mon May 23 11:32:12 2016] [error] [client 192.168.10.117] malformed header from script. Bad header=hello: testeof.cgi
[Mon May 23 11:32:18 2016] [error] [client 192.168.10.117] Directory index forbidden by Options directive: /var/www/, referer: http://192.168.10.100/
[Mon May 23 11:32:18 2016] [error] [client 192.168.10.117] Can't find string terminator "EOF" anywhere before EOF at /var/www/sandeep/testeof.cgi line 2.
[Mon May 23 11:32:18 2016] [error] [client 192.168.10.117] Premature end of script headers: testeof.cgi
I tried putting EOF in single quotes (print <<'EOF';) as shown in this answer, but the error is same. Printing by this method is working in otherr files in the same directory.
I also referred this question (Why am I getting “Can't find string terminator ”'“ anywhere before EOF at -e line 1” when I try to run a Perl one-liner on Windows?) but in that question OP is using different method to print and iam using linux(UBUNTU).
Please guide where I am doing wrong?
Your code is correct.
Ensure that your EOF is on a single line with no spaces around it.
Use quotes to designate your terminator:
print <<_EOF_;
hi $$
_EOF_
print <<"_EOF_";
hi $$
_EOF_
Both are exactly the same (print hi 12345 where 12345 is the process id of the current process), but the second one is more clear compared to single quotes:
print <<'_EOF_';
hi $$
_EOF_
This one will print hi $$, because no variable replacing is done with singles quotes.
Web-script always require a header (at least for Apache). Add this line as the first output line to get rid of the malformed header from script error:
print "Content-type: text/html\r\n\r\n";

How do I get a refresh token for command line gsutil to work?

I use gsutil to transfer files from a Windows machine to Google Cloud Storage.
I have not used it for more than 6 months and now when I try it I get:
Failure: invalid_grant
From researching this I suspect the access token is no longer valid as it has not been used for 6 months, and I need a refresh token?
I cannot seem to find how to get and use this.
thanks
Running gsutil -DD config produces the following output:
C:\Python27>python c:/gsutil/gsutil -DD config
DEBUG:boto:path=/pub/gsutil.tar.gz
DEBUG:boto:auth_path=/pub/gsutil.tar.gz
DEBUG:boto:Method: HEAD
DEBUG:boto:Path: /pub/gsutil.tar.gz
DEBUG:boto:Data:
DEBUG:boto:Headers: {}
DEBUG:boto:Host: storage.googleapis.com
DEBUG:boto:Params: {}
DEBUG:boto:establishing HTTPS connection: host=storage.googleapis.com, kwargs={'timeout': 70}
DEBUG:boto:Token: None
DEBUG:oauth2_client:GetAccessToken: checking cache for key *******************************
DEBUG:oauth2_client:FileSystemTokenCache.GetToken: key=******************************* not present (cache_file= c:\users\admini~1\appdata\local\temp\2\oauth2_client-tokencache._.ea******************************)
DEBUG:oauth2_client:GetAccessToken: token from cache: None
DEBUG:oauth2_client:GetAccessToken: fetching fresh access token...
INFO:oauth2client.client:Refreshing access_token connect: (accounts.google.com, 443)
send: 'POST /o/oauth2/token HTTP/1.1\r\nHost: accounts.google.com\r\nContent-Length: 177\r\ncontent-type: application/x- www-form-urlencoded\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.7.7 (gzip)\r\n\r\nclient_secret=******************&grant_type=refresh_token&refresh_token=****************************************&client_ id=****************.apps.googleusercontent.com' reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Content-Type: application/json; charset=utf-8 header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate header: Pragma: no-cache header: Expires: Fri, 01 Jan 1990 00:00:00 GMT header: Date: Thu, 08 May 2014 02:02:21 GMT header: Content-Disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt header: Content-Encoding: gzip header: X-Content-Type-Options: nosniff header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block header: Server: GSE header: Alternate-Protocol: 443:quic header: Transfer-Encoding: chunked
INFO:oauth2client.client:Failed to retrieve access token: { "error" : "invalid_grant" }
Traceback (most recent call last):
File "c:/gsutil/gsutil", line 83, in <module> gslib.__main__.main() File "c:\gsutil\gslib_main_.py", line 151, in main command_runner.RunNamedCommand('ver', ['-l'])
File "c:\gsutil\gslib\command_runner.py", line 95, in RunNamedCommand self._MaybeCheckForAndOfferSoftwareUpdate(command_name, debug)):
File "c:\gsutil\gslib\command_runner.py", line 181, in _MaybeCheckForAndOfferSoftwareUpdate cur_ver = LookUpGsutilVersion(suri_builder.StorageUri(GSUTIL_PUB_TARBALL))
File "c:\gsutil\gslib\util.py", line 299, in LookUpGsutilVersion obj = uri.get_key(False)
File "c:\gsutil\third_party\boto\boto\storage_uri.py", line 342, in get_key generation=self.generation)
File "c:\gsutil\third_party\boto\boto\gs\bucket.py", line 102, in get_key query_args_l=query_args_l)
File "c:\gsutil\third_party\boto\boto\s3\bucket.py", line 176, in _get_key_internal query_args=query_args)
File "c:\gsutil\third_party\boto\boto\s3\connection.py", line 547, in make_request retry_handler=retry_handler
File "c:\gsutil\third_party\boto\boto\connection.py", line 947, in make_request retry_handler=retry_handler)
File "c:\gsutil\third_party\boto\boto\connection.py", line 838, in _mexe request.authorize(connection=self)
File "c:\gsutil\third_party\boto\boto\connection.py", line 377, in authorize connection._auth_handler.add_auth(self, *********)
File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_plugin.py", line 22, in add_auth self.oauth2_client.GetAuthorizationHeader()
File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 338, in GetAuthorizationHeader return 'Bearer %s' % self.GetAccessToken().token
File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 309, in GetAccessToken access_token = self.FetchAccessToken()
File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 435, in FetchAccessToken credentials.refresh(http)
File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 516, in refresh self._refresh(http.request)
File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 653, in _refresh self._do_refresh_request(http_request)
File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 710, in _do_refresh_request raise AccessTokenRefreshError(error_msg) oauth2client.client.AccessTokenRefreshError: invalid_grant
You can ask gsutil to configure itself. Go to the directory with gsutil and run this:
c:\gsutil> python gsutil config
Gsutil will lead you through the steps to setting up your credentials.
That said, access tokens only normally last about a half hour. It's more likely that the previously-configured refresh token was revoked for some reason. Alternately, you can only request new tokens at a certain rate. It's possible your account has been requesting many, many refresh tokens for some reason and has been temporarily rate limited by the access service.
The command to authenticate is now
$ gcloud auth login
That should refresh your grant and get you going again.
You may also want to run
$ gcloud components update
to update your installation.
Brandon Yarbrough gave me suggestions which solved this problem. He suspected that the .boto file was corrupted and suggested I delete it and run gsutil config again. I did this and it solved the problem.

Swift TempAuth returned 404 when HEAD the account

I am a newbie to Swift, but was trying to install it on my CentOS 6.5 VM. I have done
Installing lasted Swift release (1.12.0) and python-swiftclient (2.0.2) and their dependencies
Preparing and mounting my drive (a separated device formated as xfs) at /svr/node/d1
Creating the rings and adding the device to the rings (account, container, object)
Building the rings, which generates one .ring.gz file for each ring. Placed them in /etc/swift
Configuring hash_path_prefix for proxy
Setting up TempAuth and adding a new user 'myaccount:me' with password 'pa'
Starting proxy and account.
I would expect to successfully do
swift -A http://localhost:8080/auth/v1.0 -U myaccount:me -K pa stat
but the command told me 'Account not found'. To see detailed information, I did
swift --debug -v -A http://localhost:8080/auth/v1.0 -U myaccount:me -K pa stat
the output is
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
DEBUG:requests.packages.urllib3.connectionpool:"GET /auth/v1.0 HTTP/1.1" 200 0
DEBUG:swiftclient:REQ: curl -i http://localhost:8080/auth/v1.0 -X GET
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: [('content-length', '0'), ('x-trans-id', 'tx88b6b6b71ec14c3393248-00530de039'), ('x-auth-token', 'AUTH_tkdc7e842046e9469da324f2ec82c80a92'), ('x-storage-token', 'AUTH_tkdc7e842046e9469da324f2ec82c80a92'), ('date', 'Wed, 26 Feb 2014 12:38:17 GMT'), ('x-storage-url', 'http://localhost:8080/v1/AUTH_myaccount'), ('content-type', 'text/html; charset=UTF-8')]
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
DEBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_myaccount HTTP/1.1" 404 0
INFO:swiftclient:REQ: curl -i http://localhost:8080/v1/AUTH_myaccount -I -H "X-Auth-Token: AUTH_tkdc7e842046e9469da324f2ec82c80a92"
INFO:swiftclient:RESP STATUS: 404 Not Found
INFO:swiftclient:RESP HEADERS: [('date', 'Wed, 26 Feb 2014 12:38:17 GMT'), ('content-length', '0'), ('content-type', 'text/html; charset=UTF-8'), ('x-trans-id', 'tx553c40e63c69470e9d146-00530de039')]
ERROR:swiftclient:Account HEAD failed: http://localhost:8080:8080/v1/AUTH_myaccount 404 Not Found
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 1192, in _retry
rv = func(self.url, self.token, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 469, in head_account
http_response_content=body)
ClientException: Account HEAD failed: http://localhost:8080:8080/v1/AUTH_myaccount 404 Not Found
Account not found
I figured out my self: in proxy-server.conf, add these two lines
allow_account_management = true
account_autocreate = true