Perl utf8 not quite working with my script - perl

I'm having a problem with utf-8 support on a perl script I'm writing. The script is meant to send html email messages. The html messages are saved in UTF-8 format inside a PostgreSQL database. Everything seems to be working but still I get corruption sometimes when I receive an email from the script - "�".
In the beginning of the script I have:
#!/usr/bin/perl -w
use utf8;
use Encode;
use MIME::Base64;
use MIME::Lite;
my $connection = DBI->connect('dbi:Pg:dbname='.$db_name.';host='.$db_host.'', $db_user,$db_pass, { AutoCommit=>1, PrintError => 1, pg_enable_utf8 => 1 });
my $fetchHtml = $connection->prepare('SELECT * FROM emails ORDER BY n_id DESC LIMIT 1');
$fetchHtml->execute();
my $message = $fetchHtml->fetchrow_hashref();
my $sendMsg = MIME::Lite->build(
Encoding => 'quoted-printable',
Type => 'multipart',
To => '<atesting#address.com>',
From => '<destination#address.com>',
Subject => encode("MIME-B", $message->{'title'}),
Data => decode_entities($message->{'html'})
);
$sendMsg->attr("Content-Type" => "text/html; charset=utf-8;");
$sendMsg->send_by_smtp('127.0.0.1', Timeout =>30, Debug => 0, SkipBad => 1);
I'm wondering what I'm doing wrong and why do I keep on getting the cool "�" sign ? :)
Another thing is that I get this exception when I execute the script:
Uncaught exception from user code:
Wide character in subroutine entry at /usr/lib/perl5/site_perl/5.10.0/MIME/Lite.pm line 2259.
at /usr/lib/perl5/site_perl/5.10.0/MIME/Lite.pm line 2259
MIME::Lite::print_simple_body('MIME::Lite=HASH(0xa51b9c8)', 'MIME::Lite::SMTP=GLOB(0xa5b8888)', 1) called at /usr/lib/perl5/site_perl/5.10.0/MIME/Lite.pm line 2191
MIME::Lite::print_body('MIME::Lite=HASH(0xa51b9c8)', 'MIME::Lite::SMTP=GLOB(0xa5b8888)', 1) called at /usr/lib/perl5/site_perl/5.10.0/MIME/Lite.pm line 2126
MIME::Lite::print_for_smtp('MIME::Lite=HASH(0xa51b9c8)', 'MIME::Lite::SMTP=GLOB(0xa5b8888)') called at /usr/lib/perl5/site_perl/5.10.0/MIME/Lite.pm line 2897
MIME::Lite::send_by_smtp('MIME::Lite=HASH(0xa51b9c8)', 'bla.example.com', 'Timeout', 30, 'Debug', 0, 'SkipBad', 1) called at ./advanced-daemon.pl line 354
main::send_mail('Subject Title' <webmaster#testing>', 'spam#spam.com', 'HASH(0xa518630)') called at ./advanced-daemon.pl line 225
main::sendEmailsToSubscribers('DBI::db=HASH(0xa517f40)', 24, 'HASH(0xa518630)') called at ./advanced-daemon.pl line 136
I can't understand what exactly is the problem but I think it's related to the utf8..
Any help would be pretty much appreciated.. :)

First you need use utf8 only if you have unicode character in the quellcode. Then decode_entities($message->{'html'}) ist also wrong. Use only $message->{'html'}.
The database must be utf8 by default. Then add Encoding => '8bit'. That works nice for me.
Your MIME::Lite is false: see on http://www.perlmonks.org/?node_id=105262 for a nice example

In my code I use the multipart type and here is the version which seems to work correctly.
my $mail_htm = 'Text of the email with utf8 characters like these: ľščťžýáíúäô';
my $msg = MIME::Lite->new(
From =>'me#me.com',
'Reply-To'=>'me#me.com',
To =>'you#you.com',
Subject => 'Simple ascii non-utf8 subject',
Type =>'multipart/mixed'
);
my $msg_body = MIME::Lite->new(
Type =>'multipart/alternative'
);
$msg_body->attach(
Type =>'text/html; charset=UTF-8',
Encoding=>'quoted-printable',
Data =>$mail_htm
);
$msg->attach($msg_body);
I didn't play much with utf8 encoding for the subject as I didn't need it that much and several solutions I found around didn't work.

Related

Create Email object and update body_str afterwards in Perl

I am trying to update a script so that it can send an email using the Email::MIME object.
I create email object, and then later on update the body of the message, as the message body can change depending on different circumstances.
I have the following code
my $message = Email::MIME->create(
header_str => [
From => 'someone#example.com',
To => 'someoneelse#example.com',
Subject => 'This is the subject'
],
attributes => {
encoding => 'quoted-printable',
charset => 'ISO-8859-1'
}
);
if ( $i > 1 ) {
$message->body_str = "Here's one message";
}
else {
$message->body_str = "Here's seconds message";
}
When I do the above I get the following error:
Can't modify non-lvalue subroutine call
The error line is referencing where I am updating body_str
$message->body_str is a get method, not set.
What you are looking for is body_set or probably body_str_set.
#!/usr/bin/perl
use strict;
use warnings;
use Email::MIME;
my $message = Email::MIME->create(
header_str => [
From => 'someone#example.com',
To => 'someoneelse#example.com',
Subject => 'This is the subject'
],
attributes => {
encoding => 'quoted-printable',
charset => 'ISO-8859-1'
}
);
#put your conditionals here
print $message->body_set('Blah');
According to the documentation, you must use body_set or body_str_set to change the value of the body text
Use
$message->body_set("Here's one message")
if you want to pass a simple 7-bit ASCII string, or if your string is already encoded to conform with the message's character set. The module will automatically encode it according to the message's character set before storing it
Or use
$message->body_str_set("Here's one message")
if you want to pass a general unencoded Unicode string. You will need to be careful about any literal strings you use, as the result depends on the encoding your editor adopts when it writes the file
It is generally best to add use utf8 at the top of your program, and ensure that your editor writes UTF-8 data. That will cause Perl to read your source code and decode it as UTF-8, storing your strings as basic Unicode data that you can pass to body_str_set.

Perl Net::Telnet::Cisco Bad named parameter

I'm trying to get some scripting finished to deploy changes en masse to about 400 Cisco devices. I've got a perl script modified from MrAudit that's using Net::Telnet::Cisco and for the life of me, I can't figure out the named parameter component.
In the documentation, they have:
$ok = $obj->cmd($string);
$ok = $obj->cmd(String => $string,
[Output => $ref,]
[Prompt => $match,]
[Timeout => $secs,]
[Cmd_remove_mode => $mode,]);
#output = $obj->cmd($string);
#output = $obj->cmd(String => $string,
[Output => $ref,]
[Prompt => $match,]
[Timeout => $secs,]
[Cmd_remove_mode => $mode,]
[Normalize_cmd => $boolean,]);
And my code is:
$testString is the test command I'm running against the device, $userTest1 is an array being cast where I want the output to be stored.
$::OPENRTR->cmd(String=>$testString,[Timeout=>5,Output=>$userTest1,]);
And every single time, no matter which component I modify or try and write it a different way, I get a variation of the error:
Odd number of elements in hash assignment at(filename)
bad named parameter "ARRAY(0x2e46460)" given to Net::Telnet::Cisco::cmd() at mrAudit-TACACSMod.pl line 279
I know it has to be something simple, but it's just flying right by. Any help would be appreciated.
I think the square brackets in the documentation just show the arguments are optional, you shouldn't use them in real code:
$OPENRTR->cmd( String => $testString,
Timeout => 5,
Output => $userTest1);

Perl CGI script print (utf-8 encoded - japanese html) over http (apache httpd) getting truncated

Environment settings
OS : RHEL 6.6 (kernel 2.6.32) - x86_64
httpd : httpd-2.2.15-39
perl : 5.10.1-136
CGI API :
perl-CGI-3.15-136
perl-CGI-Session-4.35-6
I am using a static html page with Perl-CGI defined variables in the static html. This html is read in through perl, and then passed to a perl CGI script for eval.
Note:
While reading the static html, I am using UTF-8 encoding like
open( IN ,"<:encoding(UTF-8)", $file_path )
After reading the status HTML page, the output is passed back to the CGI script through a variable and then pressed in to eval to evaluate the variables.
Finally, the eval(uated) output from CGI is print which can be read through http daemon.
In the CGI script I am using
binmode(STDIN, ':encoding(UTF-8)');
binmode(STDOUT, ':encoding(UTF-8)');
The static HTML looks something like this
When I check the output of print in the CGI script, I see the complete output as desired, like this
However on the Browser, the hidden input fields are getting truncated in an unwanted manner. Like this
When I checked the wireshark output for the text/html, which is being printed back from the server to the browser, this is also getting truncated.
Like this
The HTML header has proper Content-Type and charset declaration.
The same code is working fine with EN language
The same code is working fine with zh(chinese) language as well.
When the language is set to japanese in the browser, and we read from HTTP_ACCEPT_LANGUAGE that it is 'ja', than we print the japanese specific data.
We are not using cgid module of apache.
Are we supposed to use some special encoding for japanese language??
Or it is a double encoding issue. I have tried removing the encoding when I am reading the static html file, however, that also did not help.
The same code is working fine with RHEL 5.x (2.6.18-308), and perl-CGI-Session (4.42-2), perl (5.8.8-38) httpd (2.2.3-63), there was no perl-CGI in RHEL-5.x.
I kind of found the solution to the problem. In our code we used to add a data in the CGI::Session object, which had the language. In the following form
$session->param( KEY_SESSION_LANG, $code ).
Where $session is a CGI::Session object and KEY_SESSION_LANG is 'language' and $code is something that we get from HTTP_ACCEPT_LANGUAGE. When we see 'en' we used to set it as perl constant 'en', when we got 'ja' we used to set it as perl constant 'ja'.
When we used to form the session object we used to get a session file as (perl version RHEL 5.x (2.6.18-308) & perl-CGI-Session-4.42-2 ) for JA language we used to get the following in the cgi file
cat cgisess_e31c8d21af82b59fd064babc7ca25c01
$D = {'_SESSION_ID' => 'e31c8d21af82b59fd064babc7ca25c01','_SESSION_ETIME' => 6000,'language' => 'ja','permit' => 'yes','_SESSION_REMOTE_ADDR' => '192.168.101.1','_SESSION_CTIME' => 1441090386,'execute' => 'yes','_SESSION_ATIME' => 1441090387,'_SESSION_EXPIRE_LIST' => {}};*a = \undef;;$D
For perl CGI Session in RHEL 6.6 this is coming out to be
cat cgisess_e31c8d21af82b59fd064babc7ca25c01
$D = {'_SESSION_ID' => 'e31c8d21af82b59fd064babc7ca25c01','_SESSION_ETIME' => 6000,'language' => *a,'permit' => 'yes','_SESSION_REMOTE_ADDR' => '192.168.101.1','_SESSION_CTIME' => 1441090386,'execute' => 'yes','_SESSION_ATIME' => 1441090387,'_SESSION_EXPIRE_LIST' => {}};*a = \undef;;$D
language data for ja is becoming *a. The same is also reflected when we use perl dumper for getting in memory data.
I checked the /usr/share/perl5/vendor_perl/CGI/Session.pm and it had following information in it
=head1 A Warning about UTF8
Trying to use UTF8 in a program which uses CGI::Session has lead to problems. See RT#21981 and RT#28516.
In the first case the user tried "use encoding 'utf8';" in the program, and in the second case the user tried "$dbh->do(qq|set names 'utf8'|);".
Until this problem is understood and corrected, users are advised to avoid UTF8 in conjunction with CGI::Session.
For details, see: http://rt.cpan.org/Public/Bug/Display.html?id=28516 (and ...id=21981).
=head1 TRANSLATIONS
This document is also available in Japanese.
Now when I used the perl dumper following things happened. I quote below from my offical analysis presented on our local development portal
I think the problem is because of perl-CGI-Session OSS package, please see the analysis below.
Some inputs from the CGI session source code.
## Inputs ##
From the file /usr/share/perl5/vendor_perl/CGI/Session.pm
## Following are the status of CGI session, set internally after modification to any of the parameters ##
sub STATUS_NEW () { 1 } # denotes session that's just created
sub STATUS_MODIFIED () { 2 } # denotes session that needs synchronization
sub STATUS_DELETED () { 4 } # denotes session that needs deletion
sub STATUS_EXPIRED () { 8 } # denotes session that was expired.
--snip --
I::Session - persistent session data in CGI applications
=head1 SYNOPSIS
# Object initialization:
use CGI::Session;
$session = new CGI::Session();
$CGISESSID = $session->id();
We are setting the "language" parameter in the session object. (We create a CGI object, set cookie to it, to get sid, and through sid get the session object). For setting up the language parameter we do
$session->param( 'language', ); ---> language_value = en(english) or ja(japanese). When we have completed the printing of the HTML page in /opt/packageManager/pm_gui/cgi/status.cgi file, I checked the cgi session object it is as follows
For EN Language
Before executing session flush
[09/10/2015 16:13:41] [23722] <ERROR> status.cgi : 267 :
$VAR1 = bless( {
'_STATUS' => 2,
'_DATA' => {
'_SESSION_ETIME' => 6000,
'_SESSION_ID' => '995d11334f2c39b95b3fdb86cecd9655',
'permit' => 'yes',
'language' => 'en',
Then after this when I flush the session as $session->flush() and check the session object it is
[09/10/2015 16:13:41] [23722] <ERROR> status.cgi : 270 :
$VAR1 = bless( {
'_STATUS' => 0,
'_DATA' => {
'_SESSION_ETIME' => 6000,
'_SESSION_ID' => '995d11334f2c39b95b3fdb86cecd9655',
'permit' => 'yes',
'language' => 'en',
Inference 1: session status changed after doing flush. This is good, and should be done.
For JP Language
Before executing session flush
[09/10/2015 16:14:54] [31910] status.cgi : 267 :
$VAR1 = bless( {
'_STATUS' => 2,
'_DATA' => {
'_SESSION_ID' => '1cd1b7860af4c71264f3969fe74e7a44',
'_SESSION_ETIME' => 6000,
'language' => *a
Then after this, when I flush the session as $session->flush() and check the session object it is NOT THERE. SCRIPT CRASHES HERE IT SELF
Inference 2 : Doing flush with language JP, is terminating the session, and that is why the session gets destroyed. And that is why, ending data in response is truncated
Due to the wrong value being set in memory, in session object, and then the implicit flush by CGI session is failing on the disk. Which results in termination of the session object, and in between termination of session, and and data loss of HTML.
I checked the actual code in sessions.pm file and it seems to be coming in from here
sub param {
my ($self, #args) = #_;
--snip--
# USAGE: $s->param($name, $value);
# USAGE: $s->param($name1 => $value1, $name2 => $value2 [,...]);
# DESC: updates one or more **public** records using simple syntax
if ((#args % 2) == 0) {
my $modified_cnt = 0;
ARG_PAIR:
while (my ($name, $val) = each %args) {
if ( $name =~ m/^_SESSION_/) {
carp "param(): attempt to write to private parameter";
next ARG_PAIR;
}
$self->{_DATA}->{ $name } = $val; ----> HERE
++$modified_cnt;
}
$self->_set_status(STATUS_MODIFIED);
return $modified_cnt;
}
As a solution, we stopped putting 'ja' value as a perl constant, but now are putting it as a string "ja" and it seems to be working fine now.

Sending email to multiple recipients

I've moved some old code from an old unix box to our new unix box, and I'm having some difficulty with a perl script sending email to multiple recipients. It works on the old box.
Old box perl: version 5.004_04 built for PA-RISC2.0
New box perl: v5.8.8 built for IA64.ARCHREV_0-thread-multi-LP64
Here's the basics of the script (stripped-down):
use Net::SMTP::Multipart;
$to = "sam\#bogus.com tom\#foo.com";
$smtp = Net::SMTP::Multipart->new($smtpserver);
$smtp->Header(To => $to,
From => "junk\#junk.com",
Subj => "This is a test.");
$smtp->Text("Hello, world!\n");
$smtp->End();
This works if I change it to $to = "justOneEmail\#address.com", but if I have two or more email addresses (separated by spaces), it no longer works. I don't get an error message, but no message shows up.
Any ideas why?
Do it like this:
use Net::SMTP::Multipart;
$to1 = "sam\#bogus.com";
$to2 = 'tom#foo.com';
$smtp = Net::SMTP::Multipart->new($smtpserver);
$smtp->Header(To => [ $to1, $to2, 'another_email#server.com' ],
From => "junk\#junk.com",
Subj => "This is a test.");
$smtp->Text("Hello, world!\n");
$smtp->End();
Notice that if you use double-quotes, you should escape the # in the email addresses, or perl may try to interpret it as an array interpolation.
Instead of separating the email addresses with spaces, use a comma with no intervening spaces. This works for me..
Declare an array and put all the email id's like
#MailTo = ('mail1#demomail.com', 'mail2#demomail.com', ...., 'mailn#demomail.com')
Now use the Net::SMTP module to send out the emails
$smtp->to(#MailTo);

base64-Encoding breaks smime-encrypted emaildata

I'm using Mime::Lite to create and send E-Mails. Now I need to add support for S/Mime-encryption and finally could encrypt my E-Mail (the only Perllib I could install seems broken, so I'm using a systemcall and openssl smime), but when I try to create a mime-object with it, the E-Mail will be broken as soon as I set the Content-Transfer-Encoding to base64. To make it even more curious, it happens only if I set it via $myMessage->attr. If I'm using the constructor ->new everything is fine, besides a little warning which I suppress by using MIME::Lite->quiet(1);
Is it a bug or my fault? Here are the two ways how I create the mime-object.
Setting the Content-Transfer-Encoding via construtor and suppress the warning:
MIME::Lite->quiet(1);
my $msgEncr = MIME::Lite->new(From =>'me#myhost.com',
To => 'you#yourhost.com',
Subject => 'SMIME Test',
Data => $myEncryptedMessage,
'Content-Transfer-Encoding' => 'base64');
$msgEncr->attr('Content-Disposition' => 'attachment');
$msgEncr->attr('Content-Disposition.filename' => 'smime.p7m');
$msgEncr->attr('Content-Type' => 'application/x-pkcs7-mime');
$msgEncr->attr('Content-Type.smime-type' => 'enveloped-data');
$msgEncr->attr('Content-Type.name' => 'smime.p7m');
$msgEncr->send;
MIME::Lite->quiet(0);
Setting the Content-Transfer-Encoding via $myMessage->attr which breaks the encrypted Data, but won't cause a warning:
my $msgEncr = MIME::Lite->new(From => 'me#myhost.com',
To => 'you#yourhost.com',
Subject => 'SMIME Test',
Data => $myEncryptedMessage);
$msgEncr->attr('Content-Disposition' => 'attachment');
$msgEncr->attr('Content-Disposition.filename' => 'smime.p7m');
$msgEncr->attr('Content-Type' => 'application/x-pkcs7-mime');
$msgEncr->attr('Content-Type.smime-type' => 'enveloped-data');
$msgEncr->attr('Content-Type.name' => 'smime.p7m');
$msgEncr->attr('Content-Transfer-Encoding' => 'base64');
$msgEncr->send;
I just don't get why my message is broken when I'm using the attribute-setter. Thanks in advance for your help!
Besides that i'm unable to attach any file to this E-Mail without breaking the encrypted message again.
To debug this
Make a script call showmail.pl
#!/usr/bin/perl
while (<STDIN>) { print $_; }
Test it like
use MIME::Lite;
use Net::SMTP;
use MIME::Base64;
$myEncryptedMessage = encode_base64("This is not valid encrypted message\n");
MIME::Lite->send('sendmail', "./showmail.pl"); ## Add this for debugging.
MIME::Lite->quiet(1); my $msgEncr = MIME::Lite->new(From =>'me#localhost',
To => 'you#localhost',
Subject => 'SMIME Test',
Data => $myEncryptedMessage,
'Content-Transfer-Encoding' => 'base64');
$msgEncr->attr('Content-Disposition' => 'attachment');
$msgEncr->attr('Content-Disposition.filename' => 'smime.p7m');
$msgEncr->attr('Content-Type' => 'application/x-pkcs7-mime');
$msgEncr->attr('Content-Type.smime-type' => 'enveloped-data');
$msgEncr->attr('Content-Type.name' => 'smime.p7m');
$msgEncr->send();
you should see something like.
MIME-Version: 1.0
Content-Disposition: attachment; filename="smime.p7m"
Content-Length: 49
Content-Type: application/x-pkcs7-mime; name="smime.p7m"; smime-type="enveloped-data"
X-Mailer: MIME::Lite 3.028 (F2.74; B3.07; Q3.07)
Date: Mon, 23 Mar 2012 10:40:51 -0400
From: me#localhost
To: you#localhost
Subject: SMIME Test
Content-Transfer-Encoding: base64
VGhpcyBpcyBub3QgdmFsaWQgZW5jcnlwdGVkIG1lc3NhZ2UK
The message is encoded base64, but the real message still needs to be correctly
encypted. You need to make sure that is the case since $myEncryptedMessage is
passed in. With the debug output, you can compare with a known good encrypted mail
and see if the headers are good, as far as I can see the headers are fine, it is probably
the data that is not valid.
I am not able to test this with a real mail client, but this is what I think may work for multi-parts.
use MIME::Lite;
use Net::SMTP;
use MIME::Base64;
MIME::Lite->send('sendmail', "./showmail.pl"); ## <---- for testing only
my $from_address = "nobody#localhost";
my $to_address = "somebody#localhost";
my $mail_host = "localhost";
my $subject = "Subject list";
my $message_body = "Attachment list";
my #files = ("crypt.data1","crypt.data2");
$msg = MIME::Lite->new (
From => $from_address,
To => $to_address,
Subject => $subject,
Type =>'multipart/mixed'
) or die "Error creating multipart container: $!\n";
foreach $c(#files) {
$msg->attach (
Disposition => 'attachment',
Type => "application/x-pkcs7-mime; name=smime.p7m; smime-type=enveloped-data",
Path => $c,
) or die "Error adding $c: $!\n";
}
$msg->send;
As I said in one comment the difference in setting the encoding in the construtor of the mimeobject or with the ->attr-Setter is, that the construtor just sets the encoding in the mimeheader. By using the ->attr-Setter mime encodes the data with base64.
So in my case, my previously generated mimeobject - which is base64-encoded and with s/mime encrypted - read from a file needs to set the encoding in the construtor (and suppress the warning) so no more encoding will be done by mime. Otherwise mime will encode the data again and therefore break the encryption and the email itself.
I finally got attachments to work. To achieve this I create a normal multipart/mixed mimeobject, print this object into a normal file, encrypt this file with openssl smime, read this whole file (except the 6 headerlines) into a variable and use this as the datainput. Additionally I set the Content-Transfer-Encoding to base64 using the construtor (so no encoding is done to my data).
I hope this will help someone else then me ;)
Replace $myEncryptedMessage with encode_base64($myEncryptedMessage)
and use MIME::Base64;