well the problem is, I can see the robots.txt file here :
http://persian.cc
(please put a /robots.txt after the domain name)
But google cant !
I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine and this robots.txt is a virtual one being made by WordPress.
Now how to stop wordpress doing that?!
OR
If I cannot stop wordpress showing that virtual robots.txt file, how can I stop google looking for it on my website? maybe a .htaccess code or something?
thanks
Don't know if this helps or not. But I promised to maintain an article on robots.txt for WordPres / Joomla / PHPBB. It's still in working condition. Here it is:
http://mast3rpee.tk/?p=127
Basically what I'm doing is setting a default robot.txt and also modifying apache using .haccess file to force loading of a custom robots.php. Why? Well this solves most problems not just the one here. It also works on free hosting which can create all sorts of issues (some ban txt files! some overide robots.txt etc).
HERE IS THE CODE
.htaccess
# BEGIN Robots
<IfModule mod_rewrite.c>
<FilesMatch "^robots.(txt|php)$">
Header Set Last-Modified "Tue, 01 Jan 2013 12:00:00 GMT"
</FilesMatch>
RewriteEngine On
RewriteBase /
RewriteRule ^(robots)\.txt$ /$1.php [L]
</IfModule>
# END Robots
robots.txt
User-agent: *
Disallow: /cgi-bin/
Disallow: /feed/
Disallow: /wp-admin/
Disallow: /wp-content/plugins/
Sitemap: http://{PUT YOUR DOMAIN}/sitemap.xml
Crawl-delay: 4
robots.php
<?php
$start = "2013/01/01"; // Date you started your blog YYYY/MM/DD
$average = 30; // Number of posts you make per month
$sitemap = "http://{PUT YOUR DOMAIN}/sitemap.xml";
// Is blog old enough
$old = ($average/30)*(time()-strtotime($start)) > 3600*24*360? true : false;
// Output proper headers
header ('Content-Type: text/plain');
header ('Cache-Control: private, pre-check=0, post-check=0, max-age=36000');
header ('Expires: ' . gmstrftime('%a, %d %b %Y %H:%M:%S GMT', time() + 36000));
header ('Last-Modified: ' . gmstrftime('%a, %d %b %Y %H:%M:%S GMT', time() - 36000));
if ($old) { $custom = date("Y/m");
echo <<< ROBOTS
User-agent: *
Disallow: /cgi-bin/
Disallow: /feed/
Disallow: /wp-admin/
Disallow: /wp-content/plugins/
Disallow: /$custom
Sitemap: $sitemap
Crawl-delay: 4
ROBOTS;
} else { $custom = date("Y");
echo <<< ROBOTS
User-agent: *
Disallow: /cgi-bin/
Disallow: /feed/
Disallow: /wp-admin/
Disallow: /wp-content/plugins/
Disallow: /archives/
Disallow: /tag/
Disallow: /$custom
Sitemap: $sitemap
Crawl-delay: 4
ROBOTS;
}
exit; ?>
Related
I would like to create a rule for Apache to block a massive logins according this type of log:
93.176.51.15 - - [21/Nov/2019:00:02:40 +0100] "GET /wordpress/wp-login.php HTTP/1.1" 200 5485
What's the exactly regex that I need? I use this:
^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) .*/wp-login.php.*$
Thanks in advance
More "precise" regex would look like:
^<HOST> \S+ \S+ [^"]*"[A-Z]{3,15}\s+\S*/wp-login\.php\b
It is anchored (^.+ is not an anchor), does not have catch-all's (like .* but especially non-greedy like .+?), and covers all http-methods as well as from intruder side supplied user-name (if no auth expected and web server would log it instead of -).
And if you have fail2ban >= 0.10 use <ADDR> instead of <HOST> (more faster, safe and precise if only IPs are logged).
I would like to send message from Unix server. I use command 'mail':
echo "MESSAGE_BODY" | mail -s "MESSAGE_TITLE" somebody#gmail.com
It's ok with it.
After that I want to send message with different colors. I tried this command:
echo "<font color="red">MESSAGE_BODY</font>" | mail -s "MESSAGE_TITLE" somebody#gmail.com
But it didn't help me. How to use colors ?
There have already been a "one-liner" that have posted the correct answer.
I do still feel that it's better to post how and why.
The reason why you can't just echo HTML code directly into your mail is that the receiver (Client) don't know how to display it. So it will most likely just fallback to clear text and all you would see was your HTML code when viewing the message.
What you need, is to tell the client that the content of your message is composed in HTML. You do this by adding the correct MIME header to the message.
Content-Type: text/html; charset=UTF-8
MIME-Version: 1.0
Notice you can also set charset information.
The MIME version is there for better compatibility also some SMTP servers will give you a higher spam score if you don't obey the RFC :)
But with these headers set now all "BODY" content will be treated like HTML content.
I don't just want to provide you with a "one-liner" I think showing more in a script is better to make it easier to read.
So how about this
(
echo "From: my#email.tld";
echo "To: some#email.tld";
echo "Subject: Test html mail";
echo "Content-Type: text/html";
echo "MIME-Version: 1.0";
echo "";
echo "<strong>Testing</strong><br><font color=\"blue\">I'm Blue :)</font>";
) | sendmail -t
Well technically it's still a one-liner :) But it just looks nicer and you can see what's going on!
Bonus information
If you want to have both HTML and TEXT bodies you need to look into Multipart content type bodies. I have included an example but you would properly need to read up on this if you don't know much about multipart types.
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="--0001boundary text--"
--0001boundary text--
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
The TEXT body goes here
--0001boundary text--
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
<strong>HTML code goes here</strong>
--0001boundary text--
As you can see it's no longer some simple mail body.
But I thought I wanted to show you how it was done in case you wanted to give it a go.
echo "<font color="green">Message body</font>" | mail -s "$(echo -e "Message title\nContent-Type: text/html")" somebody#gmail.com
Facebook wants you to use the og:locale:alternate meta tag to declare what languages your site is available in. Unforunetly instead of having a refrence in the tag when they want to use the alternate local they tack &fb_locale=es_ES to the end of the URL(or what ever language code they want). My problem is my site is static so I can't read this extra info easily so was woundering if there was a way in the .htaccess file to remap the url.
so
(.*)(?|&)fb_locale=(.*) to [first 2 letters of $3]/[everything but first 2 letters of $1]
so
en/test.html?fb_locale=es_ES
should go to
es/test.html
RewriteCond %{THE_REQUEST} /([^/]+)/(.*)[\?&]fb_locale=([^_]+)_(.*) [NC]
RewriteRule ^ /%3/%2 [NC,L,R]
almost works but for some reason the value after ? still gets appended.
Try :
RewriteCond %{THE_REQUEST} /([^/]+)/([^?&]+)[\?&]fb_locale=([^_]+)_([^\s]+) [NC]
RewriteRule ^ /%3/%2? [NC,L,R]
Empty question mark at the end is important as it discards the orignal query strings.
In the last capture group ([^\s]+) in THE_REQUEST RewriteCond ,we exclude matching spaces \s and tell the regex to match limited chars , because the regex is greedy and it tries to capture the whole remaining part of the request string spaceHTTP/1.1 .
Here is an example of The_Request string :
GET /en/index.html?q=foo HTTP/1.1
And the correct regex pattern without capture groups to match against it :
/en/index\.html\?q=foo
Pattern with regex capture groups :
/([^/]+)/([^?]+)\?q=([^\s]+)
The first capture group captures /en , the second captures /index.html and the third capture group captures the value of key.
You can use the following code in root/.htaccess :
RewriteEngine On
#if the requested url is /en/test.html?fb_locale=foo_bar
RewriteCond %{THE_REQUEST} /en/test\.html\?fb_locale=([^_]+)_[^\s]+ [NC]
#Then redirect the request to /es/test.html?fb_locale=foo
RewriteRule ^ /es/test.html?fb_locale=%1 [NC,L,R]
I'm using perl's Mail::Sendmail to send email notifications, but I'm having trouble displaying Japanese. Mail::Sendmail changes the Japanese to html numeric character references, but one email client I use (claws mail) doesn't display those in html mode, while none do in text mode. Mail::Sendmail also changes newlines to <br>. I want neither. What am I doing wrong? I use basic options, like these for plain text:
my %mail = (smtp => 'localhost:25',
to => $to,
from => $from,
'content-type' => 'text/plain; charset=UTF-8',
subject => $subject,
message => $text);
sendmail(%mail) or warn $Mail::Sendmail::error;
output is like:
....
Subject: test
Date: Fri, 19 Dec 2014 14:59:31 +0100
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
To: me#foo.com
Content-Transfer-Encoding: quoted-printable
From: info#foo.com
....
content: test this: 怖い <br>end test.=20
I would like to get rid of the numeric character references and html, how do I do that?
To answer my own question: The assumption that Mail::Sendmail was adding the entities was wrong. I should have checked the other code of the program I was working on more thoroughly.
I am writing a little Perl app to send emails internally where I work, but I am having troubles with attachments. Now, I can hear all of you screaming "use MIME::Lite", but it aint that easy - management rules tell me I am unable to use anything from the cpan...
Anyhow, I am using MIME::Base64 to encode any attachements I am sending. I am using Net::SMTP for the email side of things.
encoding is done as such (straight out of the Perldoc page):
my $encoded = "";
use MIME::Base64 qw(encode_base64);
open (FILE, "7857084216_9816ae9bec_b.jpg") or die "$!\n";
while (read(FILE, $buf, 60*57)) {
$encoded .= encode_base64($buf);
}
The relevant Net::SMTP code is as follows:
use Net::SMTP;
$smtp = Net::SMTP->new("mailserver",
Hello => 'somedomain.com.au',
Timeout => 60,
Debug => 0,
);
$smtp->mail(<services\#somedomain.com.au>);
$smtp->to(<me\#somedomain.com.au>);
$smtp->cc(<another\#somedomain.com.au>);
$smtp->data();
$smtp->datasend("Subject: testing 123\n");
$smtp->datasend("To: me\#somedomain.com.au\n");
$smtp->datasend("CC: another\#somedomain.com.au\n");
$smtp->datasend("MIME-Version: 1.0\n");
$smtp->datasend("Content-type: multipart/mixed;\n\tboundary=\"$boundary\"\n");
$smtp->datasend("--$boundary\n");
$smtp->datasend("Content-type: text/html; charset=utf-8\n");
$smtp->datasend("\n");
$smtp->datasend("<p> Test Email!</p>\n");
$smtp->datasend("\n");
$smtp->datasend("--$boundary\n");
$smtp->datasend("Content-type: image/jpeg; name=\"7857084216_9816ae9bec_b.jpg\"\n");
$smtp->datasend("Content-ID: \n");
$smtp->datasend("Content-Disposition: attachment; filename=\"7857084216_9816ae9bec_b.jpg\"\n");
$smtp->datasend("Content-transfer-encoding: base64\n");
$smtp->datasend("\n");
$smtp->datasend("$encoded");
$smtp->datasend("\n");
$smtp->datasend("--$boundary--\n");
$smtp->dataend;
$smtp->quit;
At this moment, I am directing these emails through to myself and reading them using Outlook 2010
When sending images (such as jpegs), I am not having any issues at all - everything seems to decode OK byte for byte as far as I can see.
When I am sending docx type files with just plain text, everything seems fine.
But, when I am sending docx files with images inserted, the files are corrupting.
Not being an expert in mail sending and attaching, I am at a bit of a loss. How should I be encoding docx files for attaching to emails? Any help would be appreciated!
Also forgot to mention, I have attempted to set the content-type accordingly: Content-type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
I've seen this happen before and usually comes down to just the incorrect mime type or mime structure. You mention you tried adding the mime type, so that might indicate that the mime structure might be a little off. I've seen this with xslx and csv files. CSV would come in find since the decoder assumes text, but if you don't have the right Mime for binary data it will always attempt to convert to ascii. I realize this around an xslx vs docx, but I think the same principles apply.
Here is a snip of an sample email I'm generating (Using Mime:Lite), but might help.
From: me#example.com
To: example#example.com
Subject: Example
Content-Transfer-Encoding: binary
Content-Type: multipart/mixed; boundary="_----------=_13433203262384120"
--_----------=_13433203262384120
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html
Content-Disposition: inline
<body>Hello,<br>
<p>EMAIL BODY</p>
<p>Thanks,<br> Blah</body>
--_----------=_13433203262384120
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet; name="sample.xlsx"
Content-Disposition: attachment; filename="sample.xlsx"
Content-Transfer-Encoding: base64
UEsDBBQAAAAIAAFk+kDm/EEvVgEAACQFAAATAAAAW0NvbnRlbnRfVHlwZXNdLnhtbMVUS28CIRC+
N+l/IFybXdRD0zSuHvo4tia1P4DC6BJZIAxa/fed3a32EesjmvQCgfleQ8j0h8vKsgVENN4VvJt3
OAOnvDZuWvDX8WN2wxkm6bS03kHBV4B8OLi86I9XAZAR22HBy5TCrRCoSqgk5j6Ao8rEx0omOsap
CFLN5BREr9O5Fsq7BC5lqdbgg/49TOTcJvawpOs2SQSLnN21wNqr4DIEa5RMVBcLp3+5ZJ8OOTEb
DJYm4BUBOBNbLZrSnw5r4jM9TjQa2EjG9CQrggnt1Sj6gIII+W6ZLUH9ZGIUkMa8IkoOdSINOgsk
CTEZ+Eq901z5CMe7r5+pZh9qubQC08oCntwshghSYwmQKpu3ovusE30qaNfuyQEamX2O7z7O3ryf
.... <Snipped>
ZRQyppbEwDToT079O3NfFSgofVlmcbvM34PqkQQdCmrLiWYxleokviz25OPYvpRw/s64avTwn+uh
SSg4ctedMMaTkj47g3IXX1BLAQIUAxQAAAAIAAFk+kDm/EEvVgEAACQFAAATAAAAAAAAAAEAAAC2
gQAAAABbQ29udGVudF9UeXBlc10ueG1sUEsBAhQDFAAAAAgAAWT6QPWzn3yRAQAAQAMAABAAAAAA
AAAAAQAAALaBhwEAAGRvY1Byb3BzL2FwcC54bWxQSwECFAMUAAAACAABZPpAkg6xSFoBAAC2AgAA
EQAAAAAAAAABAAAAtoFGAwAAZG9jUHJvcHMvY29yZS54bWxQSwECFAMUAAAACAABZPpAyfySzFpy
BAAcqRgAFAAAAAAAAAABAAAAtoHPBAAAeGwvc2hhcmVkU3RyaW5ncy54bWxQSwECFAMUAAAACAAB
ZPpARQtWMQgCAABaBQAADQAAAAAAAAABAAAAtoFbdwQAeGwvc3R5bGVzLnhtbFBLAQIUAxQAAAAI
AABk+kCVKtbcsAEAAFgDAAAPAAAAAAAAAAEAAAC2gY55BAB4bC93b3JrYm9vay54bWxQSwECFAMU
AAAACAABZPpA8XOrpLUFAABVGwAAEwAAAAAAAAABAAAAtoFrewQAeGwvdGhlbWUvdGhlbWUxLnht
bFBLAQIUAxQAAAAIAO1j+kD+7E73UA8AAM1zAAAYAAAAAAAAAAEAAAC2gVGBBAB4bC93b3Jrc2hl
ZXRzL3NoZWV0MS54bWxQSwECFAMUAAAACAAAZPpA7/olcLO/MgAHxsABGAAAAAAAAAABAAAAtoHX
kAQAeGwvd29ya3NoZWV0cy9zaGVldDIueG1sUEsBAhQDFAAAAAgAAWT6QCFo34b0AAAATgMAABoA
AAAAAAAAAQAAALaBwFA3AHhsL19yZWxzL3dvcmtib29rLnhtbC5yZWxzUEsBAhQDFAAAAAgAAWT6
QIB6g4TsAAAAUQIAAAsAAAAAAAAAAQAAALaB7FE3AF9yZWxzLy5yZWxzUEsFBgAAAAALAAsAxgIA
AAFTNwAAAA==
--_----------=_13433203262384120--
I have found the issue and it does appear that the 'content-type' does seem to make a difference. According to what I found, I replaced all the Content-type calls, that include Base64 encoded data with the following:
Content-type: application/octet-stream;
This seemed to have sent it straight through, no issues with either Outlook 2010 and 2003. I have tested it against binary files, plain text, ect and all seems to work OK in my current environment.
Cheers