I have small website with some pages in the root folder, some folders (js, css, images) and sitemap.
/index.php
/whoweare.php
/contacts.php
/sitemap.xml
/images/
/css/
/js/
I want Google to crawl only pages ( index.php, whoweare.php, contacts.php, sitemap.xml ), images folder and the sitemap.
This is my robots.txt, I'm not sure is correct:
User-agent: *
Allow: /$
Allow: /images/
Allow: /sitemap.xml
Should I remove Allow: /$ in this case would Google crawl php pages in root folder (I don't have any Disallow rule)?
Related
after upgrading to 9.5.17 i get in the reports the following security messages:
Server Response on static files:
www.mydomain.de/typo3temp/assets/43cd7f07.tmp/2500f854.html.wrong
unexpected content-type text/html
www.mydomain.de/typo3temp/assets/43cd7f07.tmp/2500f854.1.svg.wrong
unexpected content-type image/svg+xml
www.mydomain.de/typo3temp/assets/43cd7f07.tmp/2500f854.php.wrong
unexpected content PHP content
www.mydomain.de/typo3temp/assets/43cd7f07.tmp/2500f854.php.txt
unexpected content PHP content
what does this mean?
I inspected the folder /typo3temp/assets/ - there is no folder 43cd7f07.tmp
Thanks!
The error messages you are receiving are part of a security feature that has been integrated into recent TYPO3 v9.5.17 and v10.4.2 releases, see https://docs.typo3.org/c/typo3/cms-core/master/en-us/Changelog/9.5.x/Feature-91354-IntegrateServerResponseSecurityChecks.html
Basically it means that your current server system
is evaluating files like test.php.txt (.php not at the end of the filename) still as PHP content - this can cause a security vulnerability in case somebody manages to upload a similar file (which might be considered as text/plain file, but is actually executable PHP code)
potentially remote code execution
is serving files like test.html.wrong (.html not at the end of the filename) still as text/html which triggers the browser to execute HTML tags and potential dangerous <script> tags
potentially cross-site scripting
Call for action
In case this is a live and in production server, you should adjust your web server configuration.
The fix is to limit those web server mime-type mapping only to those files having e.g. .html at the very end, like shown in this example for the Apache HTTP web server
<FilesMatch ".+\.html?$">
AddType text/html .html .htm
</FilesMatch>
Find more details and explanation in the TYPO3 security guidelines for server admins at https://docs.typo3.org/m/typo3/reference-coreapi/10.4/en-us/Security/GuidelinesAdministrators/Index.html#file-extension-handling
Update May 17th, 2020
https://gist.github.com/ohader/11d737de95895f8ca16495a8b7001c45 contains examples how to adjust an .htaccess file in case settings cannot be changed on a (shared) hosting environment.
<IfModule mod_mime.c>
RemoveType .html .htm
<FilesMatch ".+\.html?$">
AddType text/html .html
AddType text/html .htm
</FilesMatch>
RemoveType .svg .svgz
<FilesMatch ".+\.svgz?$">
AddType image/svg+xml .svg
AddType image/svg+xml .svgz
</FilesMatch>
RemoveHandler .php
<FilesMatch ".+\.php$">
# IMPORTANT: `php-fcgid` is using in THIS example
# Most probably is different for each individual configuration
SetHandler php-fcgid
# SetHandler php-script
# SetHandler application/x-httpd-php
</FilesMatch>
</IfModule>
Current handler identifier php-fcgid was identified for the example above using a phpinfo(); and searching for $_SERVER[REDIRECT_HANDLER]:
$_SERVER['REDIRECT_HANDLER'] php-fcgid
For shared hosting it can be quite hard to find out the correct handler for php.
some specialty for 1&1 Ionos, might be even special to this particular shared hosting package:
shared hosting with php 7.3 (confirmed in phpinfo), but $_SERVER['REDIRECT_HANDLER'] gives "x-mapp-php5" (not sure why, could be that the hosting is running for many years and was upgraded to php 7 and they somehow alias it for whatever reason)
The working solution for me was:
<IfModule mod_mime.c>
RemoveType .html .htm
<FilesMatch ".+\.html?$">
AddType text/html .html
AddType text/html .htm
</FilesMatch>
RemoveType .svg .svgz
<FilesMatch ".+\.svgz?$">
AddType image/svg+xml .svg
AddType image/svg+xml .svgz
</FilesMatch>
RemoveHandler .php
RemoveType .php
<FilesMatch ".+\.php$">
AddType x-mapp-php5 .php
AddHandler x-mapp-php5 .php
</FilesMatch>
</IfModule>
I had to remove both the handler/type and add them again within the filesmatch.
Took me quite a while to get this working, hope this helps.
For host-europe $_SERVER['REDIRECT_HANDLER'] was empty, php7.4:
<IfModule mod_mime.c>
....
RemoveHandler .php
RemoveType .php
<FilesMatch ".+\.php$">
# only this handler seems to work
AddType application/x-httpd-php .php
AddHandler application/x-httpd-php .php
</FilesMatch>
</IfModule>
The following solution was recommended to me by the support team of ALL-INKL.COM.
I had to contact them, because the remove statements (RemoveHandler .php) did not work.
<FilesMatch "\.(php[0-9,x]*|phtml)\.">
SetHandler text/plain
</FilesMatch>
Thanks to the ALL-INKL.COM-Support-Team
Here is some Domainfactory speciality.
Mind the ForceType directive (set your specific PHP version there). If not used, its webserver would still use mimetype-sniffing.
To be used on the bottom of the newest .htaccess template (10.4, 9.5) which includes the strict handling for .svg[z]/.htm[l] already
# DomainFactory-special:
# 1) remove mimetype-sniffing anything for PHP
# 2) force PHP 7.3 mimetype on .php files
<IfModule mod_mime.c>
RemoveType .php
<FilesMatch ".+\.php$">
ForceType application/x-httpd-php73
</FilesMatch>
</IfModule>
This works for JWEILAND, WEBGO and PHP:
<IfModule mod_mime.c>
RemoveHandler .php
RemoveType .php
<FilesMatch ".+\.php$">
SetHandler application/x-httpd-php
AddType application/x-httpd-php .php
AddHandler application/x-httpd-php .php
</FilesMatch>
</IfModule>
I would like to:
Redirect www.olddomain.com to www.newdomain.com/page
Redirect www.olddomain.com/article1 to www.newdomain.com/article1
Here is what I put into my htaccess file:
redirect 301 / http://www.newdomain.com/page
redirect 301 /article1/ http://www.newdomain.com/article1
The first redirect works, however, the second one does NOT work, since www.olddomain.com/article1 redirects to www.newdomain.com/page/article1, which does not exist and thus returns a 404 error.
How to redirect the old domain home page (root) to a new domain subdirectory, without affecting all other redirects?
In .htaccess code should look like that:
RewriteEngine On
RewriteRule ^/$ http://www.newdomain.com/page [L,R=301]
RewriteRule ^/article1/$ http://www.newdomain.com/article1 [L,R=301]
how to do a 301 redirect from domain.com/subdirectory/url.htm to domain.com/url.htm so that it redirects all the urls that were previously in the subdirectory but are now in the root directory. I have this
RedirectMatch 301 ^/subdirectory/(.*)$ http://www.domain.com/
But it is redirecting all the pages that were in the subdirectory to the homepage and I want it to redirect to the matching url.
For example redirect
domain.com/subdirectory/blue.htm to domain.com/blue.htm
domain.com/subdirectory/yellow.htm to domain.com/yellow.htm
domain.com/subdirectory/red.htm to domain.com/red.htm
and so on.
Use mod_rewrite:
RewriteEngine on
RewriteRule ^subdirectory/(.*)$ /$1 [R=301,L]
R=301 causes the 301 redirect. L tells the parser to ignore any following rule. You can access the parts matched inside the parentheses by $1, $2, $3 and so forth.
after installing digitalus i created the following rule (.htaccess ) file in the root directory
<VirtualHost *:80>
ServerName ecn.local
DocumentRoot /home/speshu/Development/ecn
SetEnv APPLICATION_ENV tinsae
<Directory /home/speshu/Development/ecn>
DirectoryIndex index.php
AllowOverride All
Order allow,deny
Allow from all
</Directory>
and put 127.0.0.1 ecn.local in /etc/hosts
but when i type http://ecn.local/scripts or http://etc.local/library or some existing folder in the document root rather than displaying a not found (404 message ) it lists all the folders in there how can i restrict this
in may earlier projects i remember having such restrictions on zend-framework what is the problem with digitalus since it's built on the same framework ..............
with out modifying in the digitalus
just add Options -Indexes to your wanted folder and it will hide the files (aka 'forbidden message ') like :
Forbidden
You don't have permission to access /library on this server.
for example , if you need to hide the library folder
just create .htaccess file inside of it and write down Options -Indexes
and it will do the job :)
I am trying to make the iPhone cache a HTML5 web application such that I can be offline when I use it. The web application is at www.prism.gatech.edu/~gtg880f and I did not make it. I am borrowing it just to try it out.
There are only 3 files:
index.html
index.js
style.css
I modified the index.html to include <html manifest="offline2.manifest">
and <meta content="yes" name="apple-mobile-web-app-capable" /> so that it will look full screen as an offline web app.
My offline2.manifest file are as follow:
CACHE MANIFEST
index.html
index.js
style.css
debug.js
NETWORK:
CACHE:
PS: debug.js is from Jonathan Stark.
When I use firefox, it caches it properly and I was able to use the web app offline. However, it fails in both chrome and safari.
In Chrome, I get the following debug message:
Application Cache Checking event
Application Cache Error event: Invalid manifest mime type (text/plain) http://www.prism.gatech.edu/~gtg880f/offline2.manifest
I googled manifest mime type and it mentions something about .htaccess and what not and I am actually not too sure what that means. Following instructions, I went to etc/apache2/httpd.conf and change the ALLOWOVERIDE ALL from none.
That does not seem to fix anything though and I still get the same error message.
In a nutshell, what I want to be able to do is use my safari browser on iPhone to www.prism.gatech.edu/~gtg880f and save it to my home screen. Then, turn off 3G and wifi and still use the web app.
EDIT: Tried the 1st answer from roryf:
Still does not work. Am I suppose to edit the httpd.conf file in /etc/apache2/httpd.conf? I am using Mac OSX. I added it under this section
<IfModule mime_module>
#
# TypesConfig points to the file containing the list of mappings from
# filename extension to MIME-type.
#
TypesConfig /private/etc/apache2/mime.types
#
# AddType allows you to add to or override the MIME configuration
# file specified in TypesConfig for specific file types.
#
#AddType application/x-gzip .tgz
#
# AddEncoding allows you to have certain browsers uncompress
# information on the fly. Note: Not all browsers support this.
#
#AddEncoding x-compress .Z
#AddEncoding x-gzip .gz .tgz
#
# If the AddEncoding directives above are commented-out, then you
# probably should define those extensions to indicate media types:
#
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType text/cache-manifest manifest # added to allow HTML5 offline caching
Try changing the file extention to something different.
I had the same problem and when I saved it as cache.manifesto - changed the .htaccess to
AddType text/cache-manifest .manifesto
and pointed it in the html files as
<html manifest="cache.manifesto" >
it wоrked just fine.
I just checked and it looks like your manifest file is still getting served as text/plain. Here are the steps you can take to fix it.
Create a new file called .htaccess in the same directory as the manifest file (sometimes the only way to create a file with a name that starts with a dot is to do so on the command line)
Edit the file and insert the following line to it:
AddType text/cache-manifest manifest
Go to http://web-sniffer.net/ and insert the path to your manifest file to confirm it's being served with the right mime type. It appears the path you need to use here is http://www.prism.gatech.edu/~gtg880f/offline2.manifest
This is what I did to achieve to work with my offline storage in mac
Open httpd.conf
Take a backup.
Find "AllowOverride" and change the Value from "None" to "All"
Somewhere close to line # 198
Options Indexes FollowSymLinks
AllowOverride None
Looks like you need to set the MIME type for .manifest files to text/cache-manifest in your Apache config, which is probably what you read about .htaccess (one way to do this).
Adding this to your .htaccess file should work:
AddType text/cache-manifest manifest
My working web app clipping duplicates the filenames in the CACHE: portion of the .manifest file. Like this:
CACHE MANIFEST
index.html
index.js
style.css
debug.js
CACHE:
index.html
index.js
style.css
debug.js
NETWORK:
I also included this in a .htaccess file in the same web server directory as the manifest:
AddType text/cache-manifest .manifest manifest
I had same troubles debugging an offline web app on an iPhone as well. The app behaved correctly in Chrome and Safari (both for Windows). A reboot on the iPhone finally did the trick. Hope this helps.
Or, you could simply make a file called: manifest.php and put this content in it;
<?php
header('Content-Type: text/cache-manifest');
echo "CACHE MANIFEST\n\n";
echo "CACHE:\n";
$hashes = "";
$dir = new RecursiveDirectoryIterator(".");
foreach(new RecursiveIteratorIterator($dir) as $file) {
if ($file->IsFile() &&
$file != "./manifest.php" &&
substr($file->getFilename(), 0, 1) != ".") {
echo $file . "\n";
$hashes .= md5_file($file);
}
}
echo "\n# Hash: " . md5($hashes) . "\n";
?>