i need a nginx rewrite rule for the following problem:
I have Urls that include several hyphen and eventually underscores
Example request: http://www.example.com/cat/cat2/200-AB---a-12_12-123.312/cat-_-cat/cat/dog---I
would give a 404 error
so in need a 301- redirect to:
http://www.example.com/cat/cat2/200-AB-a-12-12-123.312/cat-cat/cat/dog-I
So all underscores should be replaced with hyphens and there should be only one hyphen a time.
short version:
replace --- with - and replace _ with -
but by replacing _ with - this -_- will become --- and rule one would have to be called again.
Is it possible to to that in one rule? and if not how to do it any other way :)i have absolutely no idea how to do that with nginx
any help appreciated :)
% nginx -c $PWD/test.conf
% curl -I localhost:8080/cat/cat2/200-AB---a-12_12-123.312/cat-_-cat/cat/dog---I
HTTP/1.1 301 Moved Permanently
Server: nginx/1.3.13
Date: Wed, 20 Feb 2013 00:09:50 GMT
Content-Type: text/html
Content-Length: 185
Location: http://localhost:8080/cat/cat2/200-AB-a-1212-123.312/cat-cat/cat/dog-I
Connection: keep-alive
% cat test.conf
events { }
#error_log logs/error.log debug;
http {
server {
listen 8080;
location /cat/cat2/ {
# replace up to 3 inconsecutive
# uderscores per internal redirect
rewrite "^(.+?)_+(?:(.+?)_+)?(?:(.+?)_+)?(.+)$" $1$2$3$4 last;
# replace up to 3 inconsecutive multiple
# hyphens per internal redirect
rewrite "^(.+?-)-+(?:(.+?-)-+)?(?:(.+?-)-+)?(.+)$" $1$2$3$4 last;
return 301 $uri;
}
}
}
Related
I am trying to rewrite a domain to include a language path but without the trailing slash.
So
www.example.com => www.example.com/en
www.example.com/page/ => www.example.com/en/page
www.example.com/page => www.example.com/en/page
I am currently using this config, but it is not working as expected.
server {
listen 80;
server_name www.example.com;
root /var/www/example.com/public;
rewrite ^/(.*)/$ /$1 permanent;
index index.php index.html;
location = / {
return 301 http://www.example.com/en$request_uri;
}
}
Mainly,
www.example.com => www.example.com/en/ => www.example.com/en
www.example.com/page => www.example.com/page
This code is also playing havoc on some of the http_post requests.
In the case of a POST, the redirect is downgraded to a GET (this is normal behaviour). The 307 response can be used to repeat a POST with the new URI. See this page for more.
In which case, you will need to rewrite your configuration to use return 307 statements. You can use regular expression location blocks to capture the URI without its trailing /. See this document for more.
One way to identify URIs that do not begin with /en, is to use a location ^~ /en block to process all URIs that do begin with /en, and use the regular expression location block to capture everything else.
For example:
location = / {
return 307 /en;
}
location ~ ^(/.*?)/?$ {
return 307 /en$1;
}
location ^~ /en {
...
}
Set-Cookie: SESSIONID=836cfc64b5856712b040a0b1b3bf4237; Secure; HttpOnly
Set-Cookie: Watson-DPAT=gBJQjAG%2FYflxpHKCwJVswQPEBuUmikj38zzFm8UZNTbOERxbeXS4WVxBIT5JetJBkeO1RT16PNz6%2BI17oFrEqvxjny%2FifZRorvBxXVzDmFkRpfRLxxj6ZNvFCvuRL1DtfW3nL8Ne1QDwpuKQmNt8%2BD9vFk7bGjlaziHT0ZFhNffWJT7FRCWbuJAyjKd%2BQui2WTIl6B8KglPi6GG1buh5UPDE%2Bc8OvrqyAfJRfYOApRdx7kHhtHdxIV7g%2FzNExXhafScqxi4cWEa5Kg9YGcypr8SIO%2FD7WOq0KyERHUDkbZatH53CCii25it5XD0plnt3cVc4bWs8tXkMT82V9DwCYULto64L%2BgNh30iTpyv72xOIfHeZTt2KISfhXMy6z86ueaJZzNd4nS6rwc7s2E8ldxwYLXrCU996xsLmzPYbGSzaeFLplG7c%2BCxzTlAll5fn8eMMbGn30W%2BrXLNtcaJ3lRK2nvzQCim1GhMdqoOvOcSvPWiJoVBrF8lc75eGSr8C%2Fovq20fOk3NDw4f0UPfBEGZYuAtXjonU7QdRhSgLRXxKyGvcYHEWeWUOQ2kvtI2m%2FRD%2BMRx9384p1v6uu8XfaU16IqoidV0Vew3MLPW4fxcOWRqnWKy0iIYbIJrWVcigloIy%2FNxgO7oHW7aacgH1u8IluAURz5AiE1Bej4l%2FjAI91IUTEssbg6fsXd3AqmlkixDglDJBgTEtMoXhXVyDjvJSaVUqdFTokP3YcRNhlTzqDQ3vG8txTLzECsyQHZ7DgWp%2B98P3zjvtad9xB%2BDzXhF4CaUB7ve99bWO5FO1DU3KRhx7pEAKGselDCoxTOkjIhEMMJbeQrbC1QWJ4uR9KlBPBdIbShd3; path=/speech-to-text/api; secure; HttpOnly
The request to create the Speech to Text session works.
{
"recognize": "https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/836cfc64b5856712b040a0b1b3bf4237/recognize",
"recognizeWS": "wss://stream.watsonplatform.net/speech-to-text/api/v1/sessions/836cfc64b5856712b040a0b1b3bf4237/recognize",
"observe_result": "https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/836cfc64b5856712b040a0b1b3bf4237/observe_result",
"session_id": "836cfc64b5856712b040a0b1b3bf4237",
"new_session_uri": "https://stream.watsonplatform.net/speech-to-text/api/v1/sessions/836cfc64b5856712b040a0b1b3bf4237"
}
Then I try to get the status of the session to make sure the state is "initialized" but I get a "Malformed Session ID Cookie" error.
GET /speech-to-text/api/v1/sessions/836cfc64b5856712b040a0b1b3bf4237/recognize HTTP/1.1\x0d
Content-Length: 0\x0d
Accept-Encoding: gzip\x0d
Authorization: Basic OGIyMTk0MDYtYWYzYS00YTFhLWExYmMtZDA3ZjNlNTY2Y2JmOm1lYmpBaG1ndkhMSw==\x0d
Cookie: SESSIONID=836cfc64b5856712b040a0b1b3bf4237; Watson-DPAT=gBJQjAG%2FYflxpHKCwJVswQPEBuUmikj38zzFm8UZNTbOERxbeXS4WVxBIT5JetJBkeO1RT16PNz6%2BI17oFrEqvxjny%2FifZRorvBxXVzDmFkRpfRLxxj6ZNvFCvuRL1DtfW3nL8Ne1QDwpuKQmNt8%2BD9vFk7bGjlaziHT0ZFhNffWJT7FRCWbuJAyjKd%2BQui2WTIl6B8KglPi6GG1buh5UPDE%2Bc8OvrqyAfJRfYOApRdx7kHhtHdxIV7g%2FzNExXhafScqxi4cWEa5Kg9YGcypr8SIO%2FD7WOq0KyERHUDkbZatH53CCii25it5XD0plnt3cVc4bWs8tXkMT82V9DwCYULto64L%2BgNh30iTpyv72xOIfHeZTt2KISfhXMy6z86ueaJZzNd4nS6rwc7s2E8ldxwYLXrCU996xsLmzPYbGSzaeFLplG7c%2BCxzTlAll5fn8eMMbGn30W%2BrXLNtcaJ3lRK2nvzQCim1GhMdqoOvOcSvPWiJoVBrF8lc75eGSr8C%2Fovq20fOk3NDw4f0UPfBEGZYuAtXjonU7QdRhSgLRXxKyGvcYHEWeWUOQ2kvtI2m%2FRD%2BMRx9384p1v6uu8XfaU16IqoidV0Vew3MLPW4fxcOWRqnWKy0iIYbIJrWVcigloIy%2FNxgO7oHW7aacgH1u8IluAURz5AiE1Bej4l%2FjAI91IUTEssbg6fsXd3AqmlkixDglDJBgTEtMoXhXVyDjvJSaVUqdFTokP3YcRNhlTzqDQ3vG8txTLzECsyQHZ7DgWp%2B98P3zjvtad9xB%2BDzXhF4CaUB7ve99bWO5FO1DU3KRhx7pEAKGselDCoxTOkjIhEMMJbeQrbC1QWJ4uR9KlBPBdIbShd3\x0d
User-Agent: Mojolicious (Perl)\x0d
Host: stream.watsonplatform.net\x0d
HTTP/1.1 400 Bad Request\x0d
X-Backside-Transport: FAIL FAIL\x0d
Connection: Keep-Alive\x0d
Transfer-Encoding: chunked\x0d
X-Error-Cause: Zuul Error: Malformed Session ID Cookie\x0d
Content-Type: application/json\x0d
Date: Wed, 01 Jun 2016 19:41:26 GMT\x0d
Server: -\x0d
X-Global-Transaction-ID: 237895544\x0d
X-DP-Watson-Tran-ID: stream-dp01-c0182762-b9fe-4533-acab-7fbeb02b63dd\x0d
The code is using a single instance of Mojo::UserAgent so the cookies are maintained on each request.
when using sessions you receive a SESSIONID cookie when creating the session, that cookie you need to send back to the service on every call you make after creating the session. Please note that the value of that cookie does not equal "session_id": "836cfc64b5856712b040a0b1b3bf4237", it is a longer alphanumeric string.
btw. why are you using sessions? what is your use case? maybe you could benefit from sessionless calls (simpler) or websockets (better for live use cases)
Dani
Using a trailing slash is the cause of this error. The interesting part is that the "start session" POST request with the trailing slash URL will succeed and return the correct JSON data. The next request to get the session status will fail. Not really a code problem. I also demonstrated the issue with curl.
my $ua = Mojo::UserAgent->new();
$ua->proxy->detect();
$ua->inactivity_timeout(0);
# THIS URL WORKS - no trailing slash
my $start_session_url = "https://${watson_username}:${watson_password}\#stream.watsonplatform.net/speech-to-text/api/v1/sessions";
# THIS URL DOES NOT WORK - with trailing slash
# my $start_session_url = "https://${watson_username}:${watson_password}\#stream.watsonplatform.net/speech-to-text/api/v1/sessions/";
my $session_tx = $ua->post($start_session_url);
my $response;
my $recognize_url;
if($response = $session_tx->success) {
print Dumper($response->json);
$recognize_url = $response->json->{recognize};
} else {
die "Failure to start session";
}
$recognize_url =~ s/https:\/\//https:\/\/${watson_username}:${watson_password}\#/;
# Malformed Cookie error happens here
my $status_tx = $ua->get($recognize_url);
if($response = $status_tx->success) {
print Dumper($response->json);
} else {
die "Failure to get session status";
}
I have a set of ip address in my redis server to be blocked. Now when a client makes a request,
the request must be intercepted by nginx
check if the remote_addr belongs to the blocked ip
add a header if the ip is blocked
then redirect to the actual ip address with the request_uri.
nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
lua_shared_dict ip_status 1m;
server {
listen 9080;
server_name localhost;
location ~ .* {
rewrite_by_lua_file src/ip_check.lua;
}
}
}
src/ip_check.lua
-- redis configuration
local redis_host = "127.0.0.1"
local redis_port = 6379 -- connection timeouts for redis in ms.
local redis_max_idle_timeout = 10000
local redis_pool_size = 2 -- don't set this too high!
local redis_timeout = 200
-- check a set with this key for blacklist entries
local redis_key = ngx.var.remote_addr
local ip_status = ngx.shared.ip_status
local status_unblocked = "0"
local status_blocked = "1"
-- cache lookups for this many seconds
local cache_ttl = 1800
local redirect_host = "http://192.168.12.103:8080/Spring4MVCHelloWorld1"
local header_ip_status_key = "Ip-Status"
-- value of the header to be sent when the client ip address is blocked
local header_ip_status_value = "block"
local add_header = status_unblocked
-- lookup the value in the cache
local cache_result = ip_status:get(redis_key)
if cache_result then
if cache_result == status_blocked then
add_header = status_blocked
end
else
-- lookup against redis
local resty = require "resty.redis"
local redis = resty:new()
redis:set_timeout(redis_timeout)
local connected, err = redis:connect(redis_host, redis_port)
if not connected then
ngx.log(ngx.ERR, "ip_check: could not connect to redis #"..redis_host..":"..redis_port.." - "..err)
else
ngx.log(ngx.ALERT, "ip_check: found connect to redis #"..redis_host..":"..redis_port.." - successful")
local result, err = redis:get(redis_key)
if not result then
ngx.log(ngx.ERR, "ip_check: lookup failed for "..ngx.var.remote_addr.." - "..err)
else
if result == status_blocked then
add_header = status_blocked
end
-- cache the result from redis
ip_status:set(ip, add_header, cache_ttl)
end
redis:set_keepalive(redis_max_idle_timeout, redis_pool_size)
end
end
ngx.log(ngx.ALERT, "ip_check: "..header_ip_status_key.." of "..ngx.var.remote_addr.." is "..add_header)
if add_header == status_blocked then
ngx.header[header_ip_status_key] = header_ip_status_value
ngx.req.set_header(header_ip_status_key, header_ip_status_value)
end
ngx.redirect(redirect_host..ngx.var.request_uri)
For testing purpose, i added 127.0.0.1 key to redis with value 1. So, the redirect uri should be hit with the additional header. The problem i'm facing is no matter what i use ngx.header or ngx.req.set_header the Ip-Status header is not sent to the redirect request and the end api does not recieve it.
For example if i hit http://localhost:9080/hello in the browser,
Request headers
Host:"localhost:9080"
User-Agent:"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
Accept:"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
Accept-Language:"en-US,en;q=0.5"
Accept-Encoding:"gzip, deflate"
Connection:"keep-alive"
Response headers
Connection:"keep-alive"
Content-Length:"166"
Content-Type:"text/html"
Date:"Thu, 05 May 2016 08:06:33 GMT"
Location:"http://192.168.12.103:8080/Spring4MVCHelloWorld1/hello"
Server:"openresty/1.9.7.2"
ip-status:"block"
The redirected uri is http://localhost:9080/hello,
Request headers
Host:"192.168.12.103:8080"
User-Agent:"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
Accept:"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
Accept-Language:"en-US,en;q=0.5"
Accept-Encoding:"gzip, deflate"
Cookie:"JSESSIONID=4834843FE0E76170E429028E096A66E5"
Connection:"keep-alive"
Response headers
Content-Language:"en-US"
Content-Length:"166"
Content-Type:"text/html;charset=UTF-8"
Date:"Thu, 05 May 2016 08:06:33 GMT"
Server:"Apache-Coyote/1.1"
I'm able to see Ip-Status header in response headers of the original request but not in the request headers of the redirected uri. Any help as to how to send header to the redirected uri will be very helpful.
I'm new to nginx and lua. Asking because i couldn't find any corresponding questions apologies if the question is already asked.
It is the browser that is redirecting to the redirected uri and nginx do not have control on the headers. So, i removed
ngx.redirect(redirect_host..ngx.var.request_uri)
from the src/ip_check.lua and changed nginx.conf to make a proxy call and i was able to observe that the api was able to receive the additional header.
Modified nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
lua_shared_dict ip_status 1m;
server {
listen 9080;
server_name localhost;
location ~ .* {
set $backend_host "http://192.168.12.103:8080/Spring4MVCHelloWorld1";
access_by_lua_file src/ip_check.lua;
proxy_pass $backend_host$request_uri;
}
}
}
This modified nginx.conf will make a request to $backend_host$request_uri and browser will not have any knowledge of the redirections made. Hence, the headers set by ngx.req.set_header will be sent when making the proxy call. So,
ngx.header[header_ip_status_key] = header_ip_status_value
can also be removed from src/ip_check.lua.
I'm just getting started with nginx and the HttpLuaModule. I've created a test configuration file to familiarize myself with how things work.
now I'm trying to write some logic to accept GET, POST and DELETE requests for a specific type of resource.
I would like to create a "location" entry that would match the following URI / accept the following curl calls:
curl -i -X GET http://localhost/widgets/widget?name=testname&loc=20000 -H "Accept:application/json"
This is what my current nginx.conf looks like:
server {
listen 80;
server_name nsps2;
root /var/www/;
index index.html index.htm;
#charset koi8-r;
#access_log logs/host.access.log main;
#curl http://localhost/hello?name=johndoe
location /hello {
default_type "text/plain";
content_by_lua '
local rquri = ngx.var.request_uri;
ngx.say("the uri is ", rquri ,".")
local name = ngx.var.arg_name or "Anonymous"
ngx.say("Hello, ", name, "!")
';
}
location / {
root /var/www/;
index index.html index.htm;
}
#curl -i -X GET http://localhost/widgets/widget?name=testname&loc=20000 -H "Accept:application/json"
location /widgets/widget {
root /var/www/widgets;
default_type "text/pain";
content_by_lua '
local arga,argb = ngx.arg[1], ngx.arg[2] ;
ngx.say("the arga is ", arga ,".")
ngx.say("the argb is ", argb, ".")
';
}
Using the last "location" entry, I'm trying to
1. prove that the system is getting the GET request
2. prove that I understand how to access the parameters passed in with the GET request.
I'm getting an error right now that looks like this:
2015/02/24 20:18:19 [error] 2354#0: *1 lua entry thread aborted: runtime error: content_by_lua:2: API disabled in the context of content_by_lua*
stack traceback:
coroutine 0:
[C]: in function '__index'
content_by_lua:2: in function <content_by_lua:1>, client: 127.0.0.1, server: nsps2, request: "GET /widgets/widget?name=testname?loc=20000 HTTP/1.1", host: "localhost"
I'm not too sure about what this error means / is trying to tell me.
Any tips would be appreciated.
Thank you.
EDIT 1
I think the problem was something syntactically wrong. I was reading the manual and found an example to try. I've changed the code to look like this:
location /widgets/widget {
default_type "text/pain";
content_by_lua '
local args = ngx.req.get_uri_args()
for key, val in pairs(args) do
if type(val) == "table" then
ngx.say(key, ": ", table.concat(val, ", "))
else
ngx.say(key, ": ", val)
end
end
';
}
Now when I call the app like this:
mytestdevbox2:/var/www/nsps2# curl -i -X GET http://localhost/widgets/widget?name=testname&loc=20000 -H "Accept:application/json"
-ash: -H: not found
mytestdevbox2:/var/www/nsps2# HTTP/1.1 200 OK
Server: nginx/1.6.2
Date: Tue, 24 Feb 2015 21:32:44 GMT
Content-Type: text/pain
Transfer-Encoding: chunked
Connection: keep-alive
name: testname
[1]+ Done curl -i -X GET http://localhost/widgets/widget?name=testname
After the system displays the "name: testname" stuff, it just sits there until I hit "enter". After I do that, then it proceeds to display the 1+ stuff.
I'm not too sure what it's doing.
EDIT 2:
Adding quotes to the curl call did fix the problem:
mytestdevbox2:/var/www/nsps2# curl -i -X GET 'http://localhost/widgets/widget?name=testname&loc=20000'
HTTP/1.1 200 OK
Server: nginx/1.6.2
Date: Wed, 25 Feb 2015 12:59:29 GMT
Content-Type: text/pain
Transfer-Encoding: chunked
Connection: keep-alive
loc: 20000
name: testname
mytestdevbox2:/var/www/nsps2#
The problem described in EDIT 1 doesn't have anything to do with your question about nginx and is caused by not quoting the parameters for curl when you execute the command:
curl -i -X GET http://localhost/widgets/widget?name=testname&loc=20000
This is executed as two commands separated by '&': curl -i -X GET http://localhost/widgets/widget?name=testname and loc=20000; that's why you see the output after you already got the prompt back as the first command is now executed in the background. "1+ Done" message is a confirmation that the background process is terminated; it's just shown after you press Enter.
Wrap the URL with the query string in quotes and you should see the expected behavior.
My previous question was about fetching page title in lua using the socket.http module. The question lies here. Previously, youtube pages led me to a 404 error page. Based on MattJ's help, I put up custom HOST header for the request. This is what I did and what was the result:
Code
header = { host= "youtube.com" }
local result,b,c,h = http.request{ url = "http://www.youtube.com/watch?v=_eT40eV7OiI", headers = header }
print ( result, b, c, h )
for k,v in pairs(c) do print(k,v) end
Result
1 301 table: 0047D430 HTTP/1.1 301 Moved Permanently
x-content-type-options nosniff
content-length 0
expires Tue, 27 Apr 1971 19:44:06 EST
cache-control no-cache
connection close
location http://www.youtube.com/watch?v=_eT40eV7OiI
content-type text/html; charset=utf-8
date Sat, 28 Apr 2012 04:26:21 GMT
server wiseguy/0.6.11
As far as I was able to understand from this, the error is basically because of X-Content-Type-Options valued nosniff. Reading its documentation, I got to know that the only defined value, "nosniff", prevents Internet Explorer from MIME-sniffing a response away from the declared content-type.
Please help me so that I can use custom proxy and fetch the youtube(and some other sites, as mentioned in the previous question) title from their body. Here is the complete LUA file I currently have:
local http = require "socket.http"
http.PROXY="http://<proxy address here>:8080"
header = { host= "youtube.com" }
local result,b,c,h = http.request{ url = "http://www.youtube.com/watch?v=_eT40eV7OiI", headers = header }
print ( result, b, c, h )
for k,v in pairs(c) do print(k,v) end
I believe this line should be changed:
header = { host= "youtube.com" }
To:
header = { host= "www.youtube.com" }
After that, works for me.
The solution is to install luasec and to use ssl.https module to do the request.
Answered here by Paul Kulchenko!
Example:
-- luasec version 0.4.2
require("ssl")
require("https")
-- ssl.https.request(...)