I am trying to route the traffic/connections to backend director based on url parameters which are passed on incoming request.
Here is my current config
sub vcl_recv {
if (req.url ~ "sto") {
set req.backend_hint = endpoint1.backend();
}
}
Currently, above config does static mapping/check for sto. I would like to do a dynamic lookup from map/hash/KV file and do condition check with url contents.
Here is how I do it in haproxy
use_backend %[path,map_sub(/etc/haproxy/GET.map)]
#>cat GET.map
# url paramter backend location/name
/sgtier get_backend1
/sto get_backend2
Config file
vcl 4.0;
import directors;
backend sg_gw1 {
.host = "1.8.4.4";
.port = "1043";
}
backend tss_gw1 {
.host = "1.8.13.2";
.port = "8080";
}
sub vcl_recv {
if (req.url ~ "sto") {
set req.backend_hint = sg_gw1;
}
elsif (req.url ~ "common") {
set req.backend_hint = tss_gw1;
}
}
Dynamic backends are supported in Varnish Enterprise. See https://docs.varnish-software.com/varnish-cache-plus/vmods/goto/ for more information.
An open source alternative for dynamic backends can be found on https://code.uplex.de/uplex-varnish/libvmod-backend_dyn
Related
im trying to create a service for web redirect through the application gateway using terraform.
I would like to authenticate the application gateway sl with the free certified (azurm_app_service_managed_certified) of the azure app service plan, is there an example?
Currently, thinking about the composition as follows. However, azurem_application_gateway is demanding ssl certification, so I don't know how to work.
Please let me know if there's a way to solve the problem in that way or in another way.
The problem with the script below is that if you want to use https in the application gateway, you have to use certificate, and I want to make and use free certificated in the service plan.
resource "azurerm_application_gateway" "app_gateway" {
provider = azurerm.generic
name = "${local.service_name}-app-gateway"
resource_group_name = azurerm_resource_group.rg.name
location = azurerm_resource_group.rg.location
enable_http2 = true
sku {
name = "Standard_Small"
tier = "Standard" # v1
capacity = 2
}
gateway_ip_configuration {
name = "${local.service_name}-ip-config"
subnet_id = azurerm_subnet.front_subnet.id
}
frontend_port {
name = local.frontend_port_name
port = 80
}
frontend_port {
name = local.backend_port_name
port = 443
}
frontend_ip_configuration {
name = local.frontend_ip_configuration_name
public_ip_address_id = azurerm_public_ip.pub_ip.id
}
backend_address_pool {
name = "${azurerm_virtual_network.vn.name}-beap"
fqdns = [local.host_name]
}
backend_http_settings {
name = local.http_setting_name
cookie_based_affinity = "Disabled"
port = 443
protocol = "Https"
request_timeout = 60
host_name = local.host_name
}
http_listener {
name = "${local.listener_name}-http"
frontend_ip_configuration_name = local.frontend_ip_configuration_name
frontend_port_name = local.frontend_port_name
protocol = "Http"
}
http_listener {
name = "${local.listener_name}-https"
frontend_ip_configuration_name = local.frontend_ip_configuration_name
frontend_port_name = local.backend_port_name
protocol = "Https"
}
request_routing_rule {
name = "${local.request_routing_rule_name}-http"
rule_type = "Basic"
http_listener_name = "${local.listener_name}-http"
backend_address_pool_name = local.backend_address_pool_name
backend_http_settings_name = local.http_setting_name
}
redirect_configuration {
name = local.redirect_configuration_name
redirect_type = "Permanent"
include_path = false
include_query_string = false
target_listener_name = "${local.listener_name}-https"
}
request_routing_rule {
name = "${local.request_routing_rule_name}-https"
rule_type = "Basic"
http_listener_name = "${local.listener_name}-https"
redirect_configuration_name = local.redirect_configuration_name
}
lifecycle {
ignore_changes = [
backend_address_pool,
backend_http_settings,
frontend_port,
http_listener,
request_routing_rule,
ssl_certificate,
redirect_configuration
]
}
}
resource "azurerm_dns_zone" "zone" {
provider = azurerm.generic
for_each = toset(local.dns_zone_names)
name = each.key
resource_group_name = azurerm_resource_group.rg.name
}
resource "azurerm_app_service_plan" "service_plan" {
provider = azurerm.generic
name = "${local.service_name}-service-plan"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
sku {
tier = "Basic"
size = "B1"
}
}
resource "azurerm_app_service" "service" {
provider = azurerm.generic
name = "${local.service_name}-service"
app_service_plan_id = azurerm_app_service_plan.service_plan.id
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
}
resource "azurerm_app_service_custom_hostname_binding" "service_host_bind" {
provider = azurerm.generic
count = length(local.dns_zone_names)
hostname = "${local.dns_zone_names[count.index]}"
app_service_name = azurerm_app_service.service.name
resource_group_name = azurerm_resource_group.rg.name
lifecycle {
ignore_changes = [ssl_state, thumbprint]
}
depends_on = [
azurerm_app_service.service,
azurerm_resource_group.rg
]
}
resource "azurerm_app_service_managed_certificate" "service_manage_cert" {
provider = azurerm.generic
count = length(local.dns_zone_names)
custom_hostname_binding_id = azurerm_app_service_custom_hostname_binding.service_host_bind[count.index].id
}
resource "azurerm_app_service_certificate_binding" "service_certi_bind" {
provider = azurerm.generic
count = length(local.dns_zone_names)
hostname_binding_id = azurerm_app_service_custom_hostname_binding.service_host_bind[count.index].id
certificate_id = azurerm_app_service_managed_certificate.service_manage_cert[count.index].id
ssl_state = "SniEnabled"
}
i want a service that simply directs to another website through dns using terraform, and if there is any other way, please let us know. (include http to https)
To protect and prevent website abuse, we would like to redirect multiple domains to one website.
ex : (adomain.net -> www.target.com, adomain.tv -> www.target.com, bdomain.net -> www.target.com)
Fist of all there is no support for app services managed certificate with application gateway as of now.
Yes, you can do redirection from multiple domains to one domain using system.webserver rewrite rule either inside app services web.config file or application gateway rewrite rule.
I am trying to Provision VMs with Terraform.
i give the source image for the vm
here:
source = "http://10.1.1.160/Builds/14.7.1.10_0.39637/output/KVM/14.7.1.10_0.39637-disk1.qcow2"
but this site require a user name and a password.
where and how can i specify the credentials of this site in my tf file?
this is my main.tf file:
terraform {
required_providers {
libvirt = {
source = "dmacvicar/libvirt"
}
}
}
provider "libvirt" {
uri = "qemu:///system"
}
resource "libvirt_volume" "centos7-qcow2" {
name = "centos7.qcow2"
pool = "default"
source = "http://10.1.1.160/Builds/14.7.1.10_0.39637/output/KVM/14.7.1.10_0.39637-disk1.qcow2"
format = "qcow2"
}
# Define KVM domain to create
resource "libvirt_domain" "gw" {
name = "gw"
memory = "8192"
vcpu = 4
network_interface {
network_name = "default"
}
disk {
volume_id = "${libvirt_volume.centos7-qcow2.id}"
}
console {
type = "pty"
target_type = "serial"
target_port = "0"
}
graphics {
type = "spice"
listen_type = "address"
autoport = true
}
}
when i run terraform apply i got this error:
*Error while determining image type for http://10.1.1.160/Builds/14.7.1.10_0.39637/output/KVM/14.7.1.10_0.39637-disk1.qcow2: Can't retrieve partial header of resource to determine file type: http://10.1.1.160/Builds/14.7.1.10_0.39637/output/KVM/14.7.1.10_0.39637-disk1.qcow2 - 401 Authorization Required
with libvirt_volume.centos7-qcow2,
on main.tf line 13, in resource "libvirt_volume" "centos7-qcow2":
13: resource "libvirt_volume" "centos7-qcow2" {*
thanks for helping!
I just added the password and the user to the URL, like this:
http://user:password#host/path
:)
We are using Magento 2 with Varnish cache
We only get Varnish Cache HIT on very few /catalogsearch/result/ pages, and we really can not figure out why we don't get cache HIT on all /catalogsearch/result/ pages.
Please help us in the right direction :-)
Ex.
we always get HIT on this url
https://www.babygear.dk/catalogsearch/result/?q=bog
We always get MISS on a lot of other search queries
https://www.babygear.dk/catalogsearch/result/?q=black
https://www.babygear.dk/catalogsearch/result/?q=box
https://www.babygear.dk/catalogsearch/result/?q=box
Here is our varnish.vlc
# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 5
vcl 4.0;
import std;
# The minimal Varnish version is 5.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'
backend default {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 600s;
}
acl purge {
"127.0.0.1";
}
sub vcl_recv {
if (req.method == "PURGE") {
if (client.ip !~ purge) {
return (synth(405, "Method not allowed"));
}
# To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
# has been added to the response in your backend server config. This is used, for example, by the
# capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
}
if (req.http.X-Magento-Tags-Pattern) {
ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
}
if (req.http.X-Pool) {
ban("obj.http.X-Pool ~ " + req.http.X-Pool);
}
return (synth(200, "Purged"));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
# Bypass shopping cart, checkout
if (req.url ~ "/checkout") {
return (pass);
}
# Bypass health check requests
if (req.url ~ "/pub/health_check.php") {
return (pass);
}
# Set initial grace period usage status
set req.http.grace = "none";
# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");
# collect all cookies
std.collect(req.http.Cookie);
# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
# No point in compressing these
unset req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
set req.http.Accept-Encoding = "deflate";
} else {
# unknown algorithm
unset req.http.Accept-Encoding;
}
}
# Remove all marketing get parameters to minimize the cache objects
if (req.url ~ "(\?|&)(gclid|ff|fp|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
set req.url = regsuball(req.url, "(gclid|ff|fp|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
set req.url = regsub(req.url, "[?|&]+$", "");
}
# Static files caching
if (req.url ~ "^/(pub/)?(media|static)/") {
# Static files should not be cached by default
#return (pass);
# But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
unset req.http.Https;
unset req.http.X-Forwarded-Proto;
unset req.http.Cookie;
}
return (hash);
}
sub vcl_hash {
if (req.http.cookie ~ "X-Magento-Vary=") {
hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}
# For multi site configurations to not cache each other's content
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
hash_data(req.http.X-Forwarded-Proto);
}
if (req.url ~ "/graphql") {
call process_graphql_headers;
}
}
sub process_graphql_headers {
if (req.http.Store) {
hash_data(req.http.Store);
}
if (req.http.Content-Currency) {
hash_data(req.http.Content-Currency);
}
}
sub vcl_backend_response {
set beresp.grace = 3d;
if (beresp.http.content-type ~ "text") {
set beresp.do_esi = true;
}
if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
set beresp.do_gzip = true;
}
if (beresp.http.X-Magento-Debug) {
set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}
# cache only successfully responses
if (beresp.status != 200) {
set beresp.ttl = 0s;
set beresp.uncacheable = true;
return (deliver);
} elsif (beresp.http.Cache-Control ~ "private") {
set beresp.uncacheable = true;
set beresp.ttl = 44400s;
return (deliver);
}
# validate if we need to cache it and prevent from setting cookie
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
unset beresp.http.set-cookie;
}
# If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
if (beresp.ttl <= 0s ||
beresp.http.Surrogate-control ~ "no-store" ||
(!beresp.http.Surrogate-Control &&
beresp.http.Cache-Control ~ "no-cache|no-store") ||
beresp.http.Vary == "*") {
# Mark as Hit-For-Pass for the next 2 minutes
set beresp.ttl = 120s;
set beresp.uncacheable = true;
}
return (deliver);
}
sub vcl_deliver {
if (resp.http.X-Magento-Debug) {
if (resp.http.x-varnish ~ " ") {
set resp.http.X-Magento-Cache-Debug = "HIT";
set resp.http.Grace = req.http.grace;
} else {
set resp.http.X-Magento-Cache-Debug = "MISS";
}
} #else {
# unset resp.http.Age;
# }
# Not letting browser to cache non-static files.
if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
set resp.http.Pragma = "no-cache";
set resp.http.Expires = "-1";
set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
}
unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;
}
sub vcl_hit {
if (obj.ttl >= 0s) {
# Hit within TTL period
return (deliver);
}
if (std.healthy(req.backend_hint)) {
if (obj.ttl + 3000000s > 0s) {
# Hit after TTL expiration, but within grace period
set req.http.grace = "normal (healthy server)";
return (deliver);
} else {
# Hit after TTL and grace expiration
return (miss);
}
} else {
# server is not healthy, retrieve from cache
set req.http.grace = "unlimited (unhealthy server)";
return (deliver);
}
}
It's a bit tough to judge what's really going on, because there's both a Varnish cache in front of Magento and Cloudflare as the CDN.
A no-cache/no-store Cache-Control value
What I am seeing in general for your searches, is the following Cache-Control value:
cache-control: no-store, no-cache, must-revalidate, max-age=0
Based on this value, Varnish will decide not to cache. In the response headers of must of your search results you will see that Age: 0 is set. This means that Varnish doesn't hold the value in cache.
A cached search result that shouldn't be cacheable
However, weirdly enough https://www.babygear.dk/catalogsearch/result/?q=bog does have a an Age header with a value greater than zero:
age: 38948
This means it's been in cache for 38948 seconds. But really, this shouldn't be happening, because the page is not supposed to be cacheable.
Making search results cacheable:
Please make sure you have a Cache-Control header that allows caching, if you want to cache search results.
Example:
Cache-Control: public, s-maxage=3600
Debugging using Varnishlog
If you really want to know what happens in behind the scenes in Varnish, you can perform some debugging using the varnishlog binary.
You could run the following command to get debug output:
varnishlog -g request -q "ReqUrl eq '/catalogsearch/result/\?q=bog'"
This will print some very verbose logs on how Varnish treats the URL that causes the hit. You can add this output to your question, and I can try to examine what's going on.
FYI: I wrote a very detailed blog post about varnishlog a couple of years ago. Please go to https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/ to have a look, and to learn.
What's Cloudflare doing?
All that being said, I have no clue what the impact of Cloudflare is on the cacheability of the website. The varnishlog output will give us some insight, but if those results diverge from reality, Cloudflare is probably getting in our way.
Keep this in mind while debugging.
Inside vcl_recv Add
# Bypass search requests
if (req.url ~ "/catalogsearch") {
return (pass);
}
check .htaccess file for Cache-Control setting.
I would like an array of URLs, let's say ['/about','/supported-software', '/the-team', ...] to simply redirect to /.
Do I need to write multiple location { } blocks?
I'm new to NGINX configuration so any guidance would be very much appreciated!
If the array entries are exact matches, then the following locations should give you the best performance:
location = /about { return 301 $scheme://$host; }
location = /supported-software { return 301 $scheme://$host; }
location = /the-team { return 301 $scheme://$host; }
# ... or put these in an included file
If they are not exact matches, a map may be better:
map $uri $send_home {
~^/about/? 1;
~^/supported-software/? 1;
~^/the-team/? 1;
# ... or put these in an included file
}
server {
# ...
if ($send_home) {
return 301 $scheme://$host;
}
# ...
}
The map would allow more flexible redirects like:
/about
/about/
/about/stuff
I have nginx+php-fpm and some rewrite rules for sef-urls.
The problem is that all my custom redirects/rewrites are ignored and request going to php script instead redirecting.
Part for sef-links:
if ($request_filename !~ ".(png|gif|ico|swf|jpe?g|js|css)$"){
set $rule_0 1$rule_0;
}
if (!-f $request_filename){
set $rule_0 2$rule_0;
}
if (!-d $request_filename){
set $rule_0 3$rule_0;
}
if ($rule_0 = "321"){
rewrite /. /index.php?sef_rewrite=1 last;
}
And I want to do that redirect:
location = /first.html {
return 301 /second.html;
}
You need to thing in terms of nginx's locations and rules. Then it will be much easier to write proper config. Try this one:
location / {
# replacement for last three `if`s
try_files $uri $uri/ /index.php?sef_rewrite=1;
}
# here is you redirect
location = /first.html {
return 301 /second.html;
}
# this is replacement for first `if`
location ~ \.(png|gif|ico|swf|jpe?g|js|css)$ {
# serve static files
}
# I'm sure you have this block somewhere
location ~ \.php$ {
# serve php
}
Also these articles worth to read them:
http://wiki.nginx.org/Pitfalls
http://wiki.nginx.org/IfIsEvil