I'm running an Argo workflow on a local MinIO K8s cluster. I'm setting up an Artifact Repository on MinIO where output artifacts from my workflow can be stored. I followed the instructions here https://argoproj.github.io/argo-workflows/configure-artifact-repository/#configuring-minio .
The error I'm running into is: failed to create new S3 client: Endpoint url cannot have fully qualified paths.
My MinIO endpoint is at http://127.0.0.1:52139.
Here is my workflow YAML file:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: artifactory-repository-ref-
spec:
archiveLogs: true
entrypoint: main
templates:
- name: main
container:
image: docker/whalesay:latest
command: [ sh, -c ]
args: [ "cowsay hello world | tee /tmp/hello_world.txt" ]
archiveLocation:
archiveLogs: true
outputs:
artifacts:
- name: hello_world
path: /tmp/hello_world.txt
Here is my workflow-controller-configmap YAML which is deployed in the same namespace as the workflow:
# This file describes the config settings available in the workflow controller configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: workflow-controller-configmap
data: # "config: |" key is optional in 2.7+!
artifactRepository: | # However, all nested maps must be strings
archiveLogs: true
s3:
endpoint: argo-artifacts:9000
bucket: my-bucket
insecure: true
accessKeySecret: #omit if accessing via AWS IAM
name: my-minio-cred
key: accessKey
secretKeySecret: #omit if accessing via AWS IAM
name: my-minio-cred
key: secretKey
useSDKCreds: true
I've created a secret called my-minio-cred in the same namespace where the workflow is running.
Here are the logs from the pod where the workflow is running:
time="2023-02-16T21:39:05.044Z" level=info msg="Starting Workflow Executor" version=v3.4.5
time="2023-02-16T21:39:05.047Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-02-16T21:39:05.047Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo podName=artifactory-repository-ref-5tcmt template="{\"name\":\"main\",\"inputs\":{},\"outputs\":{\"artifacts\":[{\"name\":\"hello_world\",\"path\":\"/tmp/hello_world.txt\"}]},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay:latest\",\"command\":[\"sh\",\"-c\"],\"args\":[\"cowsay hello world | tee /tmp/hello_world.txt\"],\"resources\":{}},\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"http://127.0.0.1:52897\",\"bucket\":\"my-bucket\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"my-minio-cred\",\"key\":\"accessKey\"},\"secretKeySecret\":{\"name\":\"my-minio-cred\",\"key\":\"secretKey\"},\"useSDKCreds\":true,\"key\":\"artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt\"}}}" version="&Version{Version:v3.4.5,BuildDate:2023-02-07T12:36:25Z,GitCommit:1253f443baa8ad1610d2e62ec26ecdc85fe1b837,GitTag:v3.4.5,GitTreeState:clean,GoVersion:go1.18.10,Compiler:gc,Platform:linux/arm64,}"
time="2023-02-16T21:39:05.047Z" level=info msg="Starting deadline monitor"
time="2023-02-16T21:39:08.048Z" level=info msg="Main container completed" error="<nil>"
time="2023-02-16T21:39:08.048Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2023-02-16T21:39:08.048Z" level=info msg="No output parameters"
time="2023-02-16T21:39:08.048Z" level=info msg="Saving output artifacts"
time="2023-02-16T21:39:08.048Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2023-02-16T21:39:08.048Z" level=info msg="Deadline monitor stopped"
time="2023-02-16T21:39:08.048Z" level=info msg="Staging artifact: hello_world"
time="2023-02-16T21:39:08.049Z" level=info msg="Copying /tmp/hello_world.txt from container base image layer to /tmp/argo/outputs/artifacts/hello_world.tgz"
time="2023-02-16T21:39:08.049Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/hello_world.txt.tgz -> /tmp/argo/outputs/artifacts/hello_world.tgz"
time="2023-02-16T21:39:08.049Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/hello_world.tgz, key: artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt/hello_world.tgz"
time="2023-02-16T21:39:08.049Z" level=info msg="Creating minio client using static credentials" endpoint="http://127.0.0.1:52897"
time="2023-02-16T21:39:08.049Z" level=warning msg="Non-transient error: Endpoint url cannot have fully qualified paths."
time="2023-02-16T21:39:08.049Z" level=info msg="Save artifact" artifactName=hello_world duration="282.917µs" error="failed to create new S3 client: Endpoint url cannot have fully qualified paths." key=artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt/hello_world.tgz
time="2023-02-16T21:39:08.049Z" level=error msg="executor error: failed to create new S3 client: Endpoint url cannot have fully qualified paths."
time="2023-02-16T21:39:08.049Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt/main.log"
time="2023-02-16T21:39:08.049Z" level=info msg="Creating minio client using static credentials" endpoint="http://127.0.0.1:52897"
time="2023-02-16T21:39:08.049Z" level=warning msg="Non-transient error: Endpoint url cannot have fully qualified paths."
time="2023-02-16T21:39:08.049Z" level=info msg="Save artifact" artifactName=main-logs duration="28.5µs" error="failed to create new S3 client: Endpoint url cannot have fully qualified paths." key=artifactory-repository-ref-5tcmt/artifactory-repository-ref-5tcmt/main.log
time="2023-02-16T21:39:08.049Z" level=error msg="executor error: failed to create new S3 client: Endpoint url cannot have fully qualified paths."
time="2023-02-16T21:39:08.056Z" level=info msg="Create workflowtaskresults 403"
time="2023-02-16T21:39:08.056Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:argo:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"argo\""
time="2023-02-16T21:39:08.057Z" level=info msg="Patch pods 403"
time="2023-02-16T21:39:08.057Z" level=warning msg="Non-transient error: pods \"artifactory-repository-ref-5tcmt\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-02-16T21:39:08.057Z" level=error msg="executor error: pods \"artifactory-repository-ref-5tcmt\" is forbidden: User \"system:serviceaccount:argo:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"argo\""
time="2023-02-16T21:39:08.057Z" level=info msg="Alloc=6350 TotalAlloc=12366 Sys=18642 NumGC=4 Goroutines=5"
time="2023-02-16T21:39:08.057Z" level=fatal msg="failed to create new S3 client: Endpoint url cannot have fully qualified paths."
I've tried changing the endpoint key in the workflow-controller-config.yaml from 127.0.0.1:52139 to 127.0.0.1:9000 and also argo-artifacts:9000 but it still doesn't work. argo-artifacts is the name of the LoadBalancer service thats created by the helm install argo-artifacts minio/minio command.
I got the endpoint of the MinIO bucket from
minikube service --url argo-artifacts as given in the 'Configuring MinIO' section at https://argoproj.github.io/argo-workflows/configure-artifact-repository/#configuring-minio
Everything is in the same namespace.
What could be wrong here?
I tried changing the endpoint URL of the MinIO bucket, changing namespaces for different components, and changing the namespace that the argo-artifacts service gets deployed in.
I am trying to put an ingress resource behind a whitelist using traefik 1.7.6 but it keeps returning 403 status code.
Note that Traefik is behind a Load Balancer that puts the X-Forwarded-For header to the request (https://docs.cloud.oracle.com/iaas/Content/Balance/Reference/httpheaders.htm).
Here is the traefik configuration
data:
traefik.toml: |
defaultEntryPoints = ["http"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.forwardedHeaders]
trustedIPs = ["<**load-balancer-ip**>"]
[entryPoints.traefik]
address = ":8080"
[entryPoints.traefik.auth.basic]
users = ["admin:secret"]
[kubernetes]
[kubernetes.ingressEndpoint]
publishedService = "default/mygateway"
[ping]
entryPoint = "http"
[api]
entryPoint = "traefik"
And here there is the ingress resource that should be whitelisted
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: frontend-ingress
annotations:
kubernetes.io/ingress.class: traefik
traefik.frontend.rule.type: PathPrefixStrip
ingress.kubernetes.io/whitelist-x-forwarded-for: "true"
traefik.ingress.kubernetes.io/whitelist-source-range: "<whitelisted-ips-range>"
spec:
rules:
- host:
http:
paths:
- path: /mypath
backend:
serviceName: myservice
servicePort: myserviceport
---
If I make a request, using one of the whitelisted ips I can see that Traefik isn't using the X-forwarder-header to resolve my real ip.
time="2019-02-28T14:31:51Z" level=debug msg="request &{Method:GET URL:/mypath Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Authorization:[Basic YWRtaW46YWRtaW4=] Referer:[https://<load-balancer-ip>/mypath] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9,it;q=0.8] Connection:[keep-alive] Access-Control-Allow-Origin:[*] Accept:[application/json, text/plain, */*] Dnt:[1] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:<load-balancer-ip> Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:<load-balancer-ip>:35818 RequestURI:/mypath TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000b5e5d0} - rejecting: "<load-balancer-ip>:35818\" matched none of the white list"
Adding service file.yml
---
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-controller
labels:
app: traefik-ingress-controller
annotations:
spec:
selector:
app: traefik-ingress-controller
ports:
- protocol: TCP
port: 80
targetPort: 80
nodePort: 30080
name: http
type: NodePort
externalTrafficPolicy: Local
---
UPDATE: Trefik debug logs
time="2019-03-01T13:57:15Z" level=info msg="Using TOML configuration file /config/traefik.toml"
time="2019-03-01T13:57:15Z" level=info msg="Traefik version v1.7.6 built on 2018-12-14_06:43:37AM"
time="2019-03-01T13:57:15Z" level=debug msg="Global configuration loaded {\"LifeCycle\":{\"RequestAcceptGraceTimeout\":0,\"GraceTimeOut\":10000000000},\"GraceTimeOut\":0,\"Debug\":false,\"CheckNewVersion\":true,\"SendAnonymousUsage\":false,\"AccessLogsFile\":\"\",\"AccessLog\":null,\"TraefikLogsFile\":\"\",\"TraefikLog\":null,\"Tracing\":null,\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Address\":\":80\",\"TLS\":null,\"Redirect\":null,\"Auth\":null,\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":false,\"TrustedIPs\":[\"<loadbalancer-ip-1>/32\",\"<loadbalancer-ip-2>/32\"]}},\"traefik\":{\"Address\":\":8080\",\"TLS\":null,\"Redirect\":null,\"Auth\":{\"basic\":{\"users\":[\"admin:$apr1$FV9Q2vjA$gJt2UT8bt6DT6RVDm5qI20\"]}},\"WhitelistSourceRange\":null,\"WhiteList\":null,\"Compress\":false,\"ProxyProtocol\":null,\"ForwardedHeaders\":{\"Insecure\":true,\"TrustedIPs\":null}}},\"Cluster\":null,\"Constraints\":[],\"ACME\":null,\"DefaultEntryPoints\":[\"http\"],\"ProvidersThrottleDuration\":2000000000,\"MaxIdleConnsPerHost\":200,\"IdleTimeout\":0,\"InsecureSkipVerify\":false,\"RootCAs\":null,\"Retry\":null,\"HealthCheck\":{\"Interval\":30000000000},\"RespondingTimeouts\":null,\"ForwardingTimeouts\":null,\"AllowMinWeightZero\":false,\"KeepTrailingSlash\":false,\"Web\":null,\"Docker\":null,\"File\":null,\"Marathon\":null,\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeeper\":null,\"Boltdb\":null,\"Kubernetes\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":{\"IP\":\"\",\"Hostname\":\"\",\"PublishedService\":\"default/anotherservice\"}},\"Mesos\":null,\"Eureka\":null,\"ECS\":null,\"Rancher\":null,\"DynamoDB\":null,\"ServiceFabric\":null,\"Rest\":null,\"API\":{\"EntryPoint\":\"traefik\",\"Dashboard\":true,\"Debug\":false,\"CurrentConfigurations\":null,\"Statistics\":null},\"Metrics\":null,\"Ping\":{\"EntryPoint\":\"http\"},\"HostResolver\":null}"
time="2019-03-01T13:57:15Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n"
time="2019-03-01T13:57:15Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0003550e0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2019-03-01T13:57:15Z" level=info msg="Preparing server traefik &{Address::8080 TLS:<nil> Redirect:<nil> Auth:0xc000525aa0 WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc00002a5a0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2019-03-01T13:57:15Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
time="2019-03-01T13:57:15Z" level=info msg="Starting server on :80"
time="2019-03-01T13:57:15Z" level=info msg="Starting server on :8080"
time="2019-03-01T13:57:15Z" level=info msg="Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":{\"IP\":\"\",\"Hostname\":\"\",\"PublishedService\":\"default/anotherservice\"}}"
time="2019-03-01T13:57:15Z" level=debug msg="Using Ingress label selector: \"\""
time="2019-03-01T13:57:15Z" level=info msg="ingress label selector is: \"\""
time="2019-03-01T13:57:15Z" level=info msg="Creating in-cluster Provider client"
time="2019-03-01T13:57:16Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/frontend-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/service-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/other-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/whitelist-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/traefik-dashboard-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Received Kubernetes event kind *v1.Secret"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/frontend-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/service-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/other-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/whitelist-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping updating Ingress default/traefik-dashboard-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:16Z" level=debug msg="Skipping Kubernetes event kind *v1.Secret"
time="2019-03-01T13:57:16Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"/fe\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/promo\":{\"servers\":{\"promo-api-service-7d5968c556-rrfk2\":{\"url\":\"http://10.11.3.53:9189\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/anotherservice\":{\"servers\":{\"anotherservice-7856788888-dwnjc\":{\"url\":\"http://10.11.4.4:8762\",\"weight\":1},\"anotherservice-7856788888-kczj5\":{\"url\":\"http://10.11.3.9:8762\",\"weight\":1},\"anotherservice-7856788888-sxtzt\":{\"url\":\"http://10.11.5.6:8762\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/sockjs-node\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/static\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/traefik\":{\"servers\":{\"traefik-ingress-controller-5945dd5fbf-hkt6p\":{\"url\":\"http://10.11.3.99:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"/fe\":{\"entryPoints\":[\"http\"],\"backend\":\"/fe\",\"routes\":{\"/fe\":{\"rule\":\"PathPrefixStrip:/fe\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null,\"whiteList\":{\"sourceRange\":[\"<whitelisted-ip-subnet>/24\",\"<whitelisted-ip-subnet>/24\"],\"useXForwardedFor\":true}},\"/promo\":{\"entryPoints\":[\"http\"],\"backend\":\"/promo\",\"routes\":{\"/promo\":{\"rule\":\"PathPrefix:/promo\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/anotherservice\":{\"entryPoints\":[\"http\"],\"backend\":\"/anotherservice\",\"routes\":{\"/anotherservice\":{\"rule\":\"PathPrefixStrip:/anotherservice\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/sockjs-node\":{\"entryPoints\":[\"http\"],\"backend\":\"/sockjs-node\",\"routes\":{\"/sockjs-node\":{\"rule\":\"PathPrefix:/sockjs-node\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/static\":{\"entryPoints\":[\"http\"],\"backend\":\"/static\",\"routes\":{\"/static\":{\"rule\":\"PathPrefix:/static\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/traefik\":{\"entryPoints\":[\"http\"],\"backend\":\"/traefik\",\"routes\":{\"/traefik\":{\"rule\":\"PathPrefixStrip:/traefik\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2019-03-01T13:57:16Z" level=debug msg="Wiring frontend /fe to entryPoint http"
time="2019-03-01T13:57:16Z" level=debug msg="Creating backend /fe"
time="2019-03-01T13:57:16Z" level=debug msg="configured IP white list: [<whitelisted-ip-subnet>/24 <whitelisted-ip-subnet>/24]"
time="2019-03-01T13:57:16Z" level=debug msg="Configured IP Whitelists: [<whitelisted-ip-subnet>/24 <whitelisted-ip-subnet>/24]"
time="2019-03-01T13:57:16Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /fe"
time="2019-03-01T13:57:16Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:16Z" level=debug msg="Creating server myservice-5c55585b8-cxprc at http://10.11.4.70:3000 with weight 1"
time="2019-03-01T13:57:16Z" level=debug msg="Creating route /fe PathPrefixStrip:/fe"
time="2019-03-01T13:57:16Z" level=debug msg="Wiring frontend /promo to entryPoint http"
time="2019-03-01T13:57:16Z" level=debug msg="Creating backend /promo"
time="2019-03-01T13:57:16Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /promo"
time="2019-03-01T13:57:16Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:16Z" level=debug msg="Creating server promo-api-service-7d5968c556-rrfk2 at http://10.11.3.53:9189 with weight 1"
time="2019-03-01T13:57:16Z" level=debug msg="Creating route /promo PathPrefix:/promo"
time="2019-03-01T13:57:16Z" level=debug msg="Wiring frontend /anotherservice to entryPoint http"
time="2019-03-01T13:57:16Z" level=debug msg="Creating backend /anotherservice"
time="2019-03-01T13:57:16Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /anotherservice"
time="2019-03-01T13:57:16Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:16Z" level=debug msg="Creating server anotherservice-7856788888-dwnjc at http://10.11.4.4:8762 with weight 1"
time="2019-03-01T13:57:16Z" level=debug msg="Creating server anotherservice-7856788888-kczj5 at http://10.11.3.9:8762 with weight 1"
time="2019-03-01T13:57:16Z" level=debug msg="Creating server anotherservice-7856788888-sxtzt at http://10.11.5.6:8762 with weight 1"
time="2019-03-01T13:57:16Z" level=debug msg="Creating route /anotherservice PathPrefixStrip:/anotherservice"
time="2019-03-01T13:57:17Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"/fe\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/promo\":{\"servers\":{\"promo-api-service-7d5968c556-rrfk2\":{\"url\":\"http://10.11.3.53:9189\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/anotherservice\":{\"servers\":{\"anotherservice-7856788888-dwnjc\":{\"url\":\"http://10.11.4.4:8762\",\"weight\":1},\"anotherservice-7856788888-kczj5\":{\"url\":\"http://10.11.3.9:8762\",\"weight\":1},\"anotherservice-7856788888-sxtzt\":{\"url\":\"http://10.11.5.6:8762\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/sockjs-node\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/static\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/traefik\":{\"servers\":{\"traefik-ingress-controller-5945dd5fbf-hkt6p\":{\"url\":\"http://10.11.3.99:8080\",\"weight\":1},\"traefik-ingress-controller-5945dd5fbf-xd5b6\":{\"url\":\"http://10.11.4.88:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"/fe\":{\"entryPoints\":[\"http\"],\"backend\":\"/fe\",\"routes\":{\"/fe\":{\"rule\":\"PathPrefixStrip:/fe\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null,\"whiteList\":{\"sourceRange\":[\"<whitelisted-ip-subnet>/24\",\"<whitelisted-ip-subnet>/24\"],\"useXForwardedFor\":true}},\"/promo\":{\"entryPoints\":[\"http\"],\"backend\":\"/promo\",\"routes\":{\"/promo\":{\"rule\":\"PathPrefix:/promo\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/anotherservice\":{\"entryPoints\":[\"http\"],\"backend\":\"/anotherservice\",\"routes\":{\"/anotherservice\":{\"rule\":\"PathPrefixStrip:/anotherservice\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/sockjs-node\":{\"entryPoints\":[\"http\"],\"backend\":\"/sockjs-node\",\"routes\":{\"/sockjs-node\":{\"rule\":\"PathPrefix:/sockjs-node\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/static\":{\"entryPoints\":[\"http\"],\"backend\":\"/static\",\"routes\":{\"/static\":{\"rule\":\"PathPrefix:/static\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/traefik\":{\"entryPoints\":[\"http\"],\"backend\":\"/traefik\",\"routes\":{\"/traefik\":{\"rule\":\"PathPrefixStrip:/traefik\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/other-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/whitelist-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/traefik-dashboard-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/frontend-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:17Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/whitelist-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/traefik-dashboard-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/frontend-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/service-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/other-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:17Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/frontend-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/service-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/other-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/whitelist-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping updating Ingress default/traefik-dashboard-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:17Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:18Z" level=debug msg="Wiring frontend /fe to entryPoint http"
time="2019-03-01T13:57:18Z" level=debug msg="Creating backend /fe"
time="2019-03-01T13:57:18Z" level=debug msg="configured IP white list: [<whitelisted-ip-subnet>/24 <whitelisted-ip-subnet>/24]"
time="2019-03-01T13:57:18Z" level=debug msg="Configured IP Whitelists: [<whitelisted-ip-subnet>/24 <whitelisted-ip-subnet>/24]"
time="2019-03-01T13:57:18Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /fe"
time="2019-03-01T13:57:18Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server myservice-5c55585b8-cxprc at http://10.11.4.70:3000 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating route /fe PathPrefixStrip:/fe"
time="2019-03-01T13:57:18Z" level=debug msg="Wiring frontend /promo to entryPoint http"
time="2019-03-01T13:57:18Z" level=debug msg="Creating backend /promo"
time="2019-03-01T13:57:18Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /promo"
time="2019-03-01T13:57:18Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server promo-api-service-7d5968c556-rrfk2 at http://10.11.3.53:9189 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating route /promo PathPrefix:/promo"
time="2019-03-01T13:57:18Z" level=debug msg="Wiring frontend /anotherservice to entryPoint http"
time="2019-03-01T13:57:18Z" level=debug msg="Creating backend /anotherservice"
time="2019-03-01T13:57:18Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /anotherservice"
time="2019-03-01T13:57:18Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server anotherservice-7856788888-kczj5 at http://10.11.3.9:8762 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server anotherservice-7856788888-sxtzt at http://10.11.5.6:8762 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server anotherservice-7856788888-dwnjc at http://10.11.4.4:8762 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating route /anotherservice PathPrefixStrip:/anotherservice"
time="2019-03-01T13:57:18Z" level=debug msg="Wiring frontend /sockjs-node to entryPoint http"
time="2019-03-01T13:57:18Z" level=debug msg="Creating backend /sockjs-node"
time="2019-03-01T13:57:18Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /sockjs-node"
time="2019-03-01T13:57:18Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server myservice-5c55585b8-cxprc at http://10.11.4.70:3000 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating route /sockjs-node PathPrefix:/sockjs-node"
time="2019-03-01T13:57:18Z" level=debug msg="Wiring frontend /static to entryPoint http"
time="2019-03-01T13:57:18Z" level=debug msg="Creating backend /static"
time="2019-03-01T13:57:18Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /static"
time="2019-03-01T13:57:18Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server myservice-5c55585b8-cxprc at http://10.11.4.70:3000 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating route /static PathPrefix:/static"
time="2019-03-01T13:57:18Z" level=debug msg="Wiring frontend /traefik to entryPoint http"
time="2019-03-01T13:57:18Z" level=debug msg="Creating backend /traefik"
time="2019-03-01T13:57:18Z" level=debug msg="Adding TLSClientHeaders middleware for frontend /traefik"
time="2019-03-01T13:57:18Z" level=debug msg="Creating load-balancer wrr"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server traefik-ingress-controller-5945dd5fbf-hkt6p at http://10.11.3.99:8080 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating server traefik-ingress-controller-5945dd5fbf-xd5b6 at http://10.11.4.88:8080 with weight 1"
time="2019-03-01T13:57:18Z" level=debug msg="Creating route /traefik PathPrefixStrip:/traefik"
time="2019-03-01T13:57:18Z" level=info msg="Server configuration reloaded on :80"
time="2019-03-01T13:57:19Z" level=debug msg="Configuration received from provider kubernetes: {\"backends\":{\"/fe\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/promo\":{\"servers\":{\"promo-api-service-7d5968c556-rrfk2\":{\"url\":\"http://10.11.3.53:9189\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/anotherservice\":{\"servers\":{\"anotherservice-7856788888-dwnjc\":{\"url\":\"http://10.11.4.4:8762\",\"weight\":1},\"anotherservice-7856788888-kczj5\":{\"url\":\"http://10.11.3.9:8762\",\"weight\":1},\"anotherservice-7856788888-sxtzt\":{\"url\":\"http://10.11.5.6:8762\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/sockjs-node\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/static\":{\"servers\":{\"myservice-5c55585b8-cxprc\":{\"url\":\"http://10.11.4.70:3000\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}},\"/traefik\":{\"servers\":{\"traefik-ingress-controller-5945dd5fbf-69jkg\":{\"url\":\"http://10.11.5.90:8080\",\"weight\":1},\"traefik-ingress-controller-5945dd5fbf-hkt6p\":{\"url\":\"http://10.11.3.99:8080\",\"weight\":1},\"traefik-ingress-controller-5945dd5fbf-xd5b6\":{\"url\":\"http://10.11.4.88:8080\",\"weight\":1}},\"loadBalancer\":{\"method\":\"wrr\"}}},\"frontends\":{\"/fe\":{\"entryPoints\":[\"http\"],\"backend\":\"/fe\",\"routes\":{\"/fe\":{\"rule\":\"PathPrefixStrip:/fe\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null,\"whiteList\":{\"sourceRange\":[\"<whitelisted-ip-subnet>/24\",\"<whitelisted-ip-subnet>/24\"],\"useXForwardedFor\":true}},\"/promo\":{\"entryPoints\":[\"http\"],\"backend\":\"/promo\",\"routes\":{\"/promo\":{\"rule\":\"PathPrefix:/promo\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/anotherservice\":{\"entryPoints\":[\"http\"],\"backend\":\"/anotherservice\",\"routes\":{\"/anotherservice\":{\"rule\":\"PathPrefixStrip:/anotherservice\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/sockjs-node\":{\"entryPoints\":[\"http\"],\"backend\":\"/sockjs-node\",\"routes\":{\"/sockjs-node\":{\"rule\":\"PathPrefix:/sockjs-node\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/static\":{\"entryPoints\":[\"http\"],\"backend\":\"/static\",\"routes\":{\"/static\":{\"rule\":\"PathPrefix:/static\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null},\"/traefik\":{\"entryPoints\":[\"http\"],\"backend\":\"/traefik\",\"routes\":{\"/traefik\":{\"rule\":\"PathPrefixStrip:/traefik\"}},\"passHostHeader\":true,\"priority\":0,\"basicAuth\":null}}}"
time="2019-03-01T13:57:19Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:19Z" level=debug msg="Received Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:19Z" level=debug msg="Skipping updating Ingress default/traefik-dashboard-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:19Z" level=debug msg="Skipping updating Ingress default/frontend-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:19Z" level=debug msg="Skipping updating Ingress default/service-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:19Z" level=debug msg="Skipping updating Ingress default/other-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:19Z" level=debug msg="Skipping updating Ingress default/whitelist-ingress due to service default/anotherservice having no status set"
time="2019-03-01T13:57:19Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints"
time="2019-03-01T13:57:20Z" level=debug msg="Wiring frontend /fe to entryPoint http"
time="2019-03-01T13:57:20Z" level=debug msg="Creating backend /fe"
time="2019-03-01T13:57:20Z" level=debug msg="configured IP white list: [<whitelisted-ip-subnet>/24 <whitelisted-ip-subnet>/24]"
time="2019-03-01T13:57:20Z" level=debug msg="Configured IP Whitelists: [<whitelisted-ip-subnet>/24 <whitelisted-ip-subnet>/24]"
time="2019-02-28T08:49:02Z" level=debug msg="request &{Method:GET URL:/ Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36] Dnt:[1] Accept-Encoding:[gzip, deflate, br] X-Forwarded-Prefix:[/<my-path>] Connection:[keep-alive] Cache-Control:[max-age=0] Authorization:[Basic YWRtaW46YWRtaW4=] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8] Accept-Language:[en-US,en;q=0.9,it;q=0.8]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:10.29.5.13 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:<load-balancer-ip>:50846 RequestURI:/ TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000baf440} - rejecting: \"<load-balancer-ip>:50846\" matched none of the white list