We have a self-managed gitlab instance running in docker container and an external url set as: https://subdomain.domain.com:50080
I’ve put gitlab_kas[‘enable’] = true in the docker-compose file in “GITLAB_OMNIBUS_CONFIG: |” and try to add the agent with helm in “Connect a Kubernetes cluster”, but the kasAddress does not contain the 50080 port:
–set config.kasAddress=wss://subdomain.domain.com:/-/kubernetes-agent/
and the agent pod gives this error:
{“level”:“error”,“time”:“2022-09-07T07:32:50.899Z”,“msg”:“Error handling a connection”,“mod_name”:“reverse_tunnel”,“error”:“Connect(): rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \"https://subdomain.domain.com/-/kubernetes-agent/\\\”: context deadline exceeded""}
If I add the port manually to the kasAddress in the helm command the gitlab-kas/current log gives this error:
2022-09-07_07:44:01.55475 {“level”:“error”,“time”:“2022-09-07T07:44:01.553Z”,“msg”:“AgentInfo()”,“correlation_id”:“01GCBE78S0SX2BA5B48M3813W4”,“grpc_service”:“gitlab.agent.reverse_tunnel.rpc.ReverseTunnel”,“grpc_method”:“Connect”,“error”:“Get "https://subdomain.domain.com:50080/api/v4/internal/kubernetes/agent_info\”: dial tcp PUBLIC_IP:50080: i/o timeout"}
I've changed the external_url to use the default 443 port for https but the same i/o timeout error is found in the kas log
The problem was upstream on the mikrotik router. I needed a masquerade nat rule for the srcnat chain. In other words, the gitlab was not able to reach itself on the public IP
Related
I am attempting to make a local request to the kubernetes cluster that is hosted on my server, the cluster's NodePort is listening at the following address 172.20.120.1:30280. External client in production are required to make requests to 172.20.0.1:8000 (this cannot change), so I am attempting to add a DNAT rule to nat the traffic from:
172.20.0.1:8000 -> 172.20.120.1:30280 (k8s NodePort)
I am able to make curl request to 172.20.120.1:30280 directly and get a successful response back. However, when I make a curl request to 172.20.0.1:8000 it just hangs with the following message:
# curl -vvvk https://172.20.0.1:8000/v1/my-api
* About to connect() to 172.20.0.1 port 8000 (#0)
* Trying 172.20.0.1...
* Connected to 172.20.0.1 (172.20.0.1) port 8000 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
And then it eventually times out with the following error:
...
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
curl: (35) TCP connection reset by peer
When I make a request directly to 172.20.120.1:30280 I don't get that cert error and it works. I get a successful response back.
Does anyone know why I am getting that cert error?
I tried to add user global permissions to a cluster, but fail to add, received the following error message:
Internal error occurred: failed calling webhook "rancherauth.cattle.io": Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s": dial tcp 10.X.X.X: connect: connection refused
I found nothing about this.
Thanks.
I have an application running in my GKE cluster that needs access to www.googleapis.com. I also make use of Network Policy to enhance security.
With a default deny all egress traffic in place, I cannot connect to www.googleapis.com naturally. I get the error
INFO 0827 14:33:53.313241 retry_util.py] Retrying request, attempt #3...
DEBUG 0827 14:33:53.313862 http_wrapper.py] Caught socket error, retrying: timed out
DEBUG 0827 14:33:53.314035 http_wrapper.py] Retrying request to url https://www.googleapis.com/storage/v1/b?project=development&projection=noAcl&key=AIzaSyDnac<key>bmJM&fields=nextPageToken%2Citems%2Fid&alt=json&maxResults=1000 after exception timed out
I found out that the hostname www.googleapis.com corresponds to the IP 216.58.207.36
So I went ahead an created an egress entry in my Network Policy
spec:
egress:
- ports:
- port: 443
protocol: TCP
to:
- ipBlock:
cidr: 216.58.207.36/32
And now from within the Pod, I can telnet this endpoint
$ telnet googleapis.com 443
Trying 216.58.207.36...
Connected to googleapis.com.
Escape character is '^]'.
But for some reason Im still encountering the same error
INFO 0827 14:36:15.767508 retry_util.py] Retrying request, attempt #5...
DEBUG 0827 14:36:15.768018 http_wrapper.py] Caught socket error, retrying: timed out
DEBUG 0827 14:36:15.768128 http_wrapper.py] Retrying request to url https://www.googleapis.com/storage/v1/b?project=development&projection=noAcl&key=AIzaSyDnac<key>bmJM&fields=nextPageToken%2Citems%2Fid&alt=json&maxResults=1000 after exception timed out
However if I delete the network policy, I can connect
INFO 0827 14:40:24.177456 base_api.py] Body: (none)
INFO 0827 14:40:24.177595 transport.py] Attempting refresh to obtain initial access_token
WARNING 0827 14:40:24.177864 multiprocess_file_storage.py] Credentials file could not be loaded, will ignore and overwrite.
DEBUG 0827 14:40:24.177957 multiprocess_file_storage.py] Read credential file
WARNING 0827 14:40:24.178036 multiprocess_file_storage.py] Credentials file could not be loaded, will ignore and overwrite.
DEBUG 0827 14:40:24.178090 multiprocess_file_storage.py] Read credential file
WARNING 0827 14:40:24.356631 multiprocess_file_storage.py] Credentials file could not be loaded, will ignore and overwrite.
DEBUG 0827 14:40:24.356972 multiprocess_file_storage.py] Read credential file
DEBUG 0827 14:40:24.357510 multiprocess_file_storage.py] Wrote credential file /var/lib/jenkins/.gsutil/credstore2.
connect: (www.googleapis.com, 443)
send: 'GET /storage/v1/b?project=development&fields=nextPageToken%2Citems%2Fid&alt=json&projection=noAcl&maxResults=1000 HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-length: 0\r\nauthorization: REDACTED
My Network Policy allows ALL ingress traffic by default
ingress:
- {}
podSelector: {}
Any idea what I might be missing here ? Is there some other IP address that I need to whitelist in this case ?
EDIT
When the network Policy is in place, I did a test using curl and I get
* Trying 2a00:1450:4001:80b::200a...
* TCP_NODELAY set
* Immediate connect fail for 2a00:1450:4001:80b::200a: Cannot assign requested address
* Trying 2a00:1450:4001:80b::200a...
* TCP_NODELAY set
* Immediate connect fail for 2a00:1450:4001:80b::200a: Cannot assign requested address
* Trying 2a00:1450:4001:80b::200a...
* TCP_NODELAY set
* Immediate connect fail for 2a00:1450:4001:80b::200a: Cannot assign requested address
* Trying 2a00:1450:4001:80b::200a...
* TCP_NODELAY set
* Immediate connect fail for 2a00:1450:4001:80b::200a: Cannot assign requested address
* Trying 2a00:1450:4001:80b::200a...
* TCP_NODELAY set
* Immediate connect fail for 2a00:1450:4001:80b::200a: Cannot assign requested address
* Trying 2a00:1450:4001:80b::200a...
* TCP_NODELAY set
* Immediate connect fail for 2a00:1450:4001:80b::200a: Cannot assign requested address
This does not happen when the Network Policy is deleted.
The comment from #mensi is correct, there are multiple IPs behind www.googleapis.com. You can for example see that by pinging the URL multiple times, you'll most likely get a different IP every time.
The easiest solution would be to allow all egress by default with:
spec:
podSelector: {}
egress:
- {}
policyTypes:
- Egress
You could also try allowing all of the Google API's public IP ranges, but as Google doesn't seem to publish a list of those (only the restricted.googleapis.com and private.googleapis.com here), that might be a bit tougher.
This is a fresh install on Ubuntu 16.04.
I have been able to change the port and edit the "/etc/gitlab/gitlab.rb" file.
changes;
external_url 'http://superawesomedomain.com:2345'
nginx['listen_port'] = 2345
nginx['proxy_set_headers'] = {
"Host" => "$http_host",
"X-Real-IP" => "$remote_addr",
"X-Forwarded-For" => "$proxy_add_x_forwarded_for",
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
When I try to access Gitlab from the browser, I get a 502 error "Whoops, GitLab is taking too much time to respond."
And this in the logs:
==> /var/log/gitlab/nginx/gitlab_error.log <== 2016/05/04 00:43:53 [error] 1599#0: *14 connect() to
unix:/var/opt/gitlab/gitlab-workhorse/socket failed (111: Connection
refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server:
superawesomedomain.com, request: "GET /favicon.ico HTTP/1.1", upstream:
"http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/favicon.ico",
host: "superawesomedomain.com:2345", referrer:
"http://superawesomedomain.com:2345/"
The only ports configured behind NAT to work on this machine are; 2345 and 8080.
What am I missing? Ultimately I would prefer that it be https://superawesomedomain.com:2345/
I was able to get this working by using the IP of the server instead of the URL in the config:
external_url 'http://192.168.0.20:2345'
After doing that, GitLab was accessible from the //superawesomedomain.com:2345/ address. I am not sure why this worked, but it seems this is the only way to get it working with NAT and forwarded ports.
I am trying to configure smtp for organisational mail in alfresco 4.2e.
Here is my global configuration
mail.host=smthost.abcd.com
mail.port=25
mail.username=sameer#abcd.com
mail.password=password
mail.encoding=UTF-8
mail.from.default=sameer#abcd.com
mail.smtp.auth=false
I am unable to send the mail. Here is
org.springframework.mail.MailSendException: Mail server connection failed; nested exception is javax.mail.MessagingException: Could not connect to SMTP host: smtphost.abcd.com, port: 25;
nested exception is:
java.net.SocketException: Network is unreachable: connect. Failed messages: javax.mail.MessagingException: Could not connect to SMTP host: smtphost.abcd.com, port: 25;
nested exception is:
java.net.SocketException: Network is unreachable: connect; message exception details (1) are:
Failed message 1:
javax.mail.MessagingException: Could not connect to SMTP host: smtphost.abcd.com, port: 25;
I am able to connect using
telnet smtphost.abcd.com25
and I am able to ping
Your network (most likely your gateway) is telling you it cannot reach the network of abc.efgh.ijkl.com. If you test, make sure to execute telnet from the server giving you the error. Telnet should give you same error message.
tcpdump -n -i any port 25
will show all outgoing traffic targeting SMTP and should give you further information about the problem at the network level (such as IP addresses used).
After so much search I found the issue of the problem. The problem occured because java using IPv6 by default. To fix,
I disabled IPV6 on windows
Set the Java property as setx _JAVA_OPTIONS -Djava.net.preferIPv4Stack=true