sqlx.Connect() get stuck in docker alpine:latest - postgresql

I have problem that I've managed to reduce to the following code:
package main
import (
"fmt"
"github.com/jmoiron/sqlx"
_ "github.com/lib/pq"
"os"
)
func main() {
addr := os.Getenv("DB")
fmt.Println("Postgres addr: " + addr)
_, err := sqlx.Connect("postgres", addr)
if err != nil {
fmt.Println("Could not connect...")
} else {
fmt.Println("Connecting successful")
}
}
I set up a repo with the code and more explanations at:
https://github.com/mraxus/mystery-golang-alpine
When I build and run this Go code with a valid DB url in a docker image (here golang:latest) throught docker-compose, where both the above program and the postgres db is in separate containers, the program runs as expected:
build_1 | Postgres addr: postgres://postgres#postgres/postgres?sslmode=disable
build_1 | Connecting successful
However, when I run the same program in same setup (docker-compose) with the base image alpine:latest, the program just gets stuck at the sqlx.Connect():
alpine_1 | Postgres addr: postgres://postgres#postgres/postgres?sslmode=disable
I have no idea why this is. Do you know? I have setup a project to see if others can reproduce and get the same problem as I get:
https://github.com/mraxus/mystery-golang-alpine
Love to hear some insights that can help me solve this issue.
My system details:
macOS 10.12.6 (Sierra, MBP Mid 2015 15-inch)
docker 17.06.1 1-ce-mac24

Proper solution (by realizing the actual problem)
So it turns out that the corporate network at work have a search domain set. This affects the alpine containers name resolution. However, the default golang is not.
To solve the problem, you can overwrite docker-compose containers search domain, by modifying the config:
build:
dns_search: .
image: image:build
...
alpine:
dns_search: .
image: image:alpine
...
See https://github.com/mraxus/mystery-golang-alpine/pull/4
Alternative solution (not having realized the actual problem)
By forcing go to use the cgo name resolver, there is no longer any issues:
In docker-compose.yml
alpine:
image: mamarcus.org/project:alpine
links:
- postgres
environment:
DB: "postgres://postgres#postgres/postgres?sslmode=disable"
GODEBUG: "netdns=cgo"
See https://github.com/mraxus/mystery-golang-alpine/pull/3
It should be mentioned that this solution is still iffy. In our real dev environment, containing ~20 services configured in docker-compose, some docker alpine images still did not resolve properly. But when updating the configuration with the "proper solution", everything worked like a charm.

Related

baseVariants in site definition with ports

I use docker on Mac OS X. On Mac OS X there is no possibility to map a domain name like mysite.local to something like http://localhost:8015/ etc.
So I have to call my docker container with http://localhost:8015/ The backend works as expected and shows me the applicationContext Development/Docker. The frontend works too, but only when i use the base entry in site.config:
base: http://localhost:8015/
works but:
baseVariants:
-
base: http://localhost:8015/
condition: 'applicationContext == "Development/Docker"'
does not.
Any ideas why or any workaround?
Thanks!
OK, i found the solution: it was a minor error in my configuration. In the htaccess file of my docker volume i wrote:
SetEnv TYPO3_CONTEXT=Development/Docker
this syntax is wrong but it works in the backend ... so TYPO3 showed me in the backend the context "Development/Docker". But it does not work in the frontend ... This syntax works:
SetEnv TYPO3_CONTEXT Development/Docker
Now my baseVariants runs as expected.
Sorry for the noise ... :-)

Running end to end tests with Testcontainers with docker-compose inside Gitlab-ci

We've written end to end tests with Testcontainers. A docker-compose file is loaded with the following Testcontainers method :
#Container
public static final DockerComposeContainer<?> COMPOSE_CONTAINER =
new DockerComposeContainer<>(new File("src/test/resources/docker-compose-test.yml"))
.withLocalCompose(true)
.withExposedService(ZOOKEEPER_SERVICE, ZOOKEEPER_PORT)
.withExposedService(BROKER_SERVICE, BROKER_PORT)
.withExposedService(BROKER_SERVICE, BROKER_PORT_LOCALHOST)
.withExposedService(SCHEMA_REGISTRY_SERVICE, SCHEMA_REGISTRY_PORT)
.withExposedService(VAULT_SERVICE, VAULT_PORT)
.withExposedService(ELASTICSEARCH_SERVICE, ELASTICSEARCH_PORT_1)
.withExposedService(ELASTICSEARCH_SERVICE, ELASTICSEARCH_PORT_2)
.waitingFor(ELASTICSEARCH_SERVICE, Wait.forHttp("/").forStatusCode(200))
.waitingFor(VAULT_SERVICE, Wait.forHttp("/").forStatusCode(200))
.waitingFor(SCHEMA_REGISTRY_SERVICE, Wait.forHttp("/subjects").forStatusCode(200));
It works locally but it fails in our Gitlab-CI test stage. We tried to add docker-compose to the stage but it still doesn't work. The test stage has been written as follow :
test:
stage: test
services:
- docker:dind
script:
- apk add --no-cache docker-compose
- docker-compose --version
- mvn clean test -Djavax.net.ssl.trustStore=${CI_PROJECT_DIR}/src/main/resources/cacerts -Djavax.net.ssl.trustStorePassword=${TRURST_STORE_PWD}
artifacts:
reports:
junit: ./target/surefire-reports/*.xml
The error in the pipe is :
Time elapsed: 101.927 s <<< ERROR!
org.testcontainers.containers.ContainerLaunchException: Local Docker Compose not found. Is docker-compose on the PATH?
I hope someone has already set up something similar and will help us :)
Best regards.
Likely it's because you can't just use docker:dind since that's docker in docker, you need dcind which is docker compose in docker. I'm not sure if you can try docker:dcind or if there's another dcind image you can try out that's on docker hub.

How to run a custom docker image testContainer

I have gone thru' multiple blogs and official documentation but couldn't resolve my issue. I am using testContainers-scala version 0.38.1 and scala version 2.11.
I am trying to create a simple test using testContainer-scala as below:
class MyServiceITSpec extends AnyFlatSpec with ForAllTestContainer {
override val container = GenericContainer(dockerImage="my-service",
exposedPorts = Seq(8080),
env=(HashMap[String, String]("PARAM1" -> "value1", "PARAM2" -> "value2", "PARAM3" -> "value3")),
waitStrategy = Wait.forHttp("/")
)
"GenericContainer" should "start my service and say Hello! Wassupp" in {
assert(Source.fromInputStream(
new URL(s"http://${container.containerIpAddress}:${container.mappedPort(8080)}/").openConnection().getInputStream
).mkString.contains("Hello! Wassupp"))
}
}
On the basis of the above snippet, my understanding is this (please correct if wrong):
Port 8155 is exposed by the docker container and a random host port against the same would be assigned.
We can get that assigned port as container.mappedPort
Here I am trying to assert that http:// localhost:mappedPort/ return Hello! Wassupp.
But, I get the below error:
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:498)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 18 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://localhost:32869/ should return HTTP 200)
at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:214)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:890)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:441)
... 20 more
The same image runs just fine with:
docker run -p 8081:8080 -e PARAM1=value1 -e PARAM2=value2 -e PARAM3=VALUE3 my-service
So after juggling with the errors, I found my issue. It is to do with the required Request Headers missing from the request. I am adding the reference code for anyone who runs into similar issue.
import com.dimafeng.testcontainers.{ForAllTestContainer, GenericContainer}
import org.scalatest.flatspec.AnyFlatSpec
import org.testcontainers.containers.wait.strategy.Wait
import scala.collection.immutable.HashMap
import scalaj.http.Http
class MyServiceITSpec extends AnyFlatSpec with ForAllTestContainer {
override val container = GenericContainer(dockerImage="my-service-img:tag12345",
exposedPorts = Seq(8080),
env=(HashMap[String, String]("PARAM1" -> "value1", "PARAM2" -> "value2")),
waitStrategy = Wait.forHttp("/") // or "/health" based on ur implementation
)
"My Service" should "successfully fetch the msg" in {
assert(Http(s"http://${container.containerIpAddress}:${container.mappedPort(8080)}/products/product1")
.header("HEADER1", "value1")
.header("HEADER2", "value2")
.asString.code==200)
}
}
Some explanations that I found after a lot of reading:
You give the port number that your docker application exposes as exposedPorts.
TestContainers then does a mapping of this port against a random port (this is by design to avoid port number conflicts). If you were to run this docker image directly on your machine you would write:
docker run -p 8081:8080 -e PARAM1=value1 -e PARAM2=value2 my-service-img:tag12345
Here, your exposed port is 8080 and the mapped port is 8081.
TestContainers runs the docker image by exposing the port 8080 and then mapping it against a random port. The mapped port can be container.mappedPort().
Another important thing to notice is the wait strategy. This tells the code to wait unless the / endpoint gets up. This is kind of a health check that your application exposes. You can have a better endpoint for the same - like /health. By default, it waits for 60 seconds for this endpoint to become active. Post that it would anyway run the tests and if the application has not started by then, it would cause an error. I am not sure how to override the default timeout but I think there should be a way to do that.
Lastly, I am using scalaj.http.Http to make a HTTP request(it is a pretty easy one to use - you can ).

AWS-ECS - Communication between containers - Unknown host error

I have two Docker containers.
TestWeb (Expose: 80)
TestAPI (Expose: 80)
Testweb container calls TestApi container. Host can communicate with TestWeb container from port 8080. Host can communicate with TestApi using 8081.
I can get TestWeb to call TestApi in my dev box (Windows 10) but when I deploy the code to AWS (ECS) I get "unknown host" exception. Both the containers work just fine and I can call them individually. But when I call a method that internally makes a Rest call using HttpClient to a method in Container2, it gives the error:
An error occurred while sending the request. ---> System.Net.Http.CurlException: Couldn't resolve host name.
Code:
using (var client = new HttpClient())
{
try
{
string url = "http://testapi/api/Tenant/?i=" + id;
var response = client.GetAsync(url).Result;
if (response.IsSuccessStatusCode)
{
var responseContent = response.Content;
string responseString = responseContent.ReadAsStringAsync().Result;
return responseString;
}
return response.StatusCode.ToString();
}
catch (HttpRequestException httpRequestException)
{
return httpRequestException.Message;
}
}
The following are the things I have tried:
The two containers (TestWeb, TestAPI) are in the same Task definition in AWS ECS. When I inspect the containers, I get the IP address of each of the containers. I can ping container2 from container1 with their IP address. But I can't ping using container2 name. It gives me "unknown host" error.
It appears ECS doesn't use legit docker-compose under the hood, however, their implementation does support the Compose V2 "links" feature.
Here is a portion of my compose file I just ran on ECS that needed this same functionality AND had the same "could not resolve host" error you were getting. The "links" I added fixed my hostname resolution issue on Elastic Container Service!
version: '3'
services:
appserver:
links:
- database:database
- socks-proxy:socks-proxy
This allowed my appserver to communicate TO the database and socks-proxy hostnames. The format is "SERVICE:ALIAS" and it is fine to keep them both the same as a default practice.
In your example it would be:
version: '3'
services:
testapi:
links:
- testweb:testweb
testweb:
links:
- testapi:testapi
AWS does not use Docker compose but provides a interface to add Task Definitions.
Containers that need to communicate together can be put on the same Task definition. Then we can also specify in the links section the containers that will be called from the current container. Each container can be given its container name on the "Host" section of Task Definition. Once I added the container name to the "Host" field, Container1 (TestWeb) was able to communicate with Container2 (TestAPI).

python-memcache memcached -- I installed on centos virtualbox but it get/set never seem to work

I'm using python. I did a yum install memcached followed by a easy_install python-memcached
I used the simple test program from the Help(memcache). When I wasn't getting the proper answers I threw in some print statements:
[~/test]$ cat m2.py
import memcache
mc = memcache.Client(['127.0.0.1:11211'], debug=0)
x = mc.set("some_key", "Some value")
print 'Just set a key and value into the cache (suposedly)'
value = mc.get("some_key")
print 'Just retrieved that value from the cache using the key'
print 'X %s' % x
print 'Value %s' % value
[~/test]$ python m2.py
Just set a key and value into the cache (suposedly)
Just retrieved that value from the cache using the key
X 0
Value None
[~/test]$
The question now is, what have I failed to do in my installation? It appears to be working from an API perspective but it fails to put anything into the memcache share area.
I'm using a virtualbox vm running centos
[~]# cat /proc/version
Linux version 2.6.32-358.6.2.el6.i686 (mockbuild#c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Thu May 16 18:12:13 UTC 2013
Is there a daemon that is supposed to be running? I don't see an obvious named one when I do a ps.
I tried to get pylibmc installed on my vm but was unable to find a working installation so for now will see if I can get the above stuff working first.
I discovered if i ran straight from the python console GUI i get a bit more output if I set debug=1
>>> mc = memcache.Client(['127.0.0.1:11211'], debug=1)
>>> mc.stats
{}
>>> mc.set('test','value')
MemCached: MemCache: inet:127.0.0.1:11211: connect: Connection refused. Marking dead.
0
>>> mc.get('test')
MemCached: MemCache: inet:127.0.0.1:11211: connect: Connection refused. Marking dead.
When I try to use per the example telnet to connect to the port i get a connection refused:
[root#~]# telnet 127.0.0.1 11211
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
[root#~]#
I tried the instructions I found on the net for configuring telnet so localhost wouldn't be disabled:
vi /etc/xinetd.d/telnet
service telnet
{
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/sbin/in.telnetd
log_on_failure += USERID
disable = no
}
And then ran the commands to restart the service(s):
service iptables stop
service xinetd stop
service iptables start
service xinetd start
service iptables stop
I ran with both cases (iptables started and stopped) but it has no effect. So I am out of ideas. What do I need to do to make it so the PORT will be allowed? if that is the problem?
Or is there a memcached service that needs to be running that needs to open up the port ?
well this is what it took to get it working: ( a series of manual steps )
1) su -
cd /var/run
mkdir memcached # this was missing
In the memcached file I added "-l 127.0.0.1" to the OPTIONS statement. It's apparently a listen option. Do this for steps 2 & 3. I'm not certain which file is actually used at runtime.
2) cd /etc/sysconfig
cp memcached memcached.old
vi memcached
3) cd /etc/init.d
cp memcached memcached.old
vi memcached
4) Try some commands to see if the server starts now
/etc/init.d/memcached start
/etc/init.d/memcached status
/etc/init.d/memcached stop
/etc/init.d/memcached restart
I tried opening a browser, but it never seemed to actually display anything so I don't really know how valid this approach is. I'm not running apache or anything like this so perhaps its not relevant to my cause. Perhaps I would have to supply a ?key=blah or something.
5) http://127.0.0.1:11211
6) Now it should be ready to go. If one runs the test shown with the following it should work. At least it did for me. doing the help(memcache) will display a simple program. just paste that in and it should work just fine.
[~]$ python
>>> import memcache
>>> help(memcache)