I like to use scrapyRT to connect all my scraper spiders to the Angular front-end.
The documentation recommends to run in it in production as follows:
docker run -p 9080:9080 -tid -v /home/user/quotesbot:/scrapyrt/project scrapinghub/scrapyrt
This will expose port 9080 to the outside world.
I'm concerned about the safety of this.
How can I properly secure this to avoid bots visiting the end points?
If Scrapyrt is running on the same machine as your frontend, you could make Scrapyrt listen only on localhost:
docker run -p 9080:9080 -tid -v /home/user/quotesbot:/scrapyrt/project scrapinghub/scrapyrt -i 127.0.0.1
That way it will be accessible only from that machine (e.g. from the frontend application).
Otherwise use firewall to allow communication from your frontend machine only.
Related
I have to consume an external rest API(using restTemplate.exchange) with Spring Boot. My rest API is running on port 8083 with URL http://localhost:8083/myrest (Docker command : docker run -p 8083:8083 myrest-app)
External API is available in form of public docker image and after running below command , I am able to pull and run it locally.
docker pull dockerExternalId/external-rest-api docker
run -d -p 3000:3000 dockerExternalId/external-rest-api
a) If I enter external rest API URL, for example http://localhost:3000/externalrestapi/testresource directly in chrome, then I get valid JSON data.
b) If I invoke it with myrest application from eclipse(Spring Boot Application), still I am getting valid JSON Response. (I am using Windows Platform to test this)
c) But if I run it on Docker and execute myrest service (say http://localhost:8083/myrest), then i am facing java.net.ConnectException: Connection refused
More details :
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:3000/externalrestapi/testresource": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
P.S - I am using Docker on Windows.
# The problem
You run with:
docker run -p 8083:8083 myrest-app
But you need to run like:
docker run --network "host" --name "app" myrest-app
So passing the flag --network with value host will allow you container to access your computer network.
Please ignore my first approach, instead use a better alternative that does not expose the container to the entire host network... is possible to make it work, but is not a best practice.
A Better Alternative
Create a network to be used by both containers:
docker network create external-api
Then run both containers with the flag --network external-api.
docker run --network "external-api" --name "app" -p 8083:8083 myrest-app
and
docker run -d --network "external-api" --name "api" -p 3000:3000 dockerExternalId/external-rest-api
The use of flag -p to publish the ports for the api container are only necessary if you want to access it from your computers browser, otherwise just leave them out, because they aren't needed for 2 containers to communicate in the external-api network.
TIP: docker pull is not necessary, once docker run will try to pull the image if does not found it in your computer.
Let me know how it went...
Call the External API
So in both solutions I have added the --name flag so that we can reach the other container in the network.
So to reach the external api from my rest app you need to use the url http://api:3000/externalrestapi/testresource.
Notice how I have replaced localhost by api that matches the value for --name flag in the docker run command for your external api.
From your myrest-app container if you try to access http://localhost:3000/externalrestapi/testresource, it will try to access 3000 port of the same myrest-app container.
Because each container is a separate running Operating System and it has its own network interface, file system, etc.
Docker is all about Isolation.
There are 3 ways by which you can access an API from another container.
Instead of localhost, provide the IP address of the external host machine (i.e the IP address of your machine on which docker is running)
Create a docker network and attach these two containers. Then you can provide the container_name instead of localhost.
Use --link while starting the container (deprecated)
We can use the command docker images to list the Docker images we have on local host.
Now I want to get the same information from a remote server by sending an HTTP GET request in Firefox or Chrome. Does Docker provide some REST API to do this?
I did a lot of search. For example:
Examples using the Docker Engine SDKs and Docker API
It provides a way something like this:
curl --unix-socket /var/run/docker.sock http:/v1.24/containers/json
I know a little about Unix sockets, and I don't think this is what I want. The URL (http:/v1.24/containers/json) is so weird and don't even have a server name in it. I don't think it can work on a remote server. (It does work on a local server.)
Is there any official documentation that Docker provides on this topic?
You need to expose the Docker daemon on a port.
You can configure the Docker daemon to listen to multiple sockets at the same time using multiple -H options:
listen using the default Unix socket, and on two specific IP addresses on this host.
$ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2
The Docker client will honor the DOCKER_HOST environment variable to set the -H flag for the client. Use one of the following commands:
https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option
You need to do this by creating a systemd dropin:
mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/10_docker.conf <<EOF
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// -H tcp://0.0.0.0:2376
EOF
Then reload and restart Docker:
systemctl daemon-reload
systemctl restart docker
Note: this way you would be exposing your host and you shouldn't do it this way in production. Please read more about this on the link I shared earlier.
I've created a kubenetes cluster on my Mac with docker-machine, following the documentation here:
http://kubernetes.io/docs/getting-started-guides/docker/
I can access the normal api from inside the instance on 127.0.0.1:8080, but I want to access it externally from my macbook. I know there is a secure port :6443, but I'm unsure how to set up the credentials to access this port.
There are lots of instructions on how to do it on custom installs of kubernetes, but I don't know how to do it inside the docker containers I'm running.
Likely, you will want to use Virtual Box's port forwarding capabilities. An example from the documentation:
VBoxManage modifyvm "MyVM" --natpf1 "k8srule,tcp,,6443,,6443"
This forwards port 6443 on all hosts interfaces to port 6443 of the guest. Port forwarding can also be configured through the VirtualBox UI.
It's like a workaround but most of the time, I think KubeOnDocker setup is for developper that don't need the credentials mecanism :
When you start the KubeOnDocker, --config=/etc/kubernetes/manifests point to master.json. If you look the apiserver start command, you will see that --insecure-bind-address is 127.0.0.1. If you use --config=/etc/kubernetes/manifests-multi it will point to master-multi.json, --insecure-bind-address will be 0.0.0.0 and the apiserver will be accessible from everywhere.
Note that you will need to start etcd with manifests-multi.
# Not tested start
docker run \
-d \
--net=host \
gcr.io/google_containers/etcd:2.2.1 \
/usr/local/bin/etcd \
--listen-client-urls=http://127.0.0.1:4001 \
--advertise-client-urls=http://127.0.0.1:4001 \
--data-dir=/var/etcd/data
I am trying to deploy a Scala based application to dokku, the application runs a http server and a customised sshd server.
The problem I have is it seems that dokku only supports one port for the application.
I need dokku to expose both my applications ports to the web.
In docker this is possible and quite straight forward to do, but when I implement the same technique in the dokku file, I get an error.
Any suggestions on allowing two ports to be accessible?
Since this is, after all, docker, you can use an ambassador...
You will need a line like:
docker run -t -i -link mysql:mysql -name mysql_ambassador -p 3306:3306 ctlc/ambassador
Replacing with your port and mysql with your container name (from docker images)
See https://www.ctl.io/developers/blog/post/deploying-multi-server-docker-apps-with-ambassadors
NOTE: Make sure you docker pull svendowideit/ambassador:latest before...
I want to setup serve multiple sites from one server:
1. http://www.example.org => node.js-www (running on port (50000)
2. http://files.example.org => node.js-files (running on port 50001)
Until now I only found out to have docker doing port redirect when using static ips.
Is is actual possible to use docker for port redirection via hostname?
I use a free amazon EC2 insance.
Thanks
Bo
EDIT:
I want to have multiple nodes applications running on the same port but however serving a different hostname.
As far as I'm aware docker does not have such functionality built in, nor it should.
To accomplish what you're trying to do you'd probably need some sort of reverse proxy, so node.js or nginx would do. Bouncy might be a good option: https://github.com/substack/bouncy
There is a great docker project on GitHub called nginx-proxy by jwilder.
This allows you to create a docker container that is doing a reverse-proxy by mapping only his port 80/443 to the host, instead of other containers. Then, all you have to do is for every new web container you create, provide a new environment variable VIRTUAL_HOST=some.domain.com.
An example:
Create a new nginx-proxy container
docker run -d -p 80:80 --net shared_hosting -v /var/run/docker.sock:/tmp/docker.sock:ro jwilder/nginx-proxy
Create a container for each website. For example:
docker run -d -p 80 --net shared_hosting -e VIRTUAL_HOST=hello1.domain.com tutum/hello-world
docker run -d -p 80 --net shared_hosting -e VIRTUAL_HOST=drupal.domain.com drupal
You need to make sure that the hosts you own, configured in DNS to point to the server that runs the docker container. In this example, I will add the to the /etc/hosts file:
echo "127.0.0.1 hello1.domain.com drupal.domain.com" >> /etc/hosts
Navigate to http://hello1.domain.com and then to http://drupal.domain.com, and see that they both use port 80 but give you a different pages.
An important note about this service. As you noticed, I have added --net argument, this is because all containers you want to be a part of a shared hosting (proxy and websites) must be on the same virtual network (this can be defined by the argument --net or --network to the docker run command), especially when you use docker-compose to create dockers, because docker-compose creates its own virtual network, thus makes one container not reachable by another, so make sure the network is explicitly defined in the docker-compose.yml file.
Hope it helps.
I used varnish as a docker container that worked as my reverse proxy
it's on the docker index
https://index.docker.io/u/sysdia/docker-varnish/
I know this is an old question, but ran across it and wanted to point out that there are much cleaner ways to do what was requested. Since you are using AWS, you can have each of your two hostnames pointing at their own load balancer (ELB) in Route53. You could then deploy your container into ECS, for example, listening on both ports. Each of those load balancers can redirect traffic to the appropriate listening port. Now you have accomplished what you want, and if your traffic becomes too heavy or imbalanced, you can easily split the tasks into two different ECS clusters so they can scale independently.