Flexget does not listen to my configuration - docker-compose

I have flexget in docker compose and it doesn't work for me
That is, I have this configuration that downloads files from an rss and renames and moves them. But flexget doesn't do any of that
web_server:
bind: 0.0.0.0
port: 5050
web_ui: yes
schedules:
- tasks: '*'
interval:
minutes: 1
templates:
transmissionrpc:
transmission:
host: localhost
port: 9091
username: admin
password: "123456"
clean_transmission:
host: localhost
port: 9091
username: admin
password: "123456"
transmission_seed_limits: yes
delete_files: no
enabled: Yes
tv:
thetvdb_lookup: yes
quality: 720p
series:
group:
- Criminal Minds
- Rick and Morty
tasks:
Drarbg task:
rss: http://showrss.info/user/229110.rss?magnets=true&namespaces=true&name=null&quality=null&re=null
priority: 1
all_series: yes
template:
- tv
- transmissionrpc
set:
path: /downloads/complete
sort-series:
metainfo_series: yes
accept_all: yes
filesystem:
path: /downloads/complete
regexp: '.*\.(avi|mkv|mp4)$'
recursive: yes
template: tv
series:
settings:
group:
parse_only: yes
require_field: series_name
move:
to: '/storage/Series/{{series_name}}'
rename: '{{series_name}} - S{{series_season|pad(2)}}E{{series_episode|pad(2)}}'
in the flexget.log it does not give me any error.
I've tried various configurations but it still doesn't work. I don't know what to do anymore. I've opened a help ticket on their github but no one helps me.
My docker-compose:
flexget:
image: wiserain/flexget:3
volumes:
- ${CONFIG}:/config
- ${STORAGE}/torrents:/downloads
- ${MEDIA}:/storage
ports:
- 5050:5050
environment:
- TORRENT_PLUGIN=transmission
- FG_WEBUI_PASSWD=123456
restart: unless-stopped
links:
- transmission
The web_ui does not work either, that is, when I put the ip of my machine followed by :5050 it tells me that the connection has been rejected

Related

Error 504 Gateway Timeout when trying to access a homeserver service through an SSH tunnel and traefik

Situation: I run Home Assistant on an Ubuntu server on my home LAN network. Because my home network is behind a double NAT, I have set up an SSH tunnel to tunnel the Home Assistant web interface to a VPS server running Ubuntu as well.
When I run the following on the VPS, I notice that the SSH tunnel works as expected:
$ curl localhost:8045 | grep -iPo '(?<=<title>)(.*)(?=</title>)'
Home Assistant
On the VPS, I run a bunch of web services via docker-compose and traefik. The other services (caddy, portainer) run without problems.
When I try to serve the Home Assistant service through traefik and access https://ha.mydomain.com through a web browser, I get an Error 504 Gateway Timeout.
Below are my configuration files. What am I doing wrong?
docker-compose yaml file:
version: "3.7"
services:
traefik:
container_name: traefik
image: traefik:latest
networks:
- proxy
extra_hosts:
- host.docker.internal:host-gateway
ports:
- "80:80"
- "443:443"
volumes:
- /etc/localtime:/etc/localtime:ro
- ${HOME}/docker/data/traefik/traefik.yml:/traefik.yml:ro
- ${HOME}/docker/data/traefik/credentials.txt:/credentials.txt:ro
- ${HOME}/docker/data/traefik/config:/config
- ${HOME}/docker/data/traefik/letsencrypt/acme.json:/acme.json
- /var/run/docker.sock:/var/run/docker.sock:ro
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
- "traefik.http.routers.dashboard.rule=Host(`traefik.mydomain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
- "traefik.http.routers.dashboard.tls=true"
- "traefik.http.routers.dashboard.tls.certresolver=letsencrypt"
- "traefik.http.routers.dashboard.tls.domains[0].main=traefik.mydomain.com"
- "traefik.http.routers.dashboard.tls.domains[0].sans=traefik.mydomain.com"
- "traefik.http.routers.dashboard.service=api#internal"
- "traefik.http.routers.dashboard.middlewares=auth"
- "traefik.http.middlewares.auth.basicauth.usersfile=/credentials.txt"
caddy:
image: caddy:latest
container_name: caddy
restart: unless-stopped
networks:
- proxy
volumes:
- ${HOME}/docker/data/caddy/Caddyfile:/etc/caddy/Caddyfile
- ${HOME}/docker/data/caddy/site:/srv
- ${HOME}/docker/data/caddy/data:/data
- ${HOME}/docker/data/caddy/config:/config
labels:
- "traefik.http.routers.caddy-secure.rule=Host(`vps.mydomain.com`)"
- "traefik.http.routers.caddy-secure.service=caddy"
- "traefik.http.services.caddy.loadbalancer.server.port=80"
portainer:
image: portainer/portainer-ce
container_name: portainer
networks:
- proxy
command: -H unix:///var/run/docker.sock --http-enabled
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ${HOME}/docker/data/portainer:/data
labels:
- "traefik.http.routers.portainer-secure.rule=Host(`portainer.mydomain.com`)"
- "traefik.http.routers.portainer-secure.service=portainer"
- "traefik.http.services.portainer.loadbalancer.server.port=9000"
restart: unless-stopped
networks:
# proxy is the network used for traefik reverse proxy
proxy:
external: true
traefik static configuration file:
api:
dashboard: true
insecure: false
debug: true
entryPoints:
web:
address: :80
http:
redirections:
entryPoint:
to: web_secure
web_secure:
address: :443
http:
middlewares:
- secureHeaders#file
tls:
certResolver: letsencrypt
providers:
docker:
network: proxy
endpoint: "unix:///var/run/docker.sock"
file:
filename: /config/dynamic.yml
watch: true
certificatesResolvers:
letsencrypt:
acme:
email: myname#mydomain.com
storage: acme.json
keyType: EC384
httpChallenge:
entryPoint: web
traefik dynamic configuration file:
# dynamic.yml
http:
middlewares:
secureHeaders:
headers:
sslRedirect: true
forceSTSHeader: true
stsIncludeSubdomains: true
stsPreload: true
stsSeconds: 31536000
user-auth:
basicAuth:
users:
- "username:hashedpassword"
routers:
home-assistant-secure:
rule: "Host(`ha.mydomain.com`)"
service: home-assistant
services:
home-assistant:
loadBalancer:
passHostHeader: true
servers:
- url: http://host.docker.internal:8045
tls:
options:
default:
cipherSuites:
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
minVersion: VersionTLS12

'No healthy upstream' error when Envoy proxy is set up manually

I have a very simple environment with a client, a server and an envoy proxy, each running on a separate docker, communicating over http.
When I set it using docker-compose it works.
However, when I set up the dockers and the network manually (with docker network create, setting the aliases, etc.), I get a "503 - no healthy upstream" message when the client tries to send requests to the server. curl to the network alias works from the envoy container. Any idea what is the difference between using docker-compose and setting up the network and containers manually?
envoy.yaml:
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains: ["*"]
routes:
- match: { prefix: "/" }
route: { cluster: service }
http_filters:
- name: envoy.filters.http.router
typed_config: {}
clusters:
- name: service
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: round_robin
load_assignment:
cluster_name: service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: server-stub
port_value: 5000
admin:
access_log_path: "/tmp/envoy.log"
address:
socket_address:
address: 0.0.0.0
port_value: 9901
The docker-compose file that worked (but I don't want to use docker-compose, I am using scripts that set up each docker separately):
version: "3.8"
services:
envoy:
image: envoyproxy/envoy:v1.16-latest
ports:
- "10000:10000"
- "9901:9901"
volumes:
- ./envoy.yaml:/etc/envoy/envoy.yaml
server-stub:
build:
context: .
dockerfile: Dockerfile
ports:
- "5000:5000"
I can't reproduce this. It works fine with your docker-compose file, and it works fine manually. Here are the manual steps I took:
$ docker network create test-net
$ docker container run --network test-net --name envoy -p 10000:10000 -p 9901:9901 --mount type=bind,src=/home/john/projects/tester/envoy.yaml,dst=/etc/envoy/envoy.yaml envoyproxy/envoy:v1.16-latest
$ docker run --network test-net --name server-stub johnharris85/simple-hostname-reporter:3
My sample app also listens on port 5000. I used your exact envoy config. Using Docker 20.10.8 if relevant.

Filebeat failed connect to backoff?

I have 2 servers A and B. When I run filebeat by docker-compose on the server A it's working well. But on the server B I have the following error:
pipeline/output.go:154 Failed to connect to backoff(async(tcp://logstash_ip:5044)): dial tcp logstash_ip:5044: connect: no route to host
So I think I missed some config on the server B. So how can I figure out my problem and fix them.
[Edited] Add filebeat.yml and docker-compose
Notice: I ran filebeat on the server A and got failed, so I tested it on the server B and it is still working. So I guess I have some problems with server config
filebeat.yml
logging.level: error
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/share/filebeat/mylog/**/*.log
processors:
- decode_json_fields:
fields: ['message']
target: 'json'
output.logstash:
hosts: ['logstash_ip:5044']
console.pretty: true
processors:
- add_docker_metadata:
host: 'unix:///host_docker/docker.sock'
docker-compose
version: '3.3'
services:
filebeat:
user: root
container_name: filebeat
image: docker.elastic.co/beats/filebeat:7.9.3
volumes:
- /var/run/docker.sock:/host_docker/docker.sock
- /var/lib/docker:/host_docker/var/lib/docker
- ./logs/progress:/usr/share/filebeat/mylog
- ./filebeat.yml:/usr/share/filebeat/filebeat.yml:z
command: ['--strict.perms=false']
ulimits:
memlock:
soft: -1
hard: -1
stdin_open: true
tty: true
network_mode: bridge
deploy:
mode: global
logging:
driver: 'json-file'
options:
max-size: '10m'
max-file: '50'
Thanks in advance
Assumptions:
Mentioned docker-compose file is for filebeat "concentration" server
which is running in docker on server B.
Both server are running in same network space and/or are accessible between themselves
Server B as filebeat server have correct firewall setting to accept connection on port 5044 (check with telnet from server A after starting container)
docker-compose (assuming server B)
version: '3.3'
services:
filebeat:
user: root
container_name: filebeat
ports: ##################
- 5044:5044 # <- see open port
image: docker.elastic.co/beats/filebeat:7.9.3
volumes:
- /var/run/docker.sock:/host_docker/docker.sock
- /var/lib/docker:/host_docker/var/lib/docker
- ./logs/progress:/usr/share/filebeat/mylog
- ./filebeat.yml:/usr/share/filebeat/filebeat.yml:z
command: ['--strict.perms=false']
ulimits:
memlock:
soft: -1
hard: -1
stdin_open: true
tty: true
network_mode: bridge
deploy:
mode: global
logging:
driver: 'json-file'
options:
max-size: '10m'
max-file: '50'
filebeat.yml (assuming both servers)
logging.level: error
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/share/filebeat/mylog/**/*.log
processors:
- decode_json_fields:
fields: ['message']
target: 'json'
output.logstash:
hosts: ['<SERVER-B-IP>:5044'] ## <- see server IP
console.pretty: true
processors:
- add_docker_metadata:
host: 'unix:///host_docker/docker.sock'

Allow IPs with TCP Listener using RBAC (Envoy)

I am trying to achieve the following with Envoy:
Allow TCP traffic to a Postgres service with RBAC rules to allow only a few IPs.
This is my listener setup.
- name: listener_postgres
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 54322
filter_chains:
filters:
- name: envoy.filters.network.rbac
config:
stat_prefix: rbac_postgres
rules:
action: ALLOW
policies:
"allow":
permissions:
- any: true
principals:
- source_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- source_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- source_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- name: envoy.tcp_proxy
config:
stat_prefix: tcp_postgres
cluster: database_service
I can confirm that the service is setup correctly because I can remove the RBAC rules and I can connect successfully.
When the RBAC rules are added I can not connect to the Postgres database.
But for some reason nothing seems to work, I have also tried remote_ip and direct_remote_ip in place of source_ip.
Am I doing something wrong?
Thanks
Hey I ran into the same issue and this is the configuration worked for me.
I used remote_ip attribute.
Also, check the updated filters names
- name: listener_postgres
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 54322
filter_chains:
filters:
- name: envoy_rbac
config:
stat_prefix: rbac_postgres
rules:
action: ALLOW
policies:
"allow":
permissions:
- any: true
principals:
- remote_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- name: envoy_tcp_proxy
config:
stat_prefix: tcp_postgres
cluster: database_service
It seems that setting the attribute to 'remote_ip' as suggested by Rahul Pratap worked.
Here is a working example:
- name: listener_postgres
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 54322
filter_chains:
filters:
- name: envoy.filters.network.rbac
config:
stat_prefix: rbac_postgres
rules:
action: ALLOW
policies:
"allow":
permissions:
- any: true
principals:
- remote_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- name: envoy.tcp_proxy
config:
stat_prefix: tcp_postgres
cluster: database_service

Docker Containers up and running but cannot load the localhost

I could able to run Wiki_js and MongoDB containers using docker-compose. Both are running without any errors.
This is my docker-compose file.
version: '3'
services:
wikidb:
image: mongo:3
expose:
- '27017'
command: '--smallfiles --bind_ip ::,0.0.0.0'
environment:
- 'MONGO_LOG_DIR=/dev/null'
volumes:
- $HOME/mongo/db:/data/db
wikijs:
image: 'requarks/wiki:1.0'
links:
- wikidb
depends_on:
- wikidb
ports:
- '8000:3000'
environment:
WIKI_ADMIN_EMAIL: myemail#gmail.com
volumes:
- $HOME/wiki/config.yml:/var/wiki/config.yml
This is config.yml file. I didn't set up git for this project.
title: Wiki
host: http://localhost
port: 8000
paths:
repo: ./repo
data: ./data
uploads:
maxImageFileSize: 3
maxOtherFileSize: 100
db: mongodb://wikidb:27017/wiki
git:
url: https://github.com/Organization/Repo
branch: master
auth:
type: ssh
username: marty
password: MartyMcFly88
privateKey: /etc/wiki/keys/git.pem
sslVerify: true
serverEmail: marty#example.com
showUserEmail: true
But I cannot load the localhost under the port 8000. Is there a specific reason for this?