Passing url query parameters in http liveness and readiness probe in kubernetes - kubernetes

I am trying to configure the HTTP liveness probe as follows:
livenessProbe:
httpGet:
path: /rest/sends/get?source=mwTESt2VP3Q9M99GNWYvvaLQ1owrGTTjTb #sends API to test address
port: 4000
httpHeaders:
- name: Authorization
value: Basic cnBjOnUzSGRlM0xvaWI1SGpEcTFTZGVoQktpU1NBbHE=
initialDelaySeconds: 60 #wait this period after staring fist time
periodSeconds: 30 # polling interval
timeoutSeconds: 30 # wish to receive response within this time period
Here, the URL path contains query parameters along with an authentication header (base64 encoding of username:password)
However, I get the following error:
ERROR in app: Exception on /rest/sends/get [GET] (http 500)
I checked that this indeed works with status code 200 after logging into the pod
curl http://username:password#localhost:4000/rest/sends/get?source=mwTESt2VP3Q9M99GNWYvvaLQ1owrGTTjTb
This question is probably similar to this one Kubernetes liveness probes with query string parameters
But, according to it, this should have already been fixed. I am using Kubernetes on Google cloud version: 1.10.7-gke.2 on both master and other nodes.
Am I missing something?
EDIT
In the server access log, I get the following error
10.0.2.1 - - [10/Oct/2018 03:50:45] "GET /rest/sends/get?source=mwTESt2VP3Q9M99GNWYvvaLQ1owrGTTjTb HTTP/1.1" 500 -
Exception on /rest/sends/get [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.5/dist-packages/flask_httpauth.py", line 88, in decorated
return f(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/counterparty_lib-9.55.4-py3.5.egg/counterpartylib/lib/api.py", line 813, in handle_root
response = handle_rest(rest_path, flask.request)
File "/usr/local/lib/python3.5/dist-packages/counterparty_lib-9.55.4-py3.5.egg/counterpartylib/lib/api.py", line 935, in handle_rest
file_format = flask_request.headers['Accept']
File "/usr/local/lib/python3.5/dist-packages/werkzeug/datastructures.py", line 1354, in __getitem__
return _unicodify_header_value(self.environ['HTTP_' + key])
KeyError: 'HTTP_ACCEPT'
The server is actually a counterparty-server https://github.com/CounterpartyXCP/counterparty-lib
I am not really sure what the problem is.

I added this header to the request
httpHeaders:
- name: Authorization
value: Basic cnBjOnUzSGRlM0xvaWI1SGpEcTFTZGVoQktpU1NBbHE=
- name: Accept
value: application/json
And now it's working alright.

As you can see, the server logs at the end line, in the function getitem that return "unicodify_header_value" the Class HttpHeaders.Names 'HTTP_ACCEPT' wasn't found, this what was missing in your "httpHeaders" config.
nice catch.

Related

Not able to run OpenDistro for Elastic in kubernetes as non-root -supervisord error

I am setting up OpenDistro for Elastic in Kubernetes. The cluster has pod security in place that will not allow privileged pods. When I start the cluster the logs indicated a permission issue with /usr/share/supervisor/supervisord.log
I have a securityContext set on the deployment
securityContext:
runAsUser: 1000
fsGroup: 1000
``
The error message from kubectl logs es-master-0 is
```/usr/share/elasticsearch/config/elasticsearch.yml seems to be already configured for Security. Quit.
Traceback (most recent call last):
File "/usr/bin/supervisord", line 9, in <module>
load_entry_point('supervisor==4.0.2', 'console_scripts', 'supervisord')()
File "/usr/lib/python2.7/site-packages/supervisor-4.0.2-py2.7.egg/supervisor/supervisord.py", line 358, in main
go(options)
File "/usr/lib/python2.7/site-packages/supervisor-4.0.2-py2.7.egg/supervisor/supervisord.py", line 368, in go
d.main()
File "/usr/lib/python2.7/site-packages/supervisor-4.0.2-py2.7.egg/supervisor/supervisord.py", line 70, in main
self.options.make_logger()
File "/usr/lib/python2.7/site-packages/supervisor-4.0.2-py2.7.egg/supervisor/options.py", line 1472, in make_logger
backups=self.logfile_backups,
File "/usr/lib/python2.7/site-packages/supervisor-4.0.2-py2.7.egg/supervisor/loggers.py", line 417, in handle_file
handler = RotatingFileHandler(filename, 'a', maxbytes, backups)
File "/usr/lib/python2.7/site-packages/supervisor-4.0.2-py2.7.egg/supervisor/loggers.py", line 212, in __init__
FileHandler.__init__(self, filename, mode)
File "/usr/lib/python2.7/site-packages/supervisor-4.0.2-py2.7.egg/supervisor/loggers.py", line 159, in __init__
self.stream = open(filename, mode)
IOError: [Errno 13] Permission denied: '/usr/share/supervisor/supervisord.log'

ansible k8s module failing to connect to cluster with 503 - appends /version/openshift to non openshift cluster

I'm trying to use ansible new k8s module (based ok k8_raw from 2.6) to maintain an aks k8 cluster.
While I can work with the cluster with kubectl , any command with the k8s cluster fails with a 503 error.
For example this task:
- name: deploy kured daemonset
k8s:
state: present
context: "{{ cluster_name}}"
host: "redacted"# tried specifying this, but does not help
kubeconfig: "~/.kube/config"
src: "aks/utils/kured-ds.yaml"
And failure:
Traceback (most recent call last):
File "/home/alonisser/.ansible/tmp/ansible-tmp-1549320815.98-157731551192134/AnsiballZ_k8s.py", line 113, in <module>
_ansiballz_main()
File "/home/alonisser/.ansible/tmp/ansible-tmp-1549320815.98-157731551192134/AnsiballZ_k8s.py", line 105, in _ansiballz_main
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
File "/home/alonisser/.ansible/tmp/ansible-tmp-1549320815.98-157731551192134/AnsiballZ_k8s.py", line 48, in invoke_module
imp.load_module('__main__', mod, module, MOD_DESC)
File "/tmp/ansible_k8s_payload_IYmGFG/__main__.py", line 233, in <module>
File "/tmp/ansible_k8s_payload_IYmGFG/__main__.py", line 229, in main
File "/tmp/ansible_k8s_payload_IYmGFG/ansible_k8s_payload.zip/ansible/module_utils/k8s/raw.py", line 131, in execute_module
File "/tmp/ansible_k8s_payload_IYmGFG/ansible_k8s_payload.zip/ansible/module_utils/k8s/common.py", line 172, in get_api_client
File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 103, in __init__
self.__init_cache()
File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 113, in __init_cache
self.__resources.update(self.parse_api_groups())
File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 169, in parse_api_groups
new_group[version] = self.get_resources_for_api_version(prefix, group['name'], version, preferred)
File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 181, in get_resources_for_api_version
resources_response = load_json(self.request('GET', path))['resources']
File "/home/alonisser/.local/lib/python2.7/site-packages/openshift/dynamic/client.py", line 363, in request
_return_http_data_only=params.get('_return_http_data_only', True)
File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 342, in request
headers=headers)
File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 231, in GET
query_params=query_params)
File "/home/alonisser/.local/lib/python2.7/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (503)
Reason: Service Unavailable
Ansible version: 2.7/8(dev)
What am I missing?
UPDATE:
When I've added print statement to the libs used by the module beneath I found out somewhere in the pipeline /version/openshift is appended to the host name, which of course fails, because it's a non openshift cluster
Any work around for this bug?
Answer: turned out there were two failing requests. the first is to version/openshift is catched by the client and doesn't cause the crash. the crash actually happened because of an error with my cluster metrics server, which while not really needed by the k8 client used by ansible still fails a request to it.
So if anyone bumps into it, might be helpful

elasticsearch-curator k8s Helm chart cannot connect to HTTPS

I am using the following Helm chart: https://github.com/kubernetes/charts/tree/master/incubator/elasticsearch-curator and passing the following in my values.yaml file:
config:
elasticsearch:
hosts:
- my-es-aws-endpoint
port: 443
ssl: True
In the pods logs I see the following exception:
Preparing Action ID: 1, "delete_indices"
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
It seems like it is trying to connect to HTTP, not HTTPS. I have tested the connection from my k8s cluster to es:443 and it works.
Do you know if HTTPS is not supported or am I doing something wrong?
...
It looks like I was passing the config in the wrong section and it was not picking it up properly. I passed it here and it works:
# Having config_yaml WILL override the other config
config_yml: |-
---
client:
hosts:
- my-es-aws-endpoint
port: 443
use_ssl: True

Issues with MongoDB Connector for Solr

I am trying to use the MongoDB connector with MongoDB and Solr for search purposes. I know MongoDB is working properly. I also have Solr up and running. I have added the LukeRequestHandler to the solrconfig.xml file. The issue I am having is when I run the command to start the MongoDB Connector I receive the following error:
pysolr.SolrError: [Reason: /solr/admin/luke]
Any help would e greatly appreciated. Thank you in advance.
/usr/lib/python2.6/site-packages/mongo_connector/doc_managers/solr_doc_manager.py
2014-06-22 09:16:10,404 - INFO - Beginning Mongo Connector
2014-06-22 09:16:10,461 - INFO - Starting new HTTP connection (1): localhost
2014-06-22 09:16:10,464 - INFO - Finished
'http://localhost:8080/solr/admin/luke?show=schema&wt=json' (get) with body '' in 0.010 seconds.
2014-06-22 09:16:10,468 - ERROR - [Reason: /solr/admin/luke]
Traceback (most recent call last):
File "/usr/bin/mongo-connector", line 9, in <module>
load_entry_point('mongo-connector==1.2.1-', 'console_scripts', 'mongo-connector')()
File "/usr/lib/python2.6/site-packages/mongo_connector-1.2.1_-py2.6.egg/mongo_connector/connector.py", line 720, in main
continue_on_error=options.continue_on_error
File "/usr/lib/python2.6/site-packages/mongo_connector-1.2.1_-py2.6.egg/mongo_connector/connector.py", line 154, in __init__
**docman_kwargs))
File "/usr/lib/python2.6/site-packages/mongo_connector/doc_managers/solr_doc_manager.py", line 57, in __init__
self._build_fields()
File "/usr/lib/python2.6/site-packages/mongo_connector/doc_managers/solr_doc_manager.py", line 71, in _build_fields
declared_fields = self.solr._send_request('get', ADMIN_URL)
File "/usr/lib/python2.6/site-packages/pysolr.py", line 321, in _send_request
raise SolrError(error_message)
pysolr.SolrError: [Reason: /solr/admin/luke]

"pysolr.SolrError: [Reason: /solr4/update/]" when running mongo_connector.py

As a follow on from this problem I was having before: (How long does mongo_connector.py usually take?)
I was wondering if anyone else has had this problem when running the following:
$ python /usr/local/lib/python2.7/dist-packages/mongo-connector/mongo_connector.py -m localhost:27017 --docManager /usr/local/lib/python2.7/dist-packages/mongo-connector/doc_managers/solr_doc_manager.py -t http://localhost:8080/solr4
This is the error output I get:
2012-08-20 10:24:11,893 - INFO - Beginning Mongo Connector
2012-08-20 10:24:12,971 - INFO - Starting new HTTP connection (1): localhost
2012-08-20 10:24:12,974 - INFO - Finished 'http://localhost:8080/solr4/update/?commit=true' (post) with body 'u'<commit ' in 0.017 seconds.
2012-08-20 10:24:12,983 - ERROR - [Reason: /solr4/update/]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/mongo-connector/mongo_connector.py", line 441, in <module>
auth_username=options.admin_name)
File "/usr/local/lib/python2.7/dist-packages/mongo-connector/mongo_connector.py", line 100, in __init__
unique_key=u_key)
File "/usr/local/lib/python2.7/dist-packages/mongo-connector/doc_managers/solr_doc_manager.py", line 54, in __init__
self.run_auto_commit()
File "/usr/local/lib/python2.7/dist-packages/mongo-connector/doc_managers/solr_doc_manager.py", line 95, in run_auto_commit
self.solr.commit()
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 802, in commit
return self._update(msg, waitFlush=waitFlush, waitSearcher=waitSearcher)
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 359, in _update
return self._send_request('post', path, message, {'Content-type': 'text/xml; charset=utf-8'})
File "/usr/local/lib/python2.7/dist-packages/pysolr.py", line 293, in _send_request
raise SolrError(error_message)
pysolr.SolrError: [Reason: /solr4/update/]
Reason: [Reason: /solr4/update/] is not really an output that I can even start to debug. Solr is working perfectly fine, MongoDB is working perfectly fine. What could this problem be caused by?
I have been following the instructions on this page up to now: http://loutilities.wordpress.com/2012/11/26/complementing-mongodb-with-real-time-solr-search/#comment-183. I've also seen on various websites that adding the following to my Solr's solrconfig.xml should make 'update' accessible, but this is already configured on my system:
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler">
That's about all the information I have. Any hints as to what I might be doing wrong?