I require to run a N1QL query on a couchbase cluster via Kubernetes job periodically. Can CBQ tool be used? is there any such reference implementation available?
You can use cbq shell
OR
curl -v http://queryhost:8093/query/service -H "Content-Type: application/json" -d '{"statement":"select * from default where k1 = $name&$name=1;"}'
You can use Eventing and run SQL++ (N1QL) via a periodic timer in the eventing service.
This can make you system more portable and eliminate dependencies outside of Couchbase.
Refer to:
https://docs.couchbase.com/server/current/eventing/eventing-handler-basicN1qlSelectStmt.html
https://docs.couchbase.com/server/current/eventing/eventing-examples-recurring-timer.html
Related
I'm running a task in AWS ECS using the CLI command run-task.
I'm successfully running a task as follows:
aws ecs run-task --cluster ${stackName}-cluster \
--task-definition ${stackName}-${tag} \
--launch-type="FARGATE" \
--network-configuration '{ "awsvpcConfiguration": { "assignPublicIp":"DISABLED", "securityGroups": ["sg-......"], "subnets": ["subnet-.....","subnet-.....","subnet-......"]}}' \
--count 1 \
--profile ${profile} \
--overrides file://overrides.json
They way I understand it, if you're using FARGATE you must to have NetworkMode: awsvpc in your TaskDefinition, and you need to specify the awsvpcConfiguration every time you run a task. This is all fine.
However, to make the above invocation tidier, is there a way to pass the --networkConfiguration above as an override. The documentation says you can pass environment variables, but it's not clear if this includes network.
I would be very grateful to anybody who could shed some light on this.
No you can't do that. Here's the full list of things you can specify in ECS Task Overrides. Network configuration is not in that list.
The documentation says you can pass environment variables, but it's not clear if this includes network.
The network configuration is not an environment variable.
If you just want to be able to simplify the the command line by passing in more arguments from a file, you can use the --cli-input-json
or --cli-input-yaml arguments.
My cluster have many java applications running on it, I would like to take thread dump for few microservices. But as these are distroless images, I was thinking of ephemeral containers but not sure if volume can be attached to such containers. Can someone help?
Regards,
Anjani
Are you using Spring Boot to build those microservices? If yes, then by using Spring Boot Admin you can get thread dumps, heap dumps and monitor your services from a single window.
Please check here to get more details about it.
Check Spring Boot actuator endpoints https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.endpoints
There is a dedicated endpoint for thread dumps. Note that by default it returns custom JSON format, but you could request default text format.
curl 'http://localhost:8080/actuator/threaddump' -i -X GET \
-H 'Accept: text/plain'
See for more details https://docs.spring.io/spring-boot/docs/current/actuator-api/htmlsingle/#threaddump.retrieving-text
On airflow 2.1.3, looking at the documentation for the CLI, airflow celery -h
it shows:
-H CELERY_HOSTNAME, --celery-hostname CELERY_HOSTNAME
Set the hostname of celery worker if you have multiple workers on a single machine
I am familiar with Celery and I know you can run multiple workers on the same machine. But with Airflow (and the celery executor) I don't understand how to do so.
if you do, on the same machine:
> airflow celery worker -H 'foo'
> airflow celery worker -H 'bar'
The second command will fail, complaining about the pid, so then:
> airflow celery worker -H 'bar' --pid some-other-pid-file
This will run another worker and it will sync with the first worker BUT you will get a port binding error since airflow binds the worker process to http://0.0.0.0:8793/ no matter what (unless I missed a parameter?).
It seems you are not suppose to run multiple workers per machines... Then my question is, what is the '-H' (--celery-hostname) option for? How would I use it?
The port that celery listens to is also configurable - it is used to serve logs to the webserver:
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-log-server-port
You can run multiple celery workers with different settings for that port or run them with --skip-serve-logs to not open the webserver in the first place (for example if you send logs to sentry/s3/gcs etc.).
BTW. This sounds strange by the way to run several celery workers because you can utilise the machine CPUS by using parallelism. This seems much more practical an easier to manage.
I'm trying to find a way to automate the task of getting COS bucket sizes on IBM Cloud.
I have dozens of buckets over different accounts but still couldn't find a way to get this information using IBM Cloud COS CLI, just other information, like bucket names, etc.
The COS S3 API does not return size information for buckets. Thus, the CLI, which is based on the API, won't return size information either.
But here's an indirect way to find the size of a bucket by looping through the sizes of the individual objects in the bucket
ibmcloud cos objects --bucket <BUCKET_NAME> --output JSON | jq 'reduce (.Contents[] | to_entries[]) as {$key,$value} ({}; .[$key] += $value) | .Size'
The output is in bytes
You may have to loop through the bucket names may be in a shell script. For all the buckets in an account + resource group, run the below command
ibmcloud cos buckets --output JSON
Note: Before running the above commands, remember to add the COS service CRN to the config with the below command
ibmcloud cos config crn --crn <SERVICE_CRN>
The answer that loops through the individual objects is indeed the only (and likely best) way to use the IBM Cloud CLI to find that information, but there are a few other ways worth mentioning for completion.
If you need to do this elegantly on the command line, the Minio Client provides a Linux-esque syntax:
mc du cos/$BUCKET
This returns the size of the bucket in MiB.
Additionally, the COS Resource Configuration API will directly return a bytes_used value, with no iterating over objects behind the scenes. While there's no official CLI implementation yet (although it's in the pipeline) it's relatively easy to use cURL or httpie to query the bucket.
curl "https://config.cloud-object-storage.cloud.ibm.com/v1/b/$BUCKET" \
-H 'Authorization: bearer $TOKEN'
I have a kubernetes cluster running with one master and 2 nodes. I want to run e2e tests on this cluster. How should I run it? I tried doing go run hack/e2e.go -v --test but that command wants to create a cluster first and then run the test, while I want to run the tests on my already present cluster. Any idea how should I go ahead with it or what parameters should I pass to e2e tests?
TIA.
If what you want to do is run the conformance tests and verify your cluster, you might also consider looking into the tool that Heptio created called sonobuoy, which was created specifically to run the non-destructive conformance tests for Kubernetes 1.7 (or later) in a consistent fashion. Lachlan Everson posted a 6 minute youtube video showing how to use it that I thought was pretty easy to follow, and will get you up and running with it very quickly.
It's configuration driven, so you can turn on/off tests that interest you easily, and includes some plugin driven "get more data about this cluster" sort of setup if you find you want or need to dig more in specific areas.
Use the conformance test, described here:
https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md#conformance-tests
An updated link you can find it here: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md or you can now use kubetest to run e2e tests.
Update: The easiest way to run e2e tests is by using Heptio's scanner
I use this command:
docker run -v $HOME/.kube/config:/kubeconfig \
--env KUBECONFIG=/kubeconfig \
k8s.gcr.io/conformance-amd64:v1.14.1 \
/usr/local/bin/ginkgo \
--focus="\[Conformance\]" \
--skip="Alpha|\[(Disruptive|Feature:[^\]]+|Flaky)\]" \
--noColor=false \
--flakeAttempts=2 \
/usr/local/bin/e2e.test -- \
--repo-root=/kubernetes \
--provider="skeleton" \
--kubeconfig="/kubeconfig" \
--allowed-not-ready-nodes=1
You can run the conformance e2e tests as described here:
https://github.com/cncf/k8s-conformance/blob/master/instructions.md
if your cluster is running 1.7.X or 1.8.x this approach is easy.
Basically you can run
curl -L https://raw.githubusercontent.com/cncf/k8s-conformance/master/sonobuoy-conformance.yaml | kubectl apply -f -