Can we create a dataproc cluster with custom image using Rest API or Java API. what config parameter can be used to give custom image name/uri. According to the documentation, ImageURI in instanceGroupConfig is "Output only. The Compute Engine image resource used for cluster instances. Inferred from SoftwareConfig.image_version."
[Output Only] tag on instanceGroupConfig.imageUri field is a hold over from when it was [Output Only]. As long as you set imageUri for both Master and Worker groups the setting will be honored.
In the mean time we will change the documentation.
Related
I'm trying to create a Grafana dashboard that will reflect my AWS RDS cluster metrics.
For the simplicity I've chose CloudWatch as a datasource, It works well for showing the 'direct' metrics from the RDS cluster.
Problem is that we've switched to use RDS Proxy due the high number of connections we are required to support.
Now, I'm adjusting my dashboard to reflect few metrics that are lacking, most important is number of actual connections, which in AWS CloudWatch console presented by this query:
SELECT AVG(DatabaseConnections)
FROM SCHEMA("AWS/RDS", ProxyName,Target,TargetGroup)
WHERE Target = 'db:my-db-1'
AND ProxyName = 'my-db-rds-proxy'
AND TargetGroup = 'default'
Problem is that I can't find it anywhere in the CloudWatch Grafana query editor:
The only metric with "connections" is the standard DatabaseConnections which represents the 'direct' connections to the RDS cluster and not the connections to the RDS Proxy.
Any ideas?
That UI editor is generated from hardcoded list of metrics, which may not contain all metrics and dimensions (especially if they have been added recently), so in that case UI doesn't generate them in the selectbox.
But that is not a problem, because that selectbox is not a standard selectbox. It is an input, where you can write your own metric and dimension name. Just click there, write what you need and Hit enter to add (the same is applicable for:
Pro tip: don't use UI query builder (that's for beginners), but switch to Code and write your queries directly (anyway UI builder builds that query under the hood):
It would be nice if you create a Grafana PR - add these metrics and dimensions which are missing in the UI builder to metrics.go.
So for who ever will ever get here you should use ClientConnections and use the ProxyName as the dimension (which I didn't set initially
I was using old Grafana version (7.3.5) which didn't have it built in.
I'm using Google's Cloud API to only return disk images (compute.instances.list) created after a certain date.
I'm using the following for the filter parameter: creationTimestamp > 2019-08-02 but it's not working. I'm getting Invalid value for field 'filter': 'creationTimestamp \u003e 2019-08-02'. Invalid list filter expression.
Any ideas, or is it not possible? I can have it work using a partial date & a wildcard, using creationTimeStamp = 2019-08-0*, but that's not the same as everything after this date.
This is a known issue by Google Cloud Platform, and you can follow it progress here.
As an alternative, you could use a gcloud command. You could first format the list as, for instance, a table, and then use one of the columns (that will be the creation time stamp) to filter.
The following command
gcloud compute instances list --format="table(name,creationTimestamp)" --filter="CREATION_TIMESTAMP > 2019-08-23"
will give you a list of Compute Engine instances created after 2019-08-23 and, in this case, you will only obtain pear each GCE instance their name and the creation date.
This blog is very interesting and educational about how to use filter, formats, tables and more regarding gcloud commands.
Hazelcast Configuration modifies the map configuration value for the "Time to Live" or "TTL" property at runtime either through the Hazelcast client or a RESTful API call.
I saw the Hazelcast Mancenter through which we can modify the Map TTL value at runtime without bringing the cluster down.
Is there a way to change the TTL value from the Hazelcast client program or RESTful API call?
Note: I do not want to add one more node to the cluster just to change the TTL value. I am also not interested in bringing the cluster down change the XML configuration and restart the Hazelcast cluster.
You can use an overridden IMap.put(K key, V value, long ttl, TimeUnit ttlUnit) call to specify the desired timeout value. With this, you can have ttl & ttlUnit field value changing dynamically based on your requirements.
If you want to apply a new TTL value for existing entries, you can iterate over the entries and reset TTL - refer this javadoc. This is not an optimal solution, but this is the best solution as of now. A Github ticket is already raised for this feature with Hazelcast team, you can track it here: Improve map re-config capabilities
There's an article "Tracking Every Release" which tells about displaying a vertical line on graphs for every code deployment. They are using Graphite. I would like to do something similar with Prometheus 2.2 and Grafana 5.1. More specifically I want to get an "application start" event displayed on a graph.
Grafana annotations seem to be the appropriate mechanism for this but I can't figure out what type of prometheus metric to use and how to query it.
The simplest way to do this is via the same basic approach as in the article, by having your deployment tool tell Grafana when it performs a deployment.
Grafan has a built-in system for storing annotations, which are displayed on graphs as vertical lines and can have text associated with them. It would be as simple as creating an API key in your Grafana instance and adding a curl call to your deploy script:
curl -H "Authorization: Bearer <apikey>" http://grafana:3000/api/annotations -H "Content-Type: application/json" -d '{"text":"version 1.2.3 deployed","tags":["deploy","production"]}'
For more info on the available options check the documentation:
http://docs.grafana.org/http_api/annotations/
Once you have your deployments being added as annotations, you can display those on your dashboard by going to the annotations tab in the dashboard settings and adding a new annotation source:
Then the annotations will be shown on the panels in your dashboard:
You can get the same result purely from Prometheus metrics, no need to push anything into Grafana:
If you wanted to track all restarts your search expression could be something like:
changes(start_time_seconds{job="foo",env="prod"} > 0
Or something like this if you only wanted to track version changes (and you had some sort of info metric that provided the version):
alertmanager_build_info unless max_over_time(alertmanager_build_info[1d] offset 5m)
The latter expression should only produce an output for 5 minutes whenever a new alertmanager_build_info metric appears (i.e. one with different labels such as version). You can further tweak it to only produce an output when version changes, e.g. by aggregating away all other labels.
A note here as technology has evolved. We get deployment job state information in Prometheus metrics format scraped directly from the community edition of Hashicorp's Nomad and we view this information in Grafana.
In your case, you would just add an additional query to an existing panel to overlay job start events, which is equivalent to a new deployment for us. There are a lot of related metrics "out of the box," such as for a change in job version that can be considered as well. The main point is no additional work is required besides adding a query in Grafana.
I am adding data into OTSDB from different sources. But i give metric name for each data points using XML file. Also i dont have any access to OTSDB to create Metric Name via terminal
I have reffered below links :-
API PUT
GitHub Issue
In gitHub issue, i couldn't understand how to use --auto-metirc .
I know how to create metric using Terminal :-
Here i am creating abxcs metirc using terminal.
./tsdb mkmetric abxcs
But How to create metric using API?
FYI :- Please suggest solution using JAVA
Thanks for help in advance.
In order to have metric names auto created on-the-fly, you'll need to set
tsd.core.auto_create_metrics = true
in the OpenTSDB configuration file. Ref: http://opentsdb.net/docs/build/html/user_guide/configuration.html
Whether or not a data point with a new metric will assign a UID to the metric. When false, a data point with a metric that is not in the database will be rejected and an exception will be thrown.
CLI equivalent of it is to pass --auto-metric switch while starting tsd process.