How to extract common `yq` to keep script DRY? - yq

I have the following (modified from from https://stackoverflow.com/a/70152440/807037):
yq eval-all '
.clusters = (
(
(.clusters[] | {.name: .}) as $item ireduce ({}; . * $item)
) as $uniqueMap |
( $uniqueMap | to_entries | .[]) as $item ireduce([]; . + $item.value)
) |
.contexts = (
(
(.contexts[] | {.name: .}) as $item ireduce ({}; . * $item)
) as $uniqueMap |
( $uniqueMap | to_entries | .[]) as $item ireduce([]; . + $item.value)
) |
select(fi == 0)' konfig monfig
How can the following common code be extracted so as to keep the script DRY:
.«KEY» = (
(
(.«KEY»[] | {.name: .}) as $item ireduce ({}; . * $item)
) as $uniqueMap |
( $uniqueMap | to_entries | .[]) as $item ireduce([]; . + $item.value)
)
Input files:
# konfig
apiVersion: apiVersion-keep
clusters:
- cluster:
certificate-authority-data: cad-0
server: server-0
name: name-0
- cluster:
certificate-authority-data: cad-1-discard
server: server-1-discard
name: name-1
- cluster:
certificate-authority-data: cad-2
server: server-2
name: name-2
contexts:
- context:
cluster: cluster-0
user: user-0
name: name-0
- context:
cluster: cluster-1-discard
user: user-1-discard
name: name-1
- context:
cluster: cluster-2
user: user-2
name: name-2
current-context: name-keep
# monfig
apiVersion: apiVersion-discard
clusters:
- cluster:
certificate-authority-data: cad-1-keep
server: server-1-keep
name: name-1
contexts:
- context:
cluster: cluster-1-keep
user: user-1-keep
name: name-1
current-context: name-discard
Expected:
apiVersion: apiVersion-keep
clusters:
- cluster:
certificate-authority-data: cad-0
server: server-0
name: name-0
- cluster:
certificate-authority-data: cad-1-keep
server: server-1-keep
name: name-1
- cluster:
certificate-authority-data: cad-2
server: server-2
name: name-2
contexts:
- context:
cluster: cluster-0
user: user-0
name: name-0
- context:
cluster: cluster-1-keep
user: user-1-keep
name: name-1
- context:
cluster: cluster-2
user: user-2
name: name-2
current-context: name-keep

The issue is a little trickier than it appears because a use case for |= is updating each matching left hand side node with respect to itself. .clusters results in two nodes (as with .contexts) and each of those nodes is updated independently. yq doesn't know to group the nodes together. After playing around a little I got this to work:
./yq eval-all '
. ref $r |
with( ("clusters", "contexts");
$r[.] = (
(
($r[.] | .[] | {.name: .}) as $item ireduce ({}; . * $item)
) as $uniqueMap |
( $uniqueMap | to_entries | .[]) as $item ireduce([]; . + $item.value)
)
) | select(fi==0)' file1.yaml file2.yaml
Explanation:
. ref $r Create a reference to the root context, called $r. This matches the top level nodes (file1 and file2).
Using the with operator, I can parameterise the merge expression against $r, passing in the two paths that need to be merged. Each path is run against the root context $r in $r[.].
Hope that makes sense!
Disclaimer: I wrote yq

Generally speaking (due to missing sample data), you can use the update operator |= (available since v4.3.0), enabling you to address by context . on the RHS. Then, as the absolute context only appears on the LHS, you can just list at once all contexts you want this to be applied to.
(.clusters, .contexts) |= ( .[] | ... )

Related

Subkey yq query on docker-compose.yml

What I'm looking for is a yq query that returns the service names that are using
a specified volume for a given docker-compose.yml file.
For example, in the stripped down docker-compose.yml file below, say I am looking for the names of all services that
use the volume v-app-olorin.
version: "3"
services:
arwen:
this: that
volumes:
- v-app-mithrandir:/data/mithrandir
- v-app-olorin:/data/olorin
boromir:
volumes:
- v-app-mithrandir:/data/mithrandir
- v-app-stormcrow:/data/stormcrow
cirdan:
volumes:
- v-app-mithrandir:/data/mithrandir
- v-app-olorin:/data/olorin
volumes:
v-app-mithrandir:
name: v-app-mithrandir
v-app-olorin:
name: v-app-olorin
v-app-stormcrow:
name: v-app-stormcrow
The expected response would be:
arwen
cirdan
I can match simple key values with something like this:
yq e '.services | with_entries(select(.value.this == "that")) | to_entries | .[] | .key' docker-compose.yml
arwen
But I'm having trouble matching an element of the volumes array. Thank you for any help.
here's an expression that does that:
yq '.services[] | select(.volumes[] | contains("v-app-olorin")) | key' docker-compose.yml
Explanation:
splat out the services entries into their invidiual nodes .services[]
select the ones that have "v-app-olorin" in their volumes array: select(.volumes[] | contains("v-app-olorin"))
get the key of that services entry
Disclaimer: I wrote yq

Replace string into a multiline string

I have the following yaml file:
values: |
nameOverride: my-service
fullnameOverride: ""
namespace: my-ns
containerApps:
- name: app-frontend
image_tag: xxxxxxx
- name: app-backend
image_tag: xxxxxxx
I'm looking for a way to replace e.g. xxxxxx by yyyyyy on containerApps.[app-frontend].image_tag within a multi-line value (values: |).
The output being:
values: |
nameOverride: my-service
fullnameOverride: ""
namespace: my-ns
containerApps:
- name: app-frontend
image_tag: yyyyyy
- name: app-backend
image_tag: xxxxxxx
How this can be accomplished using yq?
Any help is welcomed.
Here's a solution using mikefarah/yq. It decodes the multiline string using #yamld, makes the substitution using sub, and encodes the result back using #yaml.
yq '
.values |= (
#yamld | (
.containerApps[] | select(.name == "app-frontend") | .image_tag
) |= sub("xxxxxxx", "yyyyyy")
| #yaml
)
'
values: |
nameOverride: my-service
fullnameOverride: ""
namespace: my-ns
containerApps:
- name: app-frontend
image_tag: yyyyyy
- name: app-backend
image_tag: xxxxxxx
To update the file in-place (instead of just outputting it), use the -i flag.

Cloud Foundry bosh Error 140003: unknown resource pool

I'm attempting to setup a service broker to add postgres to our Cloud Foundry installation. We're running our system on vmWare. I'm using this release in order to do that:
cf-contrib-release
I added the release in bosh:
#bosh releases
Acting as user 'director' on 'microbosh-ba846726bed7032f1fd4'
+-----------------------+----------------------+-------------+
| Name | Versions | Commit Hash |
+-----------------------+----------------------+-------------+
| cf | 208.12* | a0de569a+ |
| cf-autoscaling | 13* | 927bc7ed+ |
| cf-metrics | 34* | 22f7e1e1 |
| cf-mysql | 20* | caa23b3d+ |
| | 22* | af278086+ |
| cf-rabbitmq | 161* | 4d298aec |
| cf-riak-cs | 10* | 5e7e46c9+ |
| cf-services-contrib | 6* | 57fd2098+ |
| docker | 23* | 82346881+ |
| newrelic_broker | 1.3* | 1ce3471d+ |
| notifications-with-ui | 18* | 490b6446+ |
| postgresql-docker | 4* | a53c9333+ |
| push-console-release | console-du-jour-203* | d2d31585+ |
| spring-cloud-broker | 1.0.0* | efd69612 |
+-----------------------+----------------------+-------------+
(*) Currently deployed
(+) Uncommitted changes
Releases total: 13
I setup my resource pools and jobs in my yaml file according to this doumentation:
http://bosh.io/docs/vsphere-cpi.html#resource-pools
This is how our cluster looks:
vmware cluster
And here is what I put in the yaml file:
resource_pools:
- name: default
network: default
stemcell:
name: bosh-vsphere-esxi-ubuntu-trusty-go_agent
version: '2865.1'
cloud_properties:
cpu: 2
ram: 4096
disk: 10240
datacenters:
- name: 'Universal City'
clusters:
- USH_UCS_CLOUD_FOUNDRY_NONPROD_01: {resource_pool: 'USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'}
jobs:
- name: gateways
release: cf-services-contrib
templates:
- name: postgresql_gateway_ng
instances: 1
resource_pool: 'USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'
networks:
- name: default
default: [dns, gateway]
properties:
# Service credentials
uaa_client_id: "cf"
uaa_endpoint: http://uaa.devcloudwest.example.com
uaa_client_auth_credentials:
username: admin
password: secret
And I'm getting an error when I run 'bosh deploy' that says:
Error 140003: Job `gateways' references an unknown resource pool `USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'
Here's my yaml file in it's entirety:
name: cf-22b9f4d62bb6f0563b71
director_uuid: fd713790-b1bc-401a-8ea1-b8209f1cc90c
releases:
- name: cf-services-contrib
version: 6
compilation:
workers: 3
network: default
reuse_compilation_vms: true
cloud_properties:
ram: 5120
disk: 10240
cpu: 2
update:
canaries: 1
canary_watch_time: 30000-60000
update_watch_time: 30000-60000
max_in_flight: 4
networks:
- name: default
type: manual
subnets:
- range: exam 10.114..130.0/24
gateway: exam 10.114..130.1
cloud_properties:
name: 'USH_UCS_CLOUD_FOUNDRY'
#resource_pools:
# - name: common
# network: default
# size: 8
# stemcell:
# name: bosh-vsphere-esxi-ubuntu-trusty-go_agent
# version: '2865.1'
resource_pools:
- name: default
network: default
stemcell:
name: bosh-vsphere-esxi-ubuntu-trusty-go_agent
version: '2865.1'
cloud_properties:
cpu: 2
ram: 4096
disk: 10240
datacenters:
- name: 'Universal City'
clusters:
- USH_UCS_CLOUD_FOUNDRY_NONPROD_01: {resource_pool: 'USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'}
jobs:
- name: gateways
release: cf-services-contrib
templates:
- name: postgresql_gateway_ng
instances: 1
resource_pool: 'USH_UCS_CLOUD_FOUNDRY_NONPROD_01_RP'
networks:
- name: default
default: [dns, gateway]
properties:
# Service credentials
uaa_client_id: "cf"
uaa_endpoint: http://uaa.devcloudwest.example.com
uaa_client_auth_credentials:
username: admin
password: secret
- name: postgresql_service_node
release: cf-services-contrib
template: postgresql_node_ng
instances: 1
resource_pool: common
persistent_disk: 10000
properties:
postgresql_node:
plan: default
networks:
- name: default
default: [dns, gateway]
properties:
networks:
apps: default
management: default
cc:
srv_api_uri: http://api.devcloudwest.example.com
nats:
address: exam 10.114..130.11
port: 25555
user: nats #CHANGE
password: secret
authorization_timeout: 5
service_plans:
postgresql:
default:
description: "Developer, 250MB storage, 10 connections"
free: true
job_management:
high_water: 230
low_water: 20
configuration:
capacity: 125
max_clients: 10
quota_files: 4
quota_data_size: 240
enable_journaling: true
backup:
enable: false
lifecycle:
enable: false
serialization: enable
snapshot:
quota: 1
postgresql_gateway:
token: f75df200-4daf-45b5-b92a-cb7fa1a25660
default_plan: default
supported_versions: ["9.3"]
version_aliases:
current: "9.3"
cc_api_version: v2
postgresql_node:
supported_versions: ["9.3"]
default_version: "9.3"
max_tmp: 900
password: secret
And here's gist with the debug output from that error:
postgres_2423_debug.txt
The docs for the jobs blocks say:
resource_pool [String, required]: A valid resource pool name from the Resource Pools block. BOSH runs instances of this job in a VM from the named resource pool.
This needs to match the name of one of your resource_pools, namely default, not the name of the resource pool in vSphere.
The only sections that have direct references to the IaaS are things that say cloud_properties. Specific names of resources (like networks, clusters, or datacenters in your vSphere, or subnets, AZs, and instance types in AWS) only show up in places that say cloud_properties.
You use that data to define "networks" and "resource pools" at a higher level of abstraction that is IaaS-agnostic, e.g. except for cloud properties, the specifications you give for resource pools is the same whether you're deploying to vSphere, AWS, OpenStack, etc.
Then your jobs reference these networks, resource pools, etc. by the logical name you've given to the abstractions. In particular, jobs don't require any IaaS-specific configuration whatsoever, just references to a logical network(s) and a resource pool that you've defined elsewhere in your manifest.

MongoTimeoutException Message: Timed out while waiting to connect after 10000 ms

I'm using gorm in grails to connect to MongoDB. I'm getting MongoTimeOutException.
I'm able to successfully connect to MongoDb using mongo java driver stand alone program.
Why am i not able to connect through Grails GORM plugin?
Can someone help?
The below is the configuration
grails {
mongo {
host = "localhost"
port = 27107
databaseName = "test"
options{
connectionsPerHost=20
}
}
}
The below is domain class
class Device {
String deviceType
String deviceId
int primary
static constraints = {
}
}
The below is the exception trace
| Error 2014-11-09 16:10:39,620 [http-bio-8080-exec-4] ERROR errors.GrailsExceptionResolver - MongoTimeoutException occurred when processing request: [GET] /devices/Device
Timed out while waiting to connect after 10000 ms. Stacktrace follows:
Message: Timed out while waiting to connect after 10000 ms
Line | Method
->> 131 | getDescription in com.mongodb.BaseCluster
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 396 | getClusterDescription in com.mongodb.DBTCPConnector
| 569 | getType . . . . . . . in ''
| 370 | isMongosConnection in ''
| 645 | isMongosConnection . in com.mongodb.Mongo
| 454 | _check in com.mongodb.DBCursor
| 546 | _hasNext . . . . . . in ''
| 571 | hasNext in ''
| 1893 | hasNext . . . . . . . in org.grails.datastore.mapping.mongo.query.MongoQuery$MongoResultList$1
| 8 | index in ewents.DeviceController
| 198 | doFilter . . . . . . in grails.plugin.cache.web.filter.PageFragmentCachingFilter
| 63 | doFilter in grails.plugin.cache.web.filter.AbstractFilter
| 895 | runTask . . . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
| 918 | run in ''
^ 695 | run . . . . . . . . . in java.lang.Thread
or could someone share sample Grails example with GORM support? Am basically new to this Grails.
I figured out the issue. It's my bad, given the port number as 27107 instead of 27017 and i took a day to figure this out. LOL
I had a similar problem, and it showed out that in my test case, I didn't get the properties from where I expected. (My host contained a quotation mark in teh string...)
My suggestion for developers with this issue in the future is to set a breakpoint in com.mongodb.mongo.java in method "boolean isMongosConnection()" and see what host and port that is actually used.

grails mongodb 3.0.2 plugin join queries on associations using createCriteria()

I am trying to search for users with particular role using createCriteria().I have three domain classes: User, Role and UserRole (they are from SpringSecurity).
class UserRole implements Serializable {
User user
Role role
static constraints = {
user nullbale: false
role nullable: false
}
static mapping = {
version: false
id composite: ['role', 'user']
}
}
I am searching on isActive, name or username fields on User Domain and authority field on Role Domain .
def c = UserRole.createCriteria()
def results = c {
user {
eq("isActive", true)
or {
ilike("name", "somename")
ilike("username", "someusername")
}
}
role {
eq("authority", "ROLE_USER")
}
}
On running this I get UnsupportedOperationException from mongo plugin. Here is stacktrace
ERROR errors.GrailsExceptionResolver - UnsupportedOperationException occurred when processing request: [GET] /users/search - parameters:
query:
Join queries are not supported by MongoDB. Stacktrace follows:
Message: Join queries are not supported by MongoDB
Line | Method
->> 162 | handle in org.grails.datastore.mapping.mongo.query.MongoQuery$2
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 142 | handle in ''
| 1091 | populateMongoQuery in org.grails.datastore.mapping.mongo.query.MongoQuery
| 993 | executeQuery in org.grails.datastore.mapping.mongo.query.MongoQuery$58
| 861 | doInDB . . . . . . in ''
| 833 | doInDB in ''
| 542 | list . . . . . . . in org.grails.datastore.mapping.query.Query
| 325 | invokeMethod in grails.gorm.CriteriaBuilder
| 17 | $tt__searchUsers . in com.themopi.apis.searchactivity.SearchService
| 29 | searchUser in com.themopi.apis.search.SearchController
| 198 | doFilter . . . . . in grails.plugin.cache.web.filter.PageFragmentCachingFilter
| 63 | doFilter in grails.plugin.cache.web.filter.AbstractFilter
| 106 | processFilterChain in com.odobo.grails.plugin.springsecurity.rest.RestTokenValidationFilter
| 72 | doFilter in ''
| 53 | doFilter . . . . . in grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 745 | run in java.lang.Thread
I found a Jira on this issue but it is unresolved.
Any work around for this till now as jira was created on 09/Mar/12 or I am missing something?
I will be using pagination and projections in this query?
Any help or guidance is highly appreciated.
Thanks in advance.
there's an excellent article from Burt on this topic.
One of the main parts of it is, that the security model should be implemented with embedded/sub-doc entities. Actually joins shouldn't be used with mongo at all, and if you need those, use a RDBMS or Graph DB.