Arm template for Azure data factory with diagnostics settings in bicep - azure-data-factory

I have the following bicep to create ADF resource:
resource dataFactory 'Microsoft.DataFactory/factories#2018-06-01' = {
name: name
identity: {
type: 'SystemAssigned'
}
properties: {
globalParameters: {
environment: {
type: 'String'
value: environmentAbbreviation
}
}
}
location: location
}
I need to add a diagnostic setting to ADF resource as follows:
How do I update the bicep?

I tried to create diagnostic settings in ADF using Bicep. Below is the code.
Bicep code for creating data factory
This code is for creating data factory. It is same as the code in question.
param settingName string='XXXXX'
param factoryName string='XXXXX'
resource datafactory 'Microsoft.DataFactory/factories#2018-06-01' = {
name: factoryName
location: resourceGroup().location
identity: {
type: 'SystemAssigned'
}
properties: {
}
}
Bicep code for adding diagnostic setting
In order to add diagnostic setting to data factory, below code is added along with the code to create data factory.
resource factoryName_microsoft_insights_settingName 'Microsoft.DataFactory/factories/providers/diagnosticSettings#2017-05-01-preview' = {
name: '${factoryName}/microsoft.insights/${settingName}'
location: resourceGroup().location
properties: {
workspaceId: 'XXXX'
logAnalyticsDestinationType: 'Dedicated'
logs: [
{
category: 'PipelineRuns'
enabled: true
retentionPolicy: {
enabled: false
days: 0
}
}
{
category: 'TriggerRuns'
enabled: true
retentionPolicy: {
enabled: false
days: 0
}
}
{
category: 'ActivityRuns'
enabled: true
retentionPolicy: {
enabled: false
days: 0
}
}
]
metrics: [
{
category: 'AllMetrics'
timeGrain: 'PT1M'
enabled: true
retentionPolicy: {
enabled: false
days: 0
}
}
]
}
dependsOn: [
datafactory
]
}
When the above both codes are combined and run, resources got deployed successfully.
The above code will enable the categories - Pipeline runs log,Trigger runs log, Pipeline activity runs log. Change the code as per the requirement.
Reference: Microsoft.Insights/diagnosticSettings - Bicep, ARM template & Terraform AzAPI reference | Microsoft Learn

Related

Bicep variable creation from module output

I-m trying to create multiple app services with bicep using a for loop on a module.
The module has the following output:
output id string = appServiceAppResource.id
output name string = appServiceAppResource.name
output vnetIntegrationOn bool = allowVnetIntegration[appServicePlan.sku.name]
output principalId string = appServiceAppResource.identity.principalId // <-- important
After this, i want to create a key vault, and provide "get" access to all the app services previously created using an access policy:
...
param appServices object
...
module appServicePlanModules 'modules/appServicePlan.bicep' = [for appServicesConfig in appServicePlans.appServicesConfig: {
name: '${appServicesConfig.name}-appserviceplan-module'
params: {
location: location
name: appServicesConfig.name
sku: appServicesConfig.sku
}
}]
...
var accessPolicy = [ for i in range(0, appServices.instanceCount) : {
objectId: appServiceModule[i].outputs.principalId
permissions: {
secrets: ['get']
}
tenantId: subscription().tenantId
}]
module keyValutModule 'modules/keyvault.bicep' = {
name: 'key-valut-module'
dependsOn: [appServiceModule]
params: {
location: location
accessPolicies: accessPolicy
publicNetworkAccess: keyvault.publicNetworkAccess
keyVaultName: keyvault.name
}
}
The problem is that when i try to create that access policy, it fails
What puzzles me is that, this is working:
var accessPolicy = [{
objectId: appServiceModule[0].outputs.principalId
permissions: {
secrets: ['get']
}
tenantId: subscription().tenantId
}
{
objectId: appServiceModule[1].outputs.principalId
permissions: {
secrets: ['get']
}
tenantId: subscription().tenantId
}]
And also this:
var accessPolicies = [ for i in range(0,1):{
objectId: '52xxxxxx-25xx-4xxf-axxx-xxdxx3axxdff'
permissions: {
secrets: ['get']
}
tenantId: subscription().tenantId
}]
Since I want to use this template for multiple env, I want it to be more generic (so that i can have 1 or 5 service apps), so that for loop wold be very useful for me.
I'm not sure why, if I use a for loop in combination of a module output this is not working.
Do you have any idea why, or of there is a workaround on this.
Thank you!
Best regards,
Dorin

Azure bicep Storage account sku is read-only

I am trying to deploy a storage account using azure bicep.
In my code:
resource storageAccounts_storageacntin_name_default 'Microsoft.Storage/storageAccounts/blobServices#2021-04-01' = {
parent: storageAccounts_storageacntin_name_resource
name: 'default'
sku: {
name: 'Standard_RAGRS'
tier: 'Standard'
}
properties: {
changeFeed: {
enabled: false
}
restorePolicy: {
enabled: false
}
containerDeleteRetentionPolicy: {
enabled: true
days: 7
}
cors: {
corsRules: []
}
deleteRetentionPolicy: {
enabled: true
days: 30
}
isVersioningEnabled: true
}
}
I get an error in the SKU. The error is the following.
The property "sku" is read-only. Expressions cannot be assigned to read-only properties.bicep(BCP073)
I don't fully understand why is this error showing up, I am still new to azure bicep and trying to move slowly from terraform deployments to azure bicep.
Please can anyone explain me why is this error is coming up and how to solve it?
Thank you so much
UPDATE CODE:
this is the error I am getting when I removed the sku
param storageAccounts array = [
'storage1'
]
resource storage_Accounts 'Microsoft.Storage/storageAccounts#2021-04-01' = [ for storageName in storageAccounts :{
name: [storageName]
location: 'westeurope'
sku: {
name: 'Standard_RAGRS'
tier: 'Standard'
}
kind: 'StorageV2'
properties: {
allowCrossTenantReplication: true
minimumTlsVersion: 'TLS1_2'
allowBlobPublicAccess: false
allowSharedKeyAccess: true
networkAcls: {
bypass: 'AzureServices'
virtualNetworkRules: []
ipRules: []
defaultAction: 'Allow'
}
supportsHttpsTrafficOnly: true
encryption: {
services: {
file: {
keyType: 'Account'
enabled: true
}
blob: {
keyType: 'Account'
enabled: true
}
}
keySource: 'Microsoft.Storage'
}
accessTier: 'Hot'
}
}]
resource storageAccounts_hamzaelaouane1_name_default 'Microsoft.Storage/storageAccounts/blobServices#2021-04-01' = [ for storageName in storageAccounts: {
parent: [storage_Accounts]
name: storageName
properties: {
changeFeed: {
enabled: false
}
restorePolicy: {
enabled: false
}
containerDeleteRetentionPolicy: {
enabled: true
days: 7
}
cors: {
corsRules: []
}
deleteRetentionPolicy: {
enabled: true
days: 30
}
isVersioningEnabled: true
}
}
]
the error is at the last 2 lines. It says that is expecting } and ] at that point. Checking line by line, I couldn't see any error on syntax
The sku field is read-only for services that are under a storage account like blobServices and fileServices.
You can (only) set the SKU on Storage Account level (Microsoft.Storage/storageAccounts#2021-04-01).
To be complete; the tier field is also read-only on the storage account, since it's based on the SKU name. Remove these fields and you should be good to go.

Include an object from another file into main bicep template

Trying to do something, I don't know if it's possible, and if it is, I am asking for some help on how to do it.
I have a file "test.bicep" that has an object:
{
name: 'testRafaelRule'
priority: 1001
ruleCollectionType: 'FirewallPolicyFilterRuleCollection'
action: {
type: 'Allow'
}
rules: [
{
name: 'deleteme-1'
ipProtocols: [
'Any'
]
destinationPorts: [
'*'
]
sourceAddresses: [
'192.168.0.0/16'
]
sourceIpGroups: []
destinationIpGroups: []
destinationAddresses: [
'AzureCloud.EastUS'
]
ruleType: 'NetworkRule'
destinationFqdns: []
}
]
}
and I have another file, in which I am trying to somehow input the object in test.bicep into a specific property called "ruleCollections":
resource fwll 'Microsoft.Network/firewallPolicies/ruleCollectionGroups#2020-11-01' = {
name: 'netrules'
properties: {
priority: 200
ruleCollections: [
**ADD_OBJECT_FROM_TEST.BICEP_HERE_HOW?**
]
}
}
any suggestions or links to useful documentation would be helpful.
I have looked at outputs and parameters, but I am trying to add just an object into an existing property, I am not adding an entire resource on its own, otherwise, I would output the resouce and consume it with the "module" keyword.
It's not possible straightforward, but you can leverage variables or module's output.
var RULE = {
name: 'testRafaelRule'
priority: 1001
(...)
}
resource fwll 'Microsoft.Network/firewallPolicies/ruleCollectionGroups#2020-11-01' = {
name 'netrules'
properties: {
ruleCollections: [
RULE
]
}
}
or
rule.bicep
output rule object = {
name: 'testRafaelRule'
priority: 1001
(...)
}
main.bicep
module fwrule 'rule.bicep' = {
name: 'fwrule'
}
resource fwll 'Microsoft.Network/firewallPolicies/ruleCollectionGroups#2020-11-01' = {
name 'netrules'
properties: {
ruleCollections: [
fwrule.outputs.rule
]
}
}

Cloudformation - how to set filter policy of SNS subscription in code?

UPDATE: Cloudformation now supports SNS Topic Filters, so this question is not relevant anymore, no custom plugins or code is needed.
I am building a system with a number of SNS topics, and a number of Lambdas which are each reading messages from their assigned SQS queue. The SQS queues are subscribed to the SNS topics, but also have a filter policy so the messages will end up in the relevant SQS queues.
It works well when I set up the subscriptions in the AWS console.
Now I'm trying to do the same in my code, but the AWS Cloudformation documentation does not describe how to add a filter policy to a subscription. Based on the python examples here, I tried the following:
StopOperationSubscription:
Type: "AWS::SNS::Subscription"
Properties:
Protocol: sqs
TopicArn:
Ref: StatusTopic
Endpoint:
Fn::GetAtt: [StopActionQueue, Arn]
FilterPolicy: '{"value": ["stop"]}'
But then I get this error:
An error occurred: StopOperationSubscription - Encountered unsupported property FilterPolicy.
How can I set the filter policy that I need, using CloudFormation? And If that's not supported, what do you suggest as an alternative?
I want it to be set up automatically when I deploy my serverless app, with no manual steps required.
Cloudformation just started to support FilterPolicy yesterday. I have been struggling for a while too :)
Syntax
JSON
{
"Type" : "AWS::SNS::Subscription",
"Properties" : {
"DeliveryPolicy" : JSON object,
"Endpoint" : String,
"FilterPolicy" : JSON object,
"Protocol" : String,
"RawMessageDelivery" : Boolean,
"Region" : String,
"TopicArn" : String
}
}
YAML
Type: "AWS::SNS::Subscription"
Properties:
DeliveryPolicy: JSON object
Endpoint: String
FilterPolicy: JSON object
Protocol: String
RawMessageDelivery: Boolean,
Region: String
TopicArn: String
Ref:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sns-subscription.html#cfn-sns-subscription-filterpolicy
https://aws.amazon.com/blogs/compute/managing-amazon-sns-subscription-attributes-with-aws-cloudformation/
I fixed it like this:
serverless.yml
plugins:
- serverless-plugin-scripts
custom:
scripts:
commands:
update-topic-filters: sls invoke local -f configureSubscriptions --path resources/lambdaTopicFilters.json
hooks:
before:deploy:finalize: sls update-topic-filters
functions:
configureSubscriptions:
handler: src/configurationLambdas/configureSubscriptions.main
# Only invoked when deploying - therefore, no permissions or triggers are needed.
configureSubscriptions.js
import AWS from 'aws-sdk'
const nameFromArn = arn => arn.split(':').pop()
const lambdaNameFromArn = arn =>
nameFromArn(arn)
.split('-')
.pop()
exports.main = async event => {
const sns = new AWS.SNS({ apiVersion: '2010-03-31' })
const params = {}
const { Topics } = await sns.listTopics(params).promise()
for (const { TopicArn } of Topics) {
const topicName = nameFromArn(TopicArn)
const filtersForTopic = event[topicName]
if (!filtersForTopic) {
continue
}
const { Subscriptions } = await sns.listSubscriptionsByTopic({ TopicArn }).promise()
for (const { Protocol, Endpoint, SubscriptionArn } of Subscriptions) {
if (Protocol === 'lambda') {
const lambdaName = lambdaNameFromArn(Endpoint)
const filterForLambda = filtersForTopic[lambdaName]
if (!filterForLambda) {
continue
}
const setPolicyParams = {
AttributeName: 'FilterPolicy',
SubscriptionArn,
AttributeValue: JSON.stringify(filterForLambda),
}
await sns.setSubscriptionAttributes(setPolicyParams).promise()
// eslint-disable-next-line no-console
console.log('Subscription filters has been set')
}
}
}
}
Top level is the different topic names, next level is the lambda names, and the third level is the filter policies for the related subscriptions:
lambdaTopicFilters.json
{
"user-event": {
"activateUser": {
"valueType": ["status"],
"value": ["awaiting_activation"]
},
"findActivities": {
"messageType": ["event"],
"value": ["awaiting_activity_data"],
"valueType": ["status"]
}
},
"system-event": {
"startStopProcess": {
"valueType": ["status"],
"value": ["activated", "aborted", "limit_reached"]
}
}
}
If you are using serverless it is now supporting sns filter natively
functions:
pets:
handler: pets.handler
events:
- sns:
topicName: pets
filterPolicy:
pet:
- dog
- cat
https://serverless.com/framework/docs/providers/aws/events/sns#setting-a-filter-policy

searchkick reindex not working in staging env

In development environment Moment.reindex and search is OK, but in staging env is error:
2.3.1 :002 > Moment.reindex
Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to parse content to map"}],"type":"parse_exception","reason":"Failed to parse content to map","caused_by":{"type":"json_parse_exception","reason":"Duplicate field 'moment'\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput#1e0d7046; line: 1, column: 2720]"}},"status":400}
staing env using same ES.
My Moment class:
class Moment
include Mongoid::Document
searchkick inheritance: true, callbacks: :async, merge_mappings: true, mappings: {
moment: {
properties: {
text: {
type: "text",
# analyzer: "ik_max_word",
fields: {
analyzed: {
type: "text",
analyzer: "ik_max_word"
}
}
}
}
}
}}
GET /_cat/indices?v
health status index
yellow open moments_development_20180223203756302
yellow open moments_staging
This was an issue with how mappings were merged. It's fixed in the latest version of Searchkick.