HTTP APIs have OpenAPI specs (and Swagger UI)
Has anybody come across a technology or approach for maintaining live documentation of SNS topics and schemas?
Use CloudFormation templates to set up, deploy, and document your Amazon SNS configuration. For example, this is how you would describe a topic in SNS:
Type: AWS::SNS::Topic
Properties:
ContentBasedDeduplication: Boolean
DisplayName: String
FifoTopic: Boolean
KmsMasterKeyId: String
Subscription:
- Subscription
Tags:
- Tag
TopicName: String
Check more on this here.
You can also share your YAML or JSON CloudFormation templates, if needed.
Related
I'm playing a bit with Monk.io and Kafka - Meltano integration.
So, I would like to create a Monk.io Kafka cluster and provision a new connection on Meltano.
I'm using https://github.com/lensesio/fast-data-dev for Kafka env.
What would be the best approach and have the most sense?
I've planed to do it this way:
Create runnables of Kafka and Meltano and create Monk actions for Meltano template.
Those actions would have custom Meltano loader that would provide pipe to Kafka.
I haven't tested the integration with Kafka but should be similar to Postgres in terms of execution in actions i.e to add a loader or extractor if those Kafka plugins exist for Meltano.
actions:
defines: actions
add:
arguments:
resource-type:
type: string
resource-name:
type: string
code: exec("meltano", "/bin/sh", "-c", `cd elt && meltano add ${args["resource-type"]} ${args["resource-name"]}`)
The workflow and template are abstracted here https://github.com/monk-io/monk-dataops so it's more a question if Meltano has a good Kafka plugin
I have deployed a Kafka cluster and a Kafka Connect cluster in kubernetes, using Strimzi and AKS. And I wanted to start reading from RSS resources to feed my Kafka cluster, so I created a connector instance of "org.kaliy.kafka.connect.rss.RssSourceConnector" which reads from a specific RSS feed, given an url, and writes to a specific topic. But my whole intention with this is to eventually have a Kafka Connect cluster able to manage a lot of external requests of new RSSs to read from; and here is where all my doubts come in:
Shoud I create an instance of Kaliy RSS connector for each RSS feed? Or would it be better to implement my own connector, so I create only one instance of it and each time I want to read a new RSS feed I would create a new Task in the connector?
Who should be resposible of assuring the Kafka Connect Cluster state is the desired one? I mean that if a Connector(in the case of 1 RSS feed : 1 Connector instance) stopped working, who should try to start it again? An external client via the Kafka Connect REST API? Kubernetes itself?
Right now, I think my best option is to rely on Kafka Connect REST API making the external client responsible of managing the state of the set of connectors, but I don't know if these was designed to recieve a lot of requests as it would be the case. Maybe these could be scaled by provisioning several listeners in the Kafka Connect REST API configuration but I do not know.
Thanks a lot!
One of the main benefits in using Kafka Connect is to leverage a configuration-driven approach, so you will lose this by implementing your own Connector. In my opinion, the best strategy is to have one Connector instance for each RSS feed. Reducing the number of instances could make sense when having a single data source system, to avoid overloading it.
Using Strimzi Operator, Kafka Connect cluster would be monitored and it will try to restore the desired cluster state when needed. This does not include the single Connector instances and their tasks, but you may leverage the K8s API for monitoring the Connector custom resource (CR) status, instead of the REST API.
Example:
$ kubetctl get kafkaconnector amq-sink -o yaml
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
# ...
status:
conditions:
- lastTransitionTime: "2020-12-07T10:30:28.349Z"
status: "True"
type: Ready
connectorStatus:
connector:
state: RUNNING
worker_id: 10.116.0.66:8083
name: amq-sink
tasks:
- id: 0
state: RUNNING
worker_id: 10.116.0.66:8083
type: sink
observedGeneration: 1
It could be late, but it could help anyone will pass by the question, It is more relevant to have a look at Kafka-connect CR (Custom Resources) as a part of Confluent For Kubernetes (CFK), it introduces a clear cut declarative way to manage and monitor Connector with health checks and auto healing.
https://www.confluent.io/blog/declarative-connectors-with-confluent-for-kubernetes/
Whenever an S3 artifact is used, the following declaration is needed:
s3:
endpoint: s3.amazonaws.com
bucket: "{{workflow.parameters.input-s3-bucket}}"
key: "{{workflow.parameters.input-s3-path}}/scripts/{{inputs.parameters.type}}.xml"
accessKeySecret:
name: s3-access-user-creds
key: accessKeySecret
secretKeySecret:
name: s3-access-user-creds
key: secretKeySecret
It would be helpful if this could be abstracted to something like:
custom-s3:
bucket: "{{workflow.parameters.input-s3-bucket}}"
key: "{{workflow.parameters.input-s3-path}}/scripts/{{inputs.parameters.type}}.xml"
Is there a way to make this kind of custom definition in Argo to reduce boilerplate?
For a given Argo installation, you can set a default artifact repository in the workflow controller's configmap. This will allow you to only specify the key (assuming you set everything else in the default config - if not everything is defined for the default, you'll need to specify more things).
Unfortunately, that will only work if you're only using one S3 config. If you need multiple configurations, cutting down on boilerplate will be more difficult.
In response to your specific question: not exactly. You can't create a custom some-keyname (like custom-s3) as a member of the artifacts array. The exact format of the YAML is defined in Argo's Workflow Custom Resource Definition. If your Workflow YAML doesn't match that specification, it will be rejected.
However, you can use external templating tools to populate boilerplate before the YAML is installed in your cluster. I've used Helm before to do exactly that with a collection of S3 configs. At the simplest, you could use something like sed.
tl;dr - for one S3 config, use default artifact config; for multiple S3 configs, use a templating tool.
I have found lots of information on how to use environment variables in Google App Engine projects.
However I am yet to find some best practice on what to do with environment variables on compute engine.
Is it possible to use Google Cloud Deployment Manager to achieve this? My main goal is to simplify deployment between prod/stag/dev.
Right now I am moving towards using dotenv files.
Stack is webpack 4, express, node 10, vuejs 2.
For Compute Engine instances I'd suggest to use custom metadata. You can find detailed documentation about this here. From within your instance, you can access your custom metadata by performing an empty request to the instance().get method, for example:
GET https://www.googleapis.com/compute/v1/projects/myproject/zones/us-central1-a/instances/example-instance
Now, to set your custom metadata, you can indeed use the Google Cloud Deployment Manager. As per the doc here, you just need to add the metadata property and the relevant metadata keys and values for your VM resource, for example:
resources:
- name: my-first-vm-template
type: compute.v1.instance
properties:
zone: us-central1-a
machineType:
...[snip]...
metadata:
items:
- key: custom-key
value: "custom-value"
I'm new to AWS CloudFormation, I've used it to deploy CloudTrail to a number of accounts without issue however I'm trying to use one centralised SNS Topic that each CloudTrail can use, if I edit via the CloudTrail GUI it works but I can't get CloudFormation to work.
---
AWSTemplateFormatVersion: '2010-09-09'
Description: Centralised CloudTrail
Resources:
CloudTrail:
Type: "AWS::CloudTrail::Trail"
Properties:
EnableLogFileValidation: 'false'
IncludeGlobalServiceEvents: 'true'
IsLogging: 'true'
IsMultiRegionTrail: 'true'
S3BucketName: company-cloudtrail-au
TrailName: Trail1
SnsTopicName: SNSTopic
I'm trying to use the ARN for the SNS Topic (example below) but I can't figure out how to tell CloudFormation to use the ARN instead of the Name, if I just put the name in then it creates a new SNS topic in each child account.
If I go to the GUI after CloudFormation is deployed I can then point it to the correct ARN and it's centralised but it's not ideal, let me know if SNS isn't meant to be shared across accounts or if there is a better way to do this.
Example ARN for SNS Topic -
arn:aws:sns:ap-southeast-1:1111111111:SNSTopic
The solution depends on how you have created the Centralized SNS. If it is created using CF template, you can export the ARN of the SNS topic as output. Ref CloudFormation Output
In the current stack where you are creating the CloudTrail you can refer to the SNS Topic's ARN using import option. Ref Fn::ImportValue.
(or)
If you have manually created the SNS Topic, you can still pass the ARN of the SNS as an parameter for the Cloud Formation create stack call. Ref Parameters.
Hope this helps.
there are number of ways to pass the Arn to the SnsTopicName:
If you have created the SNS Topic in the Template itself as - Type: AWS::SNS::Topic , then you can refer the ARN of the SNS Topic as
SnsTopicName: !GetAtt MY_SNS_Topic.Arn
If you have manually created the SNS Topic in the Console , then you can give the Arn as
SnsTopicName: !Sub "your-full-ARN-of-the-SNS-Topic"
You can pass the ARN from the Parameters with Default: value(arn)
Parameters:
SNSTopic:
Description: SNS Topic Name
Type: String
Default: arn:aws:sns:ap-southeast-1:1111111111:SNSTopic
##########
Properties:
SnsTopicName: !Ref SNSTopic
You can define the ARN in a separate config file and pass it as an parameter with the Deploy Function for the Template or while calling the Script to deploy the Template
Filename - configfile.conf
SNSTopic="arn:aws:sns:ap-southeast-1:1111111111:SNSTopic"
AWS Serverless Template
Parameters:
SNSTopic:
Type: String
##########
Properties:
SnsTopicName: !Sub "${SNSTopic}"