I have a really complicated system which use multiple languages and frameworks (Java Python Scala Bash). In each module I need to retrieve configuration values which are similar and change frequently. Currently I'm maintaining multiple conf files which holds lots of duplicates.
I wonder if there is out of the box RestAPI which can retrieve variables by demand from remote location.
All I manage to find by now are ways to load the entire file from remote source which is half a solution from me:
YAML.parse(open('https://link_to_file/file.yaml'))
My goal, which I fail to find a lead to it, is to make a direct call.
MyRemoteAPI.get("level1.level2.x")
P.S
YAML is not mandatory solution for me, I'm Open for suggestions.
I don't know about an out-of-the-box API, but it's fairly trivial to build. Make a service that will read the YAML file and traverse to the appropriate key. e.g. using a dynamic language like Ruby (+Rails), you could do something like
def value
config = YAML.load_file '/local/path/to/config.yaml'
render plain: config.dig(params[:key].split('.'))
end
dig essentially traverses a structure and safely returns nil if a key isn't found, so this returns the value at the "leaf" of the requested path.
You might also want to cache the structure in memory to prevent constantly reading from the file, e.g. could do something like ##config ||= YAML.parse(open('https://link_to_file/file.yaml')) or config = Rails.cache.fetch('config', expire_in: 1.hour) { ... }. And/or cache the API's HTTP response.
Related
I wanted writing an IDE plugin to simplify work with references to code in my documentation, but decided to first check the existence of a specification (like URI) so as not to reinvent the wheel.
I tried to find the answer to this on the internet but failed.
We know that such references work in different IDEs:
appRoot.folder.file.Class#method
appRoot.folder.file:35
appRoot/fodler/file.ext:35
But what if you need to get the path to the nested property JSON or XML?
I naively imagine that the sign # means the root and then for the JSON path I could use something like this:
appRoot/folder/jsonFile.json#oneLevelProp.twoLevelProp
But this of course does not work at the current moment in any IDE that I know of.
Our service implements different levels of access and we are using one openAPI YAML file internally.
For external documentation purposes, we would like to create multiple openAPI files, that are valid in themselves (self-sustained), but only have a partial set of the global file, e.g. based on the path or on tags.
(The same path may be used in different split-Files but I don't think that is a problem then.)
Any idea on how to achieve that? Is there some tooling around for it?
You can use a valid URI in a JSON Pointer which points to another resource. The URI can be a path to a local file, a web resource, etc.:
paths:
/user/{id}:
summary: Get a user
parameters:
- $ref: "./path/to/file#/user_id"
# And so on...
Reserved keys in the OpenAPI spec must be unique so I don't think you'd be able to create standalone OpenAPI specs without some third-party utility that could overcome that limitation.
However, you would be able to create valid standalone JSON objects defined across many files and reference them in the index document. There are many articles online providing examples:
https://davidgarcia.dev/posts/how-to-split-open-api-spec-into-multiple-files/
https://blog.pchudzik.com/202004/open-api-and-external-ref/
I ended up writing a Python script, that I have posted here.
Flow
Read the YAML File into a dictionary
Copy the dictionary to a new dictionary
Iterate through the original dictionary and
Remove items that are not tagged with the tag(s) you want to keep
Remove items that are have some keyword you want to omit in the path
Write out the dictionary to a new YAML
The GIST is available here:
https://gist.github.com/erikm30/d1f7e1cea3f18ece207ccdcf9f12354e
My objective is to understand how to include sensitive constants/secrets (eg. api token) and non-sensitive constants (eg. api route) in my code.
Example for a sensitive constant:
val apiToken = "Hushhhh!"
Example for a non-sensitive constant:
val happyUsersEndpoint = "https://happy.foo.io/v1/users"
My questions are:
In which part of my folder structure should I store non-sensitive constants?
Is there a safe way to store sensitive constants in my code? If not, what is the right way to reference them from outside within my code?
To my understanding, there are ways to do that in Java-ish code, but I am mainly looking for ways that fit well into Scala code (if possible).
It should not be in the source code and in the VCS repo.
It should be a part of the deployment/operation process.
You can use some sort of a config or retrieve it from env variables. Which is better depends on your stack. Is it a docker? Is it a Play or something from typelevel stack?
For each situation, there are would be a different appropriate method. For example:
For Play you can use just play config files in HOCON format.
In typelevel you have pureconfig.
For docker installation, it is better to use env.
If you have a simple zero dependency console app - I suggest adding command line args with needed properties.
For a simple app without play, cats and docker - just pick a lightweight simple library.
I'm defining my own Puppet class, and I was wondering if it is possible to have an array variable which contains a list of all files in a specific directory. I was wondering to have a similar syntax like below, but didn't found a way to make it work.
$dirs = Dir.entries('C:\\Program Files\\Java\\')
Does anyone how to do it in a Puppet file?
Thanks!
I was wondering if it is possible to have an array variable which contains a list of all files in a specific directory.
Information about the current state of the machine to be configured is conveyed to the catalog compiler via facts. These are available to your classes as top-scope variables, and Puppet (or Facter, actually) provides ways to define your own custom facts. That's a link into the Facter 3 manual, but similar applies to earlier versions. Do not overlook the rest of the Facter documentation, which has more relevant information on this topic.
On the other hand, information about the machine providing catalog-building services -- the master in a master / agent setup -- can be obtained by writing and calling a custom function. This is rarely what you actually want, but it's worth mentioning because you might one day want a custom function for some other purpose.
Update: TL;DR there seems to be no built-in way to achieve this, so a custom task is an easy solution.
Capistrano provides facilities to share files and directories over all releases. This is convenient and provides even some safety on files that should not be easily changed (or must remain the same across releases), e.g. a database configuration file.
But when it comes to replace or just update one of these shared files, I end up doing it manually, directly on the target machine. I would like to improve on that, for instance by asking Capistrano to overwrite some or all shared files when deploying. A kind of --force flag with some granularity.
I am not aware of any such kind of facility, and failing so far in my search. Any pointer?
Thinking about it
One of the reason why this facility does not exist (except that I did not find it!) is that it may be harder than it looks. For example, let's assume we have a shared database configuration file, and we exclude it from version control for security reason (common practice). Current release relies on version 1 of the DB configuration. The next release requires version 2 of the DB configuration. If the deployment goes well, everything's good. It gets harder when rolling back after some error with the new release (e.g. a regression), as version 1 must then be available.
Such automation would be cool and convenient, but dangerous as well. Yet I have practical use cases at hand.
I created a template method to do this. For example, I could have a task like this:
task :create_database_yml do
on roles(:app, :db) do
within(shared_path) do
template "local/path/to/database.yml.erb",
"config/database.yml",
:mode => "600"
end
end
end
And then I have a database.yml.erb template that uses things like fetch(:database_password) to fill in appropriate values. You can use the ask method in Capistrano to prompt for these values so they are never committed.
The implementation of template can be very simple: you just need to read the file, pass it through ERB, and then use Capistrano's upload! to place the results on the server.
My version is a little more complicated than yours probably needs to be, but in case you are curious:
https://github.com/mattbrictson/capistrano-mb/blob/7600440ecd3331945d03e059368b75849857f1fb/lib/capistrano/mb/dsl.rb#L104
One approach is to use a system configuration tool like Chef or Puppet to deploy the configuration files distinctly from Capistrano.
Another approach is to create a custom task to do this: https://coderwall.com/p/wgs6gw/copy-local-files-to-remote-server-using-capistrano-3
I personally don't change on-server configs often enough or on enough servers yet to have tried to automate it. Crafting an scp command which copies the desired config file to all of the required servers has sufficed in the past.