Puppet PostgresQL management - postgresql

I am trying to provision an EC2 instance using puppet. In the process I have downloaded the puppetlabs-postgresql module from puppetlabs. Since I'm fairly new to puppet, i do not want to manage my database by creating classes in my site.pp file located in /etc/puppet/manifests/site.pp. Rather I want to have a module call database in /etc/puppet/modules/database.
What I have done so far is create an init.pp file in /etc/puppet/modules/database. Below is the content of my init.pp file :
class database {
# resources
postgresql::globals{'globals':
version => '9.3',
manage_package_repo => true,
encoding => 'UTF8',
locale => 'it_IT.utf8',
}
postgresql::server{'server':
ensure => 'present',
listen_addresses => '*',
manage_firewall => true,
}
postgresql::server::contrib{'contrib':
package_ensure => 'present',
}
}
And then in my /etc/puppet/manifests/site.pp file i have included the database class as below :
node default {
include localusers
include database
}
However i keep getting an error :
Error: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type postgresql::globals at /etc/puppet.manifests/init.pp:12
Please what is the correct way to make use of the postgresql classes and resources in my own module and create a database in the module as well ?

You're on the right track, but there are a few issues with how you're using the postgresql module. The reason you're getting the Invalid resource type error is that you're trying to use postgresql::globals as a defined type when it's actually a class. You have the same issue with the other two classes you're using. Try this...
class database {
# set global defaults before creating server
class { 'postgresql::globals':
version => '9.3',
manage_package_repo => true,
encoding => 'UTF8',
locale => 'it_IT.utf8',
}->
class { 'postgresql::server':
listen_addresses => '*',
manage_firewall => true,
}
# install the postgresql contrib package
class { 'postgresql::server::contrib':
package_ensure => 'present',
}
# create database with user and default permissions
postgresql::server::db { 'my_awesome_db':
user => 'my_db_user',
password => 'puppetRocks',
}
}
In the reference section of the module documentation, there's a breakdown of the classes and resources (a.k.a. defined types). The postgresql::server::db type that I used is the simplest way to create a database, user, and permissions all at once. There are separate types available for each of those to provide more fine-grained control.

Related

Problem with PostgreSQL json object (PGobject) to logstash - type cast problem

I am trying to index data from a PostgreSQL database to elastic search using logstash. I have a column in the database which is of type JSON. When I run logstash, I get an error called "Missing Converter handling". I do understand that is java/logstash doesn't recognize the JSON type column.
I can typecast the column to "col_name::TEXT" which works fine. However, I need that column as JSON in an elastic search index. Is there any workaround?
Note: Keys in the JSON object is not fixed, it varies.
Logstash postgres query example:
input {
jdbc {
# Postgres jdbc connection string to our database, bot
jdbc_connection_string => "***"
# The user we wish to execute our statement as
jdbc_user => "***"
jdbc_password => "***"
# The path to our downloaded jdbc driver
jdbc_driver_library => "<ja_file_path>"
# The name of the driver class for Postgresql
jdbc_driver_class => "org.postgresql.Driver"
# our query
statement => "SELECT col_1, col_2, col_3, json_col_4 from my_db;"
}
}
filter{
}
output {
stdout { codec => json_lines }
}
Need JSON object from PostgreSQL to elastic search.

Puppet-hiera-Function lookup() did not find a value-Windows

I installed dsc module and added AD user to Domain controller using puppet. Code below works fine when hard-coding password as plain text. Is it possible somehow to encrypt those passwords.
I read that hiera-eyaml is solution for this so i encrypted password
[root#PUPPET puppet]# /opt/puppetlabs/puppet/bin/eyaml encrypt -p
Enter password: **********
string: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAl/+uUACl6WpGAnA1sSqEuTp39SVYfHc7J0BMvC+a2C0YzQg1V]
Then stored that encrypted pass in /etc/common.eyaml file (specified in hiera config file)
/opt/puppetlabs/puppet/bin/eyaml edit /etc/common.eyaml
I can decrypt the file successfully:
/opt/puppetlabs/puppet/bin/eyaml decrypt -f /etc/common.eyaml
Then i specified encrypted pass to manifest file
/etc/puppetlabs/code/environments/production/manifests/site.pp:
dsc_xADUser {'FirstUser':
dsc_ensure => 'present',
dsc_domainname => 'ad.contoso.com',
dsc_username => 'tfl',
dsc_userprincipalname => 'tfl#ad.contoso.com',
dsc_password => {
'user' => 'Administrator#ad.contoso.com',
'password' => Sensitive('pass')
},
dsc_passwordneverexpires => true,
dsc_domainadministratorcredential => {
'user' => 'Administrator#ad.contoso.com',
'password' => Sensitive(lookup('password'))
},
}
On windows node i got error
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Function lookup() did not find a value for the name 'password' on node windows.example.com
Hiera config file:
cat /etc/puppetlabs/puppet/hiera.yaml
---
# Hiera 5 Global configuration file
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Eyaml hierarchy"
lookup_key: eyaml_lookup_key # eyaml backend
paths:
- "/etc/common.eyaml"
options:
pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"
cat /etc/common.eyaml
password: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAl/+uUACl6WpGAnA1sSqEuTp39SVYfHc7J0BMvC+a2C0YzQg1V]
I'm new to Puppet and this hiera is confusing me
For starters, there is a typo in your Hiera config file. The path to the data should be:
paths:
- "/etc/common.eyaml"
After fixing that, you need to retrieve the value from Hiera. This is performed with the puppet lookup function. Since you have a single key value pair here in a single data file, this can be performed with a minimal number of arguments.
dsc_xADUser {'FirstUser':
dsc_ensure => 'present',
dsc_domainname => 'ad.contoso.com',
dsc_username => 'tfl',
dsc_userprincipalname => 'tfl#ad.contoso.com',
dsc_password => {
'user' => 'Administrator#ad.contoso.com',
'password' => Sensitive('pass')
},
dsc_passwordneverexpires => true,
dsc_domainadministratorcredential => {
'user' => 'Administrator#ad.contoso.com',
'password' => lookup('string'),
},
}
However, you also really want to redact that password from your logs and reports. You would want to wrap that password String in a Sensitive data type.
'password' => Sensitive(lookup('string')),
You seem to already be doing that for your other password that is being passed in as a String pass.
A side note to all of this is that Puppet has intrinsic support for lookup retrievals from Vault and Conjur in version 6, so that will become best practices instead of hiera-eyaml soon.
Ufff, after much struggling finally got it working:
cat /etc/puppetlabs/puppet/hiera.yaml
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Eyaml hierarchy"
lookup_key: eyaml_lookup_key # eyaml backend
paths:
- "nodes/%{trusted.certname}.yaml"
- "windowspass.eyaml"
options:
pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem
Created password:
/opt/puppetlabs/puppet/bin/eyaml encrypt -l 'password' -s 'Pass' --pkcs7-public-key=/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem --pkcs7-private-key=/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem
Added it to /etc/puppetlabs/puppet/data/windowspass.eyaml file:
/opt/puppetlabs/puppet/bin/eyaml edit windowspass.eyaml --pkcs7-public-key=/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem --pkcs7-private-key=/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem
cat /etc/puppetlabs/puppet/data/windowspass.eyaml
---
password: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAUopetXenh/+DN1+VesIZUI5y4k3kOTn2xa5uBrtGZP3GvGqoWfwAbYsfeNApjeMG+lg93/N/6mE9T59DPh]
Tested decryption:
/opt/puppetlabs/puppet/bin/eyaml decrypt -f windowspass.eyaml --pkcs7-public-key=/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem --pkcs7-private-key=/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem
As Matt suggested, mapped content of windowspass.eyaml to manifest file
'password' => Sensitive(lookup('password'))
Debugging command helped me a lot:
puppet master --debug --compile windows.example.com --environment=production
Thanks everyone, especially to Matt

Logstash, mongodb and jdbc

I have a problem configuring logstash. I want to be able to put in input jdbc for mongodb.
My config :
input{
jdbc{
jdbc_driver_library => "mongo-java-driver-3.2.2.jar"
jdbc_driver_class => "com.mongodb.MongoClient"
jdbc_connection_string => "jdbc:mongodb://localhost:27017"
jdbc_user => ""
}
}
output{
stdout{
}
}
The problem is :
:error=>"Java::JavaSql::SQLException: No suitable driver found for jdbc:mongodb://localhost:27017/"}
The MongoDB JDBC Driver setting is not correct. You must specify the name of the driver class, not the client class.
jdbc_driver_class => "mongodb.jdbc.MongoDriver"
Also make sure that the jdbc_driver_library contains the full absolute path to your mongo-java-driver-3.2.2.jar JAR file
More inputs would be good.
you must specify the location of the mongo-java-driver-3.2.2.jar in jdbc_driver_library.
please see the following links :
Documentation
Similar problem

Puppet and Postgres annoying warning: Passing "version" to postgresql::server is deprecated

I'm using the puppet-postgresql module to manage PostgreSQL. That part of the manifest looks like this:
class { 'postgresql::server':
postgres_password => 'postgres',
}
postgresql::server::db { $db_name:
user => $db_user,
password => postgresql_password($db_user, $db_password),
}
Works fine but I get the annoying warning:
Warning: Scope(Class[Postgresql::Server]): Passing "version" to postgresql::server is deprecated; please use postgresql::globals instead.
EDIT:
I even added the version to the globals, but I'm still getting the warning:
class { 'postgresql::globals':
version => '9.3',
}->
class { 'postgresql::server':
postgres_password => 'postgres',
}
postgresql::server::db { $db_name:
user => $db_user,
password => postgresql_password($db_user, $db_password),
}
But I'm not passing any 'version' to postgresql::server. What I'm doing wrong here?
Docs https://forge.puppetlabs.com/puppetlabs/postgresql didn't helped me in this case...
It's a bug in the puppetlabs-postgresql module in the 3.4.x series. It's since been fixed in PR 471 which will be released in the next major version (4.0.0 by the looks of it).
If you don't specify the version, a default version is selected by the module in the file manifests/globals.pp. So you can either edit this file to specify a newer version for your OS or pass the version in parameter when calling postgresql::server

Overwrite if if it exists or create if it does not

I am trying to rewrite a file using puppet with the following function.
If the file exists I still want the file to be rewrite from the source. Will this be achieved with the following method?
define setup_sysctl_conf( $dependence=File[$dummy_dependence_file] )
{
file { $name:
path => '/etc/sysctl.conf',
ensure => present,
mode => 0777,
source => '/vagrant/files/sysctl.conf',
require => $dependence,
}
}
The file: /etc/sysctl.conf will already be present on your host (created by the initscripts package).
I would recommend to modify existing files with puppet using augeas instead of replacing them.
Example (changes net.ipv4.ip_forward to 1):
class sysctl_augeas_example {
augeas{"Set net.ipv4.ip_forward to 1":
context => "/files",
changes => [
"set etc/sysctl.conf/net.ipv4.ip_forward 1",
]
}
}
include sysctl_augeas_example
Save this example as test.pp and run it with puppet apply test.pp