I am currently monitoring the database(Jdbc).
This bank refers to product sales, where at first the status of the sale receives "WAITING FOR PAYMENT" and after payment it receives the status of "PAID".
however, when the payment status changes, the log is not updated, so it only counts the first status.
I ask for help on how can I make logstash update the status field dynamically?
the idea is that it is similar to the bank, receiving updates on top of those that already exist, in this way we can mount graphics on top of these updates.
My input is like this:
input {
jdbc {
tags => ["oracle"]
jdbc_driver_library => "/usr/share/logstash/lib/ojdbc8.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "***"
jdbc_user => "***"
jdbc_password => "***"
jdbc_validate_connection => true
jdbc_paging_enabled => true
use_column_value => true
tracking_column => unix_ts_in_secs
tracking_column_type => "timestamp"
schedule => "*/1 * * * *"
statement => "SELECT PRODUCTS.ID, PRODUCTS.STATUS, TO_TIMESTAMP(PRODUCTS.CREATION_DATE) AS unix_ts_in_secs FROM PRODUCTS ON (TO_TIMESTAMP(PRODUCTS.CREATION_DATE) > :sql_last_value) ORDER BY PRODUCTS.CREATION_DATE ASC"
record_last_run => true
}
}
In output elastic configuration just update each document with "PRODUCTS.ID" value as below
output {
elasticsearch {
ssl => true
user => "***"
password => "***"
hosts => ["https://elastic-ip:9200"]
keystore => "path to cert"
keystore_password => "****"
truststore => "path to cert"
truststore_password =>"****"
index => "data-%{+yyyy.MM.dd}"
document_id => "%{PRODUCTS.ID}"
doc_as_upsert => true
action => "update"
}
}
I want to create a user in postgres who can only made SELECT in all the tables of the current database.
How can specify ALL tables in puppet?
Here an extract of my puppet file:
...
postgresql::server::database_grant { 'PnBP':
privilege => 'CREATE',
db => 'db',
role => 'role',
}
postgresql::server::table_grant { 'SELECT':
privilege => 'SELECT',
table => 'ALL',
db => 'db',
role => 'role',
}
...
But when I specify the word ALL it doesn't work.
Here the error:
Error: /Stage[main]/Main/Node[default]/Postgresql::Server::Table_grant[PnBP]/Postgresql::Server::Grant[table:PnBP]/Postgresql_psql[grant:table:PnBP]: Could not evaluate: Error evaluating 'unless' clause, returned pid 30443 exit 1: 'ERROR: relation "all" does not exist
'
I check the doc, but it doesn't specify how to apply the privileges for all the tables.
table : Specifies the table to which you are granting access.
postgresql::server::table_grant
The Puppet documentation for Postgresql describes a postgresql::server::grant option that looks more flexible than the table_grant which assumes a single table.
Looks to be something like:
postgresql::server::grant{ 'SELECT':
object_type => 'ALL TABLES IN SCHEMA',
object_name => 'public',
privilege => 'SELECT',
db => 'db',
role => 'role',
}
https://github.com/puppetlabs/puppetlabs-postgresql#postgresqlservergrant
I just tried to up a new VM managed by Puppet.
When upgrading some packages, the following messages pops up:
Setting up libssl1.0.0:amd64 (1.0.1e-2+deb7u12) ...
Checking for services that may need to be restarted...done.
Checking for services that may need to be restarted...done.
Checking init scripts...
[1;24r(B)0[m[1;24r
Package configuration┌─────────────────────┤
Configuring libssl1.0.0:amd64 ├─────────────────────┐│││
There are services installed on your system which need to be restarted ││
when certain libraries, such as libpam, libc, and libssl, are upgraded. ││
Since these restarts may cause interruptions of service for the system, ││
you will normally be prompted on each upgrade for the list of services ││
you wish to restart. You can choose this option to avoid being││ prompted;
instead, all necessary restarts will be done for you││
automatically so you can avoid being asked questions on each library││ upgrade.││││
Restart services during package upgrades without asking?││││
<Yes><No>│││└───────────────────────────────────────────────────────────────────────────┘
Failed to open terminal.debconf: whiptail output the above errors, giving up!
Setting up libgnutls26:amd64 (2.12.20-8+deb7u2) ...
dpkg: error processing libssl1.0.0:amd64 (--configure):
subprocess installed post-installation script returned error exit status 255
Setting up libkrb5support0:amd64 (1.10.1+dfsg-5+deb7u2) ...
Setting up libk5crypto3:amd64 (1.10.1+dfsg-5+deb7u2) ...
Setting up libkrb5-3:amd64 (1.10.1+dfsg-5+deb7u2) ...
Setting up libgssapi-krb5-2:amd64 (1.10.1+dfsg-5+deb7u2) ...
Setting up libmagic1:amd64 (5.11-2+deb7u5) ...
Setting up file (5.11-2+deb7u5) ...
Setting up libxml2:amd64 (2.8.0+dfsg1-7+wheezy1) ...
dpkg: dependency problems prevent configuration of libcurl3:amd64:
libcurl3:amd64 depends on libssl1.0.0 (>= 1.0.1); however:
Package libssl1.0.0:amd64 is not configured yet.
Then follow a bunch of failed package configurations leading my environment not to be as I wanted...
How can I make this work?
Thank you!
EDIT : Here's my node's manifest:
class pricing {
package { "libatlas-base-dev":
ensure => "installed" ,
require => Exec['apt-get update']
}
package { "gfortran":
ensure => "installed" ,
require => Exec['apt-get update']
}
class { 'python':
version => '2.7',
dev => true,
virtualenv => true,
pip => true,
}
class { 'postgresql::globals':
encoding => 'UTF8',
locale => 'en_GB.UTF-8',
manage_package_repo => true,
version => '9.3',
}->class { 'postgresql::client':
}->class { 'postgresql::lib::devel': }
package {"libffi-dev" : ensure => "present"}
package {"libxml2-dev" : ensure => "present"}
package {"libxslt-dev" : ensure => "present"}
if $pricing_state == "master" {
package {"rabbitmq-server" :
ensure => "present",
require => Exec['apt-get update'],
}
}
file { '/etc/boto.cfg':
source => 'puppet:///modules/pricing/boto.cfg',
}
file { "/pricing/logs/":
ensure => directory,
mode => 777,
owner => "celery",
group => "celery",
}
file { "/pricing/logs/pricing.logs":
ensure => file,
mode => 777,
owner => "celery",
group => "celery",
}
user { "celery":
ensure => present,
comment => "celery",
membership => minimum,
shell => "/bin/bash",
home => "/home/$name",
managehome => true,
}
exec { "import-gpg-dotdeb":
command => "/usr/bin/wget -q http://www.dotdeb.org/dotdeb.gpg -O -| /usr/bin/apt-key add -"
}
apt::source { 'dotdeb':
location => 'http://packages.dotdeb.org',
release => 'wheezy',
repos => 'all',
require => [Exec['import-gpg-dotdeb']]
}
class { 'redis':
package_ensure => 'latest',
conf_port => '6379',
conf_bind => '0.0.0.0',
system_sysctl => true,
conf_requirepass => '3I53G3944G9ngZC',
require => [Apt::Source['dotdeb']]
}
if $pricing_state == "master" {
if $env_small == "prod" {
include supervisord
supervisord::program { 'pricing':
ensure => present,
command => '/pricing/bin/python getprices.py',
user => 'root',
directory => '/pricing/',
numprocs => 1,
autorestart => 'true',
require => Python::Virtualenv['/pricing']
}
supervisord::program { 'listen_newprices':
ensure => absent,
command => '/pricing/bin/python listen_newprices.py',
user => 'root',
directory => '/pricing/',
numprocs => 1,
autorestart => 'true',
require => Python::Virtualenv['/pricing']
}
supervisord::program { 'getprixvente':
ensure => present,
command => '/pricing/bin/python getprixvente.py',
user => 'root',
directory => '/pricing/',
numprocs => 1,
autorestart => 'true',
require => Python::Virtualenv['/pricing']
}
supervisord::program { 'getprixachat':
ensure => present,
command => '/pricing/bin/python getprixachat.py',
user => 'root',
directory => '/pricing/',
numprocs => 1,
autorestart => 'true',
require => Python::Virtualenv['/pricing']
}
supervisord::program { 'flower':
ensure => present,
command => '/pricing/bin/celery flower --port=5555 --basic_auth=celery:celery69 --broker=amqp://celery:2xF09Ad050Ct7yb#127.0.0.1:5672//',
user => 'root',
directory => '/pricing/',
numprocs => 1,
autorestart => 'true',
require => Python::Virtualenv['/pricing']
}
exec { 'restart pricing':
command => 'supervisorctl restart pricing',
path => '/usr/bin:/usr/sbin:/bin:/usr/local/bin/',
require => Supervisord::Program['pricing']
}
exec { 'restart getprixvente':
command => 'supervisorctl restart getprixvente',
path => '/usr/bin:/usr/sbin:/bin:/usr/local/bin/',
require => Supervisord::Program['getprixvente']
}
exec { 'restart getprixachat':
command => 'supervisorctl restart getprixachat',
path => '/usr/bin:/usr/sbin:/bin:/usr/local/bin/',
require => Supervisord::Program['getprixachat']
}
}
}
if $pricing_state == "slave" {
file { "/etc/init.d/celeryd":
ensure => file,
content => template('pricing/celeryd_init.erb'),
mode => 700,
owner => "root",
group => "root",
}
file { "/etc/default/celeryd":
ensure => file,
content => template('pricing/celeryd.erb'),
mode => 640,
owner => "root",
group => "root",
}
service { 'celeryd':
name => celeryd,
ensure => running,
enable => true,
subscribe => File['/etc/default/celeryd'],
require => [
File['/etc/default/celeryd'],
File['/etc/init.d/celeryd'],
User['celery'],
Python::Virtualenv['/pricing'],
],
}
exec { 'restart celeryd':
command => 'service celeryd restart',
path => '/usr/bin:/usr/sbin:/bin:/usr/local/bin/',
require => Service['celeryd'],
}
logrotate::rule { 'celerydslavelogs':
path => '/var/log/celery/*.log',
size => '100k',
rotate => 5,
}
}
logrotate::rule { 'celerydlogs':
path => '/pricing/logs/*.log',
size => '100k',
rotate => 5,
}
python::virtualenv { '/pricing':
ensure => present,
version => '2.7',
requirements => '/puppet/modules/pricing/files/requirements.txt',
owner => $user,
group => $user,
cwd => '/pricing',
timeout => 36000,
require => [
Class['postgresql::client', 'postgresql::lib::devel', 'python'],
Package['libatlas-base-dev', 'gfortran'],
Package['libffi-dev'],
Package['libxml2-dev'],
Package['libxslt-dev'],
Class['postgresql::client', 'postgresql::lib::devel', 'python'],
],
}
}
I have the following configuration set up in my /etc/gitlab/gitlab.rb
gitlab_rails['gitlab_email_from'] = 'testing#mydom.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'secure.emailsrvr.com'
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = 'testing#mydom.com'
gitlab_rails['smtp_password'] = 'password'
gitlab_rails['smtp_domain'] = 'mydom.info'
gitlab_rails['smtp_authentication'] = 'login'
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = true
Note: I have tried setting smtp_domain to mydom.com just in case but still no change.
Under sidekiq in the admin panel I get the following errors:
Worker: Sidekiq::Extensions::DelayedMailer
Arguments:"---\n- !ruby/class 'Notify'\n- :project_access_granted_email\n- - 68\n"
Error: ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
Worker: Sidekiq::Extensions::DelayedMailer
Arguments: "---\n- !ruby/class 'Notify'\n- :group_access_granted_email\n- - 32\n"
Error: ActiveRecord::RecordNotFound: Couldn't find UsersGroup with 'id'=32
This is the equivalent content in /opt/gitlab/embedded/service/gitlab-rails/config/environments/production.rb
config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {
:address => 'secure.emailsrvr.com',
:port => 465,
:domain => 'mydom.info',
:authentication => :login,
:user_name => 'testing#mydom.com',
:password => 'password',
:enable_starttls_auto => true,
:tls => true
}
Edit
Is there anything at least I can use to test that these configurations are correct?
This was an issue with incorrect Configuration details.
I've got a puppet manifest that resists my attempts to get it working right, given I'm no expert on the puppet DSL, and I'm fairly new to Puppet, I haven't managed to figure this out.
I'm trying to install Postgres using puppetlabs posgres module, creating a default role, and fixing up the DBs to work on UTF8.
Everything runs and installs, but the role doesn't get created. But if I run the provision again, then the role gets created. I assume perhaps has to do with the execution order, but honestly I'm lost.
Here's the code I'm using on my manifest file.
user { "user_vagrant":
ensure => "present",
}->
exec { 'apt_update':
command => 'apt-get update',
path => '/usr/bin/'
}
package { ['vim','postgresql-server-dev-9.1','libmysqlclient-dev','nodejs']:
ensure => 'installed',
before => Class['postgresql::server'],
require => Exec['apt_update'],
}
class { 'postgresql::server':
ip_mask_allow_all_users => '0.0.0.0/0',
listen_addresses => '*',
ipv4acls => ['local all all md5'],
postgres_password => 'postgres',
require => User['user_vagrant'],
}
postgresql::server::role { 'vagrant':
createdb => true,
login => true,
password_hash => postgresql_password("vagrant", "vagrant"),
require => Class['postgresql::server'],
} ->
exec { 'utf8_postgres':
command => 'pg_dropcluster --stop 9.1 main ; pg_createcluster --start --locale en_US.UTF-8 9.1 main',
unless => 'sudo -u postgres psql -t -c "\l" | grep template1 | grep -q UTF',
path => ['/bin', '/sbin', '/usr/bin', '/usr/sbin'],
}
Finally found the right approach to fix both the applied order, and the UTF8 issue which forced me to try the "pg_dropcluster" to begin with. Btw, this is a known issue here's the issue url http://projects.puppetlabs.com/issues/4695
This is the whole file I use to install PostgreSQL 9.1 with UTF8, and RVM ruby. Hope this helps.
Modules:
- puppetlabs/apt - 1.4
- puppetlabs/concat - 1.0
- puppetlabs/stdlib - 4.1.0
- puppetlabs/postgresql - 3.2
- blt04/puppet-rvm - git://github.com/blt04/puppet-rvm.git
stage { 'pre':
before => Stage['main']
}
class pre_req {
user { "vagrant":
ensure => "present",
}
exec { 'apt-update':
command => 'apt-get update',
path => '/usr/bin'
}->
exec { 'install_postgres':
command => "/bin/bash -c 'LC_ALL=en_US.UTF-8; /usr/bin/apt-get -y install postgresql'",
}
}
class { 'pre_req':
stage => pre
}
package { ['postgresql-server-dev-9.1']:
ensure => 'installed',
before => Class['postgresql::server']
}
class { 'postgresql::globals':
encoding => 'UTF8',
locale => 'en_US.UTF-8'
}->
class { 'postgresql::server':
stage => main,
locale => 'en_US.UTF-8',
ip_mask_allow_all_users => '0.0.0.0/0',
listen_addresses => '*',
ipv4acls => ['local all all md5'],
postgres_password => 'postgres',
require => User['vagrant']
}->
postgresql::server::role { 'vagrant':
createdb => true,
login => true,
password_hash => postgresql_password("vagrant", "vagrant"),
}
class rvm_install {
class { 'rvm': version => '1.23.10' }
rvm::system_user { vagrant: ; }
rvm_system_ruby {
"ruby-2.0.0-p247":
ensure => "present",
default_use => false;
}
rvm_gemset {
"ruby-2.0.0-p247#plyze":
ensure => present,
require => Rvm_system_ruby['ruby-2.0.0-p247'];
}
rvm_gem {
"puppet":
name => "puppet",
ruby_version => "ruby-2.0.0-p247",
ensure => latest,
require => Rvm_system_ruby["ruby-2.0.0-p247"];
}
rvm_gem {
"bundler":
name => "bundler",
ruby_version => "ruby-2.0.0-p247",
ensure => latest,
require => Rvm_system_ruby["ruby-2.0.0-p247"];
}
}
class { 'rvm_install':
require => User['vagrant'],
}