orientdb version 2.1.11
I'm trying to setup on 3 nodes, I want to shard the data like the description of default-distribute-config.json.
write node1 -> node1, node2
write node2 -> node2, node3
write node3 -> node3, node1
{
"autoDeploy": true,
"hotAlignment": false,
"executionMode": "undefined",
"readQuorum": 1,
"writeQuorum": 2,
"failureAvailableNodesLessQuorum": false,
"readYourWrites": true,
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"index": {
},
"person_node1": {
"servers": ["node1","node2"]
},
"person_node2": {
"servers": ["node2","node3"]
},
"person_node3": {
"servers": ["node3","node1"]
},
"*": {
"servers": ["<NEW_NODE>"]
}
}
}
but when I started nodes, they didn't wok like this. Sometime they work like this (copy from log file):
"person_node1": {
"servers": ["node1"]
},
"person_node2": {
"servers": ["node2"]
},
"person_node3": {
"servers": ["node3"]
},
"*": {
"servers": ["node2","node1","node3","<NEW_NODE>"]
},
Is there any detail document describing the configuration?
thanks.
The configuration is always updated by removing the absent nodes. If you want a more sticky configuration, set "hotAlignment": true, after 2.1.10 is safe to set it to true.
We're working on it by providing a more flexible behaviour.
Related
I am trying to run Wazuh/Wazuh docker container on ECS. I was able to register task definition and launch container using Terraform. However, I am facing an issue with "Volume"(Data Volume) while registering tak definition using AWS CLI command.
Command: aws ecs --region eu-west-1 register-task-definition --family hids --cli-input-json file://task-definition.json
Error:
ParamValidationError: Parameter validation failed:
Unknown parameter in volumes[0]: "dockerVolumeConfiguration", must be one of: name, host
2019-08-29 07:31:59,195 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255
{
"containerDefinitions": [
{
"portMappings": [
{
"hostPort": 514,
"containerPort": 514,
"protocol": "udp"
},
{
"hostPort": 1514,
"containerPort": 1514,
"protocol": "udp"
},
{
"hostPort": 1515,
"containerPort": 1515,
"protocol": "tcp"
},
{
"hostPort": 1516,
"containerPort": 1516,
"protocol": "tcp"
},
{
"hostPort": 55000,
"containerPort": 55000,
"protocol": "tcp"
}
],
"image": "wazuh/wazuh",
"essential": true,
"name": "chids",
"cpu": 1600,
"memory": 1600,
"mountPoints": [
{
"containerPath": "/var/ossec/data",
"sourceVolume": "ossec-data"
},
{
"containerPath": "/etc/filebeat",
"sourceVolume": "filebeat_etc"
},
{
"containerPath": "/var/lib/filebeat",
"sourceVolume": "filebeat_lib"
},
{
"containerPath": "/etc/postfix",
"sourceVolume": "postfix"
}
]
}
],
"volumes": [
{
"name": "ossec-data",
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
},
{
"name": "filebeat_etc",
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
},
{
"name": "filebeat_lib",
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
},
{
"name": "postfix",
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
}
]
}
I tried by adding "host" parameter(however it supports Bind Mounts only). But got the same error.
"volumes": [
{
"name": "ossec-data",
"host": {
"sourcePath": "/var/ossec/data"
},
"dockerVolumeConfiguration": {
"scope": "shared",
"driver": "local",
"autoprovision": true
}
}
]
ECS should register the task definition having 4 Data Volumes and associated mount points.
Got the issue.
Removed "dockerVolumeConfiguration" parameter from "Volume" configuration and it worked.
"volumes": [
{
"name": "ossec-data",
"host": {
"sourcePath": "/ecs/ossec-data"
}
},
{
"name": "filebeat_etc",
"host": {
"sourcePath": "/ecs/filebeat_etc"
}
},
{
"name": "filebeat_lib",
"host": {
"sourcePath": "/ecs/filebeat_lib"
}
},
{
"name": "postfix",
"host": {
"sourcePath": "/ecs/postfix"
}
}
]
Can you check on your version of awscli?
aws --version
According to all the documentation, your first task definition should work fine and I tested it locally without any issues.
It might be that you are using an older aws cli version where the syntax was different or parameters were different at the time.
Could you try updating your aws cli to the latest version and try again?
--
Some additional info I found:
Checking on the aws ecs cli command, they added docker volume configuration as part of the cli in v1.80
The main aws-cli releases updates periodically to update the commands but they don't provide much info on what specific versions of each command is changed:
https://github.com/aws/aws-cli/blob/develop/CHANGELOG.rst
If you update your aws-cli version things should work
In Loopback (v3), when defining indexes in my model.json files, how do I specify different types of indexes (such as a BRIN)? Also, how do I specify index conditions (such as if I want to create a partial index)? I'm using postgres for the database, if that's relevant.
You can configure the index type via type field.
{
"name": "MyModel",
"properties": {
// ...
},
"indexes": {
"myindex": {
"columns": "name, email",
"type": "BRIN",
// ...
}
}
}
I am afraid LoopBack does not support index conditions (partial indexes) yet. Feel free to open a new issue in https://github.com/strongloop/loopback-connector-postgresql/issues.
i was trying to add in Lb4. Its pretty straightforward there (should be same for lb3 as well i hope)
#model({
name: 'tablename',
settings: {
indexes: {
idx_tablename: {
columnA : '',
columnB : '',
columnC: ''
}
}
}
})
once the build is done, the index name idx_tablename with 3 columns will get created
In PostgreSQL and Loopback 3 you can specify an index for multi-column like this.
The following loopback JSON code creates index in Postgres with fields message and type are unique together.
{
"name": "notification",
"base": "PersistedModel",
"idInjection": true,
"options": {
"validateUpsert": true
},
"properties": {
"message": {
"type": "string",
"required": true
},
"type": {
"type": "string",
"required": true
},
"seen": {
"type": "boolean",
"required": true,
"default": false
}
},
"validations": [],
"relations": {},
"acls": [],
"methods": {},
"indexes": {
"message_type_index": {
"keys": "message, type",
"options": {"unique": true}
}
}
}
I'm following this instruction to setup glusterfs on my kubernetes cluster. At heketi-client/bin/heketi-cli setup-openshift-heketi-storage part, heketi-cli tells me :
Error: volume create: heketidbstorage: failed: Host 192.168.99.25 not connected
or sometimes:
Error: volume create: heketidbstorage: failed: Staging failed on 192.168.99.26. Error: Host 192.168.99.25 not connected
heketi.json is
{
"_port_comment": "Heketi Server Port Number",
"port": "8080",
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": false,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "7319"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "7319"
}
},
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": "Execute plugin. Possible choices: mock, kubernetes, ssh",
"executor": "kubernetes",
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",
"kubeexec": {
"rebalance_on_expansion": true
},
"sshexec": {
"rebalance_on_expansion": true,
"keyfile": "/etc/heketi/private_key",
"fstab": "/etc/fstab",
"port": "22",
"user": "root",
"sudo": false
}
},
"_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
"backup_db_to_kube_secret": false
}
topology-sample.json is
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"redis-test25"
],
"storage": [
"192.168.99.25"
]
},
"zone": 1
},
"devices": [
{
"name": "/dev/sda7",
"destroydata": true
}
]
},
{
"node": {
"hostnames": {
"manage": [
"redis-test26"
],
"storage": [
"192.168.99.26"
]
},
"zone": 1
},
"devices": [
{
"name": "/dev/sda7",
"destroydata": true
}
]
},
{
"node": {
"hostnames": {
"manage": [
"redis-test01"
],
"storage": [
"192.168.99.113"
]
},
"zone": 1
},
"devices": [
{
"name": "/dev/sda7",
"destroydata": true
}
]
}
]
}
]
}
The heketi-cli is v8.0.0 and kubernetes is v1.12.3
How do I fix this problem?
Update: Just found that I missed the iptables part, but now the message becomes
Error: volume create: heketidbstorage: failed: Host 192.168.99.25 is not in 'Peer in Cluster' state
seems that one of the glusterfs pod cannot connect to others, I tried kubectl exec -i glusterfs-59ftx -- gluster peer status:
Number of Peers: 2
Hostname: 192.168.99.26
Uuid: 6950db9a-3d60-4625-b642-da5882396bee
State: Peer Rejected (Disconnected)
Hostname: 192.168.99.113
Uuid: 78983466-4499-48d2-8411-2c3e8c70f89f
State: Peer Rejected (Disconnected)
while the other one said:
Number of Peers: 1
Hostname: 192.168.99.26
Uuid: 23a0114d-65b8-42d6-8067-7efa014af68d
State: Peer in Cluster (Connected)
I solved these problems by myself.
For first part, the reason is that I didn't setup iptables in every nodes according to Infrastructure Requirements.
For second part according to this article, delete all file in /var/lib/glusterd except glusterd.info and then start over from Kubernete Deploy.
I created a 2-node OrientDB cluster by following the below steps. But while distributing it, the data present in only one of the node is accessible. Please can you help me debug this issue. The OrientDB version is 2.2.6
Steps involved :
Utilized plocal mode in ETL tool and stored part of the data in node 1 and the other part in node2. The data stored actually belongs to just one class of vertex alone. ( On checking the data from console, the data has got injested properly ).
Then executed both the nodes in distributed mode, data from only one machineis accessible.
The default-distributed-db-config.json file is specified below :
{
"autoDeploy": true,
"readQuorum": 1,
"writeQuorum": 1,
"executionMode": "undefined",
"readYourWrites": true,
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"address": {
"servers" : [ "orientmaster" ]
},
"address_1": {
"servers" : [ "orientslave1" ]
},
"*": {
"servers": ["<NEW_NODE>"]
}
}
}
There are two clusters created for the vertex named address namely address and address_1. The data in machine orientslave1 is stored using ETL tool into cluster address_1 , similarly the data in machine orientmaster is stored into the cluster address. ( I've ensured that both of these cluster ids are different at time of creation )
However when these two machines are connected together in distributed mode, the data in cluster address_1 is only visible
The ETL json is attached below :
{
"source": { "file": { "path": "/home/ubuntu/labvolume1/DataStorage/geo1_5lacs.csv" } },
"extractor": { "csv": {"columnsOnFirstLine": false, "columns":["place:string"] } },
"transformers": [
{ "vertex": { "class": "ADDRESS", "skipDuplicates":true } }
],
"loader": {
"orientdb": {
"dbURL": "plocal:/home/ubuntu/labvolume1/orientdb/databases/ETL_Test1",
"dbType": "graph",
"dbUser": "admin",
"dbPassword": "admin",
"dbAutoCreate": true,
"wal": false,
"tx":false,
"classes": [
{"name": "ADDRESS", "extends": "V", "clusters":1}
], "indexes": [
{"class":"ADDRESS", "fields":["place:string"], "type":"UNIQUE" }
]
}
}
}
Please let me know, if there is anything i'm doing wrongly
For example: I have three nodes in the same cluster ("node1","node2" and "node3"). These nodes are identical in configuration files hazelcast.xml. I want that the database "DB_1" was placed only on the first and on the second node. And database "DB_2" was placed only on the second and the third node.
I modifed the file "default-distributed-db-config.json" on the first and on the second node:
{
"autoDeploy": true,
"hotAlignment": false,
"executionMode": "undefined",
"readQuorum": 1,
"writeQuorum": 2,
"failureAvailableNodesLessQuorum": false,
"readYourWrites": true,
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"index": {
},
"*": {
"servers": ["node1","node2"]
}
}
}
I modifed the file "default-distributed-db-config.json" on the third node:
{
"autoDeploy": true,
"hotAlignment": false,
"executionMode": "undefined",
"readQuorum": 1,
"writeQuorum": 2,
"failureAvailableNodesLessQuorum": false,
"readYourWrites": true,
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"index": {
},
"*": {
"servers": ["node3"]
}
}
}
I modifed the file "distributed-config.json" in the database directory "DB_1" on the first and second node.
I removed from it all:
<NEW_NODE>
and wrote everywhere the only names of the first and second nodes:
"*":{"#type":"d","#version":0,"servers":["node1","node2"]},"orole_node2":{"#type":"d","#version":0,"servers":["node2","node1"]},"e_node2":{"#type":"d","#version":0,"servers":["node2","node1"]},"ouser_node2":{"#type":"d","#version":0,"servers":["node2","node1"]},"oschedule_node2":{"#type":"d","#version":0,"servers":["node2","node1"]},"orids_node2":{"#type":"d","#version":0,"servers":["node2","node1"]},"v_node2":{"#type":"d","#version":0,"servers":["node2","node1"]},"ofunction_node2":{"#type":"d","#version":0,"servers":["node2","node1"]}}
But nonetheless if start the third node, then for database "DB_1" will run replicating in the third node too.
(OrientDB v2.1.13)
I don't think it is possible, every node of the same cluster has to be synchronized with the others.