Github repo, but for ease here is my health.json stub file:
{
"request": {
"url": "/health",
"method": "GET"
},
"response": {
"status": 200,
"body": "{}",
"headers": {
"content-type": "application/json"
}
}
}
Setup is extremely basic: I have a health.json stub file defining a very basic request. I run the following command to start up the docker container for wiremock: docker-compose up mocks
I then get the port from docker and hit the following url in a browser: http://localhost:49158/health
The following output is what I see from the container initialization and my attempt to hit the url:
#framework:~/git/habits/mocks$ docker-compose up mocks
Starting habits-mocks ... done
Attaching to habits-mocks
habits-mocks | 08/13/2022 21:04:39 [Info] : WireMock.Net by Stef Heyenrath (https://github.com/WireMock-Net/WireMock.Net)
habits-mocks | 08/13/2022 21:04:39 [Debug] : WireMock.Net server settings {
habits-mocks | "Port": null,
habits-mocks | "UseSSL": null,
habits-mocks | "StartAdminInterface": true,
habits-mocks | "ReadStaticMappings": true,
habits-mocks | "WatchStaticMappings": false,
habits-mocks | "WatchStaticMappingsInSubdirectories": false,
habits-mocks | "ProxyAndRecordSettings": null,
habits-mocks | "Urls": [
habits-mocks | "http://*:80"
habits-mocks | ],
habits-mocks | "StartTimeout": 10000,
habits-mocks | "AllowPartialMapping": false,
habits-mocks | "AdminUsername": null,
habits-mocks | "AdminPassword": null,
habits-mocks | "RequestLogExpirationDuration": null,
habits-mocks | "MaxRequestLogCount": null,
habits-mocks | "AllowCSharpCodeMatcher": false,
habits-mocks | "AllowBodyForAllHttpMethods": false,
habits-mocks | "AllowOnlyDefinedHttpStatusCodeInResponse": false,
habits-mocks | "DisableJsonBodyParsing": false,
habits-mocks | "DisableRequestBodyDecompressing": null,
habits-mocks | "HandleRequestsSynchronously": false,
habits-mocks | "ThrowExceptionWhenMatcherFails": false,
habits-mocks | "CertificateSettings": null,
habits-mocks | "CustomCertificateDefined": false
habits-mocks | }
habits-mocks | 08/13/2022 21:04:39 [Info] : WireMock.Net server using .NET Core 3.1
habits-mocks | Hosting environment: Production
habits-mocks | Content root path: /app/
habits-mocks | Now listening on: http://0.0.0.0:80
habits-mocks | 08/13/2022 21:04:39 [Info] : Reading Static MappingFile : '/app/__admin/mappings/health.json'
habits-mocks | 08/13/2022 21:04:39 Press Ctrl+C to shut down
habits-mocks | 08/13/2022 21:04:39 WireMock.Net server running
habits-mocks | 08/13/2022 21:05:09 [Warn] : HttpStatusCode set to 404 : No matching mapping found
habits-mocks | 08/13/2022 21:05:09 [DebugRequestResponse] : Admin[False] {
habits-mocks | "Guid": "8767ab02-b710-4c10-8db6-826eb8563b74",
habits-mocks | "Request": {
habits-mocks | "ClientIP": "172.18.0.1",
habits-mocks | "DateTime": "2022-08-13T21:05:09.2509752Z",
habits-mocks | "Path": "/health",
habits-mocks | "AbsolutePath": "/health",
habits-mocks | "Url": "http://localhost:49158/health",
habits-mocks | "AbsoluteUrl": "http://localhost:49158/health",
habits-mocks | "ProxyUrl": null,
habits-mocks | "Query": {},
habits-mocks | "Method": "GET",
habits-mocks | "Headers": {
habits-mocks | "Connection": [
habits-mocks | "keep-alive"
habits-mocks | ],
habits-mocks | "Accept": [
habits-mocks | "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
habits-mocks | ],
habits-mocks | "Accept-Encoding": [
habits-mocks | "gzip, deflate"
habits-mocks | ],
habits-mocks | "Accept-Language": [
habits-mocks | "en-US,en;q=0.5"
habits-mocks | ],
habits-mocks | "Host": [
habits-mocks | "localhost:49158"
habits-mocks | ],
habits-mocks | "User-Agent": [
habits-mocks | "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0"
habits-mocks | ],
habits-mocks | "Upgrade-Insecure-Requests": [
habits-mocks | "1"
habits-mocks | ],
habits-mocks | "Sec-Fetch-Dest": [
habits-mocks | "document"
habits-mocks | ],
habits-mocks | "Sec-Fetch-Mode": [
habits-mocks | "navigate"
habits-mocks | ],
habits-mocks | "Sec-Fetch-Site": [
habits-mocks | "none"
habits-mocks | ],
habits-mocks | "Sec-Fetch-User": [
habits-mocks | "?1"
habits-mocks | ]
habits-mocks | },
habits-mocks | "Cookies": null,
habits-mocks | "Body": null,
habits-mocks | "BodyAsJson": null,
habits-mocks | "BodyAsBytes": null,
habits-mocks | "BodyEncoding": null,
habits-mocks | "DetectedBodyType": null,
habits-mocks | "DetectedBodyTypeFromContentType": null
habits-mocks | },
habits-mocks | "Response": {
habits-mocks | "StatusCode": 404,
habits-mocks | "Headers": {
habits-mocks | "Content-Type": [
habits-mocks | "application/json"
habits-mocks | ]
habits-mocks | },
habits-mocks | "BodyDestination": null,
habits-mocks | "Body": null,
habits-mocks | "BodyAsJson": {
habits-mocks | "Guid": null,
habits-mocks | "Status": "No matching mapping found"
habits-mocks | },
habits-mocks | "BodyAsBytes": null,
habits-mocks | "BodyAsFile": null,
habits-mocks | "BodyAsFileIsCached": null,
habits-mocks | "BodyOriginal": null,
habits-mocks | "BodyEncoding": null,
habits-mocks | "DetectedBodyType": 2,
habits-mocks | "DetectedBodyTypeFromContentType": 0,
habits-mocks | "FaultType": null,
habits-mocks | "FaultPercentage": null
habits-mocks | },
habits-mocks | "MappingGuid": null,
habits-mocks | "MappingTitle": null,
habits-mocks | "RequestMatchResult": null,
habits-mocks | "PartialMappingGuid": null,
habits-mocks | "PartialMappingTitle": null,
habits-mocks | "PartialRequestMatchResult": null
habits-mocks | }
habits-mocks | 08/13/2022 21:05:09 WireMock.Net server running
^CGracefully stopping... (press Ctrl+C again to force)
Stopping habits-mocks ... done
Wiremock appears to read the file fine. The url in the log matches the url in the json stub file. I don't understand what I'm missing :(
A friend set me straight. Turns out Wiremock.net and Wiremock.org are different things, and I was using the documentation for the wrong one. Fun!
Related
I want to push log files which are present in a specific folder inside a container to Cloudwatch. For this, I tried Firelens logdriver but had no luck.
As mentioned here : https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/config-file-type-file
I created a custom docker image for fluentbit and deployed it as a side car container in task definition:
FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
COPY fluentbit.conf /extra.conf
fluentbit.conf (I tried exec for debugging purpose, aim is to use tail):
[INPUT]
Name exec
Tag exec_ls
Command ls /opt/apache-tomcat-8.5.72/my-app/logs
Interval_Sec 3
Interval_NSec 0
Buf_Size 8mb
[OUTPUT]
Name cloudwatch
Match *
region ap-south-1
log_group_name fluent-bit-adapter-logs
log_stream_prefix from-fluent-bit-
auto_create_group true
log_key log
This is my ECS Fargate task definition :
{
"ipcMode": null,
"executionRoleArn": "<role>",
"containerDefinitions": [
{
"dnsSearchDomains": null,
"environmentFiles": null,
"logConfiguration": {
"logDriver": "awsfirelens",
"secretOptions": null,
"options": {
"log_group_name": "/aws/ecs/containerinsights/$(ecs_cluster)/application",
"auto_create_group": "true",
"log_key": "log",
"log_stream_prefix": "log_stream_name",
"region": "ap-south-1",
"Name": "cloudwatch"
}
},
"entryPoint": null,
"portMappings": [
{
"hostPort": 8080,
"protocol": "tcp",
"containerPort": 8080
}
],
"command": null,
"linuxParameters": null,
"cpu": 0,
"environment": [],
"resourceRequirements": null,
"ulimits": null,
"dnsServers": null,
"mountPoints": [],
"workingDirectory": null,
"secrets": null,
"dockerSecurityOptions": null,
"memory": null,
"memoryReservation": null,
"volumesFrom": [],
"stopTimeout": null,
"image": "<image-url>",
"startTimeout": null,
"firelensConfiguration": null,
"dependsOn": null,
"disableNetworking": null,
"interactive": null,
"healthCheck": null,
"essential": true,
"links": null,
"hostname": null,
"extraHosts": null,
"pseudoTerminal": null,
"user": null,
"readonlyRootFilesystem": null,
"dockerLabels": null,
"systemControls": null,
"privileged": null,
"name": "my-app"
},
{
"dnsSearchDomains": null,
"environmentFiles": null,
"logConfiguration": {
"logDriver": "awslogs",
"secretOptions": null,
"options": {
"awslogs-group": "/ecs/log_router",
"awslogs-region": "ap-south-1",
"awslogs-create-group": "true",
"awslogs-stream-prefix": "firelens"
}
},
"entryPoint": null,
"portMappings": [],
"command": null,
"linuxParameters": null,
"cpu": 0,
"environment": [],
"resourceRequirements": null,
"ulimits": null,
"dnsServers": null,
"mountPoints": [],
"workingDirectory": null,
"secrets": null,
"dockerSecurityOptions": null,
"memory": null,
"memoryReservation": 50,
"volumesFrom": [],
"stopTimeout": null,
"image": "<image-url>",
"startTimeout": null,
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/extra.conf"
}
},
"dependsOn": null,
"disableNetworking": null,
"interactive": null,
"healthCheck": null,
"essential": true,
"links": null,
"hostname": null,
"extraHosts": null,
"pseudoTerminal": null,
"user": "0",
"readonlyRootFilesystem": null,
"dockerLabels": null,
"systemControls": null,
"privileged": null,
"name": "log_router"
}
],
"placementConstraints": [],
"memory": "8192",
"taskRoleArn": "<role>",
"compatibilities": [
"EC2",
"FARGATE"
],
"taskDefinitionArn": "<my-app arn>",
"family": "equbemi",
"requiresAttributes": [
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.execution-role-awslogs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.firelens.options.config.file"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.logging-driver.awsfirelens"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.execution-role-ecr-pull"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.task-eni"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "ecs.capability.firelens.fluentbit"
}
],
"pidMode": null,
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"cpu": "4096",
"revision": 19,
"status": "ACTIVE",
"inferenceAccelerators": null,
"proxyConfiguration": null,
"volumes": []
}
For my application container, I've given logconfiguration as firelens and deployed a side-car container as mentioned in documentation. I tried the tail command in firelens config but didn't work. So just to troubleshoot, I tried exec and found out in firelens container logs that it is giving "File not found" exception. I assume it is trying to find the path in its own container(the side-car one) and not in the application container. I'm not sure how to make the firelens container access the application container. Am I missing anything here?
*
Here, the problem is that the fluent-bit container does not have access to the application container file system. You need to configure a volume so that the log path will be shared between both of these containers.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html
Add a volume at the parent level and add this volume as a mount point in both of the container definitions:
{
...
"volumes": [
{
"fsxWindowsFileServerVolumeConfiguration": null,
"efsVolumeConfiguration": null,
"name": "log_path",
"host": {
"sourcePath": null
},
"dockerVolumeConfiguration": null
}
],
...
"containerDefinitions": [
{
...
"mountPoints": [
{
"readOnly": null,
"containerPath": "/var/log",
"sourceVolume": "log_path"
}
],
...
},
{
...
"mountPoints": [
{
"readOnly": null,
"containerPath": "/var/log",
"sourceVolume": "log_path"
}
],
...
}
]
...
}
I have a json file in which i would like to change values and save again as a Json:
Values that need to be updated:
domain
repo
[
{
"name": "[concat(parameters('factoryName'), '/LS_New')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"description": "Connection",
"annotations": [],
"type": "AzureDatabricks",
"typeProperties": {
"domain": "https://url.net",
"accessToken": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "LS_vault",
"type": "LinkedServiceReference"
},
"secretName": "TOKEN"
},
"newClusterNodeType": "Standard_DS4_v2",
"newClusterNumOfWorker": "2:10",
"newClusterSparkEnvVars": {
"PYSPARK_PYTHON": "/databricks/python3/bin/python3"
},
"newClusterVersion": "7.2.x-scala2.12"
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/LS_evaKeyVault')]"
]
},
{
"name": "[concat(parameters('factoryName'), '/PIP_Log')]",
"type": "Microsoft.DataFactory/factories/pipelines",
"apiVersion": "2018-06-01",
"properties": {
"description": "Unzip",
"activities": [
{
"name": "Parse",
"description": "This notebook",
"type": "DatabricksNotebook",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"notebookPath": "/dataPipelines/main_notebook.py",
"baseParameters": {
"businessgroup": {
"value": "#pipeline().parameters.businessgroup",
"type": "Expression"
},
"project": {
"value": "#pipeline().parameters.project",
"type": "Expression"
}
},
"libraries": [
{
"pypi": {
"package": "cytoolz"
}
},
{
"pypi": {
"package": "log",
"repo": "https://b73gxyht"
}
}
]
},
"linkedServiceName": {
"referenceName": "LS_o",
"type": "LinkedServiceReference"
}
}
],
"parameters": {
"businessgroup": {
"type": "string",
"defaultValue": "test"
},
"project": {
"type": "string",
"defaultValue": "log-analytics"
}
},
"annotations": []
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/LS_o')]"
]
}
]
I tried using regex but i am only able to update 1 value :
<valuesToReplace>
<valueToReplace>
<regExSearch>(\/PIP_Log[\w\W]*?[pP]roperties[\w\W]*?[lL]ibraries[\w\W]*?[pP]ypi[\w\W]*?"repo":\s)"(.*?[^\\])"</regExSearch>
<replaceWith>__PATValue__</replaceWith>
</valueToReplace>
<valueToReplace>
<regExSearch>('\/LS_New[\w\W]*?[pP]roperties[\w\W]*?[tT]ypeProperties[\w\W]*?"domain":\s"(.*?[^\\])")</regExSearch>
<replaceWith>__LSDomainName__</replaceWith>
</valueToReplace>
</valuesToReplace>
Here is the powershell code. The loop goes through all the values that are to be replaced.
I tried using dynamic variable in select-string and looping, but it doesn't seem to work
foreach($valueToReplace in $configFile.valuesToReplace.valueToReplace)
{
$regEx = $valueToReplace.regExSearch
$replaceValue = '"' + $valueToReplace.replaceWith + '"'
$matches = [regex]::Matches($json, $regEx)
$matchExactValueRegex = $matches.Value | Select-String -Pattern """repo\D:\s*(.*)" | % {$_.Matches.Groups[1].Value}
$updateReplaceValue = $matches.Value | Select-String -Pattern "repo\D:\s\D__(.*)__""" | % {$_.Matches.Groups[1].Value}
$updateReplaceValue = """$patValue"""
$json1 = [regex]::Replace($json, $matchExactValueRegex , $updateReplaceValue)
$matchExactValueRegex1 = $matches.Value | Select-String -Pattern """domain\D:\s*(.*)" | % {$_.Matches.Groups[1].Value}
$updateReplaceValue1 = $matches.Value | Select-String -Pattern "domain\D:\s\D__(.*)__""" | % {$_.Matches.Groups[1].Value}
$updateReplaceValue1 = """$domainURL"""
$json = [regex]::Replace($json1, $matchExactValueRegex1 , $updateReplaceValue1)
}
else
{
Write-Warning "Inactive config value"
}
$json | Out-File $armFileWithReplacedValues
Where am i missing??
You should not peek and poke in serialized files (as e.g. Json files) directly. Instead deserialize the file with the ConvertFrom-Json cmdlet, make your changes to the object and serialize it again with the ConvertTo-Json cmdlet:
$Data = ConvertFrom-Json $Json
$Data[0].properties.typeproperties.domain = '_LSDomainName__'
$Data[1].properties.activities.typeproperties.libraries[1].pypi.repo = '__PATValue__'
$Data | ConvertTo-Json -Depth 9 | Out-File $armFileWithReplacedValues
Hi I'm dealing with rather difficult XML file which I'm trying to reformat and clean for some processing. I've been using Pyspark to process the data into a dataframe and I am using com.databricks.spark.xml to read the file.
My Dataframe looks like this; Each field is JSON Formatted
+----------------+---------------------------------+
| Identifier| Info|
+----------------+---------------------------------+
| JSON | Json |
| | |
| | |
+----------------+---------------------------------+
This is a sample value from the Identifier column
{
"Other": [
{
"_Type": "A",
"_VALUE": "999"
},
{
"_Type": "B",
"_VALUE": "31086"
},
{
"_Type": "C",
"_VALUE": "13123"
},
{
"_Type": "D",
"_VALUE": "32323"
},
{
"_Type": "E",
"_VALUE": "2223"
},
{
"_Type": "F",
"_VALUE": "100"
},
]
}
And this is how the Info Column looks like
{
"Demo": {
"BirthDate": "2009-09-13",
"BirthPlace": {
"_VALUE": null,
"_nil": true
},
"Rel": {
"_VALUE": null,
"_nil": true
}
},
"EmailList": {
"_VALUE": null,
"_nil": true
},
"Name": {
"LastName": "Marwan",
"FullName": {
"_VALUE": null,
"_nil": true
},
"GivenName": "Saad",
"MiddleName": null,
"PreferredFamilyName": {
"_VALUE": null,
"_nil": true
}
},
"OtherNames": {
"_VALUE": null,
"_nil": true
}
}
I am trying to create a dataframe that looks like the following
+-------+--------+-----------+------------+------------+
| F| E| LastName| GivenName | BirthDate|
+-------+--------+-----------+------------+------------+
EDIT : I finally resolved my problem, I replace the command line by :
"command": [
"mongod",
"--port",
"3001"
]
I want to run mongodb on port 3001 instead of 27017 on my ECS instance. I have one application (annonces) and one database (mongodb).
This is my Task Definition configuration :
{
"executionRoleArn": null,
"containerDefinitions": [
{
"dnsSearchDomains": null,
"logConfiguration": null,
"entryPoint": null,
"portMappings": [
{
"hostPort": 80,
"protocol": "tcp",
"containerPort": 4001
}
],
"command": null,
"linuxParameters": null,
"cpu": 0,
"environment": [
{
"name": "DB_URI",
"value": "mongodb://mongo:3001/db-annonces"
}
],
"ulimits": null,
"dnsServers": null,
"mountPoints": [],
"workingDirectory": null,
"dockerSecurityOptions": null,
"memory": 128,
"memoryReservation": null,
"volumesFrom": [],
"image": "XXXXXXXXXXX.dkr.ecr.eu-west-3.amazonaws.com/annonces-service:latest",
"disableNetworking": null,
"healthCheck": null,
"essential": true,
"links": [
"mongo"
],
"hostname": null,
"extraHosts": null,
"user": null,
"readonlyRootFilesystem": null,
"dockerLabels": null,
"privileged": null,
"name": "annonces"
},
{
"dnsSearchDomains": null,
"logConfiguration": null,
"entryPoint": [],
"portMappings": [
{
"hostPort": 3001,
"protocol": "tcp",
"containerPort": 3001
}
],
"command": [
"mongod --port 3001"
],
"linuxParameters": null,
"cpu": 0,
"environment": [],
"ulimits": null,
"dnsServers": null,
"mountPoints": [],
"workingDirectory": null,
"dockerSecurityOptions": null,
"memory": 128,
"memoryReservation": null,
"volumesFrom": [],
"image": "mongo:latest",
"disableNetworking": null,
"healthCheck": null,
"essential": false,
"links": null,
"hostname": "mongo",
"extraHosts": null,
"user": null,
"readonlyRootFilesystem": null,
"dockerLabels": null,
"privileged": null,
"name": "mongo"
}
],
"placementConstraints": [],
"memory": null,
"taskRoleArn": null,
"compatibilities": [
"EC2"
],
"taskDefinitionArn": "arn:aws:ecs:eu-west-3:XXXXXXXX:task-definition/annonces-task:17",
"family": "annonces-task",
"requiresAttributes": [
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
},
{
"targetId": null,
"targetType": null,
"value": null,
"name": "com.amazonaws.ecs.capability.ecr-auth"
}
],
"requiresCompatibilities": null,
"networkMode": "bridge",
"cpu": null,
"revision": 17,
"status": "ACTIVE",
"volumes": []
}
The problem seems to come from the line :
"command": [
"mongod --port 3001"
]
When I try to run the task, the task shut down immediately. If I remove this line, it works but the database is not on port 3001.
If i run the command docker logs on the specific container, I get the following error :
/usr/local/bin/docker-entrypoint.sh: line 340: exec: mongod --port 3001: not found
with one joinFieldName and lookup the Edge transformer works perfect. However, now two keys is required, i.e. compound index in the lookup. How can two joinFieldNames be specified?
This is the scripted(post processing) version:
Create edge Expands from (select from MC where sample=1 and mkey=6) to (select from Event where sample=1 and mcl=6).
This works, but is not suitable for production.
Can anyone help?
you can simply add 2 joinFieldName(s) like
{ "edge": { "class": "Conn",
"joinFieldName": "b1",
"lookup": "A.a1",
"joinFieldName": "b2",
"lookup": "A.a2",
"direction": "out"
}}
see below my test data:
json1.json
{
"source": { "file": { "path": "/home/ivan/Scrivania/cose/etl/stak39517796/data1.csv" } },
"extractor": { "csv": {} },
"transformers": [
{ "vertex": { "class": "A" } }
],
"loader": {
"orientdb": {
"dbURL": "plocal:/home/ivan/OrientDB/db_installati/enterprise/orientdb-enterprise-2.2.10/databases/stack39517796",
"dbType": "graph",
"dbAutoCreate": true,
"classes": [
{"name": "A", "extends": "V"},
{"name": "B", "extends": "V"},
{"name": "Conn", "extends": "E"}
]
}
}
}
json2.json
{
"source": { "file": { "path": "/home/ivan/Scrivania/cose/etl/stak39517796/data2.csv" } },
"extractor": { "csv": {} },
"transformers": [
{ "vertex": { "class": "B" } },
{ "edge": { "class": "Conn",
"joinFieldName": "b1",
"lookup": "A.a1",
"joinFieldName": "b2",
"lookup": "A.a2",
"direction": "out"
}}
],
"loader": {
"orientdb": {
"dbURL": "plocal:/home/ivan/OrientDB/db_installati/enterprise/orientdb-enterprise-2.2.10/databases/stack39517796",
"dbType": "graph",
"dbAutoCreate": true,
"classes": [
{"name": "A", "extends": "V"},
{"name": "B", "extends": "V"},
{"name": "Conn", "extends": "E"}
]
}
}
}
data1.csv
a1,a2
1,1
1,2
2,3
data2.csv
b1,b2
1,1
2,3
1,2
execution order:
json1
json2
and here is the final result:
orientdb {db=stack39517796}> select from v
+----+-----+------+----+----+-------+----+----+--------+
|# |#RID |#CLASS|a1 |a2 |in_Conn|b2 |b1 |out_Conn|
+----+-----+------+----+----+-------+----+----+--------+
|0 |#17:0|A |1 |1 |[#25:0]| | | |
|1 |#18:0|A |1 |2 |[#27:0]| | | |
|2 |#19:0|A |2 |3 |[#26:0]| | | |
|3 |#21:0|B | | | |1 |1 |[#25:0] |
|4 |#22:0|B | | | |3 |2 |[#26:0] |
|5 |#23:0|B | | | |2 |1 |[#27:0] |
+----+-----+------+----+----+-------+----+----+--------+