My data is like below,
{"empid"="100","empname="test"}
Now, my transform is,
'transforms' ='IndexName',
'transforms.IndexName.type' = 'io.confluent.connect.transforms.ExtractTopic$Value',
'transforms.IndexName.field' = 'test',
'transforms.IndexName.skip.missing.or.null' = 'true'
I am creating the sink connector for elasticsearch. while using the above transform I can create the index name like "test". But I want to customize the index name like "test100". is it possible?
You would need to use more than one transform to modify test to testblah
Add a RegexRouter after the extraction to update the topic name with static content
"transforms" = "IndexName,AddSuffix"
"transforms.IndexName.type" = ...
...
"transforms.AddSuffix.type" = ...
...
If you're trying to pull the empid=100 field, there is no current transform that can extract & concatenate more than one field, and I would question your design choices for having one Elasticsearch index per empid
Related
I am looking for an example to implement Apache beam GCP dataflow Pipeline to Update the data in Mongo DB using upsert operation i.e if the value exsit it should update the value and if not it should insert.
Syntax like below :
pipeline.apply(...)
.apply(MongoDbIO.write()
.withUri("mongodb://localhost:27017")
.withDatabase("my-database")
.withCollection("my-collection")
.withUpdateConfiguration(UpdateConfiguration.create().withUpdateKey("key1")
.withUpdateFields(UpdateField.fieldUpdate("$set", "source-field1", "dest-field1"),
UpdateField.fieldUpdate("$set","source-field2", "dest-field2"),
//pushes entire input doc to the dest field
UpdateField.fullUpdate("$push", "dest-field3") )));
Below is my Pipeline Code where i am currently inserting the document after preapring the collection like below
{"_id":{"$oid":"619632693261e80017c44145"},"vin":"SATESTCAVA74621","timestamp":"2021-11-18T10:48:59.889Z","key":"EV_CHARGE_NOW_SETTING","value":"DEFAULT"}
Now i want to Update the 'value' and 'timestamp' if the combination of 'vin' and 'key' are present, if 'vin' and 'key' combination is not present then Insert the new document using upsert.
PCollection<PubsubMessage> pubsubMessagePCollection= pubsubMessagePCollectionMap.get(topic);
pubsubMessagePCollection.apply("Convert pubsub to kv,k=vin", ParDo.of(new ConvertPubsubToKVFn()))
.apply("group by vin key",GroupByKey.<String,String>create())
.apply("filter data for alerts, status and vehicle data", ParDo.of(new filterMessages()))
.apply("converting message to document type", ParDo.of(
new ConvertMessageToDocumentTypeFn(list_of_keys_str, collection, options.getMongoDBHostName(),options.getMongoDBDatabaseName())).withSideInputs(list_of_keys_str))
.apply(MongoDbIO.write()
.withUri(options.getMongoDBHostName())
.withDatabase(options.getMongoDBDatabaseName())
.withCollection(collection));
Now if i want to use this below lines of code:
.withUpdateConfiguration(UpdateConfiguration.create().withUpdateKey("key1")
.withUpdateFields(UpdateField.fieldUpdate("$set", "source-field1", "dest-field1"),
UpdateField.fieldUpdate("$set","source-field2", "dest-field2"),
//pushes entire input doc to the dest field
UpdateField.fullUpdate("$push", "dest-field3") )));
What will be my key1, "source-field1", "dest-field1", "source-field2", "dest-field2", "dest-field3" ?
I am confused with this values. Please help !
Below code i am trying to update
MongoDbIO.write()
.withUri(options.getMongoDBHostName())
.withDatabase(options.getMongoDBDatabaseName())
.withCollection(collection)
.withUpdateConfiguration(UpdateConfiguration.create()
.withIsUpsert(true)
.withUpdateKey("vin")
.withUpdateKey("key")
.withUpdateFields(UpdateField.fieldUpdate("$set", "vin", "vin"),
UpdateField.fieldUpdate("$set", "key", "key"),
UpdateField.fieldUpdate("$set", "timestamp", "timestamp"),
UpdateField.fieldUpdate("$set", "value", "value")))
Using above code My document is not updating instead adding with id = vin , it should update based the exsiting record with vin and key match, also if insert it should insert with auto generated _id value.
Please suggest what to do here ?
upsert configuration is read from here, you can configure it with withIsUpsert(true).
In your original syntax, add the extra line to enable upsert.
pipeline.apply(...)
.apply(MongoDbIO.write()
.withUri("mongodb://localhost:27017")
.withDatabase("my-database")
.withCollection("my-collection")
.withUpdateConfiguration(
UpdateConfiguration.create()
.withIsUpsert(true)
.withUpdateKey("key1")
.withUpdateFields(
UpdateField.fieldUpdate("$set", "source-field1", "dest-field1"),
UpdateField.fieldUpdate("$set","source-field2", "dest-field2"),
//pushes entire input doc to the dest field
UpdateField.fullUpdate("$push", "dest-field3"))));
I want to get the last data created on my database, more specifically the field called: "Saldo Atual" and after this get the form data field "Valor" and make a sum between this fields (the Old one "Saldo Atual" and the new "Valor") before create a new record and add this "Valor" and the sum of "Saldo Atual"+"Valor".
I don't know how to get the database last record
var saldoAtual =
record.saldoAtual = saldoAtual + record.valor;
And I want to know if the second line it's correct to save the sum of "Saldo Atual"+"Valor"
Based on my comment, I think that your records will need to have a Date_Created field. Then in your onBeforeCreate event the following script should work (this is untested, so you need to do your own testing):
var query = app.models.YourModelTheSameAsYourEventModel.newQuery();
query.sorting.Date_Created._descending();
query.limit = 1;
var lastrecord = query.run();
record.saldoAtual = lastrecord.saldoAtual + record.valor; //You may need to use lastrecord[0].saldoAtual
record.Created_Date = new Date();
Try that and do some testing to see if this yields your desired output.
This may be a basic question, but it's my first, so please be kind :-).
I have a sap.m.table with two models, one model with transaction data (trxModel) and another model that is used to display a sap.m.select list (reasonCodeModel). The table model is set to trxModel.
The selected value key from the dropdown needs to update a value (ReasonCodeID) in the trxModel when a value from the reason code list is selected.
I can retrieve the selected key in the change event as so
var selKey = evt.getParameter("selectedItem").getKey();
Is there a simple way to find the trxModel relevant model path from the table row Select list value I've just modified? Or is it possible to bind the ReasonCodeID from the trxModel to the ReasonCodeID field in the reasonCodeModel?
Just an extra piece of info, The current row is selected and is accessible
var selItem = dtlTable.getSelectedItem();
2nd question and I guess could be kind of related, is there a way of getting the table model path based on the selected item (highlighted row) of the table? And vice a versa?
More details on Select & Table binding.
var tabTemplate = new sap.m.ColumnListItem(
{
::
new sap.m.Select(
"idReasonCodeSelect",
{
enabled : false,
change : function(evt) {
oS4View.getController().changeReasonCodeSel(evt);
}
}
),
Bind the resource code Select to the Table
// bind the reason codes to the reason code model
sap.ui.getCore().byId("idReasonCodeSelect").setModel(
oReasonCodeModel);
sap.ui.getCore().byId("idReasonCodeSelect").bindAggregation("items", "/results",
new sap.ui.core.Item({
key : "{ReasCodeID}",
text : "{ReasCodeDesc}"
}));
Per Qualiture comment, how do I bind the Select key to the table model ReasonCodeID value?
I found an approach to tackle the first part of my question above
From the change function on the Select, I can find the path of the table model using the following.
var path = evt.getSource().getParent().getBindingContext().sPath;
2nd Update:
On the selectionChange event on the table, there's a couple of options to find the associated model path or model content.
// find the model path
oModelPath = selItem.getBindingContext().getPath();
// model values
oItem = oEvent.getParameter("listItem").getBindingContext().getObject();
So my only remaining issue, While I loop through the table model results (trxModel) and I want the Select List (using setSelectedKey) to reflect the ReasonCodeID value in the trxModel.
I'm trying to insert patterns (nodes and edges) using merge. Using the demo movies graph, I'm sending the following cypher query: the movie exists, I'd like to create the User node and the edge in one query.
MERGE (top:Movie { title:'Top Gun' })<-[:viewed]-(user:User {Name:'Pierre'})
ON CREATE SET user.created = timestamp()
ON MATCH SET user.lastSeen = timestamp()
RETURN user,top;
"MERGE needs at least some part of the pattern to already be known. Please provide values for one of: user, top"
Actually, top exits, I can't figure out what's wrong in my query. Thanks for your help.
Pierre
Would this work?
MATCH (top:Movie { title:'Top Gun' })
MERGE (top)<-[:viewed]-(user:User {Name:'Pierre'})
ON CREATE SET user.created = timestamp()
ON MATCH SET user.lastSeen = timestamp()
RETURN user,top;
or this for creates:
MERGE (top:Movie { title:'Top Gun' })
MERGE (user:User {Name:'Pierre'})
ON CREATE SET user.created = timestamp()
ON MATCH SET user.lastSeen = timestamp()
MERGE (top)<-[:viewed]-(user)
RETURN user,top;
I'm new to Couchbase and am struggling to get a composite index to do what I want it to. The use-case is this:
I have a set of "Enumerations" being stored as documents
Each has a "last_updated" field which -- as you may have guessed -- stores the last time that the field was updated
I want to be able to show only those enumerations which have been updated since some given date but still sort the list by the name of the enumeration
I've created a Couchbase View like this:
function (doc, meta) {
var time_array;
if (doc.doc_type === "enum") {
if (doc.last_updated) {
time_array = doc.last_updated.split(/[- :]/);
} else {
time_array = [0,0,0,0,0,0];
}
for(var i=0; i<time_array.length; i++) { time_array[i] = parseInt(time_array[i], 10); }
time_array.unshift(meta.id);
emit(time_array, null);
}
}
I have one record that doesn't have the last_updated field set and therefore has it's time fields are all set to zero. I thought as a first test I could filter out that result and I put in the following:
startkey = ["a",2012,0,0,0,0,0]
endkey = ["Z",2014,0,0,0,0,0]
While the list is sorted by the 'id' it isn't filtering anything! Can anyone tell me what I'm doing wrong? Is there a better composite view to achieve these results?
In couchbase when you query view by startkey - endkey you're unable to filter results by 2 or more properties. Couchbase has only one index, so it will filter your results only by first param. So your query will be identical to query with:
startkey = ["a"]
endkey = ["Z"]
Here is a link to complete answer by Filipe Manana why it can't be filtered by those dates.
Here is a quote from it:
For composite keys (arrays), elements are compared from left to right and comparison finishes as soon as a element is different from the corresponding element in the other key (same as what happens when comparing strings à la memcmp() or strcmp()).
So if you want to have a view that filters by date, date array should go first in composite key.