I'm performing a logical replication from our PostgreSQL server to AWS RDS, using the database migration tool (DMS). Most tables were migrated successfully, but I'm having troubles with a couple of the tables.
When I run the task to load the table, I get this error on the logs of RDS:
ERROR: invalid input value for enum property_type: ""
CONTEXT: unnamed portal parameter $17 = ''
STATEMENT: INSERT INTO "public"."loans"("id","account_id","loan_number","created_at","updated_at","folio","mers_min","mers_status","mers_status_date","application_number","servicer","servicer_loan_number","status","primary_borrower_last_name","primary_borrower_first_name","property_number_of_units","property_type","property_address_line1","property_address_line2","property_city","property_state","property_zip","property_county_code","property_country_code","property_census_tract_code","property_parcel_id","mortgage_type","qm_loan","purpose","ltv","amortization_type","amount","interest_rate","term","lien_priority","application_date","approval_date","rejected_date","closing_date","funding_date","purchase_date","source","officer","processor","underwriter","appraiser","property_usage","fha_case_number","var_payload","approval_type","approval_message","housing_expense_ratio","total_debt_expense_ratio","subordinate_financing_amount","combined_ltv","property_appraised_value","property_purchase_price","property_appraised_date","property_year_built","credit_score","au_type","au_recommendation","lender_product","heloc_indicator","reverse_indicator","property_pud_indicator","closer","additional_financing_amount","rejected_reason","mortgage_insurance_certificate_number","mortgage_insurance_coverage_amount","mortgage_insurance_premium","day_one_certainty","first_payment_date","maturity_date","principal_and_interest_payment_amount","is_portfolio","financing_concessions_amount","sales_concessions_amount","transaction_costs_amount","is_investment_quality","application_received_date","lender_program","refinance_cash_out_type") values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$79,$80,$81,$82,$83,$84)
The type of column property_type is a nullable enum that takes as values Condominium, Cooperative, ManufacturedHome, SingleFamily, Townhouse and TwoToFourFamily.
I'm experiencing a similar issue with another table from the same database, where I get:
ERROR: malformed range literal: ""
DETAIL: Missing left parenthesis or bracket.
CONTEXT: unnamed portal parameter $3 = ''
STATEMENT: INSERT INTO "public"."selections_sampling_data"("id","sow_id","period","loan_number","field_data","selected_on","substituted_on","substitution_for_id","received_for_review_on","created_at","updated_at","selected_for","selection_reason","selections_sampling_strategy_id") values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14)
This one doesn't explicitly tell me which column generated the error, but parameter $3 should correspond with the period column, which is a daterange type.
In both cases, the DMS task seems to be querying for an empty string as value, which is not allowed by those data types.
I don't know if this is an error with my schema, or how I configured the DMS task.
[Update]
My task configuration is the following:
{
"StreamBufferSettings": {
"StreamBufferCount": 3,
"CtrlStreamBufferSizeInMB": 5,
"StreamBufferSizeInMB": 8
},
"ErrorBehavior": {
"FailOnNoTablesCaptured": true,
"ApplyErrorUpdatePolicy": "LOG_ERROR",
"FailOnTransactionConsistencyBreached": false,
"RecoverableErrorThrottlingMax": 1800,
"DataErrorEscalationPolicy": "SUSPEND_TABLE",
"ApplyErrorEscalationCount": 0,
"RecoverableErrorStopRetryAfterThrottlingMax": true,
"RecoverableErrorThrottling": true,
"ApplyErrorFailOnTruncationDdl": false,
"DataTruncationErrorPolicy": "LOG_ERROR",
"ApplyErrorInsertPolicy": "LOG_ERROR",
"EventErrorPolicy": "IGNORE",
"ApplyErrorEscalationPolicy": "LOG_ERROR",
"RecoverableErrorCount": -1,
"DataErrorEscalationCount": 0,
"TableErrorEscalationPolicy": "STOP_TASK",
"RecoverableErrorInterval": 5,
"ApplyErrorDeletePolicy": "IGNORE_RECORD",
"TableErrorEscalationCount": 0,
"FullLoadIgnoreConflicts": true,
"DataErrorPolicy": "LOG_ERROR",
"TableErrorPolicy": "SUSPEND_TABLE"
},
"ValidationSettings": {
"ValidationPartialLobSize": 0,
"PartitionSize": 10000,
"RecordFailureDelayLimitInMinutes": 0,
"SkipLobColumns": false,
"FailureMaxCount": 10000,
"HandleCollationDiff": false,
"ValidationQueryCdcDelaySeconds": 0,
"ValidationMode": "ROW_LEVEL",
"TableFailureMaxCount": 1000,
"RecordFailureDelayInMinutes": 5,
"MaxKeyColumnSize": 8096,
"EnableValidation": true,
"ThreadCount": 5,
"RecordSuspendDelayInMinutes": 30,
"ValidationOnly": false
},
"TTSettings": {
"TTS3Settings": null,
"TTRecordSettings": null,
"EnableTT": false
},
"FullLoadSettings": {
"CommitRate": 1000,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"MaxFullLoadSubTasks": 2,
"TransactionConsistencyTimeout": 600,
"CreatePkAfterFullLoad": false,
"TargetTablePrepMode": "DO_NOTHING"
},
"TargetMetadata": {
"ParallelApplyBufferSize": 0,
"ParallelApplyQueuesPerThread": 0,
"ParallelApplyThreads": 0,
"TargetSchema": "",
"InlineLobMaxSize": 0,
"ParallelLoadQueuesPerThread": 0,
"SupportLobs": true,
"LobChunkSize": 64,
"TaskRecoveryTableEnabled": false,
"ParallelLoadThreads": 0,
"LobMaxSize": 0,
"BatchApplyEnabled": true,
"FullLobMode": true,
"LimitedSizeLobMode": false,
"LoadMaxFileSize": 0,
"ParallelLoadBufferSize": 0
},
"BeforeImageSettings": null,
"ControlTablesSettings": {
"historyTimeslotInMinutes": 5,
"HistoryTimeslotInMinutes": 5,
"StatusTableEnabled": false,
"SuspendedTablesTableEnabled": false,
"HistoryTableEnabled": false,
"ControlSchema": "",
"FullLoadExceptionTableEnabled": false
},
"LoopbackPreventionSettings": null,
"CharacterSetSettings": null,
"FailTaskWhenCleanTaskResourceFailed": false,
"ChangeProcessingTuning": {
"StatementCacheSize": 50,
"CommitTimeout": 1,
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchSplitSize": 0,
"BatchApplyTimeoutMax": 30,
"MinTransactionSize": 1000,
"MemoryKeepTime": 60,
"BatchApplyMemoryLimit": 500,
"MemoryLimitTotal": 1024
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"PostProcessingRules": null
}
So, after digging through AWS documentation and contacting support, I learned that enum data types don't migrate using DMS. The task will fail no matter what.
If your database has enum columns, you need to find another solution, i.e. barman and run it on an EC2 instance.
hello i have an object Json which have some values as payload , key , card .. so i aim to get the data i need directly with key "payload" .
var dataFinal= tag.data.toString();
and this what i get if i print my data
[log] handle {nfca: {identifier: [12, 4, 18, 17], atqa: [4, 0], maxTransceiveLength: 253, sak: 8, timeout: 618}, mifareclassic: {identifier: [99, 4, 150, 17], blockCount: 64, maxTransceiveLength: 253, sectorCount: 16, size: 1024, timeout: 618, type: 0}, ndef: {identifier: [99, 4, 150, 17], isWritable: true, maxSize: 716, canMakeReadOnly: false, cachedMessage: {records: [{typeNameFormat: 1, type: [84], identifier: [], payload: [1,45,989]}]}, type: com.nxp.ndef.mifareclassic}}
how can i get the payload value ?
You can check out the function jsonDecode() which expects a string as a param and returns dynamic or Map in your case
import 'dart:convert';
Map<String,dynamic> data = jsonDecode(tag.data.toString());
print(data["nfca"]);
Currently I have this mongoose schema and model:
let mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/whatever', { useNewUrlParser: true, useUnifiedTopology : true });
var Schema = mongoose.Schema;
var categorySchema = new Schema({
name: String,
description: String,
sortorder: Number,
isactive : { type: Boolean, default: true } ,
schedule : [] ,
});
var Category = mongoose.model('Category', categorySchema);
module.exports = Category;
schedule array get days of the week and open/close hours for everyday of the week.
When I try to update the schedule array, it adds to the previous values but I want to wipe those values and write new values in the array. This is how I use the model in my express route:
app.post('/categoryschedule', (req, res) =>{
let data = req.body
let filter = {_id : data.id}
let updated = {schedule : data.schedule }
Category.findByIdAndUpdate(filter, updated, {new : true}, function (error, results){
if (error) {
console.log("-----error ------- :", error)
}
res.json({"schedule update ": "category schedule updated", "resultsfromscheduleupdate" : results})
})
})
Currently, if I enter a schedule like 1,2,3,4,5,6,7 into the array and then edit the schedule to be 8, 9, 10, 11, 12, 13, 14 I get 1 2 3 4 5 6 7 8 9 10 11 12 13 14 but I want the update to replace 1 2 3 4 5 6 7 with 8 through 14 so array after update would only include values of 8 through 14.
So every time I update the schedule, it adds the new values to whatever existed in the array before the update. I used $set and read the doc but unable to do it. It seems like once I write to the array, I can't erase the values already exited and write new ones.
Maybe I misunderstood the doc. To be clear, my question is how to replace the values already exist in the array schedule with new one - wipe the array empty every time I update the schedule - and add the new values to the array.
schedule : Array
0: {weekdayname: 'Monday', weekdaynumber: 1, openhours: '10'...}
1: {weekdayname: 'Tuesday', weekdaynumber: 2, openhours: '10', …}
2: {weekdayname: 'Wednesday', weekdaynumber: 3, openhours: '10', …}
3: {weekdayname: 'Thursday', weekdaynumber: 4, openhours: '10', …}
4: {weekdayname: 'Friday', weekdaynumber: 5, openhours: '10', …}
5: {weekdayname: 'Saturday', weekdaynumber: 6, openhours: '10' …}
6: {weekdayname: 'Sunday', weekdaynumber: 0, openhours: '10' …}
Anyone with experience in mongoose arrays or what type of datatype for store arrays and array of objects in mongoose?
If it is not possible, then how do I get to specific array element, let us say, I want to get to schedule[0].openhours
How to I update using model.findOneAndUpdate ({filter by id}, {scheudle[0].openhours : "updated value"}) it is not updating the targeted doc.
How to reference array elements in mongoose update fn?
I have a layer that renders point features in a geojson source as circles. Here's an example of one of the features:
{
type: 'Feature',
properties: {
color: 'red',
red: true,
green: false
},
geometry: {
type: 'Point',
coordinates: [-77.038659, 38.931567]
}
};
I want circle-opacity to be a product of 3 factors (some properties on the features, map zoom, and a boolean in feature-state for whether that feature should be hidden). I can't figure out a way to write an expression that accounts for all three. Restrictions around zoom rules seem to be the issue.
Here's the logic I am trying to write:
if (feature-state.hidden) {
opacity = 0;
} else if (properties.red) {
opacity = 1;
} else if (properties.green and zoom >= 10) {
opacity = 1;
} else if (zoom >= 15) {
opacity = 1;
} else {
opacity = 0;
}
I tried writing an opacity expression like this:
'circle-opacity': [
'case',
['to-boolean', ['feature-state', 'hidden']], 0,
['to-boolean', ['get', 'red']], 1,
['to-boolean', ['get', 'green']], ['case', ['>=', ['zoom'], 10], 1, 0], // I could also write a permutation of this using an ['all'] rule
['>=', ['zoom'], 15], 1,
0,
],
That gets rejected with this message: "Error: layers.places.paint.circle-opacity: "zoom" expression may only be used as input to a top-level "step" or "interpolate" expression."
So then I tried something like this:
'circle-opacity': [
'step',
['zoom'],
0,
[
'case',
['to-boolean', ['get', 'red'], false], 1,
['to-boolean', ['get', 'green'], false], 10,
15
],
1
]
That gets rejected with this message: "Error: layers.places.paint.circle-opacity[3]: Input/output pairs for "step" expressions must be defined using literal numeric values (not computed expressions) for the input values."
As you can see, I hadn't even gotten around to adding the feature-state checks yet.
Here's a JSFiddle to play with: https://jsfiddle.net/rognstad/px4c8tbe/17/
I think I can make it work by breaking each color out into its own layer with a minZoom property and then only using an opacity expression for feature-state, but that feels really clunky and will result in worse performance.
It seems like there has to be a better way. Do you have any better suggestions for how to achieve this? Thanks!
You're really close. You're getting this error:
Error: layers.places.paint.circle-opacity[3]: Input/output pairs for "step" expressions must be defined using literal numeric values (not computed expressions) for the input values.
Because you've got the order of arguments in your ['step'] expression slightly wrong. It needs to go:
['step', ['zoom'], <value at zoom 0>, <X>, <value at zoom X>, <Y>, <value at zoom Y>
So you don't want that first 0 there.
If we refactor your pseudo code a bit you should be able to get there:
step
zoom
// zoom 0 to 10
if (!feature-state.hidden && properties.red) {
1
} else {
0
}
10 // zoom level 10+
if (!feature-state.hidden && (properties.red || properties.green) {
1
} else {
0
}
15 // zoom level 15+
if (!feature-state.hidden) {
1
} else {
0
}
The expression code will look something like this. You'll have to add appropriate type conversions probably. (I haven't tested it).
['step', ['zoom'],
['case', ['all', ['!', ['feature-state', 'hidden']], ['get', 'red'], 1, 0],
10,
['case', ['all', ['!', ['feature-state', 'hidden']], ['any', ['get', 'red'], ['get', 'green']], 1, 0],
15,
['case', [['!', ['feature-state', 'hidden']], 1, 0]],