How to prevent table replication starvation in AWS Data Migration Service (DMS) - postgresql

I'm using AWS DMS to migrate one postgres db to another. I am performing a full load + ongoing replication task. The full load succeeded without issues for all of the schemas and their tables (around 120 tables). Some tables are being inserted into/updated (with acceptable latency) on the target db, but some are not. It's been 2 days of ongoing replication and one of my tables (relatively small ~800 records) has not been updated at all and has a record lag of around 3 records with replication latency of 2 days, while other tables (usually bigger) seem to have minimal record lag and replication latency closer to a couple hours.
Note that I am using a dms.r5.4xlarge replicator instance.
Is there a way to reduce record lag and replication latency for my tables/or perform replication in some sort of round robin fashion?
I'm using the following settings:
{
"Logging": {
"EnableLogging": true,
"LogComponents": [
{
"Severity": "LOGGER_SEVERITY_DEFAULT",
"Id": "TRANSFORMATION"
},
...
],
"CloudWatchLogGroup": "dms-tasks-temporary-analytics-replicator",
"CloudWatchLogStream": "dms-task-temporary-analytics-dms-v2"
},
"StreamBufferSettings": {
"StreamBufferCount": 3,
"CtrlStreamBufferSizeInMB": 5,
"StreamBufferSizeInMB": 8
},
"ErrorBehavior": {
"FailOnNoTablesCaptured": true,
"ApplyErrorUpdatePolicy": "LOG_ERROR",
"FailOnTransactionConsistencyBreached": false,
"RecoverableErrorThrottlingMax": 1800,
"DataErrorEscalationPolicy": "SUSPEND_TABLE",
"ApplyErrorEscalationCount": 0,
"RecoverableErrorStopRetryAfterThrottlingMax": true,
"RecoverableErrorThrottling": true,
"ApplyErrorFailOnTruncationDdl": false,
"DataTruncationErrorPolicy": "LOG_ERROR",
"ApplyErrorInsertPolicy": "LOG_ERROR",
"EventErrorPolicy": "IGNORE",
"ApplyErrorEscalationPolicy": "LOG_ERROR",
"RecoverableErrorCount": -1,
"DataErrorEscalationCount": 0,
"TableErrorEscalationPolicy": "STOP_TASK",
"RecoverableErrorInterval": 5,
"ApplyErrorDeletePolicy": "IGNORE_RECORD",
"TableErrorEscalationCount": 0,
"FullLoadIgnoreConflicts": true,
"DataErrorPolicy": "LOG_ERROR",
"TableErrorPolicy": "SUSPEND_TABLE"
},
"TTSettings": {
"TTS3Settings": null,
"TTRecordSettings": null,
"EnableTT": false
},
"FullLoadSettings": {
"CommitRate": 10000,
"StopTaskCachedChangesApplied": false,
"StopTaskCachedChangesNotApplied": false,
"MaxFullLoadSubTasks": 8,
"TransactionConsistencyTimeout": 600,
"CreatePkAfterFullLoad": false,
"TargetTablePrepMode": "DROP_AND_CREATE"
},
"TargetMetadata": {
"ParallelApplyBufferSize": 0,
"ParallelApplyQueuesPerThread": 0,
"ParallelApplyThreads": 0,
"TargetSchema": "",
"InlineLobMaxSize": 0,
"ParallelLoadQueuesPerThread": 0,
"SupportLobs": true,
"LobChunkSize": 0,
"TaskRecoveryTableEnabled": false,
"ParallelLoadThreads": 0,
"LobMaxSize": 102400,
"BatchApplyEnabled": false,
"FullLobMode": false,
"LimitedSizeLobMode": true,
"LoadMaxFileSize": 0,
"ParallelLoadBufferSize": 0
},
"BeforeImageSettings": null,
"ControlTablesSettings": {
"historyTimeslotInMinutes": 5,
"HistoryTimeslotInMinutes": 5,
"StatusTableEnabled": false,
"SuspendedTablesTableEnabled": false,
"HistoryTableEnabled": false,
"ControlSchema": "",
"FullLoadExceptionTableEnabled": false
},
"LoopbackPreventionSettings": null,
"CharacterSetSettings": null,
"FailTaskWhenCleanTaskResourceFailed": false,
"ChangeProcessingTuning": {
"StatementCacheSize": 50,
"CommitTimeout": 1,
"BatchApplyPreserveTransaction": true,
"BatchApplyTimeoutMin": 1,
"BatchSplitSize": 0,
"BatchApplyTimeoutMax": 30,
"MinTransactionSize": 1000,
"MemoryKeepTime": 60,
"BatchApplyMemoryLimit": 500,
"MemoryLimitTotal": 1024
},
"ChangeProcessingDdlHandlingPolicy": {
"HandleSourceTableDropped": true,
"HandleSourceTableTruncated": true,
"HandleSourceTableAltered": true
},
"PostProcessingRules": null
}

Related

How to make a single ring chart with Apache eCharts?

I'm trying to make a chart in Apache eCharts that is a single ring with a large number in the middle. I want to use it to display a website score.
This is the closest image example I can find online of what I am trying to achieve:
And I have found these polar bar charts in the Apache eCharts docs which look great:
But I don't know how to only have one large ring with a number displayed in the middle.
Do you know how I can achieve that?
Here is my current code, taken from the docs:
option = {
angleAxis: {
show: false,
max: 10
},
radiusAxis: {
show: false,
type: 'category',
data: ['AAA', 'BBB', 'CCC', 'DDD']
},
polar: {},
series: [
{
type: 'bar',
data: [3, 4, 5, 6],
colorBy: 'data',
roundCap: true,
label: {
show: true,
position: 'start',
formatter: '{b}'
},
coordinateSystem: 'polar'
}
]
};
You probably want a variant of the Gauge chart.
here is a series that would create something very close to your first image example.
series: [
{
type: 'gauge',
startAngle: 90,
endAngle: 270,
min: 0,
max: 100,
progress: {
show: true,
width: 18
},
pointer: {
show: false
},
axisLine: {
show: false
},
axisTick: {
show: false
},
splitLine: {
show: false
},
axisLabel: {
show: false
},
title: {
show: false
},
itemStyle: {
color: 'blue'
},
detail: {
formatter: '{value}',
color: 'auto',
offsetCenter: [0, '-0%'],
valueAnimation: true,
},
}
]
Here is a link to an example that is similar to your second image

How to find total number of question, total correct answer and total incorrect?

Table-Red
"user_question_id": 1
"option_one_correct": true,
"user_option_one_correct": true,
"option_two_correct": true,
"user_option_two_correct": true,
"option_three_correct": true,
"user_option_three_correct": true,
"option_four_correct": true,
"user_option_four_correct": true,
"option_five_correct": true,
"user_option_five_correct": true,
"one_incorrect": 0,
"two_correct": 1,
"two_incorrect": 0,
"three_correct": 1,
"three_incorrect": 0,
"four_correct": 1,
"four_incorrect": 0,
"five_correct": 1,
"five_incorrect": 0,
"user_question_id": 2
"option_one_correct": true,
"user_option_one_correct": true,
"option_two_correct": true,
"user_option_two_correct": true,
"option_three_correct": false,
"user_option_three_correct": true,
"option_four_correct": false,
"user_option_four_correct": true,
"option_five_correct": true,
"user_option_five_correct": true,
"one_correct": 1,
"one_incorrect": 0,
"two_correct": 1,
"two_incorrect": 0,
"three_correct": 0,
"three_incorrect": 1,
"four_correct": 0,
"four_incorrect": 1,
"five_correct": 1,
"five_incorrect": 0,
Expected Result is total result total question,total correct answer,total incorrect answer
How to find total question and total correct and incorrect answer by query. i tried many ways but didn't find any solution
Assuming that table has name table1 and field names are case sensitive (because of apostrophes)
select
count(*)*5 questions, sum("one_correct"+"two_correct"+"three_correct"+"four_correct"+"five_correct") correct_answers,
sum("one_incorrect"+"two_incorrect"+"three_incorrect"+"four_incorrect"+"five_incorrect") incorrect_answers from table1

How to match two boolean tables and count match values and unmatch values

table_1
"mcqtest_id": 1,
"option_one_correct": true,
"option_two_correct": true,
"option_three_correct": true,
"option_four_correct": true,
"option_five_correct": false,
"option_six_correct": false,
"option_seven_correct": false,
"option_eight_correct": false,
"question_id": 1
table_2
"mcqtest_id": 1,
"option_one_correct": false,
"option_two_correct": true,
"option_three_correct": false,
"option_four_correct": true,
"option_five_correct": false,
"option_six_correct": false,
"option_seven_correct": false,
"option_eight_correct": false,
"question_id": 2
I want to find the match between this two boolean tables and if match count match values and else count unmatch values with postgresql query.
help me to understand and write this query

Why are the panels in my Grafana dashboard overlapping?

I am attempting to create a Grafana Scripted Dashboard consisting of panels that indicate whether a server is UP or not. The reason for this is that we constantly have a list of agents being added and making a new panel every time is becoming repetitive.
Currently, I have this as my script (Note my JavaScript/JQuery skills are abysmal..):
'use strict';
// accessible variables in this scope
var window, document, ARGS, $, jQuery, moment, kbn;
return function(callback) {
// Setup some variables
var dashboard;
// Initialize a skeleton with nothing but a rows array and service object
dashboard = {
annotations: {
list: [
{
builtIn: 1,
datasource: "-- Grafana --",
enable: true,
hide: false,
iconColor: "rgba(0, 211, 255, 1)",
name: "Annotations & Alerts",
type: "dashboard"
}
]
},
editable: true,
gnetId: null,
graphTooltip: 0,
id: 15,
links: [],
panels : [],
refresh: false,
schemaVersion: 16,
style: "dark",
tags: [],
templating: {
list: []
},
time: {
from: "now-5m",
to: "now"
},
timepicker: {
refresh_intervals: [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
time_options: [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
timezone: "",
title: "My Agents",
//uid: "000000019",
version: 2
};
// Set a title
dashboard.title = 'Scripted dash';
// Set default time
// time can be overridden in the url using from/to parameters, but this is
// handled automatically in grafana core during dashboard initialization
dashboard.time = {
from: "now-6h",
to: "now"
};
var rows = 1;
var seriesName = 'argName';
if(!_.isUndefined(ARGS.rows)) {
rows = parseInt(ARGS.rows, 10);
}
if(!_.isUndefined(ARGS.name)) {
seriesName = ARGS.name;
}
$.ajax({
method: 'GET',
url: 'http://my_server/api/my_rest_call_to_get_agents'
})
.done(function(result) {
//console.log(JSON.stringify(result));
var x = 0;
var y = 0;
var id = 0;
$.each(result.data.result, function(index, value) {
//console.log(value.metric.instance);
var panel = {
clusterName: "My Agent",
colorMode: "Panel",
colors: {
crit: "rgba(245, 54, 54, 0.9)",
disable: "rgba(128, 128, 128, 0.9)",
ok: "rgba(50, 128, 45, 0.9)",
warn: "rgba(237, 129, 40, 0.9)"
},
cornerRadius: 0,
//datasource: "Prometheus",
displayName: "Agent_Name",
flipCard: true,
flipTime: "2",
fontFormat: "Bold",
gridPos: {
h: 3,
w: 2,
x: x,
y: y
},
id: id,
isGrayOnNoData: true,
isHideAlertsOnDisable: false,
isIgnoreOKColors: false,
links: [],
targets: [
{
aggregation: "Last",
alias: value.metric.instance,
crit: 0,
decimals: 2,
displayAliasType: "Warning / Critical",
displayType: "Regular",
displayValueWithAlias: "Never",
expr: 'expression_to_query_service',
format: "time_series",
instant: true,
intervalFactor: 1,
legendFormat: value.metric.instance,
refId: "A",
units: "short",
valueHandler: "Number Threshold"
}
],
title: value.metric.instance,
transparent: true,
type: "vonage-status-panel"
}
dashboard.panels.push(panel);
x += 2;
});
callback(dashboard);
});
}
The strangest part about all of the panels being overlapping is that every panel has different coordinates if I look at each panel's JSON.
Any help with this would be appreciated!

How to schedule timing for a JOB in Postgres using pgagent

I have created a job using pgagent which I have scheduled on every 5 mins for that below is the code:
DO $$
DECLARE
jid integer;
scid integer;
BEGIN
-- Creating a new job
INSERT INTO pgagent.pga_job(
jobjclid, jobname, jobdesc, jobhostagent, jobenabled
) VALUES (
1::integer, 'refresh_mobile'::text, ''::text, ''::text, true
) RETURNING jobid INTO jid;
-- Steps
-- Inserting a step (jobid: NULL)
INSERT INTO pgagent.pga_jobstep (
jstjobid, jstname, jstenabled, jstkind,
jstconnstr, jstdbname, jstonerror,
jstcode, jstdesc
) VALUES (
jid, 'refresh_mobile_mv_data'::text, true, 's'::character(1),
''::text, 'ICSPGD'::name, 'f'::character(1),
'SELECT refresh_materialized_views(''mobile'');'::text, 'Please check. Issue occired while refreshing Mobile Data'::text
) ;
-- Schedules
-- Inserting a schedule
INSERT INTO pgagent.pga_schedule(
jscjobid, jscname, jscdesc, jscenabled,
jscstart, jscend, jscminutes, jschours, jscweekdays, jscmonthdays, jscmonths
) VALUES (
jid, 'minutly'::text, ''::text, true,
'2017-09-04 03:36:20-07'::timestamp with time zone, '2018-12-31 02:36:20-08'::timestamp with time zone,
-- Minutes
ARRAY[false, false, false, false, false, true, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false]::boolean[],
-- Hours
ARRAY[true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true]::boolean[],
-- Week days
ARRAY[true, true, true, true, true, true, true]::boolean[],
-- Month days
ARRAY[true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true]::boolean[],
-- Months
ARRAY[true, true, true, true, true, true, true, true, true, true, true, true]::boolean[]
) RETURNING jscid INTO scid;
END
$$;
But when I am checking its statistics(in pgadmin) or via query "select * from pgagent.pga_job;" it shows that job are running at an interval of 1 hour.
For example: If the last job run time is 2017-10-05 03:05:00.703287-07 , then the next run time it shows : 2017-10-05 04:05:00-07.
Kindly help with the timing parameter. I am suppose to run this job on an interval of every 5 mins on a daily basis throughout.
Regards,
Have you tried something like this?
INSERT INTO pgagent.pga_schedule
(jscid, jscjobid, jscname, jscdesc, jscminutes, jschours, jscweekdays, jscmonthdays, jscmonths, jscenabled, jscstart, jscend)
VALUES(<SchId>, 29, 'minutly', '', '{t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f,t,f,f,f,f}', '{f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f}', '{f,f,f,f,f,f,f}', '{f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f}', '{f,f,f,f,f,f,f,f,f,f,f,f}', true, '2017-10-07 00:00:00', NULL);
Basically, what this insert is doing is marking the minutes 0,5,10,15,20,25,30,35,40,45,50 and 55 for every hour.
I don't know why you're creating it using insert statements, since the GUI offers a pretty neat way to schedule jobs. The same configuration can be done using the GUI like this:
I hope it helps :-)