How to distinct key and calculator on Flink + Pulsar - scala

Question

I am a newbie, I am working on Flink and Pulsar.
I have a task about calculator distinct data from pulsar on SlidingProcessingTimeWindows of Flink
the my windowSize: 60s and windows slide : 5s
my data consumer from topic pulsar on every seconds (recieved 2 message/ 1 seconds) :
00:
- a.example.com
- a.example-2.com
---
01:
- b.example.com
- a.example-2.com
---
02:
- c.example.com
- a.example-2.com
---
03:
- a.example.com
- a.example-2.com
---
04:
- b.example.com
- a.example-2.com
How to group key and calculator to receive results:
example.com => 3
example-2.com => 1
I have taken much time for research about that, but I can not resolve it.
And I have a problem, when the first window slide run, my job received all data from this current time to the past, I only received data by the window size.

Answer 1

Flink SQL is a good fit for this.
SELECT window_start, window_end, domain, COUNT(*)
FROM TABLE(
HOP(TABLE Events, DESCRIPTOR(time), INTERVAL '5' SECONDS, INTERVAL '60' SECONDS))
GROUP BY window_start, window_end, domain;
You can execute SQL directly from Scala via TableEnvironment#executeSql or you can use the Table API.