Parse record (PCF) from Kafka using Kafka Kusto Sink - apache-kafka

Question

I've set-up my environment using docker based on this guide.
On kafka-console-producer I will send this line:
Hazriq|27|Undegrad|UNITEN
I want this data to be ingested to Kusto like this:
+--------+-----+----------------+------------+
| Name | Age | EducationLevel | University |
+--------+-----+----------------+------------+
| Hazriq | 27 | Undegrad | UNITEN |
+--------+-----+----------------+------------+
Can this be handled by Kusto using the mapping (which I'm still trying to understand) or this should be catered by Kafka?
Tried #daniel suggestion:
.create table ParsedTable (name: string, age: int, educationLevel: string, univ:string)
.create table ParsedTable ingestion csv mapping 'ParsedTableMapping' '[{ "Name" : "name", "Ordinal" : 0},{ "Name" : "age", "Ordinal" : 1 },{ "Name" : "educationLevel", "Ordinal" : 2},{ "Name" : "univ", "Ordinal" : 3}]'
kusto.tables.topics_mapping=[{'topic': 'kafkatopiclugiaparser','db': 'kusto-test', 'table': 'ParsedTable','format': 'psv', 'mapping':'ParsedTableMapping'}]
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
but getting this instead:
+----------------------------+-----+----------------+------+
| Name | Age | EducationLevel | Univ |
+----------------------------+-----+----------------+------+
| Hazriq|27|Undergrad|UNITEN | | | |
+----------------------------+-----+----------------+------+

Answer 1

Currently, the connector passes the data as it comes (no manipulation on it on the client side), and any parsing is left to Kusto.
As such, psv format is supported by kusto, and it should be possible by setting the format to psv and providing a mapping reference.
When adding the plugin as described, you should be able to set it up like:
kusto.tables.topics_mapping=[{'topic': 'testing1','db': 'testDB', 'table': 'KafkaTest','format': 'psv', 'mapping':'KafkaMapping'}]
The mapping can be defined in Kusto as described in the Kusto docs defined like so

Answer 2

ingestion of data as you've shown using the psv format is supported (see below) - it's probably just a matter of debugging why your client-side invocation of the underlying commands aren't yielding the expected result. if you could share the full flow and code, including parameters, it may be helpful.
.create table ParsedTable (name: string, age: int, educationLevel: string, univ:string)
.ingest inline into table ParsedTable with(format=psv) <| Hazriq|27|Undegrad|UNITEN
ParsedTable:
| name | age | educationLevel | univ |
|--------|-----|----------------|--------|
| Hazriq | 27 | Undegrad | UNITEN |