Logstash translate plugin problem parsing csv - plugins

Question

I'm trying to parse all the columns of a CSV file (except the first one, obviously).
The plugin only get the second column as result of the filter.
All the other column are ignored.
It should be possible, according to this sentence getting from the documentation :
When using a CSV dictionary, multiple values in the translation must
be extracted with another filter e.g. Dissect or KV. Note that the
fallback is a string so on no match the fallback setting needs to
formatted so that a filter can extract the multiple values to the
correct fields.
Here is my logstash code :
translate {
field => "idBatch"
dictionary_path => "D:\idBatch-description.csv"
refresh_interval => 500
destination => "donneesDictionnaireExterne"
# Données par défaut en l'absence de correspondance
fallback => "Aucune correspondance trouvée,10000"
add_tag => [ "import_CSV_ok"]
}
# Mapper des données du dictionnaire externe
dissect {
mapping => {
"donneesDictionnaireExterne" => "%{descriptionBatch},%{maxDuration}"
# EXEMPLE pour GAR01B0 : Batch d'injection Archive;86408
}
}
Here is a sample of my CSV file :
"GDA08A0_SupPdc","Batch de Suppression de PDC","9999"
"GDI01A0_Parsing","Moteur de parsing des etats internes","9999"
Does anyone know why it doesn't work ?

Answer 1

The translate filter will ignore everything after the second column, you will need to change the format of your dictionary.
Your dictionary needs to be something like this.
"GDA08A0_SupPdc","Batch de Suppression de PDC;9999"
"GDI01A0_Parsing","Moteur de parsing des etats internes;9999"
Then your dissect filter will be like this one:
dissect {
mapping => {
"donneesDictionnaireExterne" => "%{descriptionBatch};%{maxDuration}"
}
}
You can also use a mutate filter to remove the donneesDictionnaireExterne field.
mutate {
remove_field => ["donneesDictionnaireExterne"]
}
Finally the output for your example is:
{
"descriptionBatch" => "Batch de Suppression de PDC",
"maxDuration" => "9999",
"#version" => "1",
"#timestamp" => 2019-04-02T02:10:45.107Z,
"idBatch" => "GDA08A0_SupPdc",
"message" => "{ \"idBatch\":\"GDA08A0_SupPdc\"}",
"tags" => [
[0] "import_CSV_ok"
],
"host" => "hostname"
}
{
"descriptionBatch" => "Moteur de parsing des etats internes",
"maxDuration" => "9999",
"#version" => "1",
"#timestamp" => 2019-04-02T02:10:45.109Z,
"idBatch" => "GDI01A0_Parsing",
"message" => "{ \"idBatch\":\"GDI01A0_Parsing\"}",
"tags" => [
[0] "import_CSV_ok"
],
"host" => "hostname"
}