Read CSV with more than 22 colums in Apache Flink - scala

Question

What I've been doing until now is read CSV as follows:
val data = env.readCsvFile[ElecNormNew](getClass.getResource("/elecNormNew.arff").getPath)
val dataSet = data map { tuple =>
val list = tuple.productIterator.toList
val numList = list map (_.asInstanceOf[Double])
LabeledVector(numList(8), DenseVector(numList.take(8).toArray))
}
Where ElecNorNew is a case class:
case class ElecNormNew(
var date: Double,
var day: Double,
var period: Double,
var nswprice: Double,
var nswdemand: Double,
var vicprice: Double,
var vicdemand: Double,
var transfer: Double,
var label: Double) extends Serializable {
}
As specified in Flink's docs. But now I am trying to read a CSV with 53 columns. Is there a way to automate this process? Do I need to create a POJO with 53 fields?
Update
After Fabian's answer, I am trying this:
val fieldTypes: Array[TypeInformation[_]] = Array(Types.STRING, Types.INT)
val rowIF = new RowCsvInputFormat(new Path(getClass.getResource("/lungcancer.csv").getPath), fieldTypes)
val csvData: DataSet[Row] = env.createInput[Row](rowIF)
val dataSet2 = csvData.map { tuple =>
???
}
But do not know how to continue, how I am suppose to use RowTypeInfo?

Answer 1

You can use a RowCsvInputFormat as follows:
val fieldTypes: Array[TypeInformation[_]] = Array(Types.STRING, Types.INT)
val rowIF = new RowCsvInputFormat(new Path("file:///myCsv"), fieldTypes)
val csvData: DataSet[Row] = env.createInput[Row](rowIF)
Row stores the data in an Array[Any]. Therefore, Flink cannot automatically infer the field types of a Row. This makes is a bit harder to use than typed tuples or case classes. You need to explicitly provide RowTypeInfo with the correct types. This can be done as implicit values or by functions that extend the ResultTypeQueryable interface.