Text file processing using spark scala - scala

Question

I want to fetch particular portion from text file using spark scala
Is there any inbuilt function?
If we can do it by using regx then how to do that?
from below line data starts
/bin/rm: cannot unlink `/fabos/link_sbin/lscfg_test': Permission denied
Non-VF
======================
Date:
Mon Jul 8 08:48:40 CEST 2019
Time Zone:
Europe/Berlin
Version:
Kernel: 2.6.14.2
Fabric OS: v7.4.2a
Made on: Thu Jun 29 17:22:14 2017
Flash: Tue Oct 10 09:27:26 2017
BootProm: 1.0.11
supportshow groups enabled:
Unknown key pm:0
os enabled
exception enabled
port enabled
fabric enabled
services enabled
security enabled
network enabled
portlog enabled
system enabled
extend disabled
filter disabled
ficon disabled
iswitch enabled
asic_db enabled
fcip disabled (not applicable to this platform)
ag enabled
dce_hsl enabled
Begin start_port_log_cmd group
Mon Jul 8 08:48:44 CEST 2019
portlogdump:
portlogdump :
time task event port cmd args
-------------------------------------------------
Mon Jul 8 03:27:51 2019
03:27:51.199 FCPH seq 13 28 00300000,00000000,00000591,00020182,00000000
03:27:51.199 PORT Rx 11 0 c0fffffd,00fffffd,0ed10335,00000001
03:27:51.200 PORT Tx 13 40 02fffffd,00fffffd,0ed3ffff,14000000
03:27:51.200 PORT Rx 13 0 c0fffffd,00fffffd,0ed329ae,00000001
03:27:59.377 PORT Rx 15 40 02fffffd,00fffffd,0336ffff,14000000
03:27:59.377 PORT Tx 15 0 c0fffffd,00fffffd,03360ed2,00000001
03:27:59.377 FCPH read 15 40 02fffffd,00fffffd,d0000000,00000000,03360ed2
03:27:59.377 FCPH seq 15 28 22380000,03360ed2,0000052b,0000001c,00000000
03:28:00.468 PORT Rx 13 40 02fffffd,00fffffd,29afffff,14000000
03:28:00.468 PORT Tx 13 0 c0fffffd,00fffffd,29af0ed5,00000001
03:28:00.469 FCPH read 13 40 02fffffd,00fffffd,66000000,00000000,29af0ed5
03:28:00.469 FCPH seq 13 28 22380000,29af0ed5,0000052b,0000001c,00000000
03:28:01.197 FCPH write 15 40 00fffffd,00fffffd,00000000,00000000,00000000
03:28:01.197 FCPH seq 15 28 00300000,00000000,00000591,00020182,00000000
03:28:01.197 PORT Tx 15 40 02fffffd,00fffffd,0ed4ffff,14000000
03:28:01.198 PORT Rx 15 0 c0fffffd,00fffffd,0ed40338,00000001
03:28:09.380 PORT Rx 11 40 02fffffd,00fffffd,033affff,14000000
03:28:09.380 PORT Tx 11 0 c0fffffd,00fffffd,033a0ed6,00000001
03:28:09.380 FCPH read 11 40 02fffffd,00fffffd,d5000000,00000000,033a0ed6
03:28:09.380 FCPH seq 11 28 22380000,033a0ed6,0000052b,0000001c,00000000
Expected Output is like below.. I want data from particular line that can be anything(here from time onwords)
+------------+----+-----+----+---+----------------------------------------
|time |task|event|port|cmd|args
+------------+----+-----+----+---+----------------------------------------
|03:27:51.199|PORT|Rx |11 |0 |c0fffffd,00fffffd,0ed10335,00000001 |
|03:27:51.200|PORT|Tx |13 |40 |02fffffd,00fffffd,0ed3ffff,14000000 |
|03:27:51.200|PORT|Rx |13 |0 |c0fffffd,00fffffd,0ed329ae,00000001 |
|03:27:59.377|PORT|Rx |15 |40 |02fffffd,00fffffd,0336ffff,14000000 |
|03:27:59.377|PORT|Tx |15 |0 |c0fffffd,00fffffd,03360ed2,00000001 |
This is sample data I want data from line starts with
time task event port cmd args

Answer 1

scala> val serverData = spark.read.textFile("serverData.txt")
//get column names from raw data
scala> val schemaArr = serverData.filter(x=>x.matches("\\btime.*\\b")).collect.mkString.split("\\s+").toList
//get required data to transform
scala> val reqData = serverData.filter(x=>x.matches("^\\d.*"))
val df = reqData.map(x=>{
val cols = x.split("\\s+")
(cols(0),cols(1),cols(2),cols(3),cols(4),cols(5))
}).toDF(schemaArr:_*)
please try above regex to extract your data and go on. Hope this will help you!