How to select columns in PySpark which do not contain strings - pyspark

Question

I had the problem on how to remove the columns with strings in Pyspark, keeping only, numerical ones and timestamps.
This is how I did it.
I had this:
full_log.printSchema()
root
|-- ProgramClassID: integer (nullable = true)
|-- CategoryID: integer (nullable = true)
|-- LogServiceID: integer (nullable = true)
|-- LogDate: timestamp (nullable = true)
|-- AudienceTargetAgeID: integer (nullable = true)
|-- AudienceTargetEthnicID: integer (nullable = true)
|-- ClosedCaptionID: integer (nullable = true)
|-- CountryOfOriginID: integer (nullable = true)
|-- DubDramaCreditID: integer (nullable = true)
|-- EthnicProgramID: integer (nullable = true)
|-- ProductionSourceID: integer (nullable = true)
|-- FilmClassificationID: integer (nullable = true)
|-- ExhibitionID: integer (nullable = true)
|-- Duration: string (nullable = true)
|-- EndTime: string (nullable = true)
|-- LogEntryDate: timestamp (nullable = true)
|-- ProductionNO: string (nullable = true)
|-- ProgramTitle: string (nullable = true)
|-- StartTime: string (nullable = true)
This will get the list of column names to filter
no_string_columns = [types[0] for types in full_log.dtypes if types[1] != 'string']
Perform the final selection
full_log_no_strings = full_log.select([*no_string_columns])

Answer 1

You can also use the schema object of the dataframe:
from pyspark.sql.types import *
string_columns = [column.name for column in full_log.schema if column.dataType != StringType()]

Answer 2

You can use below logic for this use case.
Step:1 Finding the columns which are string as datatype .
Step:2 remove those columns list
Step:3 apply your logic
Code snippet :
from pyspark.sql.types import StructType,StructField, StringType, IntegerType
data2 = [("James","","Smith","36636","M",3000),
("Michael","Rose","","40288","M",4000),
("Robert","","Williams","42114","M",4000),
("Maria","Anne","Jones","39192","F",4000),
("Jen","Mary","Brown","","F",-1)
]
schema = StructType([ \
StructField("firstname",StringType(),True), \
StructField("middlename",StringType(),True), \
StructField("lastname",StringType(),True), \
StructField("id", StringType(), True), \
StructField("gender", StringType(), True), \
StructField("salary", IntegerType(), True) \
])
df = spark.createDataFrame(data=data2,schema=schema)
df.printSchema()
df.show(truncate=False)
columnList = [item[0] for item in df.dtypes if item[1].startswith('string')]
print(columnList)
df1 = df.drop(*columnList)
display(df1)
Screen shot: