GORM getting data out of a JoinTable - postgresql

Question

I have a custom JoinTable and I'm trying to get data out of it. I've tried every possible way I can imagine to get it but to no advance.
This is what I'm trying to do
type Race struct {
ID uint `gorm:"primaryKey" json:"id"`
ChampionshipId uint `gorm:"not null" json:"championshipId"`
Name string `gorm:"size:255;not null" json:"name"`
Date time.Time `gorm:"size:255;not null" json:"date"`
Finished bool `gorm:"not null" json:"finished"`
Drivers []Driver `gorm:"many2many:race_drivers;" json:"drivers"`
}
My model Race has Drivers that is an array of drivers. And this is what Driver looks like
type Driver struct {
ID uint `gorm:"primaryKey" json:"id"`
Name string `gorm:"size:255;not null" json:"name"`
}
But because I need some extra data I made a custom JoinTable to add that
type RaceDriver struct {
RaceID uint `gorm:"primaryKey" json:"race_id"`
DriverID uint `gorm:"primaryKey" json:"driver_id"`
Position int `json:"position"`
Laps int `json:"points"`
}
func (driver *RaceDriver) BeforeSave(db *gorm.DB) error {
driver.Position = 0
driver.Laps = 0
return nil
}
But when calling DB.Preload("Drivers").First(&race, id) it just returns the race the ID's and Name of the drivers, not the data from the JoinTable
Also in my database setup I have the SetupJoinTable with the correct information before Automigrate
DB.SetupJoinTable(&Race{}, "Drivers", &RaceDriver{})
I've tried something like this
DB.Preload("Drivers", func(db *gorm.DB) *gorm.DB {
return db.Joins("INNER JOIN race_drivers ON race_drivers.drivers_id = drivers.id")
}).Find(&r)
which gives me this error
cannot convert 0xabec80 to Int4 SELECT * FROM "drivers" WHERE "drivers"."id" IN (1,2,3,4,5,6) AND "drivers"."id" = '0xabec80'
I switched all of my primary keys to uint64 | uint | int64 and int, they don't work.
I also tried with selects and joins without the preloading and with the preloading but that doesn't work because I try to get an array of objects.
It's kinda insane how long I've been stuck on this. Any tips would be helpfull!