Eloquent - empty model relation results when "orderBy desc" - eloquent

Question

I have a query fetching products with some relations, but when adding "orderBy('updated_at', 'desc'), The relation with product images becomes an empty array.
I start with:
$products = \App\Product::where('id', '>', 0)
->with('designer')
->whereIn('id', [$someIds]);
Then I continue adding some relations with "whereHas" queries (if I would remove those, the problem is still exists)
Lastly, if I do:
return $products
->with('gallery.items')
->orderBy('updated_at', 'desc')->paginate($args['count']);
the gallery.items array becomes empty. It works fine if I do 'asc' instead of 'desc'. I also tried the orderByDesc method , but it's the same thing.
If I change the order, doing orderBy before adding "with" relations, it also returns empty array for the gallery images:
return $products->orderBy('updated_at', 'desc')
->with('gallery.items')
->paginate($args['count']);
Edit:
I can make a simpler query to narrow down the problem:
return $products = \App\Product::take(30)
->orderBy('updated_at', 'desc')
->with('gallery.items')
->get();
^^ This also gives empty gallery items. If I remove ->orderBy('updated_at', 'desc'), the gallery items appear as they should. Also 'asc' works (that's the default sorting).
update:
here is the above ^^ query, logged with \DB::enableQueryLog():
{
"query": "select * from `products`
order by `updated_at` desc limit 30",
"bindings": [],
"time": 26.46
},
{
"query": "select * from `galleries` where `galleries`.`id`
in (?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
"bindings": [
3970,3971,3973,3974,3975,3976,3977,3978,3979,3980,3981,3982,3983,3984,
3985,3986,3987,3988,3989,3990,3991,3992,3993,3994,3995,3996,3997,3998,
3999,4000],
"time": 3.64
},
{
"query": "select `uploads`.*,
`gallery_items`.`gallery_id` as `pivot_gallery_id`,
`gallery_items`.`upload_id` as `pivot_upload_id`,
`gallery_items`.`order` as `pivot_order`
from `uploads`
inner join `gallery_items` on `uploads`.`id` = `gallery_items`.`upload_id`
where `gallery_items`.`gallery_id`
in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?)",
"bindings": [
3970,3971,3973,3974,3975,3976,3977,3978,3979,3980,3981,3982,
3983,3984,3985,3986,3987,3988,3989,3990,3991,3992,3993,3994,3995,
3996,3997,3998,3999,4000],
"time": 48.92
Edit 2:
The gallery.items relation on Gallery model is like this:
items(){
return $this->belongsToMany('\App\Upload', 'gallery_items','gallery_id','upload_id')
->withPivot('order');
}