No signin id passed during redirect from Custom Middleware (External Identity Provider) - identityserver3

Question

We want an external identity provider within our identityserver where user is redirected to login to this external identity provider (Idp) and complete the authentication there. The external Idp redirects the response back to our identityserver where we want to form the user claims from the response and redirect back to the calling client.
We followed the answer to this stackoverflow question. We implemented our own owin middleware following the the Anders Abel Tutorial and owin middleware's for Google Auth as external Idp.
We also followed the exact implementation of CustomHrd Sample. Everything in the middleware works perfectly, the user gets redirected to the external Idp with the state, we get the response from there back (with the state), we form the claims and on completion we finish the owin middleware pipeline and redirect to /core/callback, but the Identityserver gives the following error:
The URL call flow is available below:
/core/connect/authorize?params
/core/login?signin=3a84a4f3dfe3218b06f3aa2185415e1e
/core/ta (IDP Selector page)
/core/login?signin=3a84a4f3dfe3218b06f3aa2185415e1e (Resume Uri after Idp Selection)
/core/ta/tacall?state=
Redirect to External URL with state
Callback from External Identity provider with state: /core/ta/tacall?state=
Redirect from middleware to redirectUri after forming claims - /core/callback
Error occurs on /core/callback saying "There is an error determining which application you are signing into. Return to the application and try again."
Can you please help us figure out why would this occur?
Note: I also looked into the cookies, we are passing the cookie to /core/callback which contains the SignInMessage.