Stop Facebook/Other Apps from crawling my web application via private sharelink - facebook

Question

Edit: The suggested answer does not work as the robots are not just randomly crawling from my index, they are visiting a specific link when it is entered in a FB message.
I've created a basic chat application in Flask on App Engine. It allows the user to invite others by adding their ID or by giving them a private sharelink that auto-adds who ever goes to it (similar to youtube or google drive).
A serious flaw I have found is that if a user posts the link into a facebook message, Facebook will crawl/visit the link and by design of my system add them as a user to the conversation. All of a sudden you'll see 3 random users join the conversation.
My chat system is completely anonymous and designed to be temporary so theres no login or authentication other than a unique key for each user saved in their session.
So Facebook bots visit the link, get assigned an ID and get authenticated into the conversation because they used the users share-link, is there a way I can stop this via either Flask/Python or App Engine? Could I IP ban facebook?
Some code for the sake of code, does this for every new visitor:
def requires_session(f):
#wraps(f)
def decorated(*args, **kwargs):
if 'profile' not in session:
user_ref = fs_database.collection('users').document()
data = {
'id': user_ref.id,
'date': datetime.now(timezone.utc)
}
# add the user to the database
user_ref.set(data)
# save their id to their session
session['profile'] = data.get('id')
# create a hash for later on to create a sharelink
session['share'] = hashlib.sha256(data.get('id').encode('utf-8')).hexdigest()
return f(*args, **kwargs)
return decorated
I could maybe add a check first if Facebook-bot: return False

Answer 1

For your case I would say that you can avoid that either on your side or on Google Cloud Platform side. To be more precise, you can reject some connections in your code or you can set firewall rules to your App Engine instance to reject connections coming from certain IPs. In the public documentation you can find more information about firewall rules when using GAE:
Using flex environment.
Using standard environment.
Code-wise you can check at this github repo which is addresses the issue of blocking certain IPs to your Flask app.
The last possible option is authentication, but as the chat is anonymous I guess that's not the solution you are looking for.

Answer 2

The accepted answer lead me to this answer, I protected the route with a decorator that would get the 'user agent' of the incoming connection and see where it comes from. If it comes from Facebook, redirect it away.
def check_for_robot(f):
#wraps(f)
def decorated(*args, **kwargs):
if 'not_a_robot' not in session:
agent = request.headers.get('User-Agent')
if request.headers.getlist("X-Forwarded-For"):
ip = request.headers.getlist("X-Forwarded-For")[0]
else:
ip = request.remote_addr
# Stop robots from crawling when sharing conversation links
# Could use the IPs too
if 'facebook' in agent or 'Slackbot' in agent:
return 'No Robots Thanks'
# Real people will get to here and continue on
session['not_a_robot'] = True
return f(*args, **kwargs)
return decorated
#app.route('/')
#check_for_robot
def index()
return 'hello human'
This issue also occurs with ANY messaging service that crawls your links to display data in the chat message (WhatsApp, Slack, etc).
This also exposed a vulnerability in these messaging services as they now return the incorrect metadata back to the chat service, but embed the link you provided, ie. Phishing, Clickjacking