I am looking for a way to automate mouse clicks based on display text on screen.
The main function is the following:
- When bot detects "Hello!" on screen of Facebook Messenger, it clicks to open conversation. Then it reads the message (messages are always the same) and depending on the messages received it clicks on Saved Replies and sends the correct message.
For example:
Bot detects "hello!" it clicks and opens the conversation. Then bot reads "Amazing Offer 20% off" so it goes to Saved Replies and sends the correct message according to that offer.
*Saved replies is a canned message function that facebook for business messages offers as a function. So basically all I need is a bot that can automate clicks and detect text on screen. It will be run from a PC using Windows.
Any suggestions?
There are three options.
1) Use the Facebook API.
Facebook have spent time and effort to build a system in which developers can interact with their products. Here is a link: https://messengerplatform.fb.com
Options that might be against their policies and will get your account blocked (or worse):
2) Chrome Extension.
Through a Chrome extension, you have the ability to scan the DOM every few seconds and interact with the elements.
This needs to be maintained as Facebook might change element positions, ids or classes. https://developer.chrome.com/extensions/getstarted
3) PhantomJs.
This is a headless browser which you can run and interact with the DOM elements. This also needs to be maintained as the DOM may change at any time.
http://phantomjs.org/page-automation.html
Related
I work for a tv/radio broadcasting company, and we stream live content through various devices through a web-based API, and we also stream through internet radio (such as iHeartRadio, Tunein, etc.). The API can also return things like show titles and descriptions.
I've been tasked with creating a Google Action that can be used to retrieve information from the API such as what's playing, what's coming up next, what shows are available, etc. It would be fantastic if Google Actions supported live-streamed content, but I believe they do not.
Since we DO stream through internet radio, I would like to create an intent that allows the user to be redirected from my action to the internet radio stream for our station. How would I go about doing that? I could simply tell the user to start a new conversation (e.g., "Say, OK Google, play 'My-Awesome-Radio'"), but it would be more user-friendly not to have to start a new conversation.
I developed a actions on google app which sends a rich response. Everything works fine in the Actions on Google simulator. Now I want to test it on my Google Home Mini but my rich responses are not told by the mini. I would like to ask if it is possible to send my rich response to the google home app? The home mini says something like "Ok, I found these hotels, look at the home app" and there are the rich responses?
You can't send users to the Home app, but you can direct them to the Assistant available through their phone. The process is roughly:
At some point in the conversation (decide what is best for you, but when you have results that require display is usually good, or if the user says something like "Show me" or "Send this to my phone"), determine if they are on a device with a screen or not. You do this by using the app.getSurfaceCapabilities() method or by looking at the JSON in the originalRequest.data.surface.capabilities property. If they're using a screen, you're all set. But if not...
Make sure they have a screen they can use. You'll do this by checking out the results from app.getAvailableSurfaces() or looking at the JSON in the (not fully documented) originalRequest.data.availableSurfaces array. If they don't have a screen, you'll need to figure out your best course of action. But if they do have a screen surface (such as their phone, currently) available...
You can request to transfer them to the new surface using the app.askForNewSurface() method, passing a message explaining why you want to do the switch, a message that will appear as a notification on the device, and what surface you need (the screen).
If the user approves, they'll get the notification on their mobile device (using that device's normal notification system). When they select the notification, the Assistant will open up and will send your Action an Event called actions_intent_NEW_SURFACE. You'll need to create an Intent that handles this Event and forwards it to your webhook.
Your webhook should confirm that it is on a useful surface, and then proceed with the conversation and send the results.
You can see more about handling different surfaces at https://developers.google.com/actions/assistant/surface-capabilities
Rich responses can appear on screen-only or audio and screen experiences.
They can contain the following components:
One or two simple responses (chat bubbles)
An optional basic card
Optional suggestion chips
An optional link-out chip
An option interface (list or carousel)
So you need to make sure that the text response is containing all the details for cases like voice only (e.g. Google home/mini/max).
However, if your users are using the assistant from a device with a screen, you can offer them a better experience with the rich responses (e.g. suggestion chips, links etc').
so this is something I've been trying to think through for about 16 hours. I am coding with PHP / CuRl / etc - the bot works and everything is fine. My current issue is figuring out how to disable the bot and allow a human to begin chatting with the customer/sender.
Has anyone successfully, created a route for this ? I mean it's pretty hard from what I see, you'd have to disable etc etc. A lot of effort for my clients.
Thanks for any input.
Facebook has rolled out a "Handover Protocol" which is supposed to facilitate a combined human/bot Messenger implementation.
https://developers.facebook.com/docs/messenger-platform/handover-protocol
It is a little unclear what actually occurs in step 5:
Pass thread control: At some point in the conversation, a user may choose to do something like interact with a live agent. To handle this, pass thread control from the Primary Receiver to the Secondary Receiver. The Secondary Receiver will receive a messaging_handovers webhook event to notify it that is now controls the conversation.
This doesn't actually disable the bot (as the OP requested), and isn't in the control of the Page owner but rather of the user. It seems FB envisions the user typing something like 'I would like to chat with a human' triggering the bot to pass control...but it would be nice to let the page owner simply put the app in standby and handle the messages herself.
Once you recognize someone wants to speak to a human, set a flag that disables all actions of your bot to on.
Then, have your bot message you, or whoever will respond, that a user ID needs responding to. Have your bot continue to send all messages received from them back to you until you enable the bot again.
Create some sort of way for your bot to interact with you that allows you to send a message to a specific user, and a way to once again enable the bot interaction with the user.
Probably something like "sendMessage104012301230'Hi, sorry you couldn't find [etc]', and enableUser104012301230
There may be a better way, but those are some thoughts on how I'd do it
If you enable messages echo, whenever a human respond using the page, a echo post is sent, and inside entry->messaging->message there's no app_id.
You can use that information to disable bot replies for a certain period, or disable indefinitely until you enabled is with some admin command (that's how I'm doing)
I thought a solution could be to label the message as "unsolved". Another solution could be to have the bot mark the conversation as unread. Does anyone know if it is possibile to add a label to a conversation or mark as unread through API?
I have a facebook app that needs to allow a user to send a message to multiple friends (potentially all their friends) at once. This isn't any kind of spam, and I don't need the app to send the message incognito (behind the scenes), I just need to open a dialog with specific friends pre-populated (that the user has selected within my app in a prior step) and then send them a custom message. Is this possible? I see the api for sending a message to a single friend, and I see the API for inviting users to my app (but that's limited to some very small number of invitations per day)... what I need is a dialog that lets me send messages to as many of the user's friends as they want, but for me to control *which friends are selected... I don't want to give them control to add/remove from the friends list.
Is this possible?
No, the Send Dialog allows prefilling only a single friend, but would otherwise be the best option here if you need a custom message displayed to the recipient. You could get the user to send to several friends in a loop by prefilling this - or fire the dialog without prefilling and let the user chose who to send to - your app won't receive a callback with the recipient IDs but you could put a referrer param in the URL sent?
Failing that, the Requests dialog / 'invites' are the only thing you can prefill with multiple recipients without the expectation that your app will be shut down for spam shortly thereafter - there's no limit per day on the number of requests sent, but you don't get to set the message shown to the recipient.
In case this is helpful to anyone else, I just found this... it's a relatively new API (still in beta) that allows for (just about) exactly what I was asking for. I've tested it and it appears to work...
https://developers.facebook.com/blog/post/2012/08/31/reach-users-1-1-with-the-notifications-api/
When users open a Send dialog (e.g. http://www.facebook.com/dialog/send?app_id=123050457758183&name=People%20Argue%20Just%20to%20Win&link=http://www.nytimes.com/2011/06/15/arts/people-argue-just-to-win-scholars-assert.html&redirect_uri=http://www.example.com/response) they can add or remove people from the To: line.
Is there a way for the application to track who was in the To: line if and when the user actually pushes Send?
No.
At least not currently...
If you look at the official documentation: https://developers.facebook.com/docs/reference/dialogs/send/ then you can see that this type of dialog does not expect a callback function as a parameter, and since there's no callback function then you have no way of knowing who and/or how many recipients were, and even if the dialog is closed/canceled/sent.
What you can do, though seems very complicated and well, a headache, is to use the outbox connection of the User object in the graph api (http://developers.facebook.com/docs/reference/api/user/).
I've never used that before, but I guess that you can (if you ask for the read_mailbox permission) check the user messages before and after the dialog, and see what changed.