I am struggling to find how to let google home do a local network rest call.
I have some ESP8266 laying around with mDNS and rest api in them.
Now with the google home I want it to send a rest call to the device.
I don't want any web hooks / services like IFTTT. I don't want the communication going through these 3rd party services.
It should work like this google home gets input (google service to understand is oke). It retrieves the action (local network, url rest call with body). Google home sends the rest api call to the local device.
No need to have port forwarding / firewall changes.
The Google Home does very very little on-device processing. Sending out local network calls is not one of the things it does. Almost all processing, including IoT controls through the Smart Home API, are done through cloud-based services.
Update
I can't answer "why" it doesn't do this, since I'm not one of the engineers that built it, but I can make a lot of guesses about why.
For starters - it increases the complexity of the software and hardware on the device dramatically. Right now, the device is really little more than a microphone and a speaker, with a little logic to detect the hotword and then stream everything else to the server, and then get a result back and play it. Most of the rest of the code is likely to handle setup and configuration.
If the device has to also be a general purpose IoT hub, then it needs software and hardware for Bluetooth and possibly other signaling systems. It needs to be able to keep track of the state of other devices on the network and manage that in between power cycles of the device (or even handle interruptions in power for the device itself). Some of the implications of that may need to open up the networking on the device to receive messages, not just send them. It has to have more extensive network configuration - to understand what local networking is and not just what the local router is and how to deal with that configuration (and that configuration when it changes). These are all possible, to be sure, but increase the complexity and, in some cases, lower the security of a device.
And that might be reasonable... if there was significant value in doing so. But you've already stipulated in the question that the voice processing could be done in the cloud, so once commands are sent to the cloud and parsed there - why not also do all of the above (device and state tracking, changing, etc) in the cloud? Particularly since most IoT devices maintain cloud servers anyway because people also want to be able to control or monitor their home devices when they aren't on their home LAN. Having a dual set of commands (some for when you're local, and some when you're not) does make sense in some cases - but also dramatically increases the complexity of both the controller and devices, so most just rely on the cloud, again.
So while I understand why some people would like to have a nice little system that can just sent your play local REST server a command now and then, the reality is that to do this for a consumer system isn't that reasonable.
If you really wanted a system that can do this - you can continue in the hobbyist spirit and build something with the Assistant SDK and your favorite IoT platform.
The “local” API for Google Home is a bit limited. Here’s a doc from someone who reverse-engineered the API.
Looks like they expose Bluetooth and Alarms/Timers, and some limited configuration stuff.
https://rithvikvibhu.github.io/GHLocalApi/
Related
I'm researching and trying to building a RC car that can be controlled by the internet. I've started looking into how communication over the web works, but I seem to be going nowhere. My goal for the project is straight forward:
The RC car has an on-board camera and 4g wifi router that enables communication (driving commands, video streaming) over the internet. A Raspberry Pi will serve as the on-board computer.
I will be able to control the car with my PC even across the globe, as long as I'm connected.
I want to preferably do as much by myself as possible without relying too much on other people's code.
So here are my questions:
How does an application communicate over the internet? What is the interface between the application's logic (e.g pressing "w" to go forward), and transmitting/receiving that command over the internet?
How is video data stream handled?
I've looked into WebRTC and WebSockets for communication, but they are aimed at providing real time communication to web browsers and mobile, not something like a raspberry pi, and I'm still in the blind as for exactly what technology should I use, and in general the overview and architecture of real time communication.
All I've achieved so far was an app that sends text messages between devices through a server on my network, with very primitive reading/writing using java Socket.
In short, what does messenger/skype/zoom do in the background when you send a message or video call?
Any guidance would be greatly appreciated.
First things first. You cannot do real-time control over Internet, period. There is absolutely no way to guarantee the delivery latency. Your control commands can arrive with a delay from milliseconds to seconds, or never. No way around it.
Now, you can still do a number of reasonable steps to absorb that unpredictable latency as much as possible and safe-guard your remote robot from the consequences of the unreliable communication.
For example, instead of sending the drive commands directly - as in, acceleration, deceleration, turn angle, etc., you can send a projected trajectory that is calculated from your drive commands locally on a model. Your RC car must be sufficiently smart to do some form of localisation - at the very least, wheel odometry, and with a good enough time sync between the sender and the RC car you'll be able to control the behaviour remotely without nasty consequences of drive commands executed at an unpredictable delay.
You can add a heart-beat to your protocol, to monitor the quality of the communication line, and if hear-beat is delayed or missing, initiate emergency stop.
Also, don't bother with TCP, use UDP only and maintain your own sequence counter to monitor missing packets. Same applies to the telemetry stream, not just command channel.
I live in an area where net access is mobile or nothing. While I can occasionally get access by tethering a mobile to that network, it isn't often connected, and when it isn't connected, no local device will function on its own, no matter which protocol it uses. Why isn't there any kind of server/cloud resiliency built in where devices can communicate in a peer fashion like Apple's Bonjour (Rendezvous? I can't remember)? If I have an Echo device, I should be able to switch it on through an Alexa interface. I'm OK without speech processing which requires interpretation of commands through an AWS or Google or Apple or whatever cloud, but being able to locally control a switch seems as though the interface could be smart enough to route locally. I guess I may have just answered my question. It seems as though routes could be internally stored so as to not to definitely require a server. Can you imagine shipping a colony to Mars and all the IoT devices stop working? If you ask me, they should not require a branch variation or special programming in order to function.
From the experience of having sat down and built a few, there are a some key reasons why viable IoT gadget products for the general market typically end up having to have a cloud-mediated mode, no matter what was envisioned when the design effort originally commenced:
General consumers (at least think) they want the option to control things when outside the home
Often even at home, a mobile phone may be on the mobile network not wifi, meaning that even if the user is physically inside their home, in network terms, they are not.
Firmware updates, dynamic content, etc are easier when they don't have to be relayed through a mobile phone or PC, especially a mobile that might sometimes have to jump networks partway through the process.
Ironically, having once set out to build an IoT product that could work entirely offline, the further the project progressed, the more and more difficulties that approach presented for general users, and the more the cloud path that was added as an option, started to look preferable in terms of how things should work all the time so that it could become the exclusive focus of development efforts.
My conclusion is that it's very hard to build an offline IoT gadget. Not only the developer, but also the users and marketing people need to understand and accept what sorts of difficulties and limitations that can mean.
So where does it happen? In the situations where the "users" are the "developers" - eg. open source. If you look around a bit, you'll find plenty of gadgets either built form scratch, or more commonly reverse engineered so they can run a custom firmware. Want a local RESTful API? Done! Want could relay via MQTT over SSL to your own broker? Done!
When you control the code, you control the mode.
But with products for the general market, most customers want things to work, not a lengthy technical explanation of why the details of their network setup mean they cannot.
I'm looking for a way to managing and controlling IoT devices without the (constant) need for a remote server to be part in it and with a PWA instead of a native application on the managing device. The resulting goal is an IoT device that keeps working when a product eventually goes obsolete (no more server running/allocated) and having an application that has PWA benefits like always up-to-date, easy to replicated and cheaply hosted on something like S3.
As an example, let's say I have one or more IoT devices of different classes like an ESP8266 and a Raspberry Pi for instance. Just as with any new IoT device I want to add it to the network and from there on manage its state from an app. The app in this case is a PWA instead of full blown native app.
1) Traditionally, in the initial setup a native app would scan for wifi networks, connecting automatically to one that has the right name. Using PWA's we are limited to just the "state" of the network. So users would have to manually switch to the network of the newly connected IoT device. This is okay.
2) Next up, the user would need to enter his/her wifi credentials. There might be methods to set this up automatically using a native app, but I don't believe this is the case on a PWA. This, also, is okay.
3) The device then restarts, tries to connect to the network set up in (2). If set up correctly it should get its IP using DHCP. And now things get difficult. I want to 'discover' that network from my PWA. There used to be a way to retrieve the IP from a device using the WebRTC API. but that since has changed to show an obfuscated mdns that resolves to localhost. The ip leak would have allowed for a browser based network scan, but that is no longer possible. I always hoped that Android would default to using a users router as a DNS server, but that is not the case. The result is that simply using an mDNS isn't an option either. I do not want user having to check their router, install an app like fing or do anything else that disrupts the flow from a UX perspective.
Step (3) needs to run every time the app starts so you'd want something reliable. Scanning for devices isn't possible using a PWA, so I need to find another method. I was thinking of something like the way Docker containers can find each other in between networks (see etcd for example), using a predefined key that is shared during installation. The problem with this, is that it requires a remote server to store the IPs attached to that key. I don't want that.
Ideas on how to solve this are very much appreciated! I want to be able to offer a solution that would work even when WAN is out. That being said, I am aware that a connection to a remote server is needed if the end user would want to enable any voice assistant or wants to control a device from WAN.
I have an idea for an iPhone app that will make billions of dollars for me, and help me take over the world. I am currently designing the architecture of the application. I had originally assumed that I would need to create and host a database that would allow users to upload their data and share it with other, targeted, users. But I thought it would be better if the users could send their data directly to each other without having it stored in a database. However, I haven't been able to find anything that would suggest this is possible.
Is it possible to send data between iPhones that are not in close proximity (bluetooth) and not on a LAN? For instance...User #1, who is in Tennessee, creates some information and hits "Send to others" and it sends that information directly to User #2, who is in Arizona, and User #3, who is in Maine. Could the text messaging functionality be hacked to do that?
(I realize that the phones belonging to Users #2 and #3 would have to be on, and running the app that would receive the data)
I am new to mobile development, and I am still getting used to the functionality of mobile devices so, I am certain this is a dumb question. But, it is worth asking because, if it can be done, this would dramatically change the architecture (and maintenance costs) of this application.
Thanks, in advance, for whatever advice/pointers you can give.
This is something handled by using the publish - subscribe pattern. You may want to look into a plug 'n play service such as pubnub. That will let you do just that. Phones that should be receiving each other's messages will have listeners set up on the same pubnub channel, and will receive a notification when any of the other phones publish to that channel. Something like pubnub has a pretty simple API/SDK that you can use to get a prototype up and running pretty quickly (and for free, at least at first).
User #1's app sends the message to your server. You server does an APNS push to the phones of users #2, and #3. They don't even need to be running the app.
No need to mark this correct or up-vote it, but I will take a high-paying job in the new world order.
Amazon has an application service called Simple Queue Service (Amazon SQS) which Allows you to create queue's with messages in them tat app's can subscribe to.
To quote their page:
Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly
scalable, hosted queue for storing messages as they travel between
computers. By using Amazon SQS, developers can simply move data
between distributed components of their applications that perform
different tasks, without losing messages or requiring each component
to be always available. Amazon SQS makes it easy to build an automated
workflow, working in close conjunction with the Amazon Elastic Compute
Cloud (Amazon EC2) and the other AWS infrastructure web services.
Amazon SQS works by exposing Amazon’s web-scale messaging
infrastructure as a web service. Any computer on the Internet can add
or read messages without any installed software or special firewall
configurations. Components of applications using Amazon SQS can run
independently, and do not need to be on the same network, developed
with the same technologies, or running at the same time.
They have an iOS API to integrate it into your app, and it is free for up to 100,000 messages per month and then $0.01 per 10,000 messages after that.
More info here:
http://aws.amazon.com/sqs/
Is it possible to send data between iPhones that are not in close
proximity (bluetooth) and not on a LAN?
The two devices obviously need some kind of connectivity.
For instance...User #1, who is in Tennessee, creates some information
and hits "Send to others" and it sends that information directly to
User #2, who is in Arizona, and User #3, who is in Maine. Could the
text messaging functionality be hacked to do that?
The problems here are:
helping each device discover the correct address for the others, and...
ensuring that the devices can reach each other.
Mobile devices are constantly moving around on the network and changing their IP addresses as they go. As you're driving to work, your device may have Internet access through it's 3G connection, except for those times when you're driving through a tunnel and don't have any connectivity at all. Once you enter your building, a wifi connection becomes available, so the device switches to that. But your company's network has a firewall that blocks incoming connections, and it's impossible to know in advance which IP address you'll get from the DHCP server at any given time anyway. The same is true for the folks in Arizona and Maine and wherever else.
A server, on the other hand, is usually located at an address that's both easy to discover (thanks to the Domain Name System), easy to reach, and almost always available. For those reasons, it's common to have mobile devices communicate with each other by going through an intermediate server.
I want to send some Text plus a image from one iphone application to other iphone app but restriction is I should not use a web server in between communication,Is there any way to fulfill it ?
Details: There are two independent devices and could be far enough say out of network. My requirement one app adds some text with a image and sends it to another iphone which can be at any long distance , and the app installed in another iphone will read that info and image into itself.
Actually there is a solution that meets your needs — and that fits to bbums answer:
Create a HTTP-Server on the iPhone, using cocoahttpserver. than you will ask some webservice like whatismyip.com for your public ip. with this your iPhone can be connected worldwide.
But very likely ur wifi-network is not forwarding your port to the iPhone. Ash.
And even if: Now it gets difficult. How to publish your ip from one phone to the other? hmmm... — I got it: I will exchange the information in a centralized space! In the web!
... wait — that would be a Webserver.
You see: Without any kind of server in the Web the users would need to exchange ip manually and have full admin power and knowledge about the local network.
So IMHO bbums answer is the only way to go.
PS: I am working with http server running on iPhones. In local network that works great, especially with bonjour. And you can use them over distance network — but only with reconfiguration of your router — something you shouldn't force your user to do
There is far from enough information to provide a specific answer.
two apps on two different devices?
are the two devices on the same network?
are the two devices both on WiFi?
do you need the user to receive a notification or something if the app isn't running?
If on same device, you can define a custom URL handler in the destination app and then openURL: in the source app to pass the data over. Encode your image and text into the URL, but be careful of size limitations.
If on different devices, there are many possible solutions, but answering the above questions will be critical to actually knowing what solution is appropriate.
Given your comment -- two apps, different devices, arbitrary networks -- then you are going to have to have some kind of server in between. Note that the recently added Game Center does have the ability to rendezvous two users, but it has a very particular user experience that may not be appropriate to your needs.
I would suggest that you investigate using push notifications to notify the receiving user of the availability of content. As for moving the content between, no direct connection is possible and you will have to have some kind of store-and-forward server in between. And, yes, a web server is going to be the easiest possible solution simply because HTTP is ubiquitous these days.
If there's no network of any kind available, but both parties have amateur radio licenses, then hooking the two devices up to HF packet radios might work.
THIS is super EASY.
I would code up some software that can turn data into modem signal, like the good old dial up modem. The device would actually make those annoying buzzing sounds.
You get the phone number for your friends nearest landline and call him.
He places his iPhone near the phones receiver in listen mode and you connect to his phone using your audible modem.
Bingo, via the power of sounds you have sent data which is decoded on his device and all for the very cheap price of a phone call, there are pretty cheap these days especially if you use Skype.
Easy Way (relatively speaking)
A way two apps on different networks can communicate without setting up a web server of some sort is as follows.
Use an existing third party storage system like DropBox.
Each app would need the login and password for your DropBox. Then both apps can read and write files that the other app can see.
An existing app that does this is a shopping list app called ShopShop.
The app on my phone and my wife's phone both link to the same DropBox account and the app keeps the shopping list synced up when one of us adds something to the list.