I am trying to build a chatbot using Rasa. I have created a basic chatbot by following the steps given in documentation here. I have installed both rasa core and rasa nlu but for now i am using only rasa core as i don't need to extract any information from input.
I have added around 20 intents and their corresponding actions. But when i trying to get response it is recognising 14 intent accurately (tested even by jumbling word, by using synonyms) but for rest 6 intent it is always returning wrong response even if i enter same input as defined in intent.
At first i used spaCy + sklearn pipeline but now i am using sklearn + MITIE but still not getting accurate responses.
Is there any way to improve the accuracy of chatbot.
The best way to improve accuracy is to optimize your bot iteratively based on what conversational analytics tells you about how it interacts with users over time. It's unrealistic to expect a bot to be accurate out of the box, no matter how much NLP you bring to the table.
There are a bunch of tools available for this, with some being stronger in analytics (in addition to health metrics) than others. (I work for one called Chatbase, which is free to use and works with any bot platform.)
Related
I am working with IBM Watson Asistant for Korean and found the failure rate to detect the correct intent is so high. Therefore, I decided to check language support and I can see the important missing features that is Entity Fuzzy Matching:
Partial match - With partial matching, the feature automatically suggests substring-based synonyms present in the user-defined entities, and assigns a lower confidence score as compared to the exact entity match.
This result in the chatbot that is not very intelligent for which we need to provide synonyms for each word. Check out the example below where Watson Assistant in English can detect an intent from words that is not included in the example by any means. I tested and found it is not possible for Korean language to do so.
I wonder If I understood something wrong or there is away to workaround this issue that I do not know of?
By default, you start with IBM Watson Assistant and an untrained dialog. You can significantly improve the understood intents and entities by providing more examples and then using the dashboard to tag correctly understood conversations and to change incorrect intents / entities to the right ones. This is the preferred way and is just part of the regular development process which includes training the model.
Another method, this time as workaround, is to preprocess a dialog using Watson Natural Language Understanding which has Korean support, too.
BTW: I use German language for some of my bots and it requires training for some scenarios.
In addition to Henrik's answer, here are couple of tips while creating an intent
Provide at least five examples for each intent.
Always re-train your system
If the system does not recognize the correct intent, you can correct
it. To correct the recognized intent, select the displayed intent and
then select the correct intent from the list. After your correction is
submitted, the system automatically retrains itself to incorporate the
new data.
Remember, The Watson Assistant service scores each intent’s confidence independently, not in relation to other intents.
Avoid conflicts and if there are any resolve the conflicts - The Watson Assistant application detects a conflict when two or more intent examples in separate intents are so similar that Watson Assistant is confused as to which intent to use.
I was tasked to find the best way to create a facial recognition feature on an app with machine learning. This feature will be used to clock employees into the app. The feature will support...
multiple users per device.
continuous training (so when the mlmodel recognizes someone, it will send new images to the model on the back-end and train the model with the new recently taken images)
updates new classes (when a new user comes along and wants to use the app, the app will take pictures of them, send those images to the model training program on the back-end which will train the mlmodel to recognize the new user)
sends the newly updated models to other devices in the same store so they will recognize the employees as well
What I've tried.
I've tinkered with on-device training and Knn. But from what I understand on-device training will not work for this, because the on-device training models can only have up to 10 classes and knn isn't giving very accurate results...at all
Manual training and re-training with the createML. This is when I...
train a model with createML on my mac
download the model to the app with URLSession
add a new user with the app or take updated pictures of old users
send the images of the new user/updated old user to createML on my mac
create a whole new model with all the images I've ever taken of all the users
repeat steps 2-5 forever
This works just fine but is unbelievably expensive, time-consuming, and unfeasible for the number of eventual users the app will have, to do over and over again.
I'm still very new to machine learning and I feel like I'm going about this the wrong way. I would just like to see if anyone knows of a better/more efficient method of continuous learning so the model will remember what it's learned previously and I can just add new classes or images to it with createML... or if someone could point me in the right direction.
Take a look at Turi Create -- also from Apple: https://github.com/apple/turicreate
It can do everything Create ML does, but is in python and programmable, so you could automate the whole process on your backend. If you know how to do it in CreateML, you will find Turi Create easy to pick up.
To have an accurate result you should look into more powerful machine learning models. Here is an example of a really powerful face recognition model: https://github.com/davidsandberg/facenet.
Now, the next question becomes how would you integrate your app with this new model. This is really up to you but I would recommend you to checkout a few backend alternatives like leveraging AWS Services (EC2 compute servers , Sagemaker, API Gateway, etc) to run and coordinate the inferences. A couple of benefits to doing this is that your app would just be mainly front-end thus making it light and also scalable across different and older IOS platforms and devices. But more importantly, it gives you extra leg-space to do more sophisticated things in the future, where as using CoreML you will be mainly limited to on-device computational power and also the swift-based language.
However, leveraging cloud services would also have other cons attached like the learning curve (learning AWS Services) and potentially privacy issues.
This is just one of the ways, there are many other similar cloud providers like Google , IBM and Azure. Without knowing further your timeline, budget, technical expertise, I can only give you these options and the rest of the choice is yours to make
Can the IBM Watson Conversation / Assistant service detect more than one intention in a single sentence?
Example of input:
play music and turn on the light
Intent 1 is #Turn_on
Intent 2 is #Play
==> the answer must be simultaneous for both intents: Music played and light turned on
If so, how can I do that?
Yes, Watson Assistant returns all detected intents with their associated confidence. See here for the API definition. In the response returned by Watson Assistant is n array of intents recognized in the user input, sorted in descending order of confidence.
The documents have an example on how to deal with multiple intents and their confidence. Be also aware of a setting alternate_intents to allow even more intents with lower confidence to be returned.
While #data_henrik is correct in how to get the other intents, it doesn't mean that the second question is related.
Take the following example graph, where we map the intents versus confidence that comes back:
Here you can clearly see that there are two intents in the persons question.
Now look at this one:
You can clearly see that there is only one intent.
So how do you solve this? There are a couple of ways.
You can check if the first and second intent fall within a certain percentage of each other. This is the easiest to detect, but tricker to code to select two different intents. It can get messy, and you will sometimes get false positives.
At the application layer you can do a K-Means on the intent result. K-Means will allow you to group intents by buckets, so you create two buckets (K=2), and if there is more than one in the first bucket, you have a compound question. I wrote about this and a sample on my site.
There is a new feature you can play with in Beta called "Disambiguation". This allows you to flag intent nodes with a question to ask to get it. Then if two questions are found it will say "Did you mean? ...." and the user can select.
IS this disambiguation feature available in non production environments, on Beta?
This seems to be a common enough problem that there are a lot of entries when one googles for help but nothing has helped me yet.
I am finding that the results provided by the REST API for estimated_steps are wildly different from those that appear in the device app.
I am running a fetch task for users via cron job on a PHP/Laravel app.
I'm using this https://developers.google.com/fit/scenarios/read-daily-step-total - estimated_steps to retrieve the step count.
Some days the data is correct. Some days its wildly different. For instance, on one given day, the REST API gives step count of 5661 while the app shows 11,108. Then there are six seven days when the stream is correct.
Has anyone faced this sort of behavior? I've tested for timezone differences, logged and analyzed the response json to see if i'm making some obvious mistake, but nope.
You may check this How do I get the same step count as the Google Fit app? documentation. Be noted that even when using the right data source, your step count may still be different from that of the Google Fit app.
This could be due to one of the following reasons:
On Wear, the Fit MicroApp when connected will display step counts queried on the phone and transferred over via the Wearable APIs. Other MicroApps accessing local-only data will only get watch steps. We are working on making this easier for developers.
Sometimes the step calculation code for the Google Fit app is updated with bug fixes before we are able to release the fixes to developers (which requires a Google Play Services release). We are also working on making it possible for developers to access fixes at the same time.
The Fit app uses a specific data source for steps and it adds some functionality (which can be seen on the documentation) on top of the default merged steps stream.
You can access the "estimated" steps stream as shown here:
derived:com.google.step_count.delta:com.google.android.gms:estimated_steps
Hope this helps!
I have developed a conversational skill using API.AI and deployed to Google Home but API.AI's support seems limited and I am unable to do certain things like playing an audio file. The question I have is whether it's better to stick with API.AI or switch to Actions on Google for the long term.
Google has said that API.AI is the recommended way to build an agent for 'actions on google' for those who don't need/want to do their own NLU. They seem to expect that most developers will use API.AI because it does some of the work for you, with the NLU being the prime example, cf. Alexa where the developer is expected to specify all the different utterence variations for an intent (well, almost all - it will do some minor interpretation for you).
On the other hand, keep in mind that API.AI was created/designed before 'actions on google' existed and before they were purchased by Google - it was designed to be a generic bot creation service. So, where you gain something in creating a single bot that can fulfill many different services and having it do some of the messy work for you, you will certainly lose something compared to the power and control you have when writing to the API of one specific service - something more then just the NLU IMO, though I can't speak to playing an audio file specifically.
So, if you plan to just target the one service (and an audio bot is not relevant to most of the other services supported by API.AI) and you are finding the API.AI interface to be limiting then you should certainly consider writing your service with the 'actions on google' sdk.