Why does Google Docs operational transformation err on the side of deletion? - real-time

Tried out this experiment today: opened two offline editors for a Google document. In one, I bolded the first word. In the second, I deleted it. Regardless of which client I turn on first, the word always ends up deleted.
First off, why is this the case - my understanding of operational transformation is that ordering matters? In the simple example of two people typing "a" and "b" respectively, if the server receives "a" first, it will enforce the output of "ab" by transforming the second person's "b" event into a "pass one space, then add b" event, and vice versa.
Secondly, if ordering doesn't matter, are there technical reasons as to why Google Docs has chosen to err on the side of deletion? Or are the reasons largely simplicity for users?

Here is (5 years later I know) a graphical explanation of what why this happens. This is, in fact, what #osma describes but graphically explained:
When you bold a string in GDocs you are wrapping the string into a container, presumably <strong></strong> but they may use any other syntax. For simplicity lets just say that bold'ing a string just requires a "+" at the beginning of the word. So that, for simplicity, the text "lorem ipsum" would become lorem +ipsum and not lorem <strong>ipsum<strong>
1
Both Alice and Bob start with the text "Lorem ipsum"
2
Bob then deletes "ipsum". Notice that he sends the changeset retain(6), delete(5) to the server. A changeset is essentially a patch, Google probably used this library.
3
Now Alice bolds "ipsum" (adding "+"). She sends is the changeset retain(6), insert(+), retain(5)
4
Both changesets are traveling to the server. The server knows nothing about these sets yet.
5
Assuming the worst scenario: Bob's package arrives first and then the word will be deleted. The other scenario is obvious.
6
When Alice's package arrives, it will only add a "+" to the text because what she sent is only a single changeset.
7
Both texts are then broadcasted to the clients. This is the first one.
8
And this is the second one.
9
After patching these changesets into the original text you end up with "Lorem +". The server and all clients now have the same text. The + symbol would later be erased by an common HTML clean process which eliminates empty tags like <tag></tag>,
To test this demo go to: http://operational-transformation.github.io/visualization.html. There you can play with the texts and packages as they are sent/received.

It's not a question of erring on the side of deletion.
In cases where both clients have equality valid but differing versions of truth, Google Docs must elect to uphold one version, or else force users to resolve conflicts, something that is inherently complicated and hard to explain.
Thus, "truth" for Google Docs is consistency of the document, not discernment of intent. And consistency is best more easily achieved through destruction of information - a sort of tendency to entropy.
All this is basically my semi-philosophical BS though...

OT does not try to discern intent, it applies transformations in an order which produces a consistent result. When you apply both of those changes to a document, it does not matter which order you apply them in.
"first second" -> "first second" -> "first"
"first second" -> "first" -> "first"
In the second stream, the bold operation is performed on a zero-length string.
This is the exact same result you would get if in one of those documents you had italicized the second word: the end result would be "first second" regardless of transformation order. Delete transformation is no different.

Related

comment structure of yaml in ruamel.yaml

I'm trying to understand the comment structure in ruamel.yaml library so that I can manipulate the comments correctly. What I don't get is why the comments in a .ca are inside a 4 items list? What are those other items and why they are always None?
For example, a comment attached to a sequence is like this with the first being CommentToken and the rest are Nones:
Comment(
start=None,
items={
2: [CommentToken('\n\n########### Foo ###########\n', line: 294, col: 0), None, None, None]
})
For a comment attached to a map, it seems to always be placed at the 3rd index?
Comment(
start=None,
items={
bar: [None, None, CommentToken('\n\n########## Bar ###########\n', line: 87, col: 0), None]
})
What's the difference between these and what's the significance of the order of their placements?
It only seems to you that for a YAML mapping the comments are placed in the same position, because you tend put the comments in the same place in your YAML file (i.e. at the end of the line after a value). I did the same when starting ruamel.yaml back in 2014, but over the years issues reported towards package pointed out that some users (of YAML) tend to put their comments in different places (e.g. between a key and a value on the next line, something not originally handled at all).
The significance is where the representer, on dumping the data structure, tries to insert the comments back into the YAML output stream. The code to do that has evolved over the last years to deal with more comment placements that were originally not handled (i.e. comments were lost or replaced on round-trip).
There is no documented API for this, the code that creates the Comment instances is changed in tandem with the code that processes them, so in principle the meaning of any position might change, and those meanings actually has changed in the past. That list structure might also be replaced by a dict with keys that are more indicative of where the comment (stored int he corresponding value) came from/has to be inserted in, than indices. A dict could also do away with the None values indicating empty slots.
I can't guarantee that the source code documentation is up to date but this is what it says:
# map key (mapping/omap/dict) or index (sequence/list) to a list of
# dict: post_key, pre_key, post_value, pre_value
# list: pre item, post item
IIRC some of these positions are no longer in use.
You should pin the version number of ruamel.yaml you work with. ruamel.yaml follows semantic versioning, but it there is no API for handling comment and it is pre version 1.0, so anything can change at any time. However the minor number tends to be bumped on major (internal) changes or on dropping support for no longer maintained Python versions. So stick with ruamel.yaml<0.18 if you get your code to work with 0.17.21, and test extensively if 0.18 and later will still do what you want.
An API for handling comments will at some point be forthcoming, and apart from dealing with more (exotic) comment placements, also have a way to specify how multi-line comments needs to be handled so they don't necessarily are attached to the preceding node like they are now, but can be assigned to the following node or split between those nodes according to some rule (e.g. first empty line).

Mozilla Deep Speech SST suddenly can't spell

I am using deep speech for speech to text. Up to 0.8.1, when I ran transcriptions like:
byte_encoding = subprocess.check_output(
"deepspeech --model deepspeech-0.8.1-models.pbmm --scorer deepspeech-0.8.1-models.scorer --audio audio/2830-3980-0043.wav", shell=True)
transcription = byte_encoding.decode("utf-8").rstrip("\n")
I would get back results that were pretty good. But since 0.8.2, where the scorer argument was removed, my results are just rife with misspellings that make me think I am now getting a character level model where I used to get a word-level model. The errors are in a direction that looks like the model isn't correctly specified somehow.
Now I when I call:
byte_encoding = subprocess.check_output(
['deepspeech', '--model', 'deepspeech-0.8.2-models.pbmm', '--audio', myfile])
transcription = byte_encoding.decode("utf-8").rstrip("\n")
I now see errors like
endless -> "endules"
service -> "servic"
legacy -> "legaci"
earning -> "erting"
before -> "befir"
I'm not 100% that it is related to removing the scorer from the API, but it is one thing I see changing between releases, and the documentation suggested accuracy improvements in particular.
Short: The scorer matches letter output from the audio to actual words. You shouldn't leave it out.
Long: If you leave out the scorer argument, you won't be able to detect real world sentences as it matches the output from the acoustic model to words and word combinations present in the textual language model that is part of the scorer. And bear in mind that each scorer has specific lm_alpha and lm_beta values that make the search even more accurate.
The 0.8.2 version should be able to take the scorer argument. Otherwise update to 0.9.0, which has it as well. Maybe your environment is changed in a way. I would start in a new dir and venv.
Assuming you are using Python, you could add this to your code:
ds.enableExternalScorer(args.scorer)
ds.setScorerAlphaBeta(args.lm_alpha, args.lm_beta)
And check the example script.

Getting the user's name without directly asking for it (Watson Assistant)

I'm creating a chatbot for fun and want to implement something to collect the user's name, but only if he says something like "my name is ..." or close to that; the intention of giving the name would come from the user, the bot won't ask for it, maybe only suggest it. Kind of like in Google Assistant, I think. So, it could be given at any time the user wants.
My idea is:
1st create an intent with different ways the user would tell his name to the bot (like in the example above).
2nd use slots and, if the intent is detected, save it as a variable. So far, I've managed.
3rd is the part that I'm stuck in, since it's only an idea and I don't know how I'd do it. Before saving the whole text as a variable, I'd like to delete the part that's included in the intent (my name is) and save only the rest in the variable. So, for example, the user says "my name is XXX"; the command deletes the "my name is" part and saves only "XXX" in the $name intent.
I don't know if this'd be possible, since I don't know coding. I used some special syntax before, like to capitalize the first letter of some other variable, but I don't really know how to use the JSON editor.
Is my idea viable? I don't know how I'd delete the intent corresponding portion and save only the remaining part as the intent. Dunno what would be the command for that, nor where I'd write it.
You can suggest something else if you have an idea.
Last thing, I'm created the skill in portuguese, so there's no access to the #sys-person entity.
Thanks for reading.
I use portuguese skills and face the same issue. I see two solutions, althougt they are not perfect:
Using intents:
When the intent is identified, the bot asks the name again, telling the user that he should only say his name, e.g without "my name is", then store the whole input on a context variable, using:
<? input.text ?>
For embedding such logic inside slots, you probaly will need to use digressions.
But, this is boring to the user.
Using entities:
Entities identification carries along they start and end position in the input text, but intents not. With this, is possible to slice the input, cutting the entity:
<? input.text.substring(entities.name.location[1], input.text.length()) ?>
Entites would be "My name is", "People call me", "I'm called", "I'm baptized", "I was baptized".
So, "Hello, my name is Gustavo", would be cut after "my name is" ending, resulting in "Gustavo". Additional input in the beggining is ignored, but problems arrise with additional input after the name. Also, you need to define more "my name is" like entities, likely all possibilities, than would need if using intents, cause even with fuzzy matching, entities identification doesn't take in account synonymus and similar meaning words.
https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-methods#dialog-methods-strings-substring
https://cloud.ibm.com/docs/assistant?topic=assistant-expression-language#expression-language-access-entity

Can Watson trigger a dialog based on 12 digits pattern?

I try to display a message that contains the number inserted in the question, for example when user inserts a number of 12 digits I want to display that 12 digits and a text.
Until now:
I created an Entity with pattern (/d{12}) named #ticket_number
An Intent named #myTicket that has as example #ticket_number
Dialog that triggers when #myTicket | #ticket_number and has on context TicketNumer "<?#ticket_number.literal?>" and display a message as follows "Do you want to get info for ticket $ticketnumber ?".
The problem is that when I try it the Intent result is irrelevant, the message looks ok but I need to match the Intent. What could I do?
Can you share a picture of your node? There is no need that you match the intent; as indicating the ticket number alone, should not be an intent itself, but the entity. I'd remove the #myTicket | part of the node. And the condition should not include an OR; otherwise, if #myTicket triggers the node, and there is no ticket number, the response would fail.
As mentioned in the IBM documentation
it is not yet possible to use pattern entity in Intents.
Currently, you can only directly reference synonym entities that you
define (pattern values are ignored). You cannot use system entities.

RESTful URL design for search

I'm looking for a reasonable way to represent searches as a RESTful URLs.
The setup: I have two models, Cars and Garages, where Cars can be in Garages. So my urls look like:
/car/xxxx
xxx == car id
returns car with given id
/garage/yyy
yyy = garage id
returns garage with given id
A Car can exist on its own (hence the /car), or it can exist in a garage. What's the right way to represent, say, all the cars in a given garage? Something like:
/garage/yyy/cars ?
How about the union of cars in garage yyy and zzz?
What's the right way to represent a search for cars with certain attributes? Say: show me all blue sedans with 4 doors :
/car/search?color=blue&type=sedan&doors=4
or should it be /cars instead?
The use of "search" seems inappropriate there - what's a better way / term? Should it just be:
/cars/?color=blue&type=sedan&doors=4
Should the search parameters be part of the PATHINFO or QUERYSTRING?
In short, I'm looking for guidance for cross-model REST url design, and for search.
[Update] I like Justin's answer, but he doesn't cover the multi-field search case:
/cars/color:blue/type:sedan/doors:4
or something like that. How do we go from
/cars/color/blue
to the multiple field case?
For the searching, use querystrings. This is perfectly RESTful:
/cars?color=blue&type=sedan&doors=4
An advantage to regular querystrings is that they are standard and widely understood and that they can be generated from form-get.
The RESTful pretty URL design is about displaying a resource based on a structure (directory-like structure, date: articles/2005/5/13, object and it's attributes,..), the slash / indicates hierarchical structure, use the -id instead.
Hierarchical structure
I would personaly prefer:
/garage-id/cars/car-id
/cars/car-id #for cars not in garages
If a user removes the /car-id part, it brings the cars preview - intuitive. User exactly knows where in the tree he is, what is he looking at. He knows from the first look, that garages and cars are in relation. /car-id also denotes that it belongs together unlike /car/id.
Searching
The searchquery is OK as it is, there is only your preference, what should be taken into account. The funny part comes when joining searches (see below).
/cars?color=blue;type=sedan #most prefered by me
/cars;color-blue+doors-4+type-sedan #looks good when using car-id
/cars?color=blue&doors=4&type=sedan #also possible, but & blends in with text
Or basically anything what isn't a slash as explained above.
The formula: /cars[?;]color[=-:]blue[,;+&], though I wouldn't use the & sign as it is unrecognizable from the text at first glance if that's your thing.
** Did you know that passing JSON object in URI is RESTful? **
Lists of options
/cars?color=black,blue,red;doors=3,5;type=sedan #most prefered by me
/cars?color:black:blue:red;doors:3:5;type:sedan
/cars?color(black,blue,red);doors(3,5);type(sedan) #does not look bad at all
/cars?color:(black,blue,red);doors:(3,5);type:sedan #little difference
possible features?
Negate search strings (!)
To search any cars, but not black and red:
?color=!black,!red
color:(!black,!red)
Joined searches
Search red or blue or black cars with 3 doors in garages id 1..20 or 101..103 or 999 but not 5
/garage[id=1-20,101-103,999,!5]/cars[color=red,blue,black;doors=3]
You can then construct more complex search queries. (Look at CSS3 attribute matching for the idea of matching substrings. E.g. searching users containing "bar" user*=bar.)
Conclusion
Anyway, this might be the most important part for you, because you can do it however you like after all, just keep in mind that RESTful URI represents a structure which is easily understood e.g. directory-like /directory/file, /collection/node/item, dates /articles/{year}/{month}/{day}.. And when you omit any of last segments, you immediately know what you get.
So.., all these characters are allowed unencoded:
unreserved: a-zA-Z0-9_.-~
Typically allowed both encoded and not, both uses are then equivalent.
special characters: $-_.+!*'(),
reserved: ;/?:#=&
May be used unencoded for the purpose they represent, otherwise they must be encoded.
unsafe: <>"#%{}|^~[]`
Why unsafe and why should rather be encoded: RFC 1738 see 2.2
Also see RFC 1738#page-20 for more character classes.
RFC 3986 see 2.2
Despite of what I previously said, here is a common distinction of delimeters, meaning that some "are" more important than others.
generic delimeters: :/?#[]#
sub-delimeters: !$&'()*+,;=
More reading:
Hierarchy: see 2.3, see 1.2.3
url path parameter syntax
CSS3 attribute matching
IBM: RESTful Web services - The basics
Note: RFC 1738 was updated by RFC 3986
Although having the parameters in the path has some advantages, there are, IMO, some outweighing factors.
Not all characters needed for a search query are permitted in a URL. Most punctuation and Unicode characters would need to be URL encoded as a query string parameter. I'm wrestling with the same problem. I would like to use XPath in the URL, but not all XPath syntax is compatible with a URI path. So for simple paths, /cars/doors/driver/lock/combination would be appropriate to locate the 'combination' element in the driver's door XML document. But /car/doors[id='driver' and lock/combination='1234'] is not so friendly.
There is a difference between filtering a resource based on one of its attributes and specifying a resource.
For example, since
/cars/colors returns a list of all colors for all cars (the resource returned is a collection of color objects)
/cars/colors/red,blue,green would return a list of color objects that are red, blue or green, not a collection of cars.
To return cars, the path would be
/cars?color=red,blue,green or /cars/search?color=red,blue,green
Parameters in the path are more difficult to read because name/value pairs are not isolated from the rest of the path, which is not name/value pairs.
One last comment. I prefer /garages/yyy/cars (always plural) to /garage/yyy/cars (perhaps it was a typo in the original answer) because it avoids changing the path between singular and plural. For words with an added 's', the change is not so bad, but changing /person/yyy/friends to /people/yyy seems cumbersome.
To expand on Peter's answer - you could make Search a first-class resource:
POST /searches # create a new search
GET /searches # list all searches (admin)
GET /searches/{id} # show the results of a previously-run search
DELETE /searches/{id} # delete a search (admin)
The Search resource would have fields for color, make model, garaged status, etc and could be specified in XML, JSON, or any other format. Like the Car and Garage resource, you could restrict access to Searches based on authentication. Users who frequently run the same Searches can store them in their profiles so that they don't need to be re-created. The URLs will be short enough that in many cases they can be easily traded via email. These stored Searches can be the basis of custom RSS feeds, and so on.
There are many possibilities for using Searches when you think of them as resources.
The idea is explained in more detail in this Railscast.
Justin's answer is probably the way to go, although in some applications it might make sense to consider a particular search as a resource in its own right, such as if you want to support named saved searches:
/search/{searchQuery}
or
/search/{savedSearchName}
I use two approaches to implement searches.
1) Simplest case, to query associated elements, and for navigation.
/cars?q.garage.id.eq=1
This means, query cars that have garage ID equal to 1.
It is also possible to create more complex searches:
/cars?q.garage.street.eq=FirstStreet&q.color.ne=red&offset=300&max=100
Cars in all garages in FirstStreet that are not red (3rd page, 100 elements per page).
2) Complex queries are considered as regular resources that are created and can be recovered.
POST /searches => Create
GET /searches/1 => Recover search
GET /searches/1?offset=300&max=100 => pagination in search
The POST body for search creation is as follows:
{
"$class":"test.Car",
"$q":{
"$eq" : { "color" : "red" },
"garage" : {
"$ne" : { "street" : "FirstStreet" }
}
}
}
It is based in Grails (criteria DSL): http://grails.org/doc/2.4.3/ref/Domain%20Classes/createCriteria.html
This is not REST. You cannot define URIs for resources inside your API. Resource navigation must be hypertext-driven. It's fine if you want pretty URIs and heavy amounts of coupling, but just do not call it REST, because it directly violates the constraints of RESTful architecture.
See this article by the inventor of REST.
In addition i would also suggest:
/cars/search/all{?color,model,year}
/cars/search/by-parameters{?color,model,year}
/cars/search/by-vendor{?vendor}
Here, Search is considered as a child resource of Cars resource.
There are a lot of good options for your case here. Still you should considering using the POST body.
The query string is perfect for your example, but if you have something more complicated, e.g. an arbitrary long list of items or boolean conditionals, you might want to define the post as a document, that the client sends over POST.
This allows a more flexible description of the search, as well as avoids the Server URL length limit.
RESTful does not recommend using verbs in URL's /cars/search is not restful. The right way to filter/search/paginate your API's is through Query Parameters. However there might be cases when you have to break the norm. For example, if you are searching across multiple resources, then you have to use something like /search?q=query
You can go through http://saipraveenblog.wordpress.com/2014/09/29/rest-api-best-practices/ to understand the best practices for designing RESTful API's
Though I like Justin's response, I feel it more accurately represents a filter rather than a search. What if I want to know about cars with names that start with cam?
The way I see it, you could build it into the way you handle specific resources:
/cars/cam*
Or, you could simply add it into the filter:
/cars/doors/4/name/cam*/colors/red,blue,green
Personally, I prefer the latter, however I am by no means an expert on REST (having first heard of it only 2 or so weeks ago...)
My advice would be this:
/garages
Returns list of garages (think JSON array here)
/garages/yyy
Returns specific garage
/garage/yyy/cars
Returns list of cars in garage
/garages/cars
Returns list of all cars in all garages (may not be practical of course)
/cars
Returns list of all cars
/cars/xxx
Returns specific car
/cars/colors
Returns lists of all posible colors for cars
/cars/colors/red,blue,green
Returns list of cars of the specific colors (yes commas are allowed :) )
Edit:
/cars/colors/red,blue,green/doors/2
Returns list of all red,blue, and green cars with 2 doors.
/cars/type/hatchback,coupe/colors/red,blue,green/
Same idea as the above but a lil more intuitive.
/cars/colors/red,blue,green/doors/two-door,four-door
All cars that are red, blue, green and have either two or four doors.
Hopefully that gives you the idea. Essentially your Rest API should be easily discoverable and should enable you to browse through your data. Another advantage with using URLs and not query strings is that you are able to take advantage of the native caching mechanisms that exist on the web server for HTTP traffic.
Here's a link to a page describing the evils of query strings in REST: http://web.archive.org/web/20070815111413/http://rest.blueoxen.net/cgi-bin/wiki.pl?QueryStringsConsideredHarmful
I used Google's cache because the normal page wasn't working for me here's that link as well:
http://rest.blueoxen.net/cgi-bin/wiki.pl?QueryStringsConsideredHarmful