Emulating Python's `index(separator, start_index)` in Rust - unicode

Question

I'm currently porting a library from Python from Rust and found a line for which I'm having trouble finding the right "translation":
right = s.index(sep, left)
where right is the index of the first instance of sep found in string s that is after index left.
A simple example of this can be seen here:
Python 3
>>> s = "Hello, my name is erip and my favorite color is green."
>>> right = s.index("my", 10) # Find index of first instance of 'my' after index 10
>>> print right
27
>>> print s[27:]
my favorite color is green.
My attempt in Rust is:
// s: &str, sep: &str, left: usize
let right = s[left..].find(sep).unwrap() + left;
This will search the bytes after left for sep. This seems to work when using ASCII characters. There seems to be a problem when using Unicode, though:
Python 3
>>> s = "Hello, mÿ name is erip and mÿ favorite color is green."
>>> right = s.index("mÿ", 10)
>>> print(right)
27
Rust
fn main() {
let sep: &str = "mÿ";
let left: usize = 10;
let s: &str = "Hello, mÿ name is erip and mÿ favorite color is green.";
let right = s[left..].find(sep).unwrap() + left;
println!("{}", right); //prints 28
}
I realize that Python 2 would also give 28 because it doesn't support Unicode natively, but I'd like to mimic Python 3's results.
The problem is because usize in Rust refers to the number of bytes in a string because "mÿ" actually requires 3 bytes to encode. How can I acquire this desired behavior in Rust?
I'm using rustc 1.4.0.

Answer 1

Let's restate the problem a bit as it's unclear what the unit of index should be. Humans believe that strings are easy because we've been using them for most of our lives. However, things aren't nearly as easy as we'd like.
Rust takes the point-of-view that strings (&str or String) are UTF-8 encoded sequences of bytes. Jumping into a string using a byte offset is O(1), and you really want that level of performance guarantee to build more complicated things upon.
I don't know what Python considers that index to be. It gets hard once you get beyond simple encoding schemes like ASCII where one character is one byte. There are multiple ways to chunk a Unicode string depending on what you want. Two obvious ones are by Unicode codepoint and by grapheme.
Since codepoints can be represented in Rust using a char, that's what I assume you want. However, you are the only one that can figure that out.
Additionally, since you requested that the result be 28, that must be the number of bytes into the string. It's a little odd to skip N codepoints but return bytes, but it is what it is.
Now that we know what we are doing... let's try and do it. (See next solution where I read the desired outcome better).
The key thing you need to use is char_indices. This is an O(n) operation that walks through the string and and gives you each codepoint and its corresponding byte offset.
Then, it's just a matter of putting that together and correctly handling cases of walking off the end of the string. This is made obvious by Rust's strong types, hooray!
// `index` is the number of Unicode codepoints to skip
// The result is the number of **bytes** inside the haystack
// that the needle can be found.
fn python_index(haystack: &str, needle: &str, index: usize) -> Option<usize> {
haystack.char_indices().nth(index).and_then(|(byte_idx, _)| {
let leftover = &haystack[byte_idx..];
leftover.find(needle).map(|inner_idx| inner_idx + byte_idx)
})
}
fn main() {
let right = python_index("Hello, mÿ name is erip and mÿ favorite color is green.", "mÿ", 10);
println!("{:?}", right); // prints Some(28)
}
We do the same high-level concept as above, but once we have found the needle, we then reset back and iterate through the codepoints again. When we find the same byte offset of the substring, we terminate.
Then it's just a matter of counting the characters we saw and adding the number that we already skipped.
// `index` is the number of Unicode codepoints to skip
// The result is the number of codepoints inside the haystack
// that the needle can be found.
fn python_index(haystack: &str, needle: &str, index: usize) -> Option<usize> {
haystack.char_indices().nth(index).and_then(|(byte_idx, _)| {
let leftover = &haystack[byte_idx..];
leftover.find(needle).map(|inner_offset| {
leftover.char_indices().take_while(|&(inner_inner_offset, _)| {
inner_inner_offset != inner_offset
}).count() + index
})
})
}
fn main() {
let right = python_index("Hello, mÿ name is erip and mÿ favorite color is green.", "mÿ", 10);
println!("{:?}", right); // prints Some(27)
}
This certainly feels not super-efficient; you'd want to benchmark to see how it fares. However, the find implementation is pretty optimized, so I'd rather use it and then do a straight-shot through the characters and trust in the cache and prefetching to help me out ^_^.