Use of strtok function - matlab

Question

Based on MATLAB's code for strtok (see end):
"Here’s a more advanced example that finds the first token in a character string. A token is a set of characters delimited by whitespace or some other character. Given one input, the function assumes a default delimiter of whitespace; given two, it lets you specify another delimiter if desired. It also allows for two possible output argument lists"
I have a few questions:
1) Is a delimiter specified at the beginning or end of a token?
So for example, if I wanted to find the section of a text which gave me a certain date and the whole text was: "I like the date april 10 because it is close to May Day". I imagine the token is "april 10" but the starting delimiter would be "a" and the ending delimiter would be a digit?
You see I am confused as to what a "delimiter" is exactly in context. In MATLAB I would normally probably write the token as (\w*\s\d*) in order to locate the date (april 10) in the text since I do not know what date it would be (what letter it starts with or the digits after it). But is a delimiter that whole "april 10" or just an "a" at the beginning? How would this help if I do not know what month it is (april, may, june, etc) or does it basically just work as a "find" command?
I ran the program shown in the picture and tried it with 'hello my friend' as the string and 'o' as the delimiter and it gives:
token=hell
remainder=o my friend
So basically I am getting the impression delimiter are usually used at the end of fields or different regions in order to specify when the new field/section (remainder) begins? Basically a delimiter is commonly used as a simple one (or maybe more) character device to indicate the start of a new field or datum whereas using (/d/w*....etc) format is used for more specific extractions like dates where there is no "comma" or specific indicator in front of it? Are these two observations correct?
BUT then when I run it using "hello my fri" as delimiter (see --> running it with delimiter, it seems to arbitrarily select "I want to say hello my friend good man" as the remainder and "nd" as the token which makes no sense so I am wondering if there is a bug in this program or if it's just not set up to handle a delimiter that appear twice.
Also,
2) Can someone please explain why [9:13 32] is made the default for one input argument? If we're assuming whitespace is the delimiter, then what does that [9:13 32] mean?
3) Is there any purpose to using "any" since it is ran by a looping process? Would not it check it each iteration anyways?
function [token, remainder] = strtok(string, delimiters)
%STRTOK Find token in string.
% TOKEN = STRTOK(STR) returns the first token in the string STR delimited
% by white-space characters. STRTOK ignores any leading white space.
% If STR is a cell array of strings, TOKEN is a cell array of tokens.
%
% TOKEN = STRTOK(STR,DELIM) returns the first token delimited by one of
% the characters in DELIM. STRTOK ignores any leading delimiters.
% Do not use escape sequences as delimiters. For example, use char(9)
% rather than '\t' for tab.
%
% [TOKEN,REMAIN] = STRTOK(...) returns the remainder of the original
% string.
%
% If the body of the input string does not contain any delimiter
% characters, STRTOK returns the entire string in TOKEN (excluding any
% leading delimiter characters), and REMAIN contains an empty string.
%
% Example:
%
% s = ' This is a simple example.';
% [token, remain] = strtok(s)
%
% returns
%
% token =
% This
% remain =
% is a simple example.
%
% See also ISSPACE, STRFIND, STRNCMP, STRCMP, TEXTSCAN.
% Copyright 1984-2009 The MathWorks, Inc.
if nargin<1
error(message('MATLAB:strtok:NrInputArguments'));
end
token = ''; remainder = '';
len = length(string);
if len == 0
return
end
if (nargin == 1)
delimiters = [9:13 32]; % White space characters
end
i = 1;
while (any(string(i) == delimiters))
i = i + 1;
if (i > len),
return,
end
end
start = i;
while (~any(string(i) == delimiters))
i = i + 1;
if (i > len),
break,
end
end
finish = i - 1;
token = string(start:finish);
if (nargout == 2)
remainder = string(finish + 1:length(string));
end
EDIT: I was not aware that strtok was a built in function. I was under the assumption it was a UDF the textbook was building as an example. This is why there are many ambiguities since the book does not specify clearly what the function does.
This, for example, was not specified in the text which only stated the function found the first token in a character string. --> token = strtok(str) parses input character vector str from left to right, returning part or all of that character vector in token. Using the white-space character as a delimiter, the token output begins at the start of str, skipping any delimiters that might appear at the start, and includes all characters up to either the next delimiter or the end of the character vector. White-space characters include space (ASCII 32), tab (ASCII 9), and carriage return (ASCII 13).
Copyright 1984-2009 The MathWorks, Inc.

Answer 1

strtok is very much not going to help you here so I'm not going to answer your main question. I think you should use regular expression for this but I don't speak regex so I'll leave that to someone else.
[9:13 32]
Why is the default delimiter set to [9:13 32]. From the comments, MATLAB is claiming that those are all the white space characters. In other words then numbers 9, 10, 11, 12, 13 and 32 are the ASCII values for white space characters. For example 32 is the value of a space. Prove this to yourself by casting one to an integer:
uint8(' ') % or even ' ' + 0
I don't know what all the others are but I'm pretty sure one must be the tab character. To check the ASCII value of a tab character you can do
uint8(sprintf('\t'))
which returns 9 which is indeed in the list.
So [9:13 32] is a list of all the white space characters, as the comment implies.
Actually there are many more white space characters that this doesn't cover: https://en.wikipedia.org/wiki/Whitespace_character
any
When you say any I'm assuming you mean in lines like this: any(string(i) == delimiters). So yes, the loop ensures that only one character of string is compared at a time however there can be multiple values in delimiter for example all the white space characters as mentioned above or maybe you called strtok like this:
strtok('I like the date...', 'ad')
now both 'a' and 'd' are used as delimiters and so it returns
'I like the '
because it hit a 'd' first.