Tokenize Large String
The Tokenize Large String process segments a string on a specified delimiter. As compared with the Tokenize String process, the Tokenize Large String process overcomes the limitation of the traditional tokenizer against large inputs by setting up a loop that detects the size of the input and parses it in 10,000-character segments, appending the segments together to create a fully-parsed result.
The following table describes the process inputs for the Tokenize Large String request process.
Input | Description | Required |
|---|---|---|
string | Specifies the string to be parsed on a specified character | Yes |
delimiter | Specifies the character on which the specified string is parsed | No |
The following table describes the process output parameters for the Tokenize Large String process.
Tokenize String process output
Output | Description |
|---|---|
tokens | Specifies an XML document with a root element, <tokens>, and child elements, <token>, containing the resulting string segments |