Tokenize Large String process

The Tokenize Large String process segments a string on a specified delimiter. As compared with the Tokenize String process, the Tokenize Large String process overcomes the limitation of the traditional tokenizer against large inputs by setting up a loop that detects the size of the input and parses it in 10,000-character segments, appending the segments together to create a fully-parsed result.

The following figure displays the Tokenize String process and the Tokenize Large String process in BMC Atrium Orchestrator Development Studio.

Tokenize String process

Tokenize Large String process

The following table describes the process inputs for the Tokenize Large String request process.

Input	Description	Required
string	Specifies the string to be parsed on a specified character	Yes
delimiter	Specifies the character on which the specified string is parsed Default value: None	No

The following table describes the process output parameters for the Tokenize Large String process.

Tokenize String process output

Output	Description
tokens	Specifies an XML document with a root element, `<tokens>`, and child elements, `<token>`, containing the resulting string segments

Tokenize Large String process

Comments