What's new in TPL

Changes to TPL in BMC Helix Discovery 24.1.02

BMC Helix Discovery 24.1.02 introduces improvements to the way that reasoning works. Part of the reasoning changes make improvements to TPL compilation and execution, resulting in:

Reduced compile times.
Improved compile-time errors and warnings.
Runtime ECA Errors now include a traceback of the TPL calls.

No changes have been made to the TPL language itself, so its version remains 1.22, though as a result of the improvements, there are some fixes to the way that TPL behaves. Behavior changes are described here.

Triggers with mixed regex flags

If a pattern trigger has an OR combination of regex matches, the old compiler would combine them into a single regex, even if they had different flags. For example triggers like this:

triggers
on DiscoveredProcess where cmd matches regex "one" or cmd matches regex "(?i)two";
end triggers;

would be compiled as if it was:

triggers
on DiscoveredProcess where cmd matches regex "(?i)one|two";
end triggers;

In other words, it would incorrectly match command "ONE", when actually it should only match "one" in lower case. The windows_cmd string qualifier is equivalent to a regex involving "(?i)" flags, and so this issue would occur with triggers mixing unix_cmd and windows_cmd, for example:

triggers
on DiscoveredProcess where cmd matches unix_cmd "one" or cmd matches windows_cmd "two";
end triggers;

The new compiler no longer has this problem, meaning that patterns that previously triggered because of this will no longer trigger. In some cases, the TPL should have specified "(?i)" for a case insensitive match, but was working by mistake. Patterns may need to be updated to add the correct flags to all parts of an OR condition.

Validation of all constant regular expressions

The compiler always validated regular expressions with the regex qualifier, so it was previously a compile-time error if code was for example:

if regex.extract(value, regex "one(abc", raw "\1") matches regex "invalid[" then...

Both of those regular expressions are invalid, and the compiler would reports errors. However, if a regular expression in a matches expression or call to regex.extract was not qualified with regex, it was only a runtime error if the regular expression was invalid. Now, if a regular expression string is constant, it is checked at compile time. The above incorrect code will now cause a compile-time error even if the regex qualifiers are removed.

A likely problem is forgetting the regex qualifier, meaning that the regular expression is invalid, for example:

if path matches "abc\\def\\ghi then ... // forgot the regex qualifier

That now reports a compile-time error: "bad escape \g at position 7".

"stop" in a function really stops the pattern

The "stop" statement is defined to stop execution of the pattern. Previously, if "stop" was used in a function, it did not stop the pattern, but instead caused the function to return none. If the function was declared to return a single value, that would mean the caller received none; if the function was declared to return multiple values, it would cause an immediate ECAError. "stop" now always stops execution of the pattern. If functions were previously using "stop", relying on that returning none, they may need to be changed to "return none".

"break" in nested for loops now behaves correctly

Previously, if a pattern had a nested set of for loops that performed some discovery requests, the "break" statement would sometimes break out from too many levels of the loop, or terminate processing of the pattern altogether. Now, "break" always correctly breaks out of just the innermost loop. Patterns may have accidentally been relying on the incorrect "break" behavior, and will now encounter errors during execution when "break" behaves correctly.

Uninitialized variables

If a variable is used without being initialized, it now always causes an ECAError at runtime. Previously, some uses of uninitialized variables caused ECAErrors, but others saw the variable's value as none. Patterns may previously have been working despite invalid use of uninitialized variables, in which case they will now fail. Any such patterns should be fixed to ensure all variables are initialized before use.

The compiler now issues warnings for more uses of uninitialized variables than in previous versions.

Invalid use of key expressions with list values

Key expressions (of the form node.#role:rel:role:node.attribute) are only valid when used on individual nodes. Previously, if a variable used in a key expression contained a list instead of a node, the key expression incorrectly evaluated to an empty list. Now such usage causes an ECAError. Patterns may have been appearing to work despite having incorrect logic; they will now receive errors.

Improved compiler errors and warnings

Certain incorrect code was previously permitted, but now generates compile-time errors or warnings:

Functions that are declared to return a result but do not contain a "return" statement now produce a compile error. Previously, the compiler did not report an error, and at runtime the function would return none.
Some cases of variables that may be used uninitialized now emit a warning, where previously they did not.
Mixing variable types in incompatible ways (for example trying to add text to a number) is now sometimes identified and warned about, where previously there was no compile warning.