Make custom filters available in Mistral; Disable Decryption-by-default in st2kv function

[Mierdin]

It's been a long-held constraint that the custom Jinja filters provided by StackStorm are only available in ActionChain workflows, and unavailable to Mistral workflows.

This PR aims to change that by duplicating the filter implementations from the st2 repo in this repo. This makes the same filters available in Mistral workflows without adding a dependency on the st2common package.

NOTE that this is a duplication of the filters in the st2 repo, so every time a filter is added or changed there, a PR must be opened here to keep things consistent. This PR only makes the existing custom filters available, not future ones.

• Documentation Update: Updates to custom YAQL and Jinja filters st2docs#552
• Integration Tests: Integration Tests for Custom Jinja Filters in Mistral st2#3565

CHANGELOG for st2 was updated in StackStorm/st2@dd72546 since st2mistral doesn't really have one and that's probably where folks will look for the change anyways.

Closes #29

BREAKING CHANGE

The st2kv function here in Mistral previously decrypted keys by default. This PR adds a parameter to the underlying function that will force users to explicitly allow decryption, instead of doing it all the time. So, workflows that use st2kv to retrieve decrypted versions of ciphertext stored in datastore must now explicitly enable this decryption in the parameter, like so:

"{{ st2kv('system.secretkey', decrypt=True) }}"

Without this parameter, st2kv will not attempt decryption, and st2kv will return whatever value is stored at that key, whether or not it is ciphertext.

This change will make st2kv behavior more consistent with equivalent usage on the CLI/API and in ActionChains, where decryption is not performed by default.

Because st2kv leverages the API, which provides decryption on its own, the decrypt_kv filter was not included in the list of filters moved over to Mistral in this PR.

[m4dcoder]

I thought we agreed to duplicate the Jinja filters in st2mistral. I'm against including st2common here as a dependency. This will complicate the deployment story for Mistral.

[Mierdin]

Sorry, I must have missed that one, or assumed that "duplicate" meant implementing these redirections. I am okay with re-implementing them fully here, it just adds work to anyone wanting to create a new custom filter.

Funny enough, I didn't catch the deployment complications you allude to because of the fact I am using one virtualenv for both st2 and mistral in StackStorm/st2#3557 (which you pointed out should be done separately). I assume this is what you're talking about, since in any real deployment, mistral would not have access to st2common.

I'll adjust this PR's description and change the approach. Thanks for the catch!

[m4dcoder]

Yep in real deployment, st2 and mistral have different venvs and mistral currently do not have access to st2common.

[m4dcoder]

It is important during development to be conscious about how changes will impact CI, packaging, and deployment.

[Mierdin]

Ack - keeping that in mind, will remedy my development script for Mistral to reflect that.

[Mierdin]

So it turns out the issue I’ve been having with YAQL while writing itests for these changes in StackStorm/st2#3565 was indeed caused by a mismatch in the function signature, and the way the YAQL evaluator was trying to pass data into it, but it had nothing to do with args vs kwargs, it was only with functions that had string parameters.

Summary

Strings as input to functions are recognized as <type 'unicode'> in YAQL, and all of the custom filters that have default string values (i.e. ‘’) are <type 'str'> (https://github.com/StackStorm/st2mistral/pull/30/files#diff-bda8d34192458bf7865ba4eb5d602996R30).
The YAQL engine checks that the input type, and the type it perceives is required from the function signature (https://github.com/openstack/yaql/blob/master/yaql/language/yaqltypes.py#L110) and if they are not the same, that result filters all the way back up several layers and eventually raises NoMatchingFunctionException (https://github.com/openstack/yaql/blob/master/yaql/language/runner.py#L70), which is what I was seeing.

From the look of it, the only integration tests that use any custom filters within a YAQL statement are testing st2kv, which doesn’t have any parameters that default to a string,.
My theory is that if anyone were to try to use any custom filters using YAQL (possibly even in ActionChains, haven’t tested that yet) that do have string-default parameters, they’d run into the same thing.
Of course, in Jinja this is not a problem because it doesn’t have the same explicit check.

Solution

The easiest way of addressing this is to add the unicode marker before each default string parameter in each custom filter function. Certainly, since this PR is not merged yet, I can take care of this. Jinja should continue to work, and this will make the equivalent YAQL work as well:

def regex_match(context, value, pattern=u'', ignorecase=False):
    if not isinstance(value, six.string_types):
        value = str(value)
    flags = _get_regex_flags(ignorecase)
    return bool(re.match(pattern, value, flags))

I already have a bit on my plate so I can’t do this now, but someone should try to use one of the custom filters that have one of these parameters, inside a YAQL statement in an ActionChain. My theory is that it won’t work, but I could be wrong. Just something to follow up on.

In addition, I couldn’t get both YAQL and Jinja to work with the additional context parameter (like in st2kv function). I dug into this issue in particular for an hour or two, and it seems like it should work because of the use of partial in mistral’s expression_utils (https://github.com/StackStorm/mistral/blob/master/mistral/utils/expression_utils.py#L92) but at runtime, context is always set to what value should be, and value to what pattern should be, so it doesn’t work. Seems like YAQL is more flexible wrt passing context (probably referring to it via keyword and suppressing exception when it doesn’t exist is my guess).

Anyways, I’m ignoring this for now so I can move on but it may be worth backtracking to figure this out because it doesn’t seem to make sense given the use of partial (even my single-script testing didn’t match up with the behavior I was seeing from mistral). For now, I'm going to submit some commits to this PR that add the unicode specifier to all string parameters for the functions here.

[m4dcoder]

LGTM overall. Please use the term function instead of filter in Mistral. Also, there are coding guidelines for Mistral if you can address.

[m4dcoder]

Please use function instead of filter In Mistral. The term filter is specific to Jinja. In Mistral, these are called function. Please move all the implementation into the functions folder.

[Mierdin]

Done in be47054

[Mierdin]

Oh, and in 5291379 (updated tests accordingly)

[m4dcoder]

Please list them by alphabetical order.

[Mierdin]

Done in f49a9ff

[m4dcoder]

How about just return json.dumps(value)?

[m4dcoder]

Why not move this to data.py?

[Mierdin]

All good ideas - this was just some copypasta from the original implementation. Fixed in a164d6c

[m4dcoder]

Why the blank lines?

[Mierdin]

No reason, just copied from st2. Fixed in f49a9ff

[m4dcoder]

Why use piping to pass k1 to to_complex? I mean we can test it if we want to make sure piping is supported. But how about also testing it as to_complex(k1)?

[Mierdin]

As discussed in StackStorm/st2#3565, we're deciding (for now) to not support the filter format due to the upstream mistral issue, and requiring that users use the function format. I added context as the first positional arg (and modified testing accordingly) in 739fa88

[m4dcoder]

Please leave default to max line length to 79 to match w/ Mistral. As I explained in Slack, this is an extension project for Mistral and I have been following the coding guidelines for Mistral. Sorry if this doesn't make any sense.

[Mierdin]

No problem, I reverted this in 5dd1696 (and made sure everything passed tox -epep8 again). Following the rules of the upstream project this is a "part" of makes sense to me (though it wreaks havoc on my regex patterns)!

[Mierdin]

Style fixes for me to make:

1. The length for each line is 79.
2. Blank line is required after each conditional/logical blocks (i.e. for, if, while, etc.)
3. Blank line is required before return statement.
4. For parentheses, if the block involves multiple lines, the closing parentheses ) needs to be in a separate line.
5. For functional arguments, if too long and require separate lines, put individual argument in a separate line.

[Mierdin]

@m4dcoder So I made style corrections that Flake8 caught (after changing the max line length back to pep8 standard) but I had some questions about the other style rules you proposed (and I reposted above).

I know only of https://docs.openstack.org/hacking/latest/user/hacking.html in terms of OpenStack style standards, and I only found references to the last rule: "For functional arguments, if too long and require separate lines, put individual argument in a separate line." Is there another doc I'm missing?

Also what about automated checks? The mistral repo doesn't seem to use anything beyond what's already used in this repo, and it doesn't currently seem to be checking for any of that (except line length of course).

[m4dcoder]

Please fix the config register opts. Otherwise, changes LGTM. @lakshmi-kannan asks me not to go easy on you so I'm going to enforce the coding style loosely (except the line length of 79). Some of the Mistral coding guidelines are additions to OpenStack so there's no doc around that. These guidelines you'll just have to get used to as you contribute more to Mistral.

[m4dcoder]

Why did you move this config section? There's a reason for the config section to be up top. It'll load config which will be used in the http module.

[Mierdin]

This was to address the "module level import not at top of file" PEP8 violation, though I can see now that flake8 doesn't care for some reason (even though E402 is not being ignored).

Anyways, I can revert this, just part of the overall style fixes.

[Mierdin]

Okay, fixed in c12ff85