Create end-to-end test suite

[toddbaert]

Leverage the test-harness and evaluation.feature gherkin suite to create a set of integration tests for the Python SDK. See equivalent examples in Java and JS. Though we should call these "end-to-end" not integration tests, since that's what they are.

The suite should start the test harness container and run it in CI (again, see references in Java and JS).

NOTE: we'll need the python flagd provider first, but work can begin now with this partial version.

Definition of done:

• gherkin test suite complete and runs on CI
• updates to documentation (CONTRIBUTING.md)

[jamescarr]

@toddbaert the Java and JS links are 404 now... where do these currently live?

[jamescarr]

Took a quick look at the projects that implement this.

Java

• Maven Task
• Step Definitions
• Runner

Javascript

• Jest Targets
• Step Definitions
• Runner

Python Implementation

To implement this in python, we will need:

• Add test-harness as a submodule
• Makefile target that copies the test-harness tests for running
• Add gherkin-python dev dependency
• Step definitions for cucumber tests
• Run e2e tests as part of Makefile via make e2e.

While the python-sdk-contrib flagd provider needs to be merged for this to work "for real," we can get the mechanics in place and it should work with the NoOpProvider for now.

Thoughts?

[toddbaert]

@toddbaert the Java and JS links are 404 now... where do these currently live?

@jamescarr sorry about that. Those links are updated (we renamed the dirs and suites "e2e" from "integration", since e2e is a more accurate description).

• Add test-harness as a submodule
• Makefile target that copies the test-harness tests for running
• Add gherkin-python dev dependency
• Step definitions for cucumber tests
• Run e2e tests as part of Makefile via make e2e.

While the python-sdk-contrib flagd provider needs to be merged for this to work "for real," we can get the mechanics in place and it should work with the NoOpProvider for now.

Thoughts?

This sounds perfect to me. Simply getting the test suite (cucumber, etc), CI working while running against the no-op provider would be an excellent first step.

I will circle back to the flagd provider for python. I'm really only a python novice, but the flagd communication layer is all built automatically via .proto definitions, ideally, so I might be able to get it into a working order without much trouble.

I will circle back on that.