Existing kubeconfig file causes tests to fail

[BramMeerten]

When running a simple test, the waitUntilDefaultNamespaceAvailable check fails:

@EnableKubeAPIServer
class OutputPortCredentialsReconcilerTest {

    @KubeConfig
    static String kubeConfigYaml;

    @Test
    void testCommunication() {
        var client = new KubernetesClientBuilder()
                .withConfig(Config.fromKubeconfig(kubeConfigYaml))
                .build();
        client.resource(configMap()).createOrReplace();
        var cm = client.resource(configMap()).get();

        assertThat(cm).isNotNull();
    }

    private ConfigMap configMap() {
        return new ConfigMapBuilder()
                .withMetadata(new ObjectMetaBuilder()
                        .withName("test1")
                        .withNamespace("default")
                        .build())
                .withData(Map.of("key", "data"))
                .build();
    }
}

io.javaoperatorsdk.jenvtest.JenvtestException: Kube API Server did not start properly

	at io.javaoperatorsdk.jenvtest.process.ProcessReadinessChecker.pollWithTimeout(ProcessReadinessChecker.java:81)
	at io.javaoperatorsdk.jenvtest.process.ProcessReadinessChecker.waitUntilDefaultNamespaceAvailable(ProcessReadinessChecker.java:42)
	at io.javaoperatorsdk.jenvtest.process.KubeAPIServerProcess.waitUntilReady(KubeAPIServerProcess.java:94)
	at io.javaoperatorsdk.jenvtest.KubeAPIServer.start(KubeAPIServer.java:48)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.startApiServer(KubeAPIServerExtension.java:85)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.lambda$startIfAnnotationPresent$2(KubeAPIServerExtension.java:78)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.lambda$startIfAnnotationPresent$3(KubeAPIServerExtension.java:71)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.startIfAnnotationPresent(KubeAPIServerExtension.java:68)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.initialize(KubeAPIServerExtension.java:61)
	at io.javaoperatorsdk.jenvtest.junit.KubeAPIServerExtension.beforeAll(KubeAPIServerExtension.java:37)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$12(ClassBasedTestDescriptor.java:395)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:395)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:211)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:84)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

By debugging and reading the errorStream of the process in ProcessReadinessChecker::defaultNamespaceExists, I find the following reason:

authority=/Users/brammeerten/.jenvtest/apiserver.crt --server=https://127.0.0.1:51775 --request-timeout=5s get ns default
Error in configuration:
* client-cert-data and client-cert are both specified for rancher-desktop. client-cert-data will override.
* client-key-data and client-key are both specified for rancher-desktop; client-key-data will override

rancher-desktop refers to the context defined in my ~/.kube/config file. If I remove this file, the test works.
Or if I add @EnableKubeAPIServer(updateKubeConfigFile = true), then my test works without removing the kube config file.

But the documentation saids this flag is not advised. And it's unclear when and why I should use this flag.

In general, it is not advised but if instructed kube config file (~/kube/config) is updated by the framework. See related property in @EnableKubeAPIServer annotation. The config file is automatically cleaned up on stop..

So it's unclear if to me if this is a bug, or is this as designed and should I use the updateKubeConfigFile flag?

[scholzj]

I run into this issue as well. What seems to help is when I override the KUBECONFIG variable and point it somewhere else when running the test -> for example to ~/.jenvtest/.kube/config. But it is quite complicated to set it in all places where I run the tests ... in the IDE, in the terminals when running from through Maven directly etc.

@csviri I wonder what would you think about adding to the code a custom KUBECONFIG environment variable like I described above when updateKubeConfig is set to false. If that sounds like a reasonable workaround/fix, I can try to open a PR.

[csviri]

thx @BramMeerten and @scholzj for the issue, so if I understand when checking if the api server is up and ready, the kubectl picks up also certificates data from standard config file to query the API server.

@scholzj yes, that might be a viable workaround. Feel free to create a PR for this.