google-cloud-logging: namespace-id and container_name not registered in the default GKE installation

[mgenov]

I'm not sure whether this is a role of the stackdriver-metadata-agent but this information is required by the java logging client and it's not available in my cluster.

The k8s cluster is configured with Stackdriver Kubernetes Engine Monitoring and Workload Identity enabled. All agents are running in the kube-system:

stackdriver-metadata-agent-cluster-level-74785fffdd-79b6v        1/1     Running   0          3h46m

The logs show no errors but the information regarding container_name and namespace_id is not available inside the containers:

root@workload-identity-test:/# curl "http://metadata.google.internal/computeMetadata/v1/instance/attributes/"  -H "Metadata-Flavor: Google"
cluster-name
root@workload-identity-test:/#

e.g only cluster-name is available but the google-cloud-logging library is looking and for namespace-id and container_name.

[igorpeshansky]

@chingor13, see Stackdriver/kubernetes-configs#47 (comment). TL;DR: this was actually a bug introduced in #3887.

[mgenov]

@chingor13 can you give any information on this?

[pmakani]

@mgenov @igorpeshansky I looked at different scenario, after running both type of cluster configuration with stackdriver logging, i am able to get container-name and namespace-id/
namespace-name. also verified using google-cloud-logging library.
attaching my finding for both type of configurations.

• Container
  -- Legacy Stackdriver Logging : Enable
  -- Legacy Stackdriver Monitoring : Enable

• gke_container
  -- Api Ref https://cloud.google.com/monitoring/api/resources#tag_gke_container

resource: {
  labels: {
   cluster_name: "springboot-stackdriver"    
   container_name: "springboot"    
   instance_id: "6864435408643901574"    
   namespace_id: "default"    
   pod_id: "springboot-599fd7db48-27z8v"    
   project_id: "google-issue-6239"    
   zone: "us-central1-a"    
  }
  type: "container"
 }
 severity: "INFO"  
 textPayload: "12:23:17.592 [http-nio-8080-exec-9] INFO  - Hello World
"  
 timestamp: "2019-09-12T12:23:17.592806398Z"  
}

---

• K8S-Container
  -- Stackdriver Kubernetes Engine Monitoring : Enable
  -- Workload Identity : Enable

• k8s_container
  -- Api Ref https://cloud.google.com/monitoring/api/resources#tag_k8s_container

resource: {
  labels: {
   cluster_name: "stackdriver-kubernetes"
   container_name: "kube-log"    
   location: "us-central1-a"    
   namespace_name: "default"    
   pod_name: "kube-log-585c5cb974-xc7hl"    
   project_id: "google-issue-6239"    
  }
  type: "k8s_container"   
 }
 severity: "INFO"  
 textPayload: "10:42:21.792 [http-nio-8080-exec-1] INFO  - Hello World! Kubernetes Stackdriver
"  
 timestamp: "2019-09-13T10:42:21.796188304Z"  
}

please let me know if there is something that i am missing here.

[igorpeshansky]

@pmakani Were those ingested via the google-cloud-logging library, or printed to stdout and ingested by the logging agent? If the latter, it's expected, because the agent does have access to the namespace name and the container name. This issue is about that information not being exposed to code running inside the container.

[codyoss]

@mgenov @igorpeshansky I would consider this a dupe of #5765 . I would like to close this issue and move conversation over there if so. Let me know if I am mistaken.

[igorpeshansky]

It does look like a dupe, but see #6239 (comment) above.

[codyoss]

Noted, thank you. Like I said here #5765 (comment) I think we might need to open up an issue with GKE to expose this data. It is unfortunate though that it looks like we can get this data from the pod, but I don't think we can today. Closing this, if we have future discussion lets do it here: #5765 .