Instructions and issue tracking for Service Mesh capabilities of Cilium

Cilium Service Mesh beta program Welcome to the Cilium Service Mesh beta program! In this repo you'll find information about the current status of Cil

Information

Category: C/C++ / Miscellaneous
Watchers: 11
Star: 10
Fork: 0
Last update: Dec 21, 2021

Resource links

https://github.com/cilium/cilium-service-mesh-beta

README
Issues 12

Cilium Service Mesh beta program

Welcome to the Cilium Service Mesh beta program!

In this repo you'll find

information about the current status of Cilium Service Mesh features
Coming soon: instructions for getting started with Cilium Service Mesh
a discussions forum for your feedback on Cilium Service Mesh features and usability
an issue tracker for bug reports related to Cilium Service Mesh.

The Cilium project is developing Service Mesh capabilities in a feature branch, to give us the flexibility to make potentially non-backwards-compatible changes, for example changes to CRDs. We plan to merge Service Mesh features into Cilium release v1.12 in early 2022. (Read more about the motivations for the beta program).

Service Mesh feature status

Please note: although we’ll only change things for good reasons, there are no backward-compatibility guarantees with the configuration of Cilium Service Mesh during beta, and the code has only been through limited testing. Please be mindful of this when you choose where to deploy Cilium Service Mesh - we do not recommend using it in production or staging environments yet.

Alpha Very early release of features that have not been through extensive testing yet. No guarantee that future releases will be back compatible with CRDs or other configuration settings. Please don't be surprised if you encounter bugs.

Beta Early release of features that we expect to be stable enough for testing by end users. No guarantee that future releases will be back compatible with CRDs or other configuration settings.

v1.11 Features already merged into the Cilium v1.11 release (as well as in the Service Mesh beta-specific builds)

Feature	Status
Kubernetes Ingress support	Beta
Open Telemetry support	Alpha, v1.11
L7-aware Traffic Management	Alpha

Other features will be added as the beta progresses.

Image tags

Container images built to include the service mesh beta features:

Component	Image name
Cilium agent	quay.io/cilium/cilium-service-mesh:v1.11.0-beta.1
Cilium operator	quay.io/cilium/operator-generic-service-mesh:v1.11.0-beta.1
Hubble relay	quay.io/cilium/hubble-relay-service-mesh:v1.11.0-beta.1

You will also need Cilium CLI version 0.10.0 or greater. (Check the version with cilium version).

Getting started

Instructions coming soon!

Raising issues

If you find an issue related to Service Mesh please raise it here.

This beta program is part of the Cilium project and follows the same Code of Conduct.

0 Open More issues

Closed

Install attaches wrong hubble relay image version

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

I ran the installation instruction for Cilium Service Mesh beta according to here and it does not attach the right image version for hubble-relay.

Specifically when I run this command:

cilium install --version -service-mesh:v1.11.0-beta.1 --config enable-envoy-config=true --kube-proxy-replacement=probe
cilium hubble enable

The output indicates that it tags it incorrectly to quay.io/cilium/hubble-relay:-service-mesh:v1.11.0-beta.1: 1:

✅ Cilium was successfully installed! Run 'cilium status' to view installation health
❯ cilium hubble enable
🔑 Found CA in secret cilium-ca
✨ Patching ConfigMap cilium-config to enable Hubble...
♻️  Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
🔑 Generating certificates for Relay...
✨ Deploying Relay from quay.io/cilium/hubble-relay:-service-mesh:v1.11.0-beta.1...
⌛ Waiting for Hubble to be installed...
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         1 errors, 1 warnings
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

DaemonSet         cilium             Desired: 4, Ready: 4/4, Available: 4/4
Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Deployment        hubble-relay       Desired: 1, Unavailable: 1/1
Containers:       cilium             Running: 4
                  cilium-operator    Running: 1
                  hubble-relay       Pending: 1
Cluster Pods:     17/17 managed by Cilium
Image versions    hubble-relay       quay.io/cilium/hubble-relay:-service-mesh:v1.11.0-beta.1: 1
                  cilium             quay.io/cilium/cilium-service-mesh:v1.11.0-beta.1: 4
                  cilium-operator    quay.io/cilium/operator-aws-service-mesh:v1.11.0-beta.1: 1
Errors:           hubble-relay       hubble-relay                     1 pods of Deployment hubble-relay are not ready
Warnings:         hubble-relay       hubble-relay-79cb8b485c-hhd4v    pod is pending


Error: Unable to enable Hubble:  timeout while waiting for status to become successful: context deadline exceeded

Manually correcting it using kubectl edit solves the issue.

Cilium Version

cilium-cli: v0.10.1 compiled with go1.17.6 on darwin/amd64
cilium image (default): v1.11.1
cilium image (stable): v1.11.1
cilium image (running): -service-mesh:v1.11.0-beta.1

Kernel Version

Darwin 3c22fbbff4ba.ant.amazon.com 19.6.0 Darwin Kernel Version 19.6.0: Sun Nov 14 19:58:51 PST 2021; root:xnu-6153.141.50~1/RELEASE_X86_64 x86_64

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

🔍 Collecting Kubernetes nodes 🔍 Collect Kubernetes nodes 🔍 Collecting Kubernetes events 🔍 Collecting Kubernetes pods 🔍 Collecting Kubernetes services 🔍 Collecting Cilium network policies 🔍 Collecting Cilium egress NAT policies 🔍 Collecting Cilium endpoints 🔍 Collecting Kubernetes network policies 🔍 Collecting Kubernetes pods summary 🔍 Collecting Cilium cluster-wide network policies 🔍 Collecting Kubernetes namespaces 🔍 Collect Kubernetes version 🔍 Collecting Cilium local redirect policies 🔍 Collecting Cilium identities 🔍 Collecting Cilium nodes 🔍 Collecting Cilium etcd secret 🔍 Collecting the Cilium configuration 🔍 Collecting the Cilium daemonset 🔍 Collecting the Hubble daemonset 🔍 Collecting the Hubble Relay configuration 🔍 Collecting the Hubble Relay deployment 🔍 Collecting the Hubble UI deployment 🔍 Collecting the Cilium operator deployment 🔍 Collecting the 'clustermesh-apiserver' deployment 🔍 Collecting gops stats from Cilium pods 🔍 Collecting gops stats from Hubble pods 🔍 Collecting 'cilium-bugtool' output from Cilium pods 🔍 Collecting gops stats from Hubble Relay pods 🔍 Collecting logs from Cilium pods 🔍 Collecting logs from Cilium operator pods 🔍 Collecting logs from 'clustermesh-apiserver' pods ⚠️ Deployment "clustermesh-apiserver" not found in namespace "kube-system" - this is expected if 'clustermesh-apiserver' isn't enabled 🔍 Collecting logs from Hubble pods 🔍 Collecting logs from Hubble Relay pods 🔍 Collecting logs from Hubble UI pods 🔍 Collecting platform-specific data 🔍 Collecting Hubble flows from Cilium pods ⚠️ The following tasks failed, the sysdump may be incomplete: ⚠️ [10] Collecting Cilium egress NAT policies: failed to collect Cilium egress NAT policies: the server could not find the requested resource (get ciliumegressnatpolicies.cilium.io) ⚠️ [11] Collecting Cilium local redirect policies: failed to collect Cilium local redirect policies: the server could not find the requested resource (get ciliumlocalredirectpolicies.cilium.io) ⚠️ Please note that depending on your Cilium version and installation options, this may be expected 🗳 Compiling sysdump ✅ The sysdump has been saved to /Users/yojeo/src/microservices-demo/cilium-sysdump-20220126-131409.zip

Relevant log output

No response

Anything else?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

opened Jan 26, 2022 by youngjeong46 5

Closed

INSTALLATION.md: use correct Hubble image

null

opened Jan 26, 2022 by tklauser 4

Closed

Update CLI minimum version

Must be 0.10.2 or newer

Signed-off-by: Joe Talerico [email protected]

opened Feb 21, 2022 by jtaleric 2

Closed

Issues with ClusterName containing dot

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

I have a k8s cluster created via kops. The cluster was created with cluster name that is the k8s api FQDN. Hence installing cilium with the command produced the following error: $ cilium install --version -service-mesh:v1.11.0-beta.1 --config enable-envoy-config=true --kube-proxy-replacement=probe ℹ️ using Cilium version "-service-mesh:v1.11.0-beta.1" 🔮 Auto-detected cluster name: k8s.example.com 🔮 Auto-detected IPAM mode: cluster-pool ❌ Cluster name "k8s.example.com" cannot contain dots ↩️ Rolling back installation...

Error: Unable to install Cilium: invalid cluster name, dots are not allowed

I have tried installing with helm and there were no issues. Is it possible to install the cilium service mesh beta with Helm?

Cilium Version

service-mesh:v1.11.0-beta.1

Kernel Version

$ uname -a Linux ip-172-31-36-201 5.13.0-1017-aws #19~20.04.1-Ubuntu SMP Mon Mar 7 12:53:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

PKs-MacBook-Pro:deployments-ec2 pk$ kubectl version Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

opened Apr 10, 2022 by aelicieguo 1

Closed

Invalid CiliumEnvoyConfig interrupts reconciliation for all types, prevents cilium-agent startup

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

I created a CRD with a resource type that wasn't recognized, and reconciliation stopped. Specifically: listing/watching for CilumEnvoyConfig resources started failing at the deserailiztion step. This had the further consequence of causing a (very slow) cilium-agent failure mode where it'd block on the "waiting for caches" step of startup for 5 minutes before crashing.

This is reproducible on any kubernetes cluster running the service-mesh-beta (WARNING: will break updates for service endpoints &c!):

kubectl apply -f - <<EOF                              
apiVersion: cilium.io/v2alpha1
kind: CiliumEnvoyConfig
metadata:
  name: bad-config
spec:
  resources:
    - "@type": type.googleapis.com/envoy.config.listener.v3.Listenerz
EOF

kubectl exec -n kube-system ds/cilium -- kill 1

The second command forces a restart on some cilium agent to put it in the pre-cache-sync state.

Service can be restored with:

kubectl delete ciliumenvoyconfig bad-config

Cilium Version

This happened when using the software from the tip of the beta/service-mesh branch.

Kernel Version

$ uname -a
Linux d3abe6123727 5.10.104-linuxkit #1 SMP Wed Mar 9 19:05:23 UTC 2022 x86_64 GNU/Linux

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

Happy to grab this if it's helpful

Relevant log output

level=warning msg="github.com/cilium/cilium/pkg/k8s/watchers/cilium_envoy_config.go:92: failed to list *v2alpha1.CiliumEnvoyConfig: proto: (line 1:10): unable to resolve \"type.googleapis.com/envoy.config.listener.v3.Listenerz\": \"not found\"" subsys=klog
level=error msg=k8sError error="github.com/cilium/cilium/pkg/k8s/watchers/cilium_envoy_config.go:92: Failed to watch *v2alpha1.CiliumEnvoyConfig: failed to list *v2alpha1.CiliumEnvoyConfig: proto: (line 1:10): unable to resolve \"type.googleapis.com/envoy.config.listener.v3.Listenerz\": \"not found\"" subsys=k8s

... (something like 5 minutes later) ...

level=fatal msg="Timed out waiting for pre-existing resources to be received; exiting" subsys=k8s-watcher

Anything else?

Two half-baked ideas I had about how to address this, if they're useful:

Set up a validation webhook that checks whether the config can be recognized by the various & sundry protobuf types at creation/update time. That has the main advantage of giving direct feedback "what you're trying to do won't work", but the main disadvantage that admission control isn't infallible (especially across versions) so it's maybe not sufficient on its own.
Defer deserialization of the envoy-specific resources (via json.RawMessage or similar) to separate the kubernetes resource listing/watching from the envoy-specific config details. The main downside here is that the system seems to accept the instance of the CR but then doesn't configure envoy with it, and it wasn't clear to me where I'd go to see if convergence was successful or not (I looked for e.g. status.conditions on the CR and events before getting lucky with finding/identifying agent logs).

In either case, a few more events re: the CiliumEnvoyConfig feel like they would've been useful to help me identify what part of the system was having trouble. I was comparing against a working config, so even something as simple as a "ConfigAccepted" event would have helped me narrow down that it was a config-specific problem.

Code of Conduct

[X] I agree to follow this project's Code of Conduct

opened Mar 28, 2022 by sethp-verica 1

Closed

cilium-operator skip create CiliumEnvoyConfig CRD when set "enable-envoy-config=true"

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

when start a virtual machine by script vagrant/scripts/start.sh, and configure cilium-agent systemd service start cmdline with enable-envoy-config=true, it will be restart forever!! and print following fatal msg Mar 12 10:32:05 k8s1 cilium-agent[7161]: level=fatal msg="Unable to find all Cilium CRDs necessary within 5m0s timeout. Please ensure that Cilium Operator is running, as it's responsible for registering. it's due to cilium-operator call function AllCRDResourceNames->agentCRDResourceNames to create CRDS, but the later function only add ciliumenvoyconfigs by cilium-agent's config, not cilium-operator!! https://github.com/cilium/cilium/blob/7da6cff33bc502c91c89bf4688deb4db0eb622e8/pkg/k8s/synced/crd.go#L60-L62

	if option.Config.EnableEnvoyConfig {
		result = append(result, CRDResourceName(v2alpha1.CECName))
	}

Cilium Version

[email protected]:~/go/src/github.com/cilium/cilium$ git log commit d8460a4280ef1a34e441ce673c438897adc63be5 (HEAD -> service_mesh, zhanghe/beta/service-mesh, origin/beta/service-mesh, beta/service-mesh) Author: André Martins [email protected] Date: Thu Dec 9 16:37:33 2021 +0100

.github: add parameter to allow for image suffix

This commits adds a workflow parameter to allow for an
developer-defined image suffix. It's useful if the developers don't want
to use the default "beta" prefix for the docker image repository.

Signed-off-by: André Martins <[email protected]>

Kernel Version

[email protected]:~/go/src/github.com/cilium/cilium$ uname -a Linux k8s1 4.9.270-0409270-generic #202105261032 SMP Wed May 26 10:36:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

[email protected]:~/go/src/github.com/cilium/cilium$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s1 Ready 2d4h v1.23.0-rc.0 192.168.60.11 Ubuntu 18.04.6 LTS 4.9.270-0409270-generic containerd://1.3.4

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

opened Mar 12, 2022 by zhanghe9702 1

Open

Cilium Installation issue on GKE Cluster

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

A bug happened! While installing Cilium Service mesh, in a GKE cluster using the method described in the installation page/eCHO episode 32, I have been running into the following error. I have not been able to find a fix for this

🔮 Auto-detected Kubernetes kind: GKE ℹ️ using Cilium version "v1.11.3" 🔮 Auto-detected cluster name: gke-pp-piman-dev-us-central1-a-gentest-gke 🔮 Auto-detected IPAM mode: kubernetes 🔮 Auto-detected datapath mode: gke ❌ Unable to auto-detect GKE native routing CIDR. Is "gcloud" installed? ℹ️ You can set the native routing CIDR manually with --ipv4-native-routing-cidr ↩️ Rolling back installation...

Error: Unable to install Cilium: unable to derive region and zone from context name "gentest": not in the form gke_PROJECT_ZONE_NAME

Moreover, Even installing cilium leads to the same error

Error: Unable to install Cilium: unable to derive region and zone from context name "gentest": not in the form gke_PROJECT_ZONE_NAME

Cilium Version

cilium-cli: v0.11.1 compiled with go1.18.1 on linux/amd64 cilium image (default): v1.11.3 cilium image (stable): v1.11.4 cilium image (running): v1.10.3-gke.15

Kernel Version

Linux gke-access 4.15.0-1098-gcp #111~16.04.1-Ubuntu SMP Tue Apr 13 19:05:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.10-gke.2000", GitCommit:"0823380786b063c3f71d5e7c76826a972e30550d", GitTreeState:"clean", BuildDate:"2022-03-17T09:22:22Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

opened Apr 22, 2022 by Shikhar0411 2

Open

Update install instructions for kube-proxy options

null

opened Mar 16, 2022 by lizrice 0

Open

TLS Secrets in Ingresses aren't loaded when the Secret is created after the Ingress

I was writing the example "TLS with cert-manager" (#20) and realized that the Cilium Service Mesh with cilium v1.11.0-beta.1 does not seem to be loading the TLS Secret when the TLS Secret is not immediately present (it takes a few seconds for cert-manager to create the Secret after the Ingress is created).

Reproducing the issue:

To reproduce, you will have to install Cilium Service Mesh on a Kind cluster, and then install cert-manager. Here are the commands to create the Kind cluster (using kind v0.11.1) and install Cilium v1.11.0-beta.1:


kind create cluster --config /dev/stdin <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
EOF

cilium install --version -service-mesh:v1.11.0-beta.1 --config enable-envoy-config=true --kube-proxy-replacement=probe
cilium hubble enable

# https://kind.sigs.k8s.io/docs/user/loadbalancer/
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/master/manifests/namespace.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/master/manifests/metallb.yaml
kubectl wait -n metallb-system --for=condition=Ready pods --all
NET=$(docker network inspect kind | jq '.[].IPAM.Config[0] | select(.Gateway) | .Subnet' -r | sed 's|/16||') && \
  kubectl -n metallb-system create configmap config --dry-run=client -oyaml --from-file=config=/dev/stdin \
    <<< "address-pools: [{name: default, protocol: layer2, addresses: ['${NET/%.0/.200}-${NET/%.0/.250}']}]" | kubectl apply -f-

Then, let us install cert-manager with an issuer ca-issuer:

helm repo add jetstack https://charts.jetstack.io
helm upgrade --install cert-manager jetstack/cert-manager --version v1.7.1 --namespace cert-manager --set installCRDs=true --create-namespace
kubectl apply -f- <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: self-signed
  namespace: default
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca
  namespace: default
spec:
  isCA: true
  privateKey:
    algorithm: ECDSA
    size: 256
  secretName: ca
  commonName: ca
  issuerRef:
    name: self-signed
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-issuer
  namespace: default
spec:
  ca:
    secretName: ca
EOF

Finally, let us install a demo application, and configure your DNS to resolve the domains ookinfo.cilium.rocks and hipstershop.cilium.rocks:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/e5c1f5b/kubernetes-ingress/tls-ingress.yaml
sudo perl -ni -e 'print if !/\.cilium\.rocks$/d' /etc/hosts; sudo tee -a /etc/hosts \
  <<<"$(kubectl get svc/cilium-ingress-tls-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}') bookinfo.cilium.rocks hipstershop.cilium.rocks"
kubectl wait --for=condition=Ready pods -n default --all

At this point, demo-cert does not exist yet since we haven't annotated the Ingress resource with cert-manager.io/issuer=ca-issuer. Let's check that the Secret is missing:

$ kubectl get secret demo-cert
Error from server (NotFound): secrets "demo-cert" not found

Weridly enough, Cilium Service Mesh seems to have configured the Ingress to listen on 443 over HTTP (instead of HTTPS), which means curl complains about not receiving the Hello Server (instead, it receives HTTP/1.1 200 OK):

# Over TLS:
$ curl -sS -o /dev/null -D /dev/stdout https://bookinfo.cilium.rocks:443/details/1
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

# Over HTTP:
$ curl -sS -o /dev/null -D /dev/stdout https://bookinfo.cilium.rocks:443/details/1
HTTP/1.1 200 OK
content-type: application/json
server: envoy
date: Fri, 18 Feb 2022 14:59:56 GMT
content-length: 178
x-envoy-upstream-service-time: 2

Now, let us create the Secret demo-cert by asking cert-manager:

kubectl annotate ingress tls-ingress cert-manager.io/issuer=ca-issuer

We can check that the certificate has been properly created:

$ kubectl get secret demo-cert
NAME                               TYPE                                  DATA   AGE
demo-cert                          kubernetes.io/tls                     3      36s

I waited for a bit, expecting Cilium Service Mesh to "load" demo-cert, but it doesn't seem to be doing so and it keeps serving over HTTP on port 443.

I noticed that you need to recreate or update the Ingress resource to get Cilium Service Mesh to take demo-cert into account:

kubectl delete -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/e5c1f5b/kubernetes-ingress/tls-ingress.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/e5c1f5b/kubernetes-ingress/tls-ingress.yaml

Two questions:

Is it possible to enable a "strict" HTTPS-only mode on the Ingress? I would never expect 443 to be serving HTTP.
Is it expected that the TLS Secret demo-cert doesn't get dynamically loaded by Cilium Service Mesh when it comes up?

Versions:

Cilium v1.11.0-beta.1
Kernel 5.13.0-28-generic
Kubernetes v1.22.4 (kind v0.11.1)
Sysdump: cilium-sysdump-20220218-160816.zip

I agree to follow this project's Code of Conduct.

opened Feb 18, 2022 by maelvls 1

Open

regeneration-recovery is failing since

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

I have upgraded an existing instance of cilium in a cluster to use the new service mesh images and settings; To do this I did the following:

git clone https://github.com/cilium/cilium.git
cd cilium
git checkout origin/beta/service-mesh
helm upgrade -n kube-system cilium ./install/kubernetes/cilium --values=../<dir-path>/values.yaml

This produced a successful install helm output. I then subsequently ran a cilium status and get the following errors:

cilium status                                                                 ✔
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         47 errors
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         OK
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
Deployment        hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 6, Ready: 6/6, Available: 6/6
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Containers:       cilium             Running: 6
                  cilium-operator    Running: 2
                  hubble-relay       Running: 1
                  hubble-ui          Running: 1
Cluster Pods:     90/90 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium-service-mesh:v1.11.0-beta.1: 6
                  cilium-operator    quay.io/cilium/operator-generic-service-mesh:v1.11.0-beta.1: 2
                  hubble-relay       quay.io/cilium/hubble-relay-service-mesh:v1.11.0-beta.1: 1
                  hubble-ui          docker.io/envoyproxy/envoy:[email protected]:e5c2bb2870d0e59ce917a5100311813b4ede96ce4eb0c6bfa879e3fbe3e83935: 1
                  hubble-ui          quay.io/cilium/hubble-ui:[email protected]:018ed122968de658d8874e2982fa6b3a8ae64b43d2356c05f977004176a89310: 1
                  hubble-ui          quay.io/cilium/hubble-ui-backend:[email protected]:13a16ed3ae9749682c817d3b834b2f2de901da6fb41de7753d7dce16650982b3: 1
Errors:           cilium             cilium-sh8rz    controller endpoint-708-regeneration-recovery is failing since 2m39s (317x): regeneration recovery failed
                  cilium             cilium-sh8rz    controller endpoint-3306-regeneration-recovery is failing since 2m34s (317x): regeneration recovery failed
                  cilium             cilium-5mnm5    controller endpoint-396-regeneration-recovery is failing since 2m44s (317x): regeneration recovery failed
                  cilium             cilium-5mnm5    controller endpoint-3141-regeneration-recovery is failing since 2m44s (317x): regeneration recovery failed

This agent output continues for all agents ofcourse.

Checking the agent logs I can see it flooded with the following:

level=warning msg="libbpf: map 'cilium_calls_00058': error reusing pinned map" subsys=datapath-loader
level=warning msg="libbpf: map 'cilium_calls_00058': failed to create: Invalid argument(-22)" subsys=datapath-loader
level=warning msg="libbpf: failed to load object '58_next/bpf_lxc.o'" subsys=datapath-loader
level=warning msg="Unable to load program" subsys=datapath-loader
level=warning msg="JoinEP: Failed to load program" containerID=d0113b584c datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=58 error="Failed to load prog with tc: exit status 1" file-path=58_next/bpf_lxc.o identity=41106 ipv4=10.0.0.58 ipv6= k8sPodName=network-system/node-feature-discovery-worker-zdtdv subsys=datapath-loader veth=lxc30e9e94e930d
level=error msg="Error while rewriting endpoint BPF program" containerID=d0113b584c datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=58 error="Failed to load prog with tc: exit status 1" identity=41106 ipv4=10.0.0.58 ipv6= k8sPodName=network-system/node-feature-discovery-worker-zdtdv subsys=endpoint
level=warning msg="generating BPF for endpoint failed, keeping stale directory." containerID=d0113b584c datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=58 file-path=58_next_fail identity=41106 ipv4=10.0.0.58 ipv6= k8sPodName=network-system/node-feature-discovery-worker-zdtdv subsys=endpoint
level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=106.417467ms bpfWaitForELF="14.093µs" bpfWriteELF=5.40179ms containerID=d0113b584c datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=58 error="Failed to load prog with tc: exit status 1" identity=41106 ipv4=10.0.0.58 ipv6= k8sPodName=network-system/node-feature-discovery-worker-zdtdv mapSync="8.796µs" policyCalculation="48.87µs" prepareBuild=1.70903ms proxyConfiguration="28.481µs" proxyPolicyCalculation="176.405µs" proxyWaitForAck=0s reason="retrying regeneration" subsys=endpoint total=119.07863ms waitingForCTClean="1.204µs" waitingForLock="73.388µs"
level=error msg="endpoint regeneration failed" containerID=d0113b584c datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=58 error="Failed to load prog with tc: exit status 1" identity=41106 ipv4=10.0.0.58 ipv6= k8sPodName=network-system/node-feature-discovery-worker-zdtdv subsys=endpoint
level=error msg="Command execution failed" cmd="[tc filter replace dev cilium_host ingress prio 1 handle 1 bpf da obj 3037_next/bpf_host.o sec to-host]" error="exit status 1" subsys=datapath-loader
level=warning msg="libbpf: couldn't reuse pinned map at '/sys/fs/bpf/tc//globals/cilium_calls_hostns_03037': parameter mismatch" subsys=datapath-loader
level=warning msg="libbpf: map 'cilium_calls_hostns_03037': error reusing pinned map" subsys=datapath-loader
level=warning msg="libbpf: map 'cilium_calls_hostns_03037': failed to create: Invalid argument(-22)" subsys=datapath-loader
level=warning msg="libbpf: failed to load object '3037_next/bpf_host.o'" subsys=datapath-loader
level=warning msg="Unable to load program" subsys=datapath-loader
level=warning msg="JoinEP: Failed to load program for host endpoint (to-host)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=3037 error="Failed to load prog with tc: exit status 1" file-path=3037_next/bpf_host.o identity=1 ipv4= ipv6= k8sPodName=/ subsys=datapath-loader veth=cilium_host
level=error msg="Error while rewriting endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=3037 error="Failed to load prog with tc: exit status 1" identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=warning msg="generating BPF for endpoint failed, keeping stale directory." containerID= datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=3037 file-path=3037_next_fail identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=warning msg="Regeneration of endpoint failed" bpfCompilation=0s bpfLoadProg=143.574912ms bpfWaitForELF="20.982µs" bpfWriteELF=6.394646ms containerID= datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=3037 error="Failed to load prog with tc: exit status 1" identity=1 ipv4= ipv6= k8sPodName=/ mapSync="9.888µs" policyCalculation="27.333µs" prepareBuild=2.842736ms proxyConfiguration="33.314µs" proxyPolicyCalculation="8.815µs" proxyWaitForAck=0s reason="retrying regeneration" subsys=endpoint total=161.941749ms waitingForCTClean="1.686µs" waitingForLock="3.741µs"
level=error msg="endpoint regeneration failed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=11 endpointID=3037 error="Failed to load prog with tc: exit status 1" identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint

These logs are somewhat repetitive so I've only grabbed the last few entries.

Cilium Version

v1.11.0

Kernel Version

Linux k8s-controlplane-01 5.11.0-1007-raspi #7-Ubuntu SMP PREEMPT Wed Apr 14 22:08:05 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

It's worth mentioning this is the oldest kernel version being used on any of the fleet of nodes.

Kubernetes Version

v1.22.2

Sysdump

Relevant log output

No response

Anything else?

I started by initially doing a helm upgrade ... then performed an uninstall using helm and the CLI to ensure all things were removed and attempted a fresh install however, ended with the same results.

I also attempted a rollback however, it now continually appears to have this issue of not entirely starting - I'm not sure what the state of impact is with these logs as services continue to start and be routable through previous methods.

Code of Conduct

[X] I agree to follow this project's Code of Conduct

opened Feb 15, 2022 by xUnholy 6

Open

Problems with Cilium Service Mesh beta on AKS

Potential issues:

https://github.com/cilium/cilium-cli/issues/664
https://cilium.slack.com/archives/C02QKQDTVDX/p1640345656199700

opened Jan 31, 2022 by lizrice 0

Open

cross-node connection use L7 policy, envoy based cilium strategy.

Proposal / RFE

Is your feature request related to a problem?

Describe the solution you'd like

restful-api based Network Policy program examples and related youtube video I have read, I wonder if the CiliumEnvoyConfig can be more easy to use. In the example it seems to difficult to apply and restart a new one.

here is the L7 traffic management example in the code base.

apiVersion: cilium.io/v2alpha1
kind: CiliumEnvoyConfig
metadata:
  name: envoy-lb-listener
spec:
  services:
    - name: echo-other-node
      namespace: cilium-test
    - name: echo-same-node
      namespace: cilium-test
  resources:
    - "@type": type.googleapis.com/envoy.config.listener.v3.Listener
      name: envoy-lb-listener
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: envoy-lb-listener
                rds:
                  route_config_name: lb_route
                http_filters:
                  - name: envoy.filters.http.router
    - "@type": type.googleapis.com/envoy.config.route.v3.RouteConfiguration
      name: lb_route
      virtual_hosts:
        - name: "lb_route"
          domains: ["*"]
          routes:
            - match:
                prefix: "/"
              route:
                weighted_clusters:
                  clusters:
                    - name: "cilium-test/echo-same-node"
                      weight: 50
                    - name: "cilium-test/echo-other-node"
                      weight: 50
                retry_policy:
                  retry_on: 5xx
                  num_retries: 3
                  per_try_timeout: 1s
                regex_rewrite:
                  pattern:
                    google_re2: {}
                    regex: "^/foo.*$"
                  substitution: "/"
    - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
      name: "cilium-test/echo-same-node"
      connect_timeout: 5s
      lb_policy: ROUND_ROBIN
      type: EDS
      outlier_detection:
        split_external_local_origin_errors: true
        consecutive_local_origin_failure: 2
    - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
      name: "cilium-test/echo-other-node"
      connect_timeout: 3s
      lb_policy: ROUND_ROBIN
      type: EDS
      outlier_detection:
        split_external_local_origin_errors: true
        consecutive_local_origin_failure: 2

opened Jan 27, 2022 by wkhno 0

Meshery Adapter for Cilium Service Mesh

eBPF & Cilium Office Hours

tri-mesh - A triangle mesh data structure including basic operations

🧞‍♂️ MESH·Y is a design tool to generate beautiful & colorful mesh gradients.

The Service Mesh Orchestration Platform

Incorporate Service Mesh Patterns into your GitHub workflows using Meshery's CLI

Cilium is eBPF-based Networking, Security, and Observability

A repo that contains all the mesh keys needed for mesh backend, along with a code example of how to use them in python

The Polygon Mesh Processing Library

👹 2D Java game engine built in Java

Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows

refine a 2d mesh by splitting edges longer than a maximum length

Visual Inertial Odometry with SLAM capabilities and 3D Mesh generation.

Decentralized Issue Tracking for git

List your dependencies capabilities and monitor if updates require more capabilities.

A service mesh for Kubernetes and beyond. Main repo for Linkerd 1.x.

Meshery, the service mesh management plane

Install hubble-ui on GKE Dataplane V2

Dart Optimization Tips and Tricks

Mesh TensorFlow: Model Parallelism Made Easier

Nuitrack™ is a 3D tracking middleware developed by 3DiVi Inc.

Share this repo

Related Repos

Miscellaneous

420

For recording and retrieving metadata associated with ML developer and data scientist work...

ML Metadata ML Metadata (MLMD) is a library for recording and retrieving metadata associated with ML developer and data scientist workflows. NOTE: ML

Miscellaneous

100

Hotcaml: an interpreter with watching and reloading

Hotcaml: an OCaml interpreter with watching and reloading Hotcaml is an OCaml interpreter that starts from a source file and loads its dependencies. W

Miscellaneous

A React Native library for accessing an ArrayBuffer of a Blob instance.

react-native-blob-jsi-helper A React Native library for directly accessing an ArrayBuffer of a Blob instance. Note: This is a workaround. A long-term

Miscellaneous

This is a OOSDK port of flat's ps4_remote_pkg_installer, all credit goes to them and their...

ps4_remote_pkg_installer-OOSDK This is a OOSDK port of flat's ps4_remote_pkg_installer, all credit goes to them and their amazing work! NOTES Some fun

Miscellaneous

731

General purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVID...

General purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends). Blazing fast, mobile-enabled, asynchronous and optimized for advanced GPU data processing usecases based on Vulkan compute. Backed by the Linux Foundation.

Miscellaneous

An open source machine learning library for performing regression tasks using RVM techniqu...

Introduction neonrvm is an open source machine learning library for performing regression tasks using RVM technique. It is written in C programming la

Miscellaneous

108

Pure C ONNX runtime with zero dependancies for embedded devices

🤖 cONNXr C ONNX Runtime A onnx runtime written in pure C99 with zero dependencies focused on embedded devices. Run inference on your machine learning

Miscellaneous

125

Multi-armed bandit simulation library

BanditLib: a simple multi-armed bandit library .-. .-') ('-. .-') _ _ .-') _ .-') _ .-. .-') \ ( OO )

Miscellaneous

466

oneAPI Data Analytics Library (oneDAL)

Miscellaneous

1.4k

Library for factorization machines

libFM Library for factorization machines web: http://www.libfm.org/ forum: https://groups.google.com/forum/#!forum/libfm Factorization machines (FM) a

Miscellaneous

617

ThunderGBM: Fast GBDTs and Random Forests on GPUs

Documentations | Installation | Parameters | Python (scikit-learn) interface What's new? ThunderGBM won 2019 Best Paper Award from IEEE Transactions o

Miscellaneous

Low dependency（C++11 STL only）, good portability, header-only, deep neural networks for em...

LKYDeepNN LKYDeepNN 可訓練的深度類神經網路 (Deep Neural Network) 函式庫。輕量，核心部份只依賴 C++11 標準函式庫，低相依性、好移植，方便在嵌入式系統上使用。 Class diagram 附有訓練視覺化 demo 程式訓練視覺化程式以 OpenCV

Miscellaneous

115

Colibri core is an NLP tool as well as a C++ and Python library for working with basic lin...

Colibri Core is software to quickly and efficiently count and extract patterns from large corpus data, to extract various statistics on the extracted patterns, and to compute relations between the extracted patterns.

Miscellaneous

Frog is an integration of memory-based natural language processing (NLP) modules developed...

Frog is an integration of memory-based natural language processing (NLP) modules developed for Dutch. All NLP modules are based on Timbl, the Tilburg memory-based learning software package.

Miscellaneous

FoLiA library for C++

Libfolia: FoLiA Library for C++ Libfolia (c) CLS/ILK 2010 - 2022 Centre for Language Studies, Radboud University Nijmegen Induction of Linguistic Know