unknown service runtime.v1alpha2.ImageService

Problem:

# kubeadm config images pull

failed to pull image "registry.k8s.io/kube-apiserver:v1.25.4": output: E1126 17:51:40.858868   34828 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService" image="registry.k8s.io/kube-apiserver:v1.25.4"

time="2022-11-26T17:51:40-05:00" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"

, error: exit status 1

To see the stack trace of this error execute with --v=5 or higher

Reason:

containerd cgroup driver is not equal with kubelet cgroup driver.

/etc/containerd/config.toml

### disabled_plugins = ["cri"]

Git: Checkout Specific Folder in a Repository

Reference: https://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository

Sparse Checkout is enabled in git 1.7.0

Check out the git version:

# git version

git version 1.8.3.1

Create a folder for the repository:

# mkdir myrepo

Initiate the folder for git

# git init

Disable ssl verify if you are using self-signed certificate:

#  git config http.sslVerify "false"

Fetch all objects from Remote

#  git remote add -f origin https://bitbucket.com/scm/iu/myrepo.git

Enable sparse checkout

#  git config core.sparseCheckout true

Append the subfolder to sparse-checkout

#  echo "tcsfs/helpdesk" >> .git/info/sparse-checkout

Pull your code: 

#  git pull origin master

Docker Client and Containerd Behind Proxy in CentOS 8

# docker pull kubeimage/kube-apiserver:v1.15.0

Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 17.15.75.8:53: no such host

 

vi /usr/lib/systemd/system/docker.service with below entries

[Service]
Environment=HTTP_PROXY=http://8.8.8.8:8888
Environment=HTTPS_PROXY=http://8.8.8.8:8888

systemctl daemon-reload
systemctl restart docker

# kubeadm config images pull --v=10




I1031 22:12:20.527118   22397 version.go:187] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
exit status 1
output: E1031 22:12:20.843791   22419 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.25.3\": failed to resolve reference \"registry.k8s.io/kube-apiserver:v1.25.3\": failed to do request: Head \"https://registry.k8s.io/v2/kube-apiserver/manifests/v1.25.3\": dial tcp: lookup registry.k8s.io on 137.15.210.9:53: no such host" image="registry.k8s.io/kube-apiserver:v1.25.3"
time="2022-10-31T22:12:20-04:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/kube-apiserver:v1.25.3\": failed to resolve reference \"registry.k8s.io/kube-apiserver:v1.25.3\": failed to do request: Head \"https://registry.k8s.io/v2/kube-apiserver/manifests/v1.25.3\": dial tcp: lookup registry.k8s.io on 137.15.210.9:53: no such host"
, error

Add below entries into /usr/lib/systemd/system/containerd.service under section [Service]

[Service]
Environment=HTTP_PROXY=http://8.8.8.8:8888
Environment=HTTPS_PROXY=http://8.8.8.8:8888

systemctl daemon-reload
systemctl restart containerd

Install Minikube with Docker Engine

Install docker engine

```

yum install -y yum-utils
yum-config-manager     --add-repo     https://download.docker.com/linux/centos/docker-ce.repo
yum remove docker
yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Install minikube

https://minikube.sigs.k8s.io/docs/start/

```

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

Install kubectl

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

Set environment variables if the minikube runs behind proxy

export HTTP_PROXY=http://<proxy hostname:port>
export HTTPS_PROXY=https://<proxy hostname:port>
export NO_PROXY=localhost,127.0.0.1,192.168.49.0/24

or run below command on fly
```

export no_proxy=$no_proxy,$(minikube ip)

Open firewall rule for port 8443

 ```

firewall-cmd --permanent --add-port=8443/tcp

Start minikube

```

minikube start --driver=docker 

List porfiles

```

minikube profiles list

Delete existing profile

```

minikube delete

Delete existing profile with force

```

minikube delete --purge --all

Get cluster information

```

kubectl cluster-info

mysql-community-server-5.7 Requires: libsasl2.so.2()(64bit)

 

# yum install mysql-community-server

Loaded plugins: langpacks, ulninfo

Resolving Dependencies

--> Running transaction check

---> Package mysql-community-server.x86_64 0:5.7.40-1.el6 will be installed

--> Processing Dependency: mysql-community-client(x86-64) >= 5.7.9 for package: mysql-community-server-5.7.40-1.el6.x86_64

--> Processing Dependency: libsasl2.so.2()(64bit) for package: mysql-community-server-5.7.40-1.el6.x86_64

--> Running transaction check

---> Package mysql-community-client.x86_64 0:5.7.40-1.el6 will be installed

---> Package mysql-community-server.x86_64 0:5.7.40-1.el6 will be installed

--> Processing Dependency: libsasl2.so.2()(64bit) for package: mysql-community-server-5.7.40-1.el6.x86_64

--> Finished Dependency Resolution

Error: Package: mysql-community-server-5.7.40-1.el6.x86_64 (mysql57-community)

           Requires: libsasl2.so.2()(64bit)

 You could try using --skip-broken to work around the problem

 You could try running: rpm -Va --nofiles --nodigest

Cause:

Baseurl in repo config file is mapped to old release.

Solution:

# cat /etc/yum.repos.d/mysql.repo

[mysql57-community]

name=MySQL 5.7 Community Server

## baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/6/$basearch/

baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/7/$basearch/

enabled=1

gpgcheck=0

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

https://blog.fearcat.in/a?ID=00001-3311f896-7ed9-4220-8f2f-0e7ba81f2add

Kubernetes: Setup K8s on your Local

 https://matt-rickard.com/docker-desktop-alternatives

https://cri-o.io/

Releases: https://github.com/cri-o/cri-o/releases

Install cri-o

  dnf module list cri-o
  VERSION=1.18
  dnf module enable cri-o:$VERSION
  dnf install cri-o

export OS=CentOS_8
export VERSION=1.25




curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
yum install cri-o

https://minikube.sigs.k8s.io/docs/start/

Install kubectl

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

Linux partition operations

Reference: https://ismailyenigul.medium.com/resize-xfs-partition-without-lvm-df3ea681c6d3

# lsblk

# dnf install cloud-utils-growpart -y

# growpart /dev/sda 3

CHANGED: partition=3 start=6293504 old: size=35649536 end=41943040 new: size=203421663 end=209715167

# df -h

Filesystem      Size  Used Avail Use% Mounted on

devtmpfs        1.8G     0  1.8G   0% /dev

tmpfs           1.8G     0  1.8G   0% /dev/shm

tmpfs           1.8G  8.5M  1.8G   1% /run

tmpfs           1.8G     0  1.8G   0% /sys/fs/cgroup

/dev/sda3        17G  2.4G   15G  14% /

/dev/sda1      1014M  349M  666M  35% /boot

tmpfs           364M     0  364M   0% /run/user/0

# xfs_growfs /

meta-data=/dev/sda3              isize=512    agcount=4, agsize=1114048 blks

         =                       sectsz=512   attr=2, projid32bit=1

         =                       crc=1        finobt=1, sparse=1, rmapbt=0

         =                       reflink=1

data     =                       bsize=4096   blocks=4456192, imaxpct=25

         =                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0, ftype=1

log      =internal log           bsize=4096   blocks=2560, version=2

         =                       sectsz=512   sunit=0 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

data blocks changed from 4456192 to 25427707

# df -h

Filesystem      Size  Used Avail Use% Mounted on

devtmpfs        1.8G     0  1.8G   0% /dev

tmpfs           1.8G     0  1.8G   0% /dev/shm

tmpfs           1.8G  8.5M  1.8G   1% /run

tmpfs           1.8G     0  1.8G   0% /sys/fs/cgroup

/dev/sda3        97G  3.0G   95G   4% /

/dev/sda1      1014M  349M  666M  35% /boot

tmpfs           364M     0  364M   0% /run/user/0

Git client notes

 Save credential:

git config --global credential.helper store

refs/heads/master is set as the default branch, but this branch does not exist

Cannot evict pod

 evicting pod istio-system/istiod-5c44465489-6rsgz

error when evicting pods/"istiod-5c44465489-6rsgz" -n "istio-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

cri-o cheatsheet

 crictl ps

Error: cannot re-use a name that is still in use

 As investigated, cannot find the possible name duplication in Terraform code. Ended up, I found it is Helm release duplicated.

Update terraform.tfstate in individual workspace

Select source workspace, we use default as example here:

terraform workspace select default

terraform state pull > terraform.tfstate

ls terraform.tfstate

Select the target workspace test

terraform workspace select test

terraform workspace list

terraform state push terraform.tfstate 

terraform state list |grep jolokia2

List the Memory allocated to processes

 

Use below command to find the target process, and take action on it. i.e. restart it.

# ps -o pid,user,%mem,command ax | sort -b -k3 -r

Kill processes with same pattern:

# pkill -9 -f /usr/local/bin/jvm_stats

Jolokia + Hawtio to monitor JVM

https://github.com/hawtio/hawtio 

Git squash/rebase

You may got code conflict when a PR created, In the case you can rebase the branch:

git checkout feature/RealSense

Delete local master in case it is messed up:

git branch -D master

git checkout master

 git pull

 Check in feature branch

git checkout feature/RealSense

git push

Rebase local master 

git rebase master

To abort rebase 

git rebase --abort

Squash the multiple commits (55 limit, if you have more than 55, you can split them) into 1 commit in branch feature/RealSense

git rebase -i HEAD~2

During above process, if you choose to git rebase --skip, you will find differences between your branch and the consolidation branch.

Create a new branch feature/RealSense-squash from current branch  feature/RealSense, run below command, there should not be any difference:

git diff feature/RealSense..eature/RealSense-squash

Once you confirm no difference, you can push to remote organ:

git push --force

Rebase local master then push to origin with force 

git rebase master

git rebase --continue

git push --force

The code conflicts should be cleaned up now. 

kubectl notes

 

Set default namespace

 kubectl config set-context $(kubectl config current-context) --namespace=kube-system

Terraform: An argument or block definition is required here.

 

spec {

template {

spec {

hostAliases [

{

ip = "10.170.0.12"

hostnames = ["db2.dev.valuex.com","foo.dev.valuex.com"]

}

]

}

}

}

Error message:

│ On ../modules/mongo/cluster/main.tf line 171: An argument or block definition is required here. To set an argument, use the equals sign "=" to introduce

│ the argument value.

Correction:

spec {

template {

spec {

hostAliases = [

{

ip = "10.170.0.12"

hostnames = ["db2.dev.valuex.com","foo.dev.valuex.com"]

}

]

}

}

}

Install Ubuntu on Raspberry Pi 4

Enable WiFi 

https://www.linuxbabe.com/command-line/ubuntu-server-16-04-wifi-wpa-supplicant

Install Desktop

sudo apt-get install ubuntu-desktop

Pull Image from OCI Container Registry

 

https://www.oracle.com/webfolder/technetwork/tutorials/obe/oci/oke-and-registry/index.html

** Notes: --docker-password is the auth token rather than user password.

kubectl create secret docker-registry ocirsecret --docker-server=phx.ocir.io --docker-username='ansh81vru1zp/jdoe@acme.com' --docker-password='k]j64r{1sJSSF-;)K8'
--docker-email='jdoe@acme.com'

HostAliases for Kubernetes

https://stackoverflow.com/questions/56390226/how-to-add-extra-hosts-entries-in-helm-charts 

Kubernetes provides a DNS service that all pods get to use. In turn, you can define an ExternalName service that just defines a DNS record. Once you do that, your pods can talk to that service the same way they'd talk to any other Kubernetes service, and reach whatever server.

You could deploy a set of ExternalName services globally. You could do it in a Helm chart too, if you wanted, something like

apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-foo
spec:
  type: ExternalName
  externalName: {{ .Values.fooHostname }}

git migration

 

Migrate one repo-a into another repo-b

git clone </path/to/repo-b>

git checkout -b branch-to-merge-repo-a

git subtree add -P aws ssh://git@git.fei.com:7999/in/aws.git <revision>

git push

Then create pull request and merge the new branch into master

Import file/folder history from one repo-a to repo-b

cd repository

git log --pretty=email --patch-with-stat --reverse --full-index --binary -- cloudformation/opensearch > ~/tmp/opensearch.patch

cd ../another_repository

git am --committer-date-is-author-date < ~/tmp/opensearch.patch

git log --pretty=email --patch-with-stat --full-index --binary --reverse -- client > patch

istio troubleshootin

 

1.   istioctl proxy-status
2.   istioctl proxy-config listener istio-ingressgateway-1-n istio-system
3.   istioctl proxy-config cluster istio-ingressgateway-1 -n istio-system
4.  istioctl proxy-config route istio-ingressgateway-1 -n istio-system

D455 Development Evnironment on Raspberry

 https://github-wiki-see.page/m/acrobotic/Ai_Demos_RPi/wiki/Raspberry-Pi-4-and-Intel-RealSense-D435

 

 

Dynamic Set for Helm Chart

 

cluster_chart_values_override = {

"upgradeOptions.apply" = "Disabled"

"imagePullPolicy" = "Always"

"sharding.enabled" = false

}

dynamic "set" {

for_each = var.cluster_chart_values_override

content {

name = set.key

value = set.value

}

}

Script to grep ip and port number from netstat

Below script is to parse output from netstat -tulnp or ipvsadm -Ln. 

#!/bin/bash

input=$1

vipNportRe="TCP ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}:[0-9]{2,3}).*$"

ipNportRe=".*-> ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}:[0-9]{2,3}).*$"

netstatIpNportRe="^tcp .* 0 ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}:[0-9]{2,3}).*$"

while read line

do

  if [[ $line =~ $ipNportRe ]]; then

    echo "${BASH_REMATCH[1]}"

  elif [[ $line =~ $netstatIpNportRe ]]; then

    echo "${BASH_REMATCH[1]}"

  fi

done < $input

# Get the lines in a.txt but not in b.txt

grep -F -x -v -f b.txt a.txt