Rundeck: Inline Script Permission Denied.

Problem: 

bash: /tmp/test.goweekend.ca-dispatch-script.tmp.sh: Permission denied

Cause:

Modern Linux /tmp was mounted with the noexec option which caused this exact error case

Solution:

Change file-copy-destination-dir: to different folder /home/fei 

web.goweekend.ca:

    hostname: 192.168.1.66

    username: fei

    ssh-key-storage-path: keys/rundeckpoc/rundeckpockey

    file-copier: script-copy

    destdir: /home/fei

    file-copy-destination-dir: /home/fei

    script-copy: scp ${file-copy.file} ${node.username}@${node.hostname}:${file-copy.destination}

    script-copy-remote-filepath: ${node.destdir}/${file-copy.filename}

DNS, NSLOOKUP, dig

 nslookup -query=SOA goweekend.ca ns.goweekend.ca

Find you own public ip address:

dig +short myip.opendns.com @resolver1.opendns.com
dig TXT +short o-o.myaddr.l.google.com @ns1.google.com

AWS: Action does not apply to any resource(s) in statement

 

Error: Action does not apply to any resource(s) in statement

The policy cannot be apply to object/container), it must be applied to the contents.

Apache HTTPd log request HEADER

 LoadModule log_forensic_module /usr/lib64/httpd/modules/mod_log_forensic.so 

<IfModule log_forensic_module> 
ForensicLog /var/log/httpd/forensic_log 
</IfModule> 

Rundeck: Define remote node with resources.xml

<project>

<node name="ansible01"

  tags="opensusei, linux"

  osFamily="unix"

  username="rundeck"

  hostname="34.72.107.63"

  ssh-key-storage-path="keys/rundeck/rundeckmac"

  file-copier="ssh-copier"

  ssh-options="-o ConnectTimeout=5000" />

</project>

nmap: find the ips not been used in subnet.

 nmap -v -sP 10.170.107.0/24 -oG - | awk '/Status: Down/{print $2}'

HTTPD 2.4 Location Basic Authentication

1. Customize httpd configuration file
   

         <Location /restrict/contents>

   
   

            Require all denied

   
   

            AuthName "Protected Contents"

            AuthType Basic

            AuthUserFile /www/svn/etc/passwd

   
   

            <RequireAny>

                #Require user google

                Require valid-user

            </RequireAny>

   
   

         </Location>

2. Create password file
   The password is encrypted with MD5, example: 5he#t?+lfRos07zonon7
   

   $ htpasswd /www/svn/etc/passwd google

3. Restart/Reload httpd daemon
   # systemctl reload httpd

4. Encrypt username:password for 'Authorization:' header
   

   $ echo google:5he#t?+lfRos07zonon7 | base64

   Z29vZ2xlOjVoZSN0PytsZlJvczA3em9ub243Cg==

5. Call with curl
   

   curl --location --request POST 'https://www.goweekend.ca/restrict/contents' \

   --header 'X-Authorization-Role: AUTHENTICATED' \

   --header 'Authorization: Basic e3thdXRoLmFjY2Vzcy5rZXl9fTp7e2F1dGguYWNjZXNzLnNlY3JldH19' \
   -H 'Content-Type: application/json' -d '{ "userid":"00019317", "lastName":"Qin"}'

    

Tomcat: The Apache Tomcat Connectors - Reference Guide

 

Reference: https://tomcat.apache.org/connectors-doc/reference/workers.html

worker.list=loadbalancer,jk-status

worker.worker1.port=8008

worker.worker1.host=203.18.21.85

worker.worker1.type=ajp13

worker.worker1.lbfactor=1

worker.worker1.socket_keepalive=1

worker.worker1.socket_timeout=300

worker.worker1.ping_mode=A

worker.worker1.ping_timeout=5000

worker.worker2.port=8008

worker.worker2.host=203.18.22.85

worker.worker2.type=ajp13

worker.worker2.lbfactor=1

worker.worker2.socket_keepalive=1

worker.worker2.socket_timeout=300

worker.worker2.ping_mode=A

worker.worker2.ping_timeout=5000

worker.loadbalancer.type=lb

worker.loadbalancer.balance_workers=worker1,worker2

#worker.loadbalancer.balance_workers=off

worker.loadbalancer.sticky_session=1

worker.jk-status.type=status

Python Development on Mac

$ pip3 install xlwt

$ pip3 install openpyxl

gpg create key pairs

Reference: https://www.howtogeek.com/427982/how-to-encrypt-and-decrypt-files-with-gpg-on-linux/

Create gpg rings on both sides: you and your client
# List existing keys
$ gpg --list-keys
# Generate new key
--full-generate-key might not be available in your environment, in the case, you can use --gen-key instead.
$ gpg --full-generate-key
$ gpg --list-keys
# Export private and public keys
$ gpg --export-secret-key -a "<uid>" > myprivate.key
$ gpg --export --output fei.asc -armor <pub id>
# Export public key in ssh key
$ gpg --export-ssh-key  <pub id>! 
gpg --export-ssh-key  6352600EF7A9D369A20FF584739E316C0B08E9A5!
$ gpg --edit-key "user id"

Task:

$ ansible-galaxy init myrole

Problem: 

ModuleNotFoundError: No module named 'yaml'

Solution:

$ pip3 list pyyaml

$ pip3 install pyyaml

Set environment variable PYTHONUSERBASE

$ export PYTHONUSERBASE=/Users/aniu/Library/Python/3.7

$ python3 -m site --user-site

/Users/aniu/Library/Python/3.7/lib/python/site-packages

Mac to erase ubuntu bootable usb

$ gpt destroy /dev/disk2

gpt destroy: unable to open device '/dev/disk2': Permission denied 

Grant full access to terminal in Security and Privacy:

Format the disk with diskutil

$ diskutil zeroDisk /dev/disk2

Terraform: For and If

Reference: https://www.concurrency.com/blog/july-2019/conditionals-and-for-in-terraform

In below article, if purpose of statement or <condition>?<true>:<false> is to filter, rather than flow control

For

In the newer versions of Terraform >= 0.12, Terraform now supports for expressions. This is mostly used for parsing preexisting lists and maps rather than generating ones. For example, we are able to convert all elements in a list of strings to upper case using this expression.

[for el in var.list : upper(el)]

The For iterates over each element of the list and returns the value of upper(el) for each element in form of a list. We can also use this expression to generate maps.

{for el in var.list : el => upper(el)}

In this case, the original element from list now correspond to their uppercase version.

Lastly, we can include an if statement as a filter in for expressions. Unfortunately, we are not able to use if in logical operations like the ternary operators we used before. The following state will try to return a list of all non-empty elements in their uppercase state.

[for el in var.list : upper(el) if el != ""]

All in all, the use of for expressions are much more limited compared to imperative programming languages. The two expressions introduced in this section, for and if are exclusively used for lists rather than flow control of the program. That is to be expected since Terraform follows the declarative programming paradigm by describing the logic of the computation rather than the control flow- describing what the program must accomplish rather than how that is accomplished.

Install Terraform on MacBook

 $ asdf plugin-add terraform https://github.com/Banno/asdf-hashicorp.git

List versions of terraform

asdf list-all terraform

You can install the different version now, or when it is needed.

$ asdf install terraform 0.13.4

$ asdf install terraform 0.12.29

$ asdf global terraform 0.13.4

$ cat .tool-versions

kubectl 1.16.10

helm 3.1.1

terraform 0.13.4

$ terraform --version

Terraform v0.13.4

To integrate terraform with python, please install terraform_external_data

$ pip3 install terraform_external_data

Terraform

 

External Data Source

https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source

https://www.terraform.io/docs/configuration/syntax-json.html

https://stackoverflow.com/questions/63296646/iterate-through-list-of-dictionaries-to-create-terraform-resources-with-differen

variable query_dict {
    default = [
        {
            name = "query1"
            workgroup = "bar"
            query = "SELECT * FROM foo"   
        },
        {
            name = "query2"
            workgroup = "bar"
            query = "SELECT * FROM baz"   
        }
    ]
}

resource "aws_athena_named_query" "olap" {

  for_each = {for idx, query in var.query_dict: idx => query}
  
  name = each.value.name
  query = each.value.query
  database = "test"
  workgroup = each.value.workgroup
} 

In the above you create map with idx as a key:

{
  "0" = {
    "name" = "query1"
    "query" = "SELECT * FROM foo"
    "workgroup" = "bar"
  }
  "1" = {
    "name" = "query2"
    "query" = "SELECT * FROM baz"
    "workgroup" = "bar"
  }
}

https://github.com/hashicorp/terraform-provider-external/issues?q=is%3Aissue+is%3Aopen+list

https://technology.amis.nl/2020/02/24/set-terraform-resource-properties-from-an-element-in-a-list-retrieved-by-a-data-source-using-a-local-value/

https://stackoverflow.com/questions/51946982/dependency-on-local-file-creation

https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source

https://paulbrice.com/terraform/python/2017/12/18/external-provider-terraform.html

Loop and if

https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9

JSON Path Finder

 JSON Path

https://support.smartbear.com/alertsite/docs/monitors/api/endpoint/jsonpath.html

Rundeck on Kubernetes

 https://github.com/helm/charts/blob/master/incubator/rundeck/README.md

Enterprise Edition

https://github.com/rundeck/docker-zoo/tree/master/kubernetes

Rancher: Node disk is running full within 24 hours

 

Problem: Node disk is running full within 24 hours

Cause: predict_linear(node_filesystem_free_bytes{mountpoint!~"^/etc/(?:resolv.conf|hosts|hostname)$"}[6h], 3600 * 24) < 0

Solution:

https://forums.rancher.com/t/rancher-monitoring-is-falsely-alerting-on-var-lib-lxcfs-running-out-of-disk-space/14513

predict_linear(node_filesystem_files_free{mountpoint!~"^/(?:etc/resolv.conf|etc/hosts|etc/hostname|var/lib/lxcfs)$"}[6h], 3600 * 24)

Kubernetes Pod Readiness Probe

Readiness Probe: 

1. httpGet 

readinessProbe:

  httpGet:

    path: /api/ready

    port: 8080

2. tcpSocket:

readinessProbe:
  topSocket:
    port:3306

3. command

readinessProbe:

  exec:

    command:

       - cat

      -/app/is_ready

Liveness Probe:

 1. httpGet 

livenessProbe:
  httpGet:
    path: /api/ready
    port: 8080 
  initialDelaySeconds: 10 
  periodSeconds: 5 
  failureThreshold: 8 

 

 

2. tcpSocket:

livenessProbe:
  topSocket:
    port:3306

3. command

livenessProbe:

  exec:

    command:

       - cat

      -/app/is_ready

 

Helm to Deploy RabbitMQ Cluster

Reference: https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq

$ helm install rabbitmq-fei-01  --namespace rabbitmq \

  --set auth.username=admin,auth.password=think4me,auth.erlangCookie=mysecretcookie,persistence.enabled=false \

    bitnami/rabbitmq

$ RABBITMQ_PASSWORD="$(kubectl get secret rabbitmq-fei-01-rabbitmq -o jsonpath='{.data.rabbitmq-$ password}' | base64 --decode)"

$ RABBITMQ_ERLANG_COOKIE="$(kubectl get secret rabbitmq-fei-01-rabbitmq -o jsonpath='{.data.rabbitmq-erlang-cookie}' | base64 --decode)"

$ helm upgrade rabbitmq-fei-01 bitnami/rabbitmq \

  --set replicaCount=4 \

  --set persistence.enabled=false \

  --set auth.password="$RABBITMQ_PASSWORD" \

  --set auth.erlangCookie="$RABBITMQ_ERLANG_COOKIE" \

  --debug

$ kubectl port-forward --namespace rabbitmq svc/rabbitmq-cluster 15672:15672

Kubectl Cheatsheet

$ kubectl get svc --all-namespaces  | awk '{{gsub(/,/, ";", $6)}; a = ""; for (i = 1 ; i <= NF -1 ; i++) a = a $i "," ; print substr(a, 1, length(a)-1)}'

HAProxy Config Sample

 defaults

        timeout connect 5000ms

        timeout client 50000ms

        timeout server 50000ms

# This listener is for stats only no proxying

listen stats 192.168.88.10:80

    mode http

    stats enable

    stats hide-version

    stats realm Haproxy\ Statistics

    stats uri /

    stats auth admin:loyalty # Change this to your own username and password!

# Break Line

listen breakline 192.168.88.13:80

    mode tcp

    balance roundrobin

    # check every 5 seconds.  Take down/up after two failed attempts.

    server primary   10.156.68.6:80 weight 1 check port 80 inter 5s rise 2 fall 3

    server secondary 138.16.222.233:80 backup weight 1 check port 80 inter 5s rise 2 fall 3

Linux: PID file /var/run/sshd.pid not readable (yet?) after start.

 Problem: 

PID file /var/run/sshd.pid not readable (yet?) after start.

Red Hat Enterprise Linux Server release 7.0 (Maipo)

CentOS Linux release 7.3.1611 (Core) 

Reference:

https://bugzilla.redhat.com/show_bug.cgi?id=1415218

Kubernetes: RabbitMQ

https://github.com/bitnami/charts

https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq

https://github.com/rabbitmq/diy-kubernetes-examples.git

https://www.rabbitmq.com/blog/2020/08/10/deploying-rabbitmq-to-kubernetes-whats-involved/

$ helm repo add bitnami https://charts.bitnami.com/bitnami

$ helm search repo bitnami

$ helm install my-release bitnami/<chart>

$ helm install art-rmq --namespace rabbitmq-qa \

  --set auth.username=admin,auth.password=changeme,auth.erlangCookie=secretcookie \

    bitnami/rabbitmq

$ helm install art-rmq01 --namespace rabbitmq-qa \

  --set auth.username=admin,auth.password=changeme,auth.erlangCookie=secretcookie \

    bitnami/rabbitmq

$ helm install art-rmq02 --namespace rabbitmq-qa \

  --set auth.username=admin,auth.password=changeme,auth.erlangCookie=secretcookie,metrics.serviceMonitor.namespace=rabbitmq-qa,metrics.prometheusRule.namespace=rabbitmq-qa \

    bitnami/rabbitmq

To Access the RabbitMQ Management interface:

    echo "URL : http://127.0.0.1:15673/"

    kubectl port-forward --namespace default svc/art-rmq01-rabbitmq 15673:15672

rabbit@art-rmq-rabbitmq-0.art-rmq-rabbitmq-headless.default.svc.cluster.local

$ kubectl exec -it /art-rmq01-rabbitmq-0 bash

$ rabbitmqctl status

$ rabbitmqctl stop_app

$ rabbitmqctl join_cluster --ram rabbit@art-rmq-rabbitmq-0.art-rmq-rabbitmq-headless.default.svc.cluster.local

$ rabbitmqctl start_app

$ rabbitmqctl cluster_status