OpenShift Cheatsheet

Here some helpful OpenShift commands which work (at least) since version >= 4.11

Login

How to get a token: https://oauth-openshift.apps.ocp.example.com/oauth/token/display

You might need it for login or automatization.

$ oc login --token=... --server=https://api.ocp.example.com:6443

Use the token directly against the API:

$ curl -H "Authorization: Bearer $TOKEN" https://api.ocp.example.com:6443/apis/user.openshift.io/v1/users/~"

Login with username/password:

$ oc login -u admin -p password https://api.ocp.example.com:6443

Get console URL:

$ oc whoami --show-console

CLI tool

Enable autocompletion

oc completion bash > /etc/profile.d/oc_completion_bash.sh

Registries

• registry.access.redhat.com (login only)
• registry.redhat.io
• quay.io

Creating

$ skopeo login -u user -p password registry.redhat.io
$ skopeo list-tags docker://docker.io/nginx
$ oc run <mypod-nginx> --image docker://docker.io/nginx:stable-alpine (--env NGINX_VERSION=1.24.1)

$ skopeo inspect (--config) docker://registry.redhat.io/rhel8/httpd-24

Search Images by help of podman:

$ podman search <wordpress>

Create new app

with label and parameters

from template

$ oc new-app (--name mysql-server) -l team=red --template=mysql-persistent -p MYSQL_USER=developer -p MYSQL_PASSWORD=topsecret

from image

$ oc new-app -l team=blue --image registry.redhat.com/rhel9/mysql-80:1 -e MYSQL_ROOT_PASSWORD=redhat -e MYSQL_USER=developer -e MYSQL_PASSWORD=evenmoresecret

Set environment variables afterwards

oc set env deployment/mariadb MARIADB_DATABASE=wikidb
oc set env deployment/mariadb MARIADB_USER=mediawiki
oc set env deployment/mariadb MARIADB_PASSWORD=wikitopsecret
oc set env deployment/mariadb MARIADB_ROOT_PASSWORD=gehheim

(Not recommended for passwords; you'd better set secrets and configmaps, s. below)

Make new app available

Create service:

$ oc expose deployment <mydeployment> --name <service-mynewapp> --port 8080 --target-port 8080

Create route:

$ oc expose service <service-mynewapp> --name <route-to-mynewapp>

Afterwards the app is reachable from outside. Alernative ingress:

$ oc create ingress <ingress-mynewapp> --rule="mynewapp.ocp4.example.de/*=service-mynewapp:8080"

Create Deployment from image

$ oc create deployment demo-pod --port 3306  --image registry.ocp.example.de:8443/rhel9/mysql-80

Problem web server

In some images web servers run on port 80 which leads to permission problems in OpenShift as security context constraints do not allow to run apps on privileged ports

Error message:

(13)Permission denied: AH00072: make_sock: could not bind to address [::]:80 (13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:80

-> either choose an image where port >= 1024 is used
-> or add permissions to the corresponding service account

$ oc get pod <your pod name> -o yaml | grep -i serviceAccountName
   serviceAccountName: default

$ oc adm policy add-scc-to-user anyuid -z default

(when you want to get rid of this setting again you have to edit the annotations field of the deployment and re-create the pod)

$ oc delete pod <your pod name>

Create Job from image

$ oc create job testjob --image registry.ocp.example.de:8443/rhel9/mysql-80 -- /bin/bash -c "create database events; mysql events -e 'source /tmp/dump.sql;'"

Cronjob:

$oc create cronjob mynewjob --image registry.ocp4.example.de:8443/ubi8/ubi:latest --schedule='* * * * 5' -- /bin/bash -c "if [ $(date +%H) -gt 15 ]; then echo 'Hands up, weekend!'; fi"

Check output of job:

$ oc logs job/<name>

Create service from deployment

$ oc expose deployment/helloworld

Create Secret from String

$ oc create secret generic test --from-literal=foo=bar

Watching

Common info

General cluster/resource info:

$ oc cluster-info

Which resources are there?

$ oc api-resources (--namespaced=false)(--api-group=config.openshift.io)(00api-group=)
                 (in|without namespace)(openshift specific)(core-api-group only)

Explain resources:

$ oc explain service

Describe resources:

$ oc describe service

Inspect resources:

$ oc adm inspect deployment XYZ --dest-dir /home/student/inspection

(Attention: control resulting files for secrets, passwords, privatekeys etc. before sending somewhere)

Get all resources:

$ oc get all

(Attention: templates, secrets, configmaps and pvcs will be shown outside resources)

$ oc get template,secret,cm,pvc

List resources in context of another user/serviceaccount:

$ oc get persistentvolumeclaims -n openshift-monitoring --as=system:serviceaccount:openshift-monitoring:default

Resources which are not shown with the "oc get all" command

$ oc api-resources --verbs=list --namespaced -o name | xargs -n 1 oc get --show-kind --ignore-not-found -n mynamespace

Nodes

Get status of all nodes:

$ oc get nodes

Compare allocatable resources vs limits:

$ oc get nodes <nodename> -o jsonpath='{"Allocatable:\n"}{.status.allocatable}{"\n\n"}{"Capacity:\n"}{.status.capacity}{"\n"}'

Get resource consumption:

$ oc adm top nodes

Be careful ! Only the free memory is shown, not the allocatable memory. For a more realistic presentation do:

$ oc adm top nodes --show-capacity

( https://www.redhat.com/en/blog/using-oc-adm-top-to-monitor-memory-usage )

Machines

Show Uptime:

$ oc get machines -A

Get state paused/not paused of machineconfigpool:

$ oc get mcp worker -o jsonpath='{.spec.paused}'

Pods

Get resource consumption of all pods:

$ oc adm top pods -A --sum

Get all pods on a specific node:

$ oc get pods --field-selector spec.nodeName=ocp-abcd1-worker-0 (-l myawesomelabel)

Get only pods from deployment mysql:

$ oc get pods -l deploymentconfig=mysql

Get pods' readinessProbe:

 $ oc get pods -o jsonpath='{item[0].spec.containers[0].readinessProbe}' | jq

Connect to pod and open a shell:

$ oc exec -it <podname> -- /bin/bash

Copy file(s) to pod:

$ oc cp mysqldump.sql mysql-server:/tmp

Other Information

Sort Events by time:

$ oc get events --sort-by=lastTimestamp

Show egress IPs:

$ oc get hostsubnets

Show/edit initial configuration:

$ oc get cm cluster-config-v1 -o yaml -n kube-system
  (edit)

List alerts:

$ oc -n openshift-monitoring exec -ti alertmanager-main-0 -c alertmanager -- amtool alert --alertmanager.url=http://localhost:9093 -o extended
List silences:
$ oc -n openshift-monitoring exec -ti alertmanager-main-0 -c alertmanager -- amtool silence query [alertname=ClusterNotUpgradable] --alertmanager.url=http://localhost:9093

https://cloud.redhat.com/blog/how-to-use-amtool-to-manage-red-hat-advanced-cluster-management-for-kubernetes-alerts

User rights to resources:

$ oc adm policy who-can <verb> <resource>
$ oc adm policy who-can patch machineconfigs

Running

Projects/Namespaces

Switch namespace:

$ oc project <namespace>

quit namespace:

$ oc project -n default

Change resources

Environment variables

Set environment variables on running deployment:

$ oc set env deployment/helloworld MYSQL_USER=user1 MYSQL_PASSWORD=f00bar MYSQL_DATABASE=testdb

Change with patch command

Patch single value of resource:

$ oc patch installplan install-defgh -n openshift-operators-redhat --type merge  --patch '{"spec":{"approved":true}}'

Patch resource by help of a file:

$ oc patch --type=merge mc 99-worker-ssh --patch-file=/tmp/patch_mc-worker-ssh.yaml

Content of patch_mc-worker-ssh.yaml:

spec:
  config:
    passwd:
      users:
      - name: core
        sshAuthorizedKeys:
        - |
          ssh-rsa AAAAB3NzaZ1yc2EAAAADAQABAAABAQDOMsVGOvN3ap+MWr7eqZpBfDLTcmFdKhozJGStwXsTrP6QJYlxwP1ITZH7tPMfD0zkHu+y7XzcPqybwmnK4hPhuzxUl4qXqdTkTUUJjy3eVPk7n3RHHdsI2yS5YnlcySnTvkYAOuMStDDhN1MF6xOwxqXOq6xalzZzt7j/MtcceHxIdB19i0Fp4XYRTfv9p3UTFFkP9DoRnspNI0TtIg8YfzYcHJy/bDhEfi6+t0UBcksUqVWpVY2jX2Nco1qfC+/E2ooWalMzYUsB4ctU4OqiLd5qxmMevn9J+knPVhiWLE41d7dReVHkNyao2HZUH1r6E6B7/n/m0+XS0qJeA0Hh testy@pc01
          ssh-rsa AAABBBCCC0815....QWertzu007Xx foobar@pc02

Attention: former content of sshAuthorizedKeys will be overwritten !

Patch secret with base64 encoded data:
Create yaml file with content:

$ head /tmp/alertmanager.yaml
global:
 resolve_timeout: 5m
 smtp_from: openshift-admin@example.de
 smtp_smarthost: 'loghorst.example.de:25'
 smtp_hello: localhost
 (...)
$ tail /tmp/alertmanager.yaml
(...)
time_intervals:
 - name: work_hours
   time_intervals:
     - weekdays: ["monday:friday"]
       times:
         - start_time: "07:00"
           end_time: "17:00"
       location: Europe/Zurich

$ oc patch secret alertmanager-main -p '{"data": {"config.yaml": "'$(base64 -w0 /tmp/alertmanager.yaml)'"}}'

Examples

Set master/worker to (un)paused:

$ oc patch --type=merge --patch='{"spec":{"paused":false}}' machineconfigpool/{master,worker}

Set maximum number of unavailable workers to 2:

$ oc patch --type=merge --patch='{"spec":{"maxUnavailable":2}}' machineconfigpool/worker

(default=1)

Restart deployment after change

$ oc rollout restart deployment testdeploy

(obsolete:
the deployment resource has no rollout option -> You must patch something before it restarts e.g.:

$ oc patch deployment testdeploy --patch "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"last-restart\":\"`date +'%s'`\"}}}}}"

)

Scaling resources

Scale number of machines/nodes up/down:

$ oc scale --replicas=2 machineset <machineset> -n openshift-machine-api

Draining nodes

Empty node and put it into maintenance mode (e.g. before booting)

$ oc adm cordon <node1> (not necessary wgen you drain it - will be emptied anyway)
$ oc adm drain <node1> --delete-emptydir-data=true --ignore-daemonsets=true

After reboot:

$ oc adm uncordon <node1>

Logging

Watch logs of a certain pod (or container)

$ oc logs <podname> (-c <container>)

Debug pod (e.g. if crashloopbacked):

$ oc debug pod/<podname>

Node logs of systemunit crio:

$ oc adm node-logs master01 -u crio --tail 2

The same of all masters:

$ oc adm node-logs --role master -u crio --tail 2

Liveness/Readiness Probes of all pods in certain timestamp:

$ oc adm node-logs --role worker -u kubelet | egrep -E 'Liveness|Readiness' | grep "Aug 21 11:22"

Space allocation of logging:

$ POD=elasticsearch-cdm-<ID>
$ oc -n openshift-logging exec $POD -c elasticsearch -- es_util --query=_cat/allocation?v\&pretty=true

Watch audit logs:

$ oc adm node-logs --role=master --path=openshift-apiserver/

Watch audit.log from certain node:

$ oc adm node-logs ocp-abcdf-master-0 --path=openshift-apiserver/audit-2023-09-26T14-11-04.448.log

Search string:

$ oc adm node-logs ocp-abcdf-master-0 --path=openshift-apiserver/audit-2023-09-26T14-11-04.448.log | jq 'select(.verb == "delete")'
$ oc adm node-logs ocp-46578-master-1 --path=openshift-apiserver/audit.log | jq 'select(.verb == "delete" and .objectRef.resource != "routes" and .objectRef.resource != "templateinstances" and .objectRef.resource != "rolebindings" )' 

Source:
https://docs.openshift.com/container-platform/4.12/security/audit-log-view.html

Information gathering

https://access.redhat.com/documentation/en-us/openshift_container_platform/4.12/html/support/gathering-cluster-data#support_gathering_data_gathering-cluster-data

Must-gather

$ oc adm must-gather

-> create must-gather.local.XXXXXX

https://docs.openshift.com/container-platform/4.12/cli_reference/openshift_cli/administrator-cli-commands.html#oc-adm-inspect (evtl. delete secrets!)

SOS Report

https://access.redhat.com/solutions/4387261

Inspect

Get information resource-wise and for a certain period:

$ oc adm inspect clusteroperator/kube-apiserver --dest-dir /tmp/kube-apiserver --since 1m

Special cases

Namespace not deletable

Namespace gets stuck in status terminating

Watch out for secrets that are left over and not deletable. Set the finalizer to Null:

$ oc patch secrets $SECRET -n ocp-cluster-iam-entw  -p '{"metadata":{"finalizers":[]}}' --type=merge

Run containers as root

Should only be done as last instance or for temporary tests as attackers could theoretically break out of the containers and become root on the system.

In the deployment add following lines under the "spec" statement:

spec:
  containers:
    securityContext:
      runAsUser: 0

You must give admin privileges to the serviceaccount under which the deployment runs. If nothing is configured it is normally the default user:

# oc project <myproject>
# oc adm policy add-scc-to-user anyuid -z default

App URLs

Kibana

https://kibana-openshift-logging.apps.ocp.example.com/

ArgoCD

https://openshift-gitops-server-openshift-gitops.apps.ocp.example.com

Useful terms

IPI Installer-provisioned infrastructure cluster
Cluster installed by install command; user must only provide some information (which platform, cluster name, network, storage, ...)

UPI User provisioned infrastructure cluster

• DNS and Loadbalancing must already be there
• Installation manually, download ova file (in case of vSphere)
• master created manually
• workers recommended
• *no* keepalived

Advantages:
IPI: installation more simple, using preconfigured features
UPI: more flexibility, no loadbalancer outage during update

Change from IPI -> UPI not possible

You can get more shortcuts by typing:

$ oc api-resources

cm	config map
csv	cluster service version
dc	deploymentconfig
ds	daemonset
ip	installplan
mcp	machineconfigpool
pv	persistent volume
sa	service account
scc	security context constraints
svc	service