docs: deepen Kubernetes production safety guidance

kubernetes/references/observability.md

@@ -25,9 +25,16 @@ Events are short-lived diagnostic evidence and should be correlated by involved
 
 Use API-server deprecation metrics to locate deprecated API use. Use controller/node/workload metrics for health and capacity. Prometheus Operator, kube-state-metrics, and provider monitoring are ecosystem overlays; route to their official docs rather than pretending they are Kubernetes core.
 
+## Audit policy boundary
+
+Audit logging requires an API-server policy and a backend. The policy levels are `None`, `Metadata`, `Request`, and `RequestResponse`; the more detailed levels increase sensitivity and cost. Start from the minimum evidence needed, redact or restrict request bodies, and treat audit-policy changes as control-plane changes rather than ordinary workload configuration.
+
+Audit records have `RequestReceived`, `ResponseStarted`, `ResponseComplete`, and `Panic` stages. The policy is ordered: the first matching rule wins, and an empty rules list is invalid. Backends are currently log files or webhooks. Audit increases API-server memory use, so measure and bound high-volume rules.
+
 ## Sources
 
 - https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/
 - https://kubernetes.io/docs/concepts/cluster-administration/logging/
 - https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
 - https://kubernetes.io/docs/reference/instrumentation/metrics/
+- https://kubernetes.io/docs/concepts/security/secrets-good-practices/

kubernetes/references/scheduling-scaling-storage.md

@@ -17,6 +17,17 @@ Requests influence scheduling. CPU limits throttle; memory-limit violations can
 
 HPA needs a scalable target and a functioning metrics API. Missing resource requests can make utilization undefined. Check HPA conditions, target reference, current/desired metrics, and `metrics.k8s.io` availability before changing replicas.
 
+HPA, VPA, and Cluster Autoscaler are separate control loops. HPA changes replica count, VPA changes resource recommendations or requests, and Cluster Autoscaler changes node capacity. Combining them without a signal/ownership design can cause oscillation, request inflation, evictions, or slow reaction. Treat VPA as recommendation-only until its eviction and stateful-workload behavior is understood; treat Cluster Autoscaler as a provider/distribution overlay.
+
+```sh
+kubectl get hpa -A
+kubectl describe hpa NAME -n NAMESPACE
+kubectl top pods -n NAMESPACE
+kubectl get --raw='/apis/metrics.k8s.io/v1beta1/namespaces/NAMESPACE/pods'
+```
+
+Do not diagnose “HPA is not scaling” from replica count alone. Check metrics availability, target requests, HPA conditions, stabilization/behavior policies, pending Pods, and node-group capacity.
+
 ## Persistent storage
 
 Separate PV, PVC, StorageClass, CSI driver, attachment/mount, and application filesystem evidence:
@@ -38,3 +49,5 @@ A Pending PVC may mean no matching class/capacity. A Terminating PVC may be prot
 - https://kubernetes.io/docs/concepts/storage/volumes/
 - https://kubernetes.io/docs/concepts/storage/persistent-volumes/
 - https://kubernetes.io/docs/concepts/storage/storage-classes/
+- https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
+- https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/

kubernetes/references/security-and-policy.md

@@ -25,6 +25,32 @@ Admission webhooks can block or mutate every matching API request. Enumerate web
 
 NetworkPolicy requires an enforcing plugin. Managed providers add identity layers: EKS IRSA/Pod Identity, AKS Workload ID/OIDC/Azure RBAC, and GKE Workload Identity Federation. These are not portable Kubernetes instructions.
 
+## Privilege-escalation checks
+
+Treat these permissions as high risk during RBAC review: `escalate`, `bind`, `impersonate`, `nodes/proxy`, `serviceaccounts/token`, CSR approval, admission webhook configuration, and broad access to Secrets. `list` and `watch` on Secrets can expose their contents. Creating workloads can indirectly grant access to mounted Secrets and service-account credentials. Review `system:masters` separately because it bypasses ordinary RBAC rules.
+
+```sh
+kubectl auth can-i --list --as=system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT -n NAMESPACE
+kubectl get clusterrolebindings -o json
+kubectl auth reconcile -f rbac.yaml --dry-run=client
+```
+
+Never broaden a RoleBinding merely to make a failing workload work. Identify the denied verb/resource, grant the smallest namespace-scoped permission, and verify with `auth can-i`.
+
+## PSA rollout
+
+Pod Security Admission is namespace-label driven. A safer migration is `warn` and `audit` first, then `enforce` after violations are remediated. Pin the profile version only after checking the target cluster’s supported profile version.
+
+```yaml
+pod-security.kubernetes.io/warn: restricted
+pod-security.kubernetes.io/audit: restricted
+# Add enforce only after reviewing violations.
+```
+
+## Secrets and image supply chain
+
+Kubernetes Secrets are API objects, not a complete secrets-management or encryption-at-rest strategy. Secret values are base64-encoded, not encrypted, unless encryption at rest is configured. Avoid putting secret values in manifests, shell history, logs, or agent output. Restrict `get`, `list`, and `watch` access, disable automatic service-account token mounting when a workload does not need the API, and use short-lived workload identity where the provider supports it. External Secret Store CSI providers and admission/image-signature policy are ecosystem overlays that require separate validation. Verify image digest, registry trust, architecture, pull credentials, and admission policy separately.
+
 ## Sources
 
 - https://kubernetes.io/docs/reference/access-authn-authz/rbac/
@@ -32,6 +58,8 @@ NetworkPolicy requires an enforcing plugin. Managed providers add identity layer
 - https://kubernetes.io/docs/concepts/security/pod-security-admission/
 - https://kubernetes.io/docs/concepts/cluster-administration/admission-webhooks-good-practices/
 - https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
+- https://kubernetes.io/docs/concepts/security/rbac-good-practices/
+- https://kubernetes.io/docs/concepts/security/secrets-good-practices/
 - https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html
 - https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
 - https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity