diff --git a/kubernetes/references/observability.md b/kubernetes/references/observability.md index dc82590..3311f47 100644 --- a/kubernetes/references/observability.md +++ b/kubernetes/references/observability.md @@ -25,9 +25,16 @@ Events are short-lived diagnostic evidence and should be correlated by involved Use API-server deprecation metrics to locate deprecated API use. Use controller/node/workload metrics for health and capacity. Prometheus Operator, kube-state-metrics, and provider monitoring are ecosystem overlays; route to their official docs rather than pretending they are Kubernetes core. +## Audit policy boundary + +Audit logging requires an API-server policy and a backend. The policy levels are `None`, `Metadata`, `Request`, and `RequestResponse`; the more detailed levels increase sensitivity and cost. Start from the minimum evidence needed, redact or restrict request bodies, and treat audit-policy changes as control-plane changes rather than ordinary workload configuration. + +Audit records have `RequestReceived`, `ResponseStarted`, `ResponseComplete`, and `Panic` stages. The policy is ordered: the first matching rule wins, and an empty rules list is invalid. Backends are currently log files or webhooks. Audit increases API-server memory use, so measure and bound high-volume rules. + ## Sources - https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/ - https://kubernetes.io/docs/concepts/cluster-administration/logging/ - https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/ - https://kubernetes.io/docs/reference/instrumentation/metrics/ +- https://kubernetes.io/docs/concepts/security/secrets-good-practices/ diff --git a/kubernetes/references/scheduling-scaling-storage.md b/kubernetes/references/scheduling-scaling-storage.md index a2bda37..cefdebf 100644 --- a/kubernetes/references/scheduling-scaling-storage.md +++ b/kubernetes/references/scheduling-scaling-storage.md @@ -17,6 +17,17 @@ Requests influence scheduling. CPU limits throttle; memory-limit violations can HPA needs a scalable target and a functioning metrics API. Missing resource requests can make utilization undefined. Check HPA conditions, target reference, current/desired metrics, and `metrics.k8s.io` availability before changing replicas. +HPA, VPA, and Cluster Autoscaler are separate control loops. HPA changes replica count, VPA changes resource recommendations or requests, and Cluster Autoscaler changes node capacity. Combining them without a signal/ownership design can cause oscillation, request inflation, evictions, or slow reaction. Treat VPA as recommendation-only until its eviction and stateful-workload behavior is understood; treat Cluster Autoscaler as a provider/distribution overlay. + +```sh +kubectl get hpa -A +kubectl describe hpa NAME -n NAMESPACE +kubectl top pods -n NAMESPACE +kubectl get --raw='/apis/metrics.k8s.io/v1beta1/namespaces/NAMESPACE/pods' +``` + +Do not diagnose “HPA is not scaling” from replica count alone. Check metrics availability, target requests, HPA conditions, stabilization/behavior policies, pending Pods, and node-group capacity. + ## Persistent storage Separate PV, PVC, StorageClass, CSI driver, attachment/mount, and application filesystem evidence: @@ -38,3 +49,5 @@ A Pending PVC may mean no matching class/capacity. A Terminating PVC may be prot - https://kubernetes.io/docs/concepts/storage/volumes/ - https://kubernetes.io/docs/concepts/storage/persistent-volumes/ - https://kubernetes.io/docs/concepts/storage/storage-classes/ +- https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ +- https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/ diff --git a/kubernetes/references/security-and-policy.md b/kubernetes/references/security-and-policy.md index 38a2d48..6ebc747 100644 --- a/kubernetes/references/security-and-policy.md +++ b/kubernetes/references/security-and-policy.md @@ -25,6 +25,32 @@ Admission webhooks can block or mutate every matching API request. Enumerate web NetworkPolicy requires an enforcing plugin. Managed providers add identity layers: EKS IRSA/Pod Identity, AKS Workload ID/OIDC/Azure RBAC, and GKE Workload Identity Federation. These are not portable Kubernetes instructions. +## Privilege-escalation checks + +Treat these permissions as high risk during RBAC review: `escalate`, `bind`, `impersonate`, `nodes/proxy`, `serviceaccounts/token`, CSR approval, admission webhook configuration, and broad access to Secrets. `list` and `watch` on Secrets can expose their contents. Creating workloads can indirectly grant access to mounted Secrets and service-account credentials. Review `system:masters` separately because it bypasses ordinary RBAC rules. + +```sh +kubectl auth can-i --list --as=system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT -n NAMESPACE +kubectl get clusterrolebindings -o json +kubectl auth reconcile -f rbac.yaml --dry-run=client +``` + +Never broaden a RoleBinding merely to make a failing workload work. Identify the denied verb/resource, grant the smallest namespace-scoped permission, and verify with `auth can-i`. + +## PSA rollout + +Pod Security Admission is namespace-label driven. A safer migration is `warn` and `audit` first, then `enforce` after violations are remediated. Pin the profile version only after checking the target cluster’s supported profile version. + +```yaml +pod-security.kubernetes.io/warn: restricted +pod-security.kubernetes.io/audit: restricted +# Add enforce only after reviewing violations. +``` + +## Secrets and image supply chain + +Kubernetes Secrets are API objects, not a complete secrets-management or encryption-at-rest strategy. Secret values are base64-encoded, not encrypted, unless encryption at rest is configured. Avoid putting secret values in manifests, shell history, logs, or agent output. Restrict `get`, `list`, and `watch` access, disable automatic service-account token mounting when a workload does not need the API, and use short-lived workload identity where the provider supports it. External Secret Store CSI providers and admission/image-signature policy are ecosystem overlays that require separate validation. Verify image digest, registry trust, architecture, pull credentials, and admission policy separately. + ## Sources - https://kubernetes.io/docs/reference/access-authn-authz/rbac/ @@ -32,6 +58,8 @@ NetworkPolicy requires an enforcing plugin. Managed providers add identity layer - https://kubernetes.io/docs/concepts/security/pod-security-admission/ - https://kubernetes.io/docs/concepts/cluster-administration/admission-webhooks-good-practices/ - https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ +- https://kubernetes.io/docs/concepts/security/rbac-good-practices/ +- https://kubernetes.io/docs/concepts/security/secrets-good-practices/ - https://docs.aws.amazon.com/eks/latest/userguide/service-accounts.html - https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview - https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity