Kubectl and Nushell: Moving files the hardest way possible for fun

Yep. Why not?

For the uninitiated, kubernetes has some of the best tooling out there for platforms and operationalizing what you need. One of those tools is its official CLI: kubectl (Kube CTL for me, kube-cuttle for the weird ones... No judgement here).

Kubectl is great for having access to powerful APIs and tools that can simplify most of what you need to do in your maintenance in your cluster. I do have my favorites:

1. kubectl edit for editing your cluster desired states "the dirty way" directly in your cluster.
2. kubectl get nodes -o wide for checking up on the cluster and making sure my nodes are behaving properly.
3. kubectl top pods: the good old metrics fellow with easy to parse and see values of how much memory and CPU your pods are using.
4. Many others that I couldn't possibly fit here.

Let's overlook for a moment here that you can also install plugins into this behemoth of a CLI using krew; we are here to talk about files, volumes and moving them using NuShell!

So here is the thing: I have a new cluster that I need to migrate some things to. This new cluster lives in a different data centre and it has its own construct that defines volumes, load balancers, etc. But my files, my definitions, everything is set for this other cluster that I currently have everything running.

Once I was done moving the definitions of the deployment, and had my helm release set with my app running on the other side, how do I move my files from the running container in my previous cluster to the new one?

I've omitted one of the coolest tools of kubectl: kubectl cp. It literally does a copy of files and folders from your local machine to the cluster pod and vice versa. So why not just use that? Well, I've tried it! But here is the thing: you might not be able to use it for everything. You depend on the container you have running to use tar, and you cannot have a process locking your files that could potentially corrupt your data.

I knew what I had to do:

1. Stop the deployment that manages the files in that persistent volume
2. Create a new pod attaching the persistent volume to a place where it is not used by any process.
3. Copy the files using kubectl cp to my local machine.
4. Stop the new pod.
5. Scale back up the deployment.
6. Repeat for the new deployment in the other cluster, but just pushing the files instead.

Sounds easy, right? Almost repeatable. I mean, it should be simple enough to do it by hand. But God I knew this HAD to become something more. I wouldn't repeat this the 167 times for each persistent volume I had in my cluster!

First, I had to think how fast I wanted to churn something out. I realistically wanted only to use about 0.5-1.5 hours to do this, so I couldn't dream too much about it. Second, I needed it to run consistently in my environment, but I never really intended to push this somewhere else; I am sure someone already wrote something that potentially had a better thought process behind it.

Alright, that's settled! Let's write a quick NuShell script to figure this out:

def main [--deployment (-d): string, --volume (-v): string, --path (-p): path, --namespace (-n): string, --files, --wait (-w): duration = 2sec, --wait-times: int = 10] {
  print $"Moving ($path) to the volume ($volume) from the deployment ($deployment)"
  mut $deployment_object = {}
  if ($namespace != null) {
    $deployment_object = (kubectl get deployment -n $namespace -o yaml $deployment | from yaml)
    if ($deployment_object | is-empty) {
      error make {msg: $"Deployment ($deployment) not found in the namespace ($namespace)", label: {text: "couldn't find this deployment", span: (metadata $deployment).span}, help: "Check the name of the deployment and if the namespace is right"}
    }
  } else {
    $deployment_object = (kubectl get deployment -o yaml $deployment | from yaml)
    if ($deployment_object | is-empty) {
      error make {msg: $"Deployment ($deployment) not found in the current namespace", label: {text: "couldn't find this deployment", span: (metadata $deployment).span}, help: "Check the name of the deployment and if the namespace is right"}
    }
  }
  let $dobj = $deployment_object
  print "searching for the volume in the deployment object"
  let $volume_information = ($deployment_object | get spec.template.spec.volumes | where name == $volume)

  if ($volume_information | is-empty) {
    error make {msg: $"Volume ($volume) was not found in the deployment ($deployment)", label: {text: "this volume was not found in the given deployment", span: (metadata $volume).span}, help: "Check the name of the volumes in the deployment"}
  }

  let $volume_pvc_name = $volume_information.0.persistentVolumeClaim.claimName

  let temp_pod = $"
apiVersion: v1
kind: Pod
metadata:
  name: nuguish-volupload-($deployment)
spec:
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - mountPath: /data
      name: volume
  volumes:
  - name: volume
    persistentVolumeClaim:
      claimName: ($volume_pvc_name)
  "
  print $"scaling down the deployment ($deployment)"
  kubectl scale deployment $deployment --replicas 0
  print $"creating new temporary pod for moving data: nuguish-volupload-($deployment)"
  $temp_pod | kubectl apply -f -
  print $"waiting for pod to be running..."
  mut $not_running = (kubectl get pod $"nuguish-volupload-($deployment)" -o yaml | from yaml | get status.phase) != "Running"
  mut $counter = 0
  let $limit = $wait_times
  mut $error = false
  while $not_running {
    sleep $wait
    $counter += 1
    $not_running = (kubectl get pod $"nuguish-volupload-($deployment)" -o yaml | from yaml | get status.phase) != "Running"
    if ($counter >= $limit) {
      $error = true
      break
    }
  }
  if ($error) {
    print $"There was an error while trying to get the temporary pod running, reverting everything back..."
    cleanup $deployment_object $namespace
    error make {msg: "There was an issue while trying to create the temporary pod. Please verify your kubernetes events", help: "You can view your kuberentes events by running 'kubectl get events'"}
  }

  print $"Moving data from ($path) to the created temp container..."
  try {
    if ($files) {
      ls $path | each {|ea|
        print $"Copying ($ea.name)"
        kubectl cp $ea.name $"nuguish-volupload-($deployment):/data"
      }
    } else {
      kubectl cp $path $"nuguish-volupload-($deployment):/data"
    }
  } catch {|err|
    cleanup $dobj $namespace
    error make {msg: "There was an error while copying the files over.", }
  }

  print "Everything completed! Reverting things to normal"
  cleanup $deployment_object $namespace
}


def cleanup [deployment_object: record, namespace?: string] {
  let $deployment = ($deployment_object | get metadata.name)
  if ($namespace != null) {
    kubectl scale deployment  -n $namespace --replicas ($deployment_object | get spec.replicas)
    kubectl delete pod -n $namespace $"nuguish-volupload-($deployment)"
  } else {
    kubectl scale deployment $deployment --replicas ($deployment_object | get spec.replicas)
    kubectl delete pod $"nuguish-volupload-($deployment)"
  }
}

It's a nice thing we can rely on fancy error messages from the error make command and it's auxiliary objects for knowing exactly where the error comes from:

.config/nushell/scripts on  main [!?]
nu ⌁ : ./kcpvolupload.nu -d 'kuma' -n kube-system
Moving  to the volume  from the deployment kuma
Error from server (NotFound): deployments.apps "kuma" not found
Error:   × Deployment kuma not found in the namespace kube-system
   ╭─[<commandline>:1:6]
 1 │ main -d kuma -n kube-system
   ·      ───┬───
   ·         ╰── couldn't find this deployment
   ╰────
  help: Check the name of the deployment and if the namespace is right

Pretty cool, isn't it? Goes without saying that you don't really need this, but if you want to check this out, I will be uploading this script as a function soon in the nuguish module for NuShell.