Development pipeline as machine learning engineer

Our company integrated a Kubernetes cluster into our network. Our work is mostly research and development so we don’t really need to deploy our models. I wanted to ask what’s the best workflow for using Docker and Kubernetes enviorment for development purposes? I would normally have gpus on a server and create python scripts to train and evaluate models. Can I still be writing and debugging code all within a Docker container tasked to Kubernetes? Should I just develop with Jupter Notebook within the Docker container? Would love to hear what others do for this workflow since I’m so accustomed to working on just a server with GPUs.