[dovholuknf]

Interesting idea. I'm not very well-versed in training models or LLMs or even Jupyter Notebooks, but the comment about port forwarding SSH caught my eye since I work on a free, open source zero-trust overlay network (OpenZiti). I tried to find some information about moonglow under the hood / how it worked but didn't succeed.

If you're interested, you might find embedding OpenZiti into Moonglow a pretty compelling alternative to port forwarding and it might open even crazier ideas once your connectivitiy is embedded into the app. You can find the cheapest compute for people and just connect them to that cheapest compute using your extension... Might be interesting? Anyway, I'd be happy to discuss some time if that sounds neat... Until then, good luck with your launch!

[bblcla]

Cool! We actually don't do port forwarding over SSH, we do it over an ngrok-like solution that we forked/modified. I looked at a few options while we were designing this, including Tailscale and ngrok, but none of them exactly suited our needs, and the pricing would have been prohibitive for something that's a pretty core part of our product.

OpenZiti looks really cool though - I'll take a look!

[williamstein]

Is it possible to use OpenZiti with Runpod? Their execution environment is very locked down, which might make ssh the only option.

[qrkourier]

At a glance, the RunPod's serverless and pod options would probably work well with OpenZiti. I didn't explore their vLLM option.

Using OpenZiti w/ Serverless probably means integrating an OpenZiti SDK with your serverless application. That way, it'll connect to the OpenZiti network every time it spawns.

The SDK option works anywhere you can deploy your application because it doesn't need any sidecar, agent, proxy, etc, so it's definitely the most flexible and I can give you some examples if you mention the language or framework you're using.

The pod option says "container based" so it'll take some investigation to find out if an OpenZiti sidecar or other tunneling proxy is an option. Would you be looking to publish something running in RunPod (the server is in RunPod), or access something elsewhere from a RunPod pod (the client is in RunPod), or both?

[dovholuknf]

I poked at it a bit but there was no free trial period. I know a bunch of people are using OpenZiti and zrok for Jupyter notebooks in general... Here's a blog I saw not long back that might help but I wasn't able to prove/test/try it... (sorry)

https://www.pogs.cafe/software/tunneling-sagemaker-kaggle

[dovholuknf]

I don't actually know. I'll go poke with Runpod for a few and see :)