Easier deployment on local network #119
Labels
enhancement
New feature or request
Comments
|
@hummuscience when you first connect to the remote machine and launch the app, do you see that the other ports assigned are 7502, 7503, etc.? When we run the app in the cloud we actually have to specify usage on those ports for similar security reasons, but the assigned ports are always monotonically increasing from 7501 - it sounds like that's not the case for you though? |
|
This is not the case for me. I get a random bag of ports that doesn't seem to fit any range. From what I understand, the code also currently doesn't fix these ports. Instead its just Streamlit: Tensorboard: Line 265 in edfe0e5 Fiftyone: Line 270 in edfe0e5 label-studio: |
|
You are correct, the ports are not fixed. It appears you can, however, fix these ports in the construction of the lighting works: https://pytorch-lightning.readthedocs.io/en/2.2.5/app/core_api/lightning_work/lightning_work.html#lightningwork It might be worth trying this with a single work first (like LabelStudio) and sending it to a port that you're able to access |
|
I will give this a try and report. Thank you! |
|
I am able to set the host to 0.0.0.0 and fixed ports on all services except for Streamlit. Somehow, PoseApp starts 4 streamlit instances at once. Still getting random ports every time I run the app... Any idea what might be causing this? |
|
I'm not really sure how to fix the ports for Streamlit. There should indeed be 4 different Streamlit instances starting at once:
There are actually 2 more pages that use streamlit (image/video diagnostics) but these are bundled with the LP code instead of the app and so they are launched differently. Can I ask what your use-case is for running the app remotely? If you want to label data and/or look at diagnostics then the app is definitely handy, if you just want to train a model or run inference on a bunch of videos then launching those jobs from the command line may be easier. |
|
I need to extract frames, label data, check diagnostics etc. I trained already many models but need to refine the models and add model labelled frames. And going through DLC all the time it tedious. |
|
@hummuscience I got in touch with the folks at Lightning, according to them the behavior that I see in the studio can be replicated by setting an environment variable in your shell before launching the app. So can you try running and then launching the app? You should see the different tabs launched on ports 7501, 7502, 7503, etc. Please let me know if that works! |
|
Thanks for checking! I tried the fix out and got the following error: Traceback (most recent call last): No matter is I set the variable to 0 or 0.0.0.0, "0.0.0.0". I also don't seem to find a documentation of this environment variable. Any idea where I can find more infos about it? |
|
Okay, we are getting somwhere. I also had to set |
|
I am able to access the 7501 page but I am only shown "lightning is intializing..." briefly and a blank page. |
|
can you share more of the terminal output? The EmptyPage error is not an issue. After that it appears the app is importing the demo dataset, which can indeed take a few minutes. The 'lightning.app.storage.Path' warning is also just a warning, not an error, so I don't think that's an issue. How long was the app blank for? |
|
Here is the whole output: |
|
Developer toosl on the website is showing the following errors: |
|
Hmm what if you change the address back to 127.0.0.1? Is that the default? [Unfortunately we are bumping up against the edge of my network knowledge here] |
|
This line here seems to be the issue: https://github.com/Lightning-AI/pytorch-lightning/blob/ac3f1ee0d32b11feebb5a051d64ed8eea99484ec/src/lightning/app/runners/backends/mp_process.py#L84 |
|
When I check LIGHTNING_CLOUDSPACE_HOST in the actual studio I get |
|
Yeah, I set it to 0.0.0.0 since =0 didnt work |
|
I see. Does switching everything back to 127.0.0.1 work? Was that the default host? |
|
I am creating a clean environment right now and will try from scratch. Something was set somewhere that is confusing me. Will report :) |
|
Somehow I am unable to get back to the original way everything works. I have a feeling there there is some .config file that was changed that always sends things to 0.0.0.0 instead. I think what is happening is that the app is always thinking that its "in the cloud" end uses this logic to create the project page (which in my case is always http://7503-<LIGHTNING_CLOUDSPACE_HOST> which seems to be a valid address in the cloud but not in a local network |
|
So, here is the summary: If I If I I will link this issue in the lightning-pytorch repo and see if someone has an idea... |
|
So, I got the main page working but its not pretty. I had to change the lightning pytorch code where the weird URL is generated and hard-code the machines hostname into it. like so: |
|
Oof that's not pretty. Did this work for the other pages as well, or just the project management page? |
|
The project management page is the only one that gets this weird URL, all the others get typical 0.0.0.0:port URLs Trying out the app today and will report. |
|
The username and password for label-studio are not working. When I try a random set of username and password I get the following: This is the output in the terminal when I attemp to login with the correct credentials: |
|
Had to set |
|
interesting - ok I'll test this in the lightning studio and see if everything still works fine. If so I can update the repo |
themattinthehatt
added a commit
that referenced
this issue
Dec 8, 2024
|
@hummuscience has this issue been fully addressed now? There is still the issue with the way lightning sets the ports, but I want to make sure that everything else is fine. |
|
Yes. From my side I would say it is solved. The "fix" is not pretty and I wonder if there will be another way to get this working. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I managed to get a linux machine with a GPU to get the Pose App running.
I need to be able to access the App from the network (the machine is not generally accessible). So I tried to open up the appropriate port (7501) on the machine. It seems to have worked, as I see the browser tab change to "lightning.ai" and the tab going straight to
http://<server-ip>:7501/view/Manage Project.But now I get errors that other ports can't be accessed (localhost:44429 for example).
If I understand this correctly, these are the ports on which the multiple servers required by the Pose App are running. two streamlit servers, one label-studio and one fiftyone. These ports are also currently not fixed and change every time I re-start the app.
Opening up all the ports on the machine is not an option for our IT (too much attack surface).
I am currently thinking of editing the App code and hard-code the $host and $port variables. This is a temporary solution, as it will probably be overwritten or needs to be manually adjusted every time there is a Pose App update.
Is there an easier way of doing so?
The text was updated successfully, but these errors were encountered: