Merge in current hosting branch, which is our active server by dcdenu4 · Pull Request #147 · natcap/urban-online-workflow

dcdenu4 · 2025-02-20T15:52:37Z

Mostly set up of an nginx proxy server.

… CORS origin list

…-online-workflow into feature/vm-hosting

…non 443 ports. The endpoint 9000 is handled with a proxy pass to fileserver:9000

…ons endpoint, see comment note

…play with certbot on the image itself.

… caussing https ssl issues, so now 443 just does a proxy pass to localhost:8000

…hey were useful or not. Moving port 8000 and 9000 calls to 443 location proxy_pass redirects. Lots of SSL configuration and setup. Lots of things added to try and work and MixedContent errors and SSL erros.

…nginx. nginx when handling the request drops the trailing slash, which was causing a http redirect, which caused a https / http conflict

… cert issues. Commenting out return calls from GET / POST that were interfering with FastAPI returns, which caused a json error on the frontend.

…cleanup.

…en renewing certs, since all traffic is being redirected to 443.

…seeing an error when trying to dump to a json file. It was related to trying to serialize a float, so I casted to a string.

…because I was seeing a malformed URL request in dev mode. 'localhost/9000scenarios/...'. I'm not sure why this is showing up with localhost and not in production.

davemfish

Just a few misc things that caught my eye.

davemfish · 2025-02-25T13:51:20Z

backend-worker/invest_results.py

    for output, output_path in carbon_outputs.items():
        carbon_results[output] = pygeoprocessing.raster_reduce(
            lambda total, block: total + numpy.sum(block), (output_path, 1), 0)
+        carbon_results[output] = f"{carbon_results[output]}"


Hmm, did something just break that required this? There's a json.dump(carbon_results) below. If there was something un-serializable in that data I wonder if it's a sign of a larger problem?

Yes, I was seeing some breaking behavior in the logs of invest_results.py I hadn't before (that I can remember). It was reporting that 'float32' was not serializable. It looks like numpy.float32 is NOT serializable, but numpy.float64 is. It looks like, np.float64 is a subclass of float which is why it's serializable but np.float32 is not. We could also cast to float here, which is maybe what I'll do instead.

https://stackoverflow.com/questions/53082708/typeerror-object-of-type-float32-is-not-json-serializable

https://stackoverflow.com/questions/27050108/convert-numpy-type-to-python

Casting to float here: 35bd170

davemfish · 2025-03-07T16:38:45Z

frontend/Dockerfile

 WORKDIR /opt/frontend

 ADD ./package.json /opt/frontend/
+RUN yarn add puppeteer --ignore-scripts


puppeteer is already listed in the package.json, so I wouldn't expect to need to add it here everytime.

When running docker compose up –build the frontend container would hang indefinitely trying to build packages as part of yarn install.
It appears this was an issue with puppeteer, perhaps downloading Chromium
forever building fresh packages :( · Issue #5268 · yarnpkg/yarn · GitHub
Following the link above, a user suggested this, which worked right away for me.

Let me know your thoughts on this and including yarn.lock. That puppeteer Chromium thing was a real blocker for me in a docker set up unfortunately.

davemfish · 2025-03-07T16:41:19Z

frontend/package.json

        "start": "vite",
-        "start-docker": "cross-env MODE=docker vite --host 0.0.0.0 --port 3000",
+        "start-docker-prod": "cross-env MODE=docker VITE_DEVMODE=false vite --host 0.0.0.0 --port 3000",
+        "start-docker-dev": "cross-env MODE=docker VITE_DEVMODE=true vite --host 0.0.0.0 --port 3000",


technically I think we're always using the vite dev server, even in production. So VITE_DEVMODE might be a bit misleading. Is it better as APP_DEV_MODE, or just DEVMODE?

It is confusing, but I found I had to use VITE_ in conjunction with import_meta.env, otherwise the environment variable was not known.

https://vite.dev/guide/env-and-mode.html

To prevent accidentally leaking env variables to the client, only variables prefixed with VITE_ are exposed to your Vite-processed code.

So, we could rename it: VITE_UO_DEV_MODE or VITE_URBANONLINE_DEVMODE...

Updated to VITE_URBANONLINE_DEVMODE here: dca5e9e

Happy to change this to something else.

davemfish · 2025-03-07T16:47:40Z

frontend/Dockerfile

@@ -5,7 +5,9 @@ RUN mkdir /opt/frontend
 WORKDIR /opt/frontend

 ADD ./package.json /opt/frontend/


I wonder if we should also be adding the yarn.lock file, to speedup the yarn install.

That could definitely be handy. See my note below on Chromium really slowing down / stopping the yarn install step when getting puppeteer.

davemfish · 2025-03-07T16:53:10Z

frontend/Dockerfile

+#RUN yarn cache clean && yarn install --production=false

 EXPOSE 3000



I guess it's not used during docker compose, but the last (next) line in this Dockerfile could be updated to yarn start-docker-dev

Right, or we just comment out / remove it all together. I'll update it for now.

Changed default CMD here: 18aaefb

dcdenu4 added 30 commits August 30, 2024 09:33

Add localhost 8080 to CORS list

20b0ab0

Add nginx reverse proxy container

7b345d7

Merge remote-tracking branch 'upstream/develop' into feature/vm-hosting

1ea4dff

handling user permission for writing to opt/data

994f7c8

listen on port 80; listen on 9000 for api server requests.

4c8be23

updating IP endpoints to match static IP

13fec0a

update from localhost to static IP

293f2b8

explicitly use port 8000 for fastapi; add static IP to fastapi server…

2a44ebb

… CORS origin list

lots of fiddling wtih nginx conf

d477443

matching updates made on server

f0db5dc

Merge branch 'develop' into feature/vm-hosting

a66763a

Merge branch 'feature/vm-hosting' of https://github.com/dcdenu4/urban…

12edbf9

…-online-workflow into feature/vm-hosting

typo fixes

7a7047b

Merge branch 'develop' into feature/vm-hosting

d3f1b4d

Merge branch 'feature/vm-hosting' of https://github.com/dcdenu4/urban…

7c66910

…-online-workflow into feature/vm-hosting

working version of hosting

18ea504

listen on 433 and vol mount ssl cert locations

3d31d7b

update IP to domain name for allow origin

1f26ea8

instead of using port 9000, use endpoint 9000 to avoid SSL issues on …

2ee370a

…non 443 ports. The endpoint 9000 is handled with a proxy pass to fileserver:9000

added new origins for https domain. removed trailing slash from sessi…

1983e7d

…ons endpoint, see comment note

updated to latest image vs alpine to get more tooling. was trying to …

83376bd

…play with certbot on the image itself.

cleaning up nginx-proxy dockerfile

6ccb135

setting fastapi url with domain and 8000 endpoint. Using the port was…

ca8789e

… caussing https ssl issues, so now 443 just does a proxy pass to localhost:8000

Lots of changes including some that need to be isolated for whether t…

56ad117

…hey were useful or not. Moving port 8000 and 9000 calls to 443 location proxy_pass redirects. Lots of SSL configuration and setup. Lots of things added to try and work and MixedContent errors and SSL erros.

remove port 8080

0fd1eae

remove trailing '/' from endpoints because of redirect conflict fron …

6c805ae

…nginx. nginx when handling the request drops the trailing slash, which was causing a http redirect, which caused a https / http conflict

moving fileserver calls from port 9000 to endpoint 9000, to avoid ssl…

7027e00

… cert issues. Commenting out return calls from GET / POST that were interfering with FastAPI returns, which caused a json error on the frontend.

removing commented out entries in nginx

62f6739

updating allow origin paths to be domain specific and not '*'

7269169

using regex to redirect fileserver image requests. other indentation …

e08ba57

…cleanup.

dcdenu4 added 5 commits October 1, 2024 13:17

troubleshooting pupeteer dep. so it doesn't hang yarn install.

a2c8cc9

small updates

3cd8929

add well-known/acme-challenge to 443 server block. This is to help wh…

bd7a6eb

…en renewing certs, since all traffic is being redirected to 443.

Adding ssl certot renew script

5b58171

Cleaning up commented out code

a7111b6

dcdenu4 requested a review from davemfish February 20, 2025 15:52

dcdenu4 added 14 commits February 20, 2025 14:02

adding server key to vite config

f80a38e

remove redundant fileserver endpoint with regex

0e4324d

first stab at dev / prod setup.

14cae58

Add nginx dev config

4a5a969

clean up docker compose. Fix env for JS

7b8bbce

Debugging dev mode

597dfce

updating readme for dev and prod mode

4f20461

Fixing json serialization error. When returning carbon results I was …

aee8671

…seeing an error when trying to dump to a json file. It was related to trying to serialize a float, so I casted to a string.

Creating dev/prod version of fileserver

3da2aa5

Cleaning up Origins in server

e9b604e

Dev/prod mode for frontend. I added a trailing slash to the dev path …

fa59bf8

…because I was seeing a malformed URL request in dev mode. 'localhost/9000scenarios/...'. I'm not sure why this is showing up with localhost and not in production.

changing fileserver-prod name

0a8d0ee

pinning sqlite because of known bug

042bfb8

removing not needed config and renaming fileserver

d55497b

davemfish reviewed Mar 7, 2025

View reviewed changes

dcdenu4 added 5 commits March 10, 2025 09:33

Add trailing forward slash for fileserver URL

5836121

pin libsqlite to be safe

07168fd

cast carbon results to float

35bd170

change default CMD to dev specific start

18aaefb

Updating devmode env name

dca5e9e

dcdenu4 requested a review from davemfish March 10, 2025 14:28

davemfish approved these changes Mar 11, 2025

View reviewed changes

davemfish merged commit 6d00341 into natcap:develop Mar 11, 2025
1 check failed

		@@ -5,7 +5,9 @@ RUN mkdir /opt/frontend
		WORKDIR /opt/frontend

		ADD ./package.json /opt/frontend/

		#RUN yarn cache clean && yarn install --production=false

		EXPOSE 3000

Conversation

dcdenu4 commented Feb 20, 2025

Uh oh!

davemfish left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants