You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,17 +1,18 @@
1
1
# 1940s.nyc
2
-
1940 New York City Street View
3
2
3
+
1940 New York City Street View
4
4
5
-
This is a monorepo containing these modules:
5
+
This is a monorepo. See the README in each submodule for more information. `npm install` at the root level can be used to install all submodule dependencies.
6
6
7
7
## backend
8
-
The serverless API for search, photo metadata, and payments.
8
+
9
+
The serverless API for search, photo metadata, and payments.
9
10
Also, s3 event lambdas used for processing images as part of ingesting from the initial photo scrape.
This is a serverless app deployed to AWS lambda. It is written in Node.js and uses the Serverless framework.
4
+
5
+
## Pre-requisites
6
+
7
+
### Credentials
8
+
9
+
First, have AWS credentials on your machine in [a way the serverless framework can use](https://www.serverless.com/framework/docs/providers/aws/guide/credentials#recommended-using-local-credentials) such as in `~/.aws`. These credentials are used to fetch secrets via SSM to connect to the database and other services.
10
+
11
+
### Node.js
12
+
13
+
This project was developed with node version specified in in .nvmrc
14
+
15
+
With `nvm` installed, run `nvm use`
16
+
17
+
## Development
18
+
19
+
When building for the first time or if dependencies have changed, run
20
+
21
+
```
22
+
npm install
23
+
```
24
+
25
+
To develop,
26
+
27
+
```
28
+
npm start
29
+
```
30
+
31
+
## Environments
32
+
33
+
There are three backend environments (stages), two database environments, unlimited frontend environments, and one Mapbox map.
34
+
35
+
On the backend, there's production, staging, and your local development environment. Production uses the `fourtiesnyc` database, while other environments use `fourtiesnyc_staging`. This means your local server uses a shared database with staging. This is not ideal and came about because originally the app had only static photo data; it should be improved.
36
+
37
+
On the frontend, there are ephemeral environments created for each pull request, these all share `staging` as the backend.
38
+
39
+
There is one Mapbox map that is shared. This means you may see story labels on the map that are not in the staging database and can't be read.
40
+
41
+
## Deployment
42
+
43
+
Deployment should _not_ be done from your local machine, but only by Github Actions.
44
+
45
+
Deployments to production are automated upon merges to `master`.
46
+
47
+
To deploy to staging, use the [backend-deploy-manual](https://github.com/jboolean/1940s.nyc/actions/workflows/backend-deploy-manual.yml) action and select stage=staging.
48
+
49
+
## Schema changes
50
+
51
+
There is no schema change management. Schema changes and migrations are made manually and carefully. It's suggested to try out your schema changes in staging, then create a DDL of the final schema and apply it to production.
52
+
53
+
Ideally, either TypeORM migrations or something like [Flyway](https://flywaydb.org/) would be used to manage schema changes.
54
+
55
+
Use the [clone-db](https://github.com/jboolean/1940s.nyc/actions/workflows/clone-db.yml) Action to clone production back to staging if staging becomes too outdated or broken and needs to be reset.
56
+
57
+
## Frameworks and patterns
58
+
59
+
### Serverless
60
+
61
+
Serverless framework is used to manage packaging and deployment.
62
+
63
+
### Express
64
+
65
+
Express is used, but most routes are defined with the `tsoa` framework.
66
+
67
+
### tsoa
68
+
69
+
The [tsoa](https://github.com/lukeautry/tsoa) API framework is used to define a REST API with type safety and OpenAPI spec generation. `*Controller` classes are scanned, annotations are parsed, and routes are generated. It also validates requests and can return validation messages to the client.
70
+
71
+
Use `npm run routes` to regenerate the routes file.
72
+
73
+
### TypeORM
74
+
75
+
TypeORM is used as a typed ORM. It's used to interact with the database.
76
+
77
+
### Directory structure
78
+
79
+
The classic layered architecture is used: api, business, persistence. Dependencies must go downward only.
80
+
81
+
-`api` - Contains API interaction including Tsoa controllers and request/response types.
82
+
-`cron` - Entry points for cron jobs. At the same "layer" as `api`.
83
+
-`business` - Contains business logic. `*Service` files are defined to provide facades to various subsystems to perform business functions. There are also utils.
84
+
-`entities` - Contains TypeORM entities corresponding to tables and views in the database.
85
+
-`repositories` - TypeORM automatically generates repositories for each entity, but extensions with custom queries can be defined here.
86
+
-`enum` - Enums can be shared between the frontend and backend. They are defined here.
87
+
88
+
### Cron jobs
89
+
90
+
Each cron job should be defined as a function in the `cron` directory, then registered in `/cron.ts` and in `serverless.yml` as a lambda function with a cron schedule.
91
+
92
+
## Features of note
93
+
94
+
### Photos and geocodes
95
+
96
+
The core of this app is photo metadata and geocoding records. Each `Photo` has many `GeocodeResult`s from geocoding services run when the app was built. The `effective_geocodes_view` is a materialized view represented by the `EffectiveGeocode` entity which contains the geocode we use for each photo. The query for that view uses a priority order of geocode services, and "corrections" submitted by users (`GeocodeCorrection`). The view is refreshed by a cron job to incorporate corrections.
97
+
98
+
### Stories
99
+
100
+
Stories was the first user-generated content feature and is pretty self-contained. It is not connected to Users or authentication as it was built before those features. Stories have their own authentication via jwt-based magic links sent via email which users can use to edit a specific story. Emails give users confirmation that their story was received and later published.
101
+
102
+
Stories are moderated by admins. All stories are submitted to the moderation queue at http://1940s.nyc/admin/review-stories.
103
+
104
+
Automatic moderation was considered but most of the rejected stories are not spam that could be detected by spam filters or CAPTCHAs but trolls or users who misuse the feature.
105
+
106
+
### Users and authentication
107
+
108
+
For a seamless user experience and to reduce costs, a custom authentication system was built instead of using a third party service like Auth0.
109
+
110
+
For any route tagged with `@Security('user-token')`, a user is created if one does not exist. The user is then considered logged in immediately without any information and is considered anonymous. Later, the user can provide, and optionally verify an email address. Specific features can check for verified email addresses if that level of security is needed. Corrections to geocodes, for example, require a verified email address. Colorization, however, can be done by anonymous users, allowing a limited "free trial" experience.
111
+
112
+
**Log-in process**: The user supplies an email address: if an account exists the user is sent a magic link to log in, or the email is associated with the current possibly anonymous account they are logged in with. Magic links are JWT tokens.
113
+
114
+
Notably, **Stories** are not connected to Users because Stories were built first with their own authentication mechanism. Users primarily support the Colorization feature for payment and credit ledgering and the Corrections feature for safety.
115
+
116
+
### Credit ledgering
117
+
118
+
Users can purchase "Color tokens" to colorize images. This is managed via the Ledger system, which is just a ledger of credits issued and images colorized. Users can go negative as we allow one free colorization per day as a trial.
119
+
120
+
`LedgerService.withMeteredUsage` is a wrapper to wrap code that requires and consumes credits.
121
+
122
+
### Admin users
123
+
124
+
Admin users actually _do_ use a third-party authentication service: Netlify Identity. This was built before end-user authentication and could potentially be updated to use the same system.
125
+
126
+
Admin users have roles defined in Identity.
127
+
128
+
A route tag for an admin route looks like `@Security('netlify', ['moderator'])`. This requires an Identity user to be logged in and have the role `moderator`.
129
+
130
+
### Email campaigns
131
+
132
+
1940s.nyc has an occasional newsletter. To avoid the costs of a third-party service like Mailchimp, a simple system was built into the app.
133
+
134
+
It's pretty simple. When a campaign is enqueued via the API it just creates a sending record for every list member. A cron job sends out the unsent emails in batches.
135
+
The email content lives in Postmark templates.
136
+
137
+
There's currently no sign-up form for the mailing list. Addresses are just added whenever they provide an email via other app features like Stories.
138
+
139
+
### Map sync
140
+
141
+
Every night, a cron job refreshes the materialized views, regenerates a set of geojson, pushes it to Mapbox, and triggers Mapbox's regeneration of the map tiles. Note that regenerating map tiles costs money, which is why it's not done continuously.
142
+
143
+
This important process supports geocode corrections and user story labels.
144
+
145
+
## Third party services
146
+
147
+
- Mapbox - For the map
148
+
- AWS SSM - For secrets
149
+
- Postmark - For transactional and campaign emails
This repo is my starting place for new frontend apps.
3
-
- Webpack
4
-
- React
5
-
- LESS CSS
6
-
- Typescript
7
-
- Linting and Prettier
1
+
# 1940s.nyc frontend
8
2
9
3
## Pre-requisites
10
4
@@ -24,21 +18,24 @@ To develop,
24
18
25
19
```
26
20
npm run watch
27
-
```
21
+
```
28
22
29
-
Starts a development server at `localhost:8080`.
23
+
Starts a development server at http://dev.1940s.nyc:8080.
24
+
The backend must also be running for many functions to work.
30
25
31
26
## Build
32
27
33
28
```
34
29
npm run build
35
30
```
36
31
37
-
bundles javascript and css into `/dist`.
32
+
bundles javascript and css into `/dist`.
38
33
39
34
## Directory structure
40
35
41
-
The files and directories in `src` are strictly organized into four standard directories. The contents of each directory are limited.
36
+
The files and directories in `src` are strictly organized into standard directories. The contents of each directory are limited.
37
+
38
+
(Example names from another project are used for illustration)
42
39
43
40
-`screens` - Contains only sub-directories named for full pages in the UI. If pages have a hierarchy, those sub-directories can contain another `screens` folder for the pages in the hierarchy.
44
41
```
@@ -50,32 +47,100 @@ The files and directories in `src` are strictly organized into four standard dir
50
47
- Show
51
48
```
52
49
-`components` - Contains only sub-directories named for UI elements that relate to the screen of the parent directory. A component's subdirectory should contain an `index.jsx` file for the component and a `ComponentName.less` file for styles. If the component has a higher-order connecting component to maintain state, that is in the `index.jsx` file, and `ComponentName.jsx` is a pure underlying component. If there is only one file, a directory is not needed.
50
+
53
51
```
54
52
- components
55
53
- Schedule
56
-
- index.jsx // Manages state, loades data from the API
54
+
- index.jsx // Manages state, loads data from the API
57
55
- Schedule.jsx // A pure React component that renders a Schedule
58
56
- Schedule.less // styles used by Schedule.jsx
59
57
```
60
58
61
-
-`utils` - Contains only files that export utility functions, objects, or constants. May contain any subdirectories to organize the utilities.
62
-
-`shared` - A special type of directory, configured in Webpack to make its contents easily importable by decendants in the hierarchy. When importing, webpack will automatically look up the tree in shared folders to find a match, so instead of `import ../../shared/utils/myUtil`, just `import 'utils/myUtil'` will suffice. `shared` can contain `components`, `screens` or `utils`. `shared` directories can exist on any level to make contents available below that level.
63
-
- The top-level `src/shared` directory contains code _not_ specific to WOWD, such as a generic calendar layout component with no knowledge of radio show data, a generic track manager, and types to store date and time data.
64
-
Anything at this level could potentially be moved out to another repository and npm package.
65
-
-`src/wowd/shared` contains components and utils used on many screens that _are_ specific to WOWD such as a "show card", a play button, data types for Shows, Djs and Playlists, and an API client.
59
+
-`utils` - Contains only files that export utility functions, objects, or constants. May contain any subdirectories to organize the utilities.
60
+
-`stores` - Contains zustand stores
61
+
-`shared` - A special type of directory, configured in Webpack to make its contents easily importable by descendants in the hierarchy. When importing, webpack will automatically look up the tree in shared folders to find a match, so instead of `import ../../shared/utils/myUtil`, just `import 'utils/myUtil'` will suffice. `shared` can contain `components`, `screens` or `utils`. `shared` directories can exist on any level to make contents available below that level.
62
+
- The top-level `src/shared` directory contains code _not_ specific to this application, that could potentially become separate packages, such as UI components.
66
63
67
64
## Frameworks and patterns
68
65
69
-
### React / state management
70
-
This project is based on React.
66
+
### React
67
+
68
+
This project is based on React.
69
+
70
+
### State management
71
+
72
+
Where state management is needed, [zustand](https://github.com/pmndrs/zustand) is used. It was added to this app in 2023, so legacy code uses Context.
73
+
74
+
Zustand is an unopinionated framework, but this project has opinions.
75
+
76
+
- Stores should be structured and separate from rendering components
77
+
- Stores should use Immer for easier immutability
78
+
- Zustand does not natively support computeds, so we have a separate hook co-located with the store
79
+
- Stores should clearly define their Actions and State, separately, using typescript interfaces
80
+
81
+
A store must be created in a file called \*Store.ts in a `stores` directory and exported as the default export. Ad-hoc stores shouldn't be created in components.
This template above demonstrates how to structure a Zustand store with Immer for immutability and a separate utility for computed state.
71
133
72
-
Bring your own flux implementation if desired.
134
+
Key Features:
135
+
• State and Actions: Define state and behavior in the same store.
136
+
• Computed State: Use a dedicated custom hook (useStoreComputeds) to derive and expose computed values from the store within the same file.
73
137
74
138
### LESS
75
139
76
-
LESS CSS ([lesscss.org](http://lesscss.org)) is used for styles with Webpack's [CSS Modules](https://github.com/webpack-contrib/css-loader#modules). The tl;dr of this is: 1) All less files can be imported into javascript as objects and 2) All the class names within a `:local` block in the less file, which should be the entire less file, gets replace with a random string. The exported javascript object is a map from the original class name to the randomized class name.
140
+
LESS CSS ([lesscss.org](http://lesscss.org)) is used for styles with Webpack's [CSS Modules](https://github.com/webpack-contrib/css-loader#modules). The tl;dr of this is: 1) All less files can be imported into javascript as objects and 2) All the class names within a `:local` block in the less file, which should be the entire less file, gets replace with a random string. The exported javascript object is a map from the original class name to the randomized class name.
This keeps styles local to a component, prevents them from being used elsewhere unintentionally, and allows for using common classnames like `.body` or `.title` without fear of duplication.
95
161
96
162
## Code style
97
-
This project follows Squarespace's JavaScript Styles. Rather than write a Style Guide, I refer to the [Squarespace .eslintrc](https://github.com/Squarespace/eslint-config-squarespace/blob/master/vanilla/.eslintrc), which has been imported into this project.
163
+
164
+
This project follows [eslint-config-jboolean](https://github.com/jboolean/eslint-config-jboolean), based on eslint recommended rules and `prettier`.
98
165
99
166
```
100
167
npm run lint
101
168
```
102
169
103
-
will find style errors, and
104
-
170
+
will find style errors, and
105
171
106
172
```
107
173
npm run lint-fix
@@ -111,25 +177,14 @@ will fix many of them.
111
177
112
178
lint errors will be automatically fixed when saving changes while running `npm run watch` as well.
113
179
114
-
## Types
115
-
This project uses [Flow](https://flow.org) to add static type checking to Javascript.
116
-
117
-
**All** files should begin with the comment `//@flow ` to enable type checking unless there is a compelling reason to turn off type checking.
118
-
119
-
Run
120
-
121
-
```
122
-
npm run flow
123
-
```
180
+
Linting is also run on commit via `husky`.
124
181
125
-
to check for type errors. It is recommended to install Flow integration into your editor.
182
+
### Typescript
126
183
127
-
### Updating Flow's libdefs
128
-
When dependencies are updated, type definities for third-party libraries must also be updated.
184
+
This project uses Typescript.
129
185
130
-
```
131
-
npm run install-libdefs
132
-
```
186
+
### API Client
133
187
134
-
Sometimes the new libdefs are not backwards compatible, so Flow itself may also need to be upgraded at this time.
188
+
API calls are written manually in utils files and use `axios`, with named exports matching API functions.
135
189
190
+
There is no automatic generation of API clients, but this could be done in a future project based on the spec generated by the backend.
0 commit comments