Skip to content

Commit adddba0

Browse files
jbooleanjboolean
committed
Update READMEs
1 parent 86bf78f commit adddba0

File tree

3 files changed

+248
-44
lines changed

3 files changed

+248
-44
lines changed

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,18 @@
11
# 1940s.nyc
2-
1940 New York City Street View
32

3+
1940 New York City Street View
44

5-
This is a monorepo containing these modules:
5+
This is a monorepo. See the README in each submodule for more information. `npm install` at the root level can be used to install all submodule dependencies.
66

77
## backend
8-
The serverless API for search, photo metadata, and payments.
8+
9+
The serverless API for search, photo metadata, and payments.
910
Also, s3 event lambdas used for processing images as part of ingesting from the initial photo scrape.
1011

1112
![backend deploy](https://github.com/jboolean/1940s.nyc/actions/workflows/backend-deploy.yml/badge.svg)
1213

1314
## frontend
15+
1416
The frontend, a React app.
1517

1618
[![Netlify Status](https://api.netlify.com/api/v1/badges/29e76d8d-f9ba-4afa-bc07-c89fc03e570a/deploy-status)](https://app.netlify.com/sites/1940s-nyc/deploys)
17-

‎backend/README.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# 1940s.nyc backend
2+
3+
This is a serverless app deployed to AWS lambda. It is written in Node.js and uses the Serverless framework.
4+
5+
## Pre-requisites
6+
7+
### Credentials
8+
9+
First, have AWS credentials on your machine in [a way the serverless framework can use](https://www.serverless.com/framework/docs/providers/aws/guide/credentials#recommended-using-local-credentials) such as in `~/.aws`. These credentials are used to fetch secrets via SSM to connect to the database and other services.
10+
11+
### Node.js
12+
13+
This project was developed with node version specified in in .nvmrc
14+
15+
With `nvm` installed, run `nvm use`
16+
17+
## Development
18+
19+
When building for the first time or if dependencies have changed, run
20+
21+
```
22+
npm install
23+
```
24+
25+
To develop,
26+
27+
```
28+
npm start
29+
```
30+
31+
## Environments
32+
33+
There are three backend environments (stages), two database environments, unlimited frontend environments, and one Mapbox map.
34+
35+
On the backend, there's production, staging, and your local development environment. Production uses the `fourtiesnyc` database, while other environments use `fourtiesnyc_staging`. This means your local server uses a shared database with staging. This is not ideal and came about because originally the app had only static photo data; it should be improved.
36+
37+
On the frontend, there are ephemeral environments created for each pull request, these all share `staging` as the backend.
38+
39+
There is one Mapbox map that is shared. This means you may see story labels on the map that are not in the staging database and can't be read.
40+
41+
## Deployment
42+
43+
Deployment should _not_ be done from your local machine, but only by Github Actions.
44+
45+
Deployments to production are automated upon merges to `master`.
46+
47+
To deploy to staging, use the [backend-deploy-manual](https://github.com/jboolean/1940s.nyc/actions/workflows/backend-deploy-manual.yml) action and select stage=staging.
48+
49+
## Schema changes
50+
51+
There is no schema change management. Schema changes and migrations are made manually and carefully. It's suggested to try out your schema changes in staging, then create a DDL of the final schema and apply it to production.
52+
53+
Ideally, either TypeORM migrations or something like [Flyway](https://flywaydb.org/) would be used to manage schema changes.
54+
55+
Use the [clone-db](https://github.com/jboolean/1940s.nyc/actions/workflows/clone-db.yml) Action to clone production back to staging if staging becomes too outdated or broken and needs to be reset.
56+
57+
## Frameworks and patterns
58+
59+
### Serverless
60+
61+
Serverless framework is used to manage packaging and deployment.
62+
63+
### Express
64+
65+
Express is used, but most routes are defined with the `tsoa` framework.
66+
67+
### tsoa
68+
69+
The [tsoa](https://github.com/lukeautry/tsoa) API framework is used to define a REST API with type safety and OpenAPI spec generation. `*Controller` classes are scanned, annotations are parsed, and routes are generated. It also validates requests and can return validation messages to the client.
70+
71+
Use `npm run routes` to regenerate the routes file.
72+
73+
### TypeORM
74+
75+
TypeORM is used as a typed ORM. It's used to interact with the database.
76+
77+
### Directory structure
78+
79+
The classic layered architecture is used: api, business, persistence. Dependencies must go downward only.
80+
81+
- `api` - Contains API interaction including Tsoa controllers and request/response types.
82+
- `cron` - Entry points for cron jobs. At the same "layer" as `api`.
83+
- `business` - Contains business logic. `*Service` files are defined to provide facades to various subsystems to perform business functions. There are also utils.
84+
- `entities` - Contains TypeORM entities corresponding to tables and views in the database.
85+
- `repositories` - TypeORM automatically generates repositories for each entity, but extensions with custom queries can be defined here.
86+
- `enum` - Enums can be shared between the frontend and backend. They are defined here.
87+
88+
### Cron jobs
89+
90+
Each cron job should be defined as a function in the `cron` directory, then registered in `/cron.ts` and in `serverless.yml` as a lambda function with a cron schedule.
91+
92+
## Features of note
93+
94+
### Photos and geocodes
95+
96+
The core of this app is photo metadata and geocoding records. Each `Photo` has many `GeocodeResult`s from geocoding services run when the app was built. The `effective_geocodes_view` is a materialized view represented by the `EffectiveGeocode` entity which contains the geocode we use for each photo. The query for that view uses a priority order of geocode services, and "corrections" submitted by users (`GeocodeCorrection`). The view is refreshed by a cron job to incorporate corrections.
97+
98+
### Stories
99+
100+
Stories was the first user-generated content feature and is pretty self-contained. It is not connected to Users or authentication as it was built before those features. Stories have their own authentication via jwt-based magic links sent via email which users can use to edit a specific story. Emails give users confirmation that their story was received and later published.
101+
102+
Stories are moderated by admins. All stories are submitted to the moderation queue at http://1940s.nyc/admin/review-stories.
103+
104+
Automatic moderation was considered but most of the rejected stories are not classic spam that could be detected by spam filters or CAPTCHAs but rather trolls and confused people.
105+
106+
### Users and authentication
107+
108+
I didn't want to pay for an authentication service, and it seemed simple enough to build my own.
109+
I wanted a more seamless user experience without usernames and passwords, and often even without doing anything.
110+
111+
For any route tagged with `@Security('user-token')`, a user is created if one does not exist. The user is then considered logged in immediately without any information and is considered anonymous. Later, the user can provide, and optionally verify an email address. Specific features can check for verified email addresses if that level of security is needed. Corrections to geocodes, for example, require a verified email address. Colorization, however, can be done by anonymous users, allowing a "free trial" experience.
112+
113+
**Log-in process**: The user supplies an email address: if an account exists the user is sent a magic link to log in, or the email is associated with the current possibly anonymous account they are logged in with. Magic links are JWT tokens.
114+
115+
Notably, **Stories** are not connected to Users because Stories were built first. Users primarily support the Colorization feature for payment and credit ledgering and the Corrections feature for safety.
116+
117+
### Credit ledgering
118+
119+
Users can purchase "Color tokens" to colorize images. This is managed via the Ledger system, which is just a ledger of credits issued and images colorized. Users can go negative as we allow one free colorization per day.
120+
121+
`LedgerService.withMeteredUsage` is a wrapper to wrap code that uses requires and consumes credits.
122+
123+
### Admin users
124+
125+
Admin users actually _do_ use an authentication service: Netlify Identity. This was built before end-user authentication and could potentially be updated to use the same system.
126+
127+
Admin users have roles defined in Identity.
128+
129+
A route tag for an admin route looks like `@Security('netlify', ['moderator'])`. This requires an Identity user to be logged in and have the role `moderator`.
130+
131+
### Email campaigns
132+
133+
I was also too cheap to pay for a mailing list service like Mailchimp so I built one.
134+
135+
It's pretty simple. When a campaign is enqueued via the API it just creates a record for every list member. A cron job sends out the emails in batches.
136+
137+
There's currently no sign-up form for the mailing list. Addresses are just added whenever they provide an email via other app features like Stories.
138+
139+
### Map sync
140+
141+
Every night, a cron job refreshes the materialized views, regenerates a set of geojson, pushes it to Mapbox, and triggers Mapbox's regeneration of the map tiles. Note that regenerating map tiles costs money, which is why it's not done continuously.
142+
143+
This important process supports geocode corrections and user story labels.
144+
145+
## Third party services
146+
147+
- Mapbox - For the map
148+
- AWS SSM - For secrets
149+
- Postmark - For transactional and campaign emails
150+
- Netlify Identity - For admin users
151+
- Stripe - For payments
152+
- OpenAI - For story title generation
153+
- Palette - For colorization

‎frontend/README.md‎

Lines changed: 90 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,4 @@
1-
# Julian's Base App
2-
This repo is my starting place for new frontend apps.
3-
- Webpack
4-
- React
5-
- LESS CSS
6-
- Typescript
7-
- Linting and Prettier
1+
# 1940s.nyc frontend
82

93
## Pre-requisites
104

@@ -24,21 +18,24 @@ To develop,
2418

2519
```
2620
npm run watch
27-
```
21+
```
2822

29-
Starts a development server at `localhost:8080`.
23+
Starts a development server at http://dev.1940s.nyc:8080.
24+
The backend must also be running for many functions to work.
3025

3126
## Build
3227

3328
```
3429
npm run build
3530
```
3631

37-
bundles javascript and css into `/dist`.
32+
bundles javascript and css into `/dist`.
3833

3934
## Directory structure
4035

41-
The files and directories in `src` are strictly organized into four standard directories. The contents of each directory are limited.
36+
The files and directories in `src` are strictly organized into standard directories. The contents of each directory are limited.
37+
38+
(Example names from another project are used for illustration)
4239

4340
- `screens` - Contains only sub-directories named for full pages in the UI. If pages have a hierarchy, those sub-directories can contain another `screens` folder for the pages in the hierarchy.
4441
```
@@ -50,6 +47,7 @@ The files and directories in `src` are strictly organized into four standard dir
5047
- Show
5148
```
5249
- `components` - Contains only sub-directories named for UI elements that relate to the screen of the parent directory. A component's subdirectory should contain an `index.jsx` file for the component and a `ComponentName.less` file for styles. If the component has a higher-order connecting component to maintain state, that is in the `index.jsx` file, and `ComponentName.jsx` is a pure underlying component. If there is only one file, a directory is not needed.
50+
5351
```
5452
- components
5553
- Schedule
@@ -58,24 +56,86 @@ The files and directories in `src` are strictly organized into four standard dir
5856
- Schedule.less // styles used by Schedule.jsx
5957
```
6058

61-
- `utils` - Contains only files that export utility functions, objects, or constants. May contain any subdirectories to organize the utilities.
62-
- `shared` - A special type of directory, configured in Webpack to make its contents easily importable by decendants in the hierarchy. When importing, webpack will automatically look up the tree in shared folders to find a match, so instead of `import ../../shared/utils/myUtil`, just `import 'utils/myUtil'` will suffice. `shared` can contain `components`, `screens` or `utils`. `shared` directories can exist on any level to make contents available below that level.
63-
- The top-level `src/shared` directory contains code _not_ specific to WOWD, such as a generic calendar layout component with no knowledge of radio show data, a generic track manager, and types to store date and time data.
64-
Anything at this level could potentially be moved out to another repository and npm package.
65-
- `src/wowd/shared` contains components and utils used on many screens that _are_ specific to WOWD such as a "show card", a play button, data types for Shows, Djs and Playlists, and an API client.
59+
- `utils` - Contains only files that export utility functions, objects, or constants. May contain any subdirectories to organize the utilities.
60+
- `stores` - Contains zustand stores
61+
- `shared` - A special type of directory, configured in Webpack to make its contents easily importable by descendants in the hierarchy. When importing, webpack will automatically look up the tree in shared folders to find a match, so instead of `import ../../shared/utils/myUtil`, just `import 'utils/myUtil'` will suffice. `shared` can contain `components`, `screens` or `utils`. `shared` directories can exist on any level to make contents available below that level.
62+
- The top-level `src/shared` directory contains code _not_ specific to this application, that could potentially become separate packages, such as UI components.
6663

6764
## Frameworks and patterns
6865

69-
### React / state management
70-
This project is based on React.
66+
### React
67+
68+
This project is based on React.
69+
70+
### State management
71+
72+
Where state management is needed, [zustand](https://github.com/pmndrs/zustand) is used. It was added to this app in 2023, so legacy code uses Context.
73+
74+
Zustand usage in this app follows a particular pattern:
75+
76+
A store must be created in a file called \*Store.ts and exported. Ad-hoc stores shouln't be created in components.
7177

72-
Bring your own flux implementation if desired.
78+
```typescript
79+
import create from 'zustand';
80+
import { immer } from 'zustand/middleware/immer';
81+
82+
interface State {
83+
isOpen: boolean;
84+
value: number | null;
85+
}
86+
87+
interface ComputedState {
88+
isValueSet: boolean;
89+
}
90+
91+
interface Actions {
92+
setValue: (value: number) => void;
93+
reset: () => void;
94+
}
95+
96+
const useStore = create(
97+
immer<State & Actions>((set) => ({
98+
isOpen: false,
99+
value: null,
100+
101+
setValue: (value) => {
102+
set((draft) => {
103+
draft.isOpen = true;
104+
draft.value = value;
105+
});
106+
},
107+
108+
reset: () => {
109+
set((draft) => {
110+
draft.isOpen = false;
111+
draft.value = null;
112+
});
113+
},
114+
}))
115+
);
116+
117+
export function useStoreComputeds(): ComputedState {
118+
const { value } = useStore();
119+
return {
120+
isValueSet: value !== null,
121+
};
122+
}
123+
124+
export default useStore;
125+
```
126+
127+
This template above demonstrates how to structure a Zustand store with Immer for immutability and a separate utility for computed state.
128+
129+
Key Features:
130+
• State and Actions: Define state and behavior in the same store.
131+
• Computed State: Use a dedicated custotm hook (useStoreComputeds) to derive and expose computed values from the store within the same file.
73132

74133
### LESS
75134

76-
LESS CSS ([lesscss.org](http://lesscss.org)) is used for styles with Webpack's [CSS Modules](https://github.com/webpack-contrib/css-loader#modules). The tl;dr of this is: 1) All less files can be imported into javascript as objects and 2) All the class names within a `:local` block in the less file, which should be the entire less file, gets replace with a random string. The exported javascript object is a map from the original class name to the randomized class name.
135+
LESS CSS ([lesscss.org](http://lesscss.org)) is used for styles with Webpack's [CSS Modules](https://github.com/webpack-contrib/css-loader#modules). The tl;dr of this is: 1) All less files can be imported into javascript as objects and 2) All the class names within a `:local` block in the less file, which should be the entire less file, gets replace with a random string. The exported javascript object is a map from the original class name to the randomized class name.
77136

78137
MyGreatComponent.less
138+
79139
```css
80140
:local {
81141
.myGreatClass {
@@ -85,6 +145,7 @@ MyGreatComponent.less
85145
```
86146

87147
MyGreatComponent.jsx
148+
88149
```javascript
89150
import stylesheet from './MyGreatComponent.less'; // { 'myGreatClass' : 'MyGreatComponent-myGreatClass-x1f2'}
90151

@@ -94,14 +155,14 @@ export <div className={stylesheet.myGreatClass} />;
94155
This keeps styles local to a component, prevents them from being used elsewhere unintentionally, and allows for using common classnames like `.body` or `.title` without fear of duplication.
95156

96157
## Code style
97-
This project follows Squarespace's JavaScript Styles. Rather than write a Style Guide, I refer to the [Squarespace .eslintrc](https://github.com/Squarespace/eslint-config-squarespace/blob/master/vanilla/.eslintrc), which has been imported into this project.
158+
159+
This project follows [eslint-config-jboolean](https://github.com/jboolean/eslint-config-jboolean), based on eslint recommended rules and `prettier`.
98160

99161
```
100162
npm run lint
101163
```
102164

103-
will find style errors, and
104-
165+
will find style errors, and
105166

106167
```
107168
npm run lint-fix
@@ -111,25 +172,14 @@ will fix many of them.
111172

112173
lint errors will be automatically fixed when saving changes while running `npm run watch` as well.
113174

114-
## Types
115-
This project uses [Flow](https://flow.org) to add static type checking to Javascript.
175+
Linting is also run on commit via `husky`.
116176

117-
**All** files should begin with the comment `//@flow ` to enable type checking unless there is a compelling reason to turn off type checking.
177+
### Typescript
118178

119-
Run
179+
This project uses Typescript.
120180

121-
```
122-
npm run flow
123-
```
124-
125-
to check for type errors. It is recommended to install Flow integration into your editor.
126-
127-
### Updating Flow's libdefs
128-
When dependencies are updated, type definities for third-party libraries must also be updated.
129-
130-
```
131-
npm run install-libdefs
132-
```
181+
### API Client
133182

134-
Sometimes the new libdefs are not backwards compatible, so Flow itself may also need to be upgraded at this time.
183+
API calls are written manually in utils files and use `axios`, with named exports matching API functions.
135184

185+
There is no automatic generation of API clients, though it is possible to do, since the backend does produce an OpenAPI spec.

0 commit comments

Comments
 (0)