Skip to content

Commit ccf28cc

Browse files
vict0rschvict0rsch
committed
♻ refactor: Remove gulp altogether and replace with Rollup
1 parent 36494cc commit ccf28cc

File tree

11 files changed

+4615
-127
lines changed

11 files changed

+4615
-127
lines changed

‎contributing.md‎

Lines changed: 124 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -6,45 +6,39 @@ PaperMemory is pure JS+HTML with minimal dependencies: no framework, (almost) no
66

77
The only external deps. are [`select2.js`](https://select2.org/) which requires `JQuery` and some of the latter here and there (but I'm working on getting rid of it, replacing it with a simple set of helper functions in `src/shared/utils/miniquery.js`).
88

9-
`npm` and `gulp` are here to make the dev+release lifecycle easier, if you don't want to set it up there still are a lot of things you can help with in the raw source code (just don't bother with the `min` files)
9+
The project uses modern ES modules with Rollup for bundling to make development contributor-friendly.
1010

1111
## Set-up
1212

1313
1. [Install `yarn`](https://classic.yarnpkg.com/lang/en/docs/install): Node's package manager
14-
2. [Install `gulp`](https://gulpjs.com/): a build tool
15-
3. Install dependencies: from the root of this repo `$ yarn install`
16-
4. Watch file changes: `$ gulp watch`
17-
5. Edit files!
18-
19-
`gulp` mainly runs the concatenation of files into a single one (especially for css and js) and its minification.
20-
21-
In `popup.html` you will notice:
22-
23-
```html
24-
<!-- @if DEV -->
25-
<script defer src="../../shared/utils/miniquery.js"></script>
26-
<script defer src="../../shared/utils/config.js"></script>
27-
<script defer src="../../shared/utils/functions.js"></script>
28-
<script defer src="../../shared/utils/parsers.js"></script>
29-
<script defer src="../js/handlers.js"></script>
30-
<script defer src="../js/templates.js"></script>
31-
<script defer src="../js/memory.js"></script>
32-
<script defer src="../js/popup.js"></script>
33-
<!-- @else -->
34-
<script defer src="../../shared/utils.min.js"></script>
35-
<script defer src="popup.min.js"></script>
36-
<!-- @endif -->
37-
```
14+
2. Install dependencies: from the root of this repo `$ yarn install`
15+
3. Start development: `$ npm run dev`
16+
4. Edit files!
17+
18+
The build system uses Rollup to bundle ES modules for browser compatibility. In development mode, you get:
19+
20+
- Hot reloading when files change
21+
- Source maps for debugging
22+
- Unminified code for easier debugging
3823

39-
Those `@` commands are meant for `gulp` (using the `preprocess` package) to choose whether to use raw, un-minified files for development (`$ gulp watch`) or concatenated and minified ones for production (`$ gulp build`)
24+
## Build Commands
4025

41-
Note: the file loaded in the popup is `src/popup/min/popup.min.html`, _not_ `src/popup/popup.html` so you _have_ to use `gulp watch` to see changes you make to `popup.html` reflected in the actual popup.
26+
```bash
27+
npm run dev # Development build (one-time)
28+
npm run dev:watch # Development build with file watching
29+
npm run build # Production build
30+
```
31+
32+
For active development, use `npm run dev:watch` which will automatically rebuild files when you save changes.
4233

4334
### Refreshing the extension
4435

45-
Once you load the local extension as an unpackaged extension, changes that affect the popup will directly take effect, no need to refresh anything.
36+
**Chrome Extensions automatically reload** when you make changes to the source files:
37+
38+
- **Popup changes**: Take effect immediately, no refresh needed
39+
- **Content script changes**: Require refreshing the extension in `chrome://extensions/` and then refreshing any web pages where you want to see the changes
4640

47-
**Content scripts** however, are loaded and not binded to the source so you _have to_ refresh the extension in the settings (and then any web page you want to see changes on) for those to be taken into account.
41+
The `npm run dev:watch` command will automatically rebuild files when you save changes.
4842

4943
## Loading in Firefox
5044

@@ -54,49 +48,108 @@ https://extensionworkshop.com/documentation/develop/temporary-installation-in-fi
5448

5549
### File structure
5650

57-
(slightly deprecated since some files have moved but names are unique enough for you to still understand I hope)
58-
5951
```tree
60-
├── jsconfig.json ➤➤➤ vscode js config
61-
├── manifest.json ➤➤➤ the extension's configuration file for the browser
62-
└── src ➤➤➤ actual code
63-
├── background ➤➤➤ background code
64-
│   └── background.js ➤➤➤ background.js can interact with some browser apis content_scripts can't use
65-
├── content_scripts
66-
│   ├── content_script.css ➤➤➤ styling for pages modified by the content_script
67-
│   ├── content_script.js ➤➤➤ the code run at the opening of a page matched in manifest.json
68-
├── popup
69-
│   ├── css ➤➤➤ The popup's style
70-
│   │   ├── dark.css ➤➤➤ Dark mode style sheet
71-
│   │   ├── options.css ➤➤➤ Sliding checkboxes style
72-
│   │   ├── popup.css ➤➤➤ Main style
73-
│   │   ├── select2.min.css ➤➤➤ Style for the select2 dropdowns (don't modify)
74-
│   ├── js ➤➤➤ The javascript files specific to the popup, minified together in this order into js.min.js
75-
│   │   ├── handlers.js ➤➤➤ Event handlers
76-
│   │   ├── memory.js ➤➤➤ Memory-specific functions
77-
│   │   ├── popup.js ➤➤➤ Main execution
78-
│   │   ├── theme.js ➤➤➤ The first thing that is executed when the popup is opened: selecting dark/light theme based on user preferences
79-
│   │   ├── select2.min.js ➤➤➤ JQuery-based tagging lib for paper tags
80-
│   │   └── templates.js ➤➤➤ HTML string templates: memory items and paper popup
81-
│ ├── min
82-
│   │   └── minified scripts
83-
│   ├── popup.html ➤➤➤ Main HTML file
84-
└── shared
85-
├── jquery.min.js ➤➤➤ JQuery lib. Do not modify.
86-
├── utils ➤➤➤ Shared utility functions minified together in this order into utils.min.js
87-
│   ├── bibtexParser.js ➤➤➤ Class to parse bibtex strings into objects
88-
│   ├── config.js ➤➤➤ Constants / State variables used throughout out the code
89-
│   ├── data.js ➤➤➤ Data/Memory manipulation (migrations, paper validation, overwrite etc.)
90-
│   ├── functions.js ➤➤➤ Utility functions, relying on config.js
91-
│   ├── logTrace.js ➤➤➤ Single var script to include the log stack trace in dev (gulp watch)
92-
│   ├── miniquery.js ➤➤➤ Custom vanilla js replacement for JQuery (working towards removing that dependency)
93-
│   ├── parsers.js ➤➤➤ Parsing functions to create papers
94-
│   ├── paper.js ➤➤➤ Single-paper-related functions (isPaper, paperToAbs, paperToPDF)
95-
│   └── state.js ➤➤➤ State-related functions (init, custom title function, addOrUpdatePaper etc.)
96-
├── utils.min.js ➤➤➤ Concatenation and minification of all files in src/shared/utils/
97-
└── loader.css ➤➤➤ the style for the loader before the BibTex entry is displayed on arxiv.org/abs/*
52+
├── jsconfig.json ➤➤➤ VS Code config with ES modules and path aliases (@pm, @pmu)
53+
├── rollup.config.js ➤➤➤ Build configuration that bundles ES modules for browsers
54+
├── manifest.json ➤➤➤ Extension configuration for Chrome/Firefox
55+
└── src ➤➤➤ Source code (all ES modules)
56+
├── background ➤➤➤ Service worker (runs in background)
57+
│ ├── background.js ➤➤➤ Main background script - handles browser APIs, sync, parsing
58+
│ ├── octokit.js ➤➤➤ GitHub API client for sync functionality
59+
│ └── background.bundle.js ➤➤➤ [Generated] Bundled for browser
60+
├── content_scripts ➤➤➤ Scripts injected into web pages
61+
│ ├── content_script.js ➤➤➤ Runs on paper websites - detects/parses papers automatically
62+
│ ├── content_script.css ➤➤➤ Styles for notifications and UI injected into pages
63+
│ └── content.bundle.js ➤➤➤ [Generated] Bundled for browser
64+
├── data ➤➤➤ JSON configuration files
65+
│ ├── journal-abbreviations.json ➤➤➤ Journal name mappings for citations
66+
│ ├── iso4-journals.json ➤➤➤ Standard journal abbreviations
67+
│ ├── art.json ➤➤➤ Article types for parsing
68+
│ └── cell.json ➤➤➤ Cell journal specific configurations
69+
├── popup ➤➤➤ Extension popup interface (main UI)
70+
│ ├── css ➤➤➤ Stylesheet sources
71+
│ │ ├── popup.css ➤➤➤ Main popup styling
72+
│ │ ├── dark.css ➤➤➤ Dark mode theme
73+
│ │ ├── options.css ➤➤➤ Settings toggles and checkboxes
74+
│ │ └── select2.min.css ➤➤➤ [External] Dropdown styling library
75+
│ ├── html ➤➤➤ HTML templates and components
76+
│ │ ├── popup.html ➤➤➤ Main popup HTML structure
77+
│ │ ├── menu.html ➤➤➤ Settings menu template
78+
│ │ ├── modals/ ➤➤➤ Dialog templates (user guide, warnings, etc.)
79+
│ │ └── svgs/ ➤➤➤ SVG icon components
80+
│ ├── js ➤➤➤ ES modules for popup functionality
81+
│ │ ├── popup.js ➤➤➤ Main entry point - initializes popup, handles paper display
82+
│ │ ├── handlers.js ➤➤➤ Event handlers for buttons, keyboard shortcuts, user actions
83+
│ │ ├── memory.js ➤➤➤ Memory display logic - table rendering, search, sorting
84+
│ │ ├── templates.js ➤➤➤ HTML string templates for dynamic content
85+
│ │ └── select2.min.js ➤➤➤ [External] Tag selection library
86+
│ └── min/ ➤➤➤ [Generated] Build output
87+
├── options ➤➤➤ Advanced settings page
88+
│ ├── options.js ➤➤➤ Settings page logic - preferences, data management, sync setup
89+
│ ├── options.html ➤➤➤ Settings page HTML
90+
│ ├── options.css ➤➤➤ Settings page styling
91+
│ ├── github*.css ➤➤➤ Code syntax highlighting themes
92+
│ ├── highlight.min.js ➤➤➤ [External] Code highlighting library
93+
│ └── options.bundle.js ➤➤➤ [Generated] Bundled for browser
94+
├── bibMatcher ➤➤➤ BibTeX matching tool
95+
│ ├── bibMatcher.js ➤➤➤ Tool to match ArXiv papers to published versions
96+
│ ├── bibMatcher.html ➤➤➤ BibTeX matcher page HTML
97+
│ ├── testBibs.js ➤➤➤ Sample BibTeX entries for testing
98+
│ └── bibMatcher.bundle.js ➤➤➤ [Generated] Bundled for browser
99+
├── fullMemory ➤➤➤ Full-page memory view
100+
│ ├── fullMemory.js ➤➤➤ Dedicated tab for browsing your paper memory
101+
│ ├── fullMemory.html ➤➤➤ Full-page memory HTML
102+
│ └── fullMemory.bundle.js ➤➤➤ [Generated] Bundled for browser
103+
└── shared ➤➤➤ Shared utilities and resources
104+
├── css/ ➤➤➤ Shared stylesheets (variables, utilities, loading animations)
105+
├── js/
106+
│ ├── theme.js ➤➤➤ Theme detection (runs first, before ES modules)
107+
│ └── utils/ ➤➤➤ Core ES modules (the heart of PaperMemory)
108+
│ ├── config.js ➤➤➤ Global state, constants, and configuration
109+
│ ├── functions.js ➤➤➤ Utility functions used throughout the app
110+
│ ├── miniquery.js ➤➤➤ DOM utilities (custom jQuery-like functions)
111+
│ ├── data.js ➤➤➤ Data management - storage, migrations, validation
112+
│ ├── paper.js ➤➤➤ Paper operations - creation, updates, URL conversions
113+
│ ├── parsers.js ➤➤➤ Website parsers for different paper sources (ArXiv, Nature, etc.)
114+
│ ├── state.js ➤➤➤ App state management - memory initialization, sorting
115+
│ ├── sync.js ➤➤➤ GitHub sync functionality for data backup
116+
│ ├── urls.js ➤➤➤ URL parsing and paper ID extraction
117+
│ ├── files.js ➤➤➤ Local file detection and PDF management
118+
│ ├── bibtexParser.js ➤➤➤ BibTeX parsing and formatting
119+
│ ├── logTrace.js ➤➤➤ Development logging utilities
120+
│ └── octokit.bundle.js ➤➤➤ [Generated] GitHub API client bundle
121+
└── min/ ➤➤➤ [Generated] Shared build output
98122
```
99123

124+
**Key Source Files for Contributors:**
125+
126+
**Core Logic** (`shared/js/utils/`):
127+
128+
- `config.js` - Global state and settings. Start here to understand data structures.
129+
- `functions.js` - Utility functions used everywhere. Common operations like string parsing, clipboard access.
130+
- `data.js` - How papers are stored, validated, and migrated between versions.
131+
- `parsers.js` - Add new paper sources here. Contains website-specific parsing logic.
132+
- `paper.js` - Paper object operations. How papers are created, updated, and linked.
133+
134+
**User Interface** (`popup/js/`):
135+
136+
- `popup.js` - Main popup logic. How the extension popup initializes and displays papers.
137+
- `memory.js` - Memory table functionality. Search, sorting, and display of saved papers.
138+
- `handlers.js` - User interactions. Button clicks, keyboard shortcuts, form handling.
139+
140+
**Background Processing**:
141+
142+
- `background/background.js` - Handles sync, notifications, and browser API calls.
143+
- `content_scripts/content_script.js` - Automatically detects papers when browsing websites.
144+
145+
**Build System Features:**
146+
147+
- **ES modules**: All `.js` files use modern `import`/`export` syntax
148+
- **Path aliases**: `@pm/*` and `@pmu/*` for clean, readable imports
149+
- **Automatic bundling**: Rollup generates optimized `.bundle.js` files for browsers
150+
- **Smart CSS processing**: Unminified CSS in development, minified in production
151+
- **Direct file editing**: Edit source files directly, no preprocessing required
152+
100153
### Prettier
101154

102155
TODO
@@ -144,10 +197,10 @@ The following functions and constants should be updated:
144197
1. Changes in the memory should be reflected in the popup and vice-versa
145198
2. Document functions (docstrings & comments)
146199
3. Bump version
147-
4. Run `gulp build`
200+
4. Run `npm run build`
148201
5. Create a Github Release
149202
1. At least use the auto-complete release feature from PRs
150-
2. Add the Archive generated by `gulp build`
203+
2. Add the Archive generated by `npm run build`
151204
6. Upload the new package to Chrome & Firefox web stores 7. There's now a [Github action](https://github.com/vict0rsch/PaperMemory/actions/workflows/submit.yml) for that thanks to @louisgv in [#51](https://github.com/vict0rsch/PaperMemory/pull/51)
152205
7. If necessary update Github and stores visuals
153206

‎package.json‎

Lines changed: 16 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -9,26 +9,25 @@
99
"src"
1010
],
1111
"scripts": {
12-
"pretest": "echo 'Make sure to be in dev mode ($gulp dev)'",
12+
"pretest": "echo 'Make sure to be in dev mode (npm run dev)'",
1313
"test": "./node_modules/.bin/mocha test/test-*.js",
14-
"pretest-cov": "echo 'Make sure to be in dev mode ($gulp dev)'",
14+
"pretest-cov": "echo 'Make sure to be in dev mode (npm run dev)'",
1515
"test-cov": "./node_modules/.bin/nyc --reporter=text ./node_modules/.bin/mocha test/test-*.js",
16-
"pretest-storage": "echo 'Make sure to be in dev mode ($gulp dev)'",
16+
"pretest-storage": "echo 'Make sure to be in dev mode (npm run dev)'",
1717
"test-storage": "./node_modules/.bin/mocha test/test-storage.js",
18-
"pretest-duplicates": "echo 'Make sure to be in dev mode ($gulp dev)'",
18+
"pretest-duplicates": "echo 'Make sure to be in dev mode (npm run dev)'",
1919
"test-duplicates": "./node_modules/.bin/mocha test/test-duplicates.js",
20-
"pretest-sync": "echo 'Make sure to be in dev mode ($gulp dev)'",
20+
"pretest-sync": "echo 'Make sure to be in dev mode (npm run dev)'",
2121
"test-sync": "./node_modules/.bin/mocha test/test-sync.js",
22-
"pretest-no-browser": "echo 'Make sure to be in dev mode ($gulp dev)'",
22+
"pretest-no-browser": "echo 'Make sure to be in dev mode (npm run dev)'",
2323
"test-no-browser": "./node_modules/.bin/mocha 'test/test-!(storage|duplicates|sync)*.js'",
24-
"pretest-no-browser-cov": "echo 'Make sure to be in dev mode ($gulp dev)'",
24+
"pretest-no-browser-cov": "echo 'Make sure to be in dev mode (npm run dev)'",
2525
"test-no-browser-cov": "./node_modules/.bin/nyc --reporter=text ./node_modules/.bin/mocha 'test/test-!(storage|duplicates|sync)*.js'",
26-
"prescreenshots": "echo 'Make sure to be in dev mode ($gulp dev)'",
26+
"prescreenshots": "echo 'Make sure to be in dev mode (npm run dev)'",
2727
"screenshots": "./node_modules/.bin/mocha test/screenshots.js",
28-
"build:modules": "rollup -c",
29-
"build:modules:watch": "rollup -c -w",
30-
"build:modules:prod": "NODE_ENV=production rollup -c",
31-
"build:prod": "NODE_ENV=production gulp build"
28+
"dev": "rollup -c",
29+
"dev:watch": "rollup -c -w",
30+
"build": "NODE_ENV=production rollup -c"
3231
},
3332
"repository": {
3433
"type": "git",
@@ -47,29 +46,24 @@
4746
"@rollup/plugin-babel": "^6.0.4",
4847
"@rollup/plugin-commonjs": "^28.0.6",
4948
"@rollup/plugin-node-resolve": "^16.0.1",
49+
"@rollup/plugin-replace": "^6.0.2",
5050
"@rollup/plugin-terser": "^0.4.4",
5151
"browserify": "^17.0.0",
5252
"esmify": "^2.1.1",
5353
"expect": "^29.7.0",
5454
"glob": "^10.3.10",
55-
"gulp": "^4.0.0",
56-
"gulp-clean-css": "^4.3.0",
57-
"gulp-concat": "^2.6.1",
58-
"gulp-debug": "^5.0.1",
59-
"gulp-html-minifier-terser": "^7.1.0",
60-
"gulp-include": "^2.4.1",
61-
"gulp-preprocess": "^4.0.2",
62-
"gulp-rename": "^2.0.0",
63-
"gulp-uglify": "^3.0.2",
64-
"gulp-zip": "^6.0.0",
6555
"heap": "^0.2.7",
6656
"jsdom": "^23.0.1",
6757
"mocha": "^10.2.0",
6858
"nyc": "^15.1.0",
6959
"ora": "^5.4.1",
60+
"postcss": "^8.5.6",
7061
"puppeteer": "^18.2.1",
7162
"readline-sync": "^1.4.10",
7263
"rollup": "^4.45.0",
64+
"rollup-plugin-copy": "^3.5.0",
65+
"rollup-plugin-delete": "^3.0.1",
66+
"rollup-plugin-postcss": "^4.0.2",
7367
"uuid": "^9.0.1",
7468
"yaml": "^2.3.4"
7569
}

0 commit comments

Comments
 (0)