Add ability to override existing docs with yuidoc from local

Author: sivakumar-kailasam

README.md

@@ -1,6 +1,6 @@
 # Ember JSON API Docs [![Build Status](https://travis-ci.org/ember-learn/ember-jsonapi-docs.svg?branch=master)](https://travis-ci.org/ember-learn/ember-jsonapi-docs)
 
-If are looking for the app behind https://emberjs.com/api/, visit 
+If are looking for the app behind https://emberjs.com/api/, visit
 [ember-api-docs](https://github.com/ember-learn/ember-api-docs) instead. This ember-jsonapi-docs
 repository is internal tooling that is not required to run the ember-api-docs app locally.
 
@@ -11,42 +11,52 @@ The script pulls yuidoc build output from all Ember versions from Amazon S3, con
 
 ## Running the app
 
-1. Fork/Clone [ember-jsonapi-docs](https://github.com/ember-learn/ember-jsonapi-docs)
-1. Run `yarn` or `npm install` (Needs node 6+)
-1. Set up AWS access
+1.  Fork/Clone [ember-jsonapi-docs](https://github.com/ember-learn/ember-jsonapi-docs)
+1.  Run `yarn` or `npm install` (Needs node 6+)
+1.  Set up AWS access
+
     ```shell
     export AWS_ACCESS_KEY=xxxxxx
     export AWS_SECRET_KEY=xxxxx
     ```
+
     The app accesses builds.emberjs.com (an Amazon S3 bucket) in read-only mode, which is public. This requires any valid AWS credentials.
 
     You can get your credentials by logging into your [AWS console](https://console.aws.amazon.com) and navigating to "_My Security Credentials_" under your profile name. You can generate a new pair under the "_Access Keys (Access Key ID and Secret Access Key)_" section.
-1. To test your changes in the app run,
-   ```node index.js```
-   Once complete, if no errors you should see a docs.tar file inside the `tmp` folder. The app tries to process all
-   ember & ember-data versions since 1.0 which takes high memory & time to complete. If you intend it, then run `node --max_old_space_size=8192 index.js`.
-   You are setting your node max heap space to 8GB, so make sure you have that much space available on your machine.
 
+1.  To test your changes in the app run,
+    `node index.js`
+    Once complete, if no errors you should see a docs.tar file inside the `tmp` folder. The app tries to process all
+    ember & ember-data versions since 1.0 which takes high memory & time to complete. If you intend it, then run `node --max_old_space_size=8192 index.js`.
+    You are setting your node max heap space to 8GB, so make sure you have that much space available on your machine.
 
 ## To Generate docs for a specific project and/or version for development
 
 You can do this by passing `--project ember/ember-data --version 2.11.1` as an argument to the index script. e.g., `yarn start -- --project ember --version 2.11.0`.
-Setting `export SKIP_S3_SYNC=yes` will stop the generator from syncing s3 content. You need an additional flag `AWS_SHOULD_PUBLISH=true` for publishing the docs.
+You need an additional flag `AWS_SHOULD_PUBLISH=true` for publishing the docs.
+
+## To override a specific version of a doc with a different yuidoc from your machine (For core contributors)
+
+- Read this section first!
+  - We assume you have the keys to the kingdom before you start doing this (AWS keys to publish to api-docs.emberjs.com & all necessary env variables that need to be set) 😄
+  - In its present form this should be used only when there aren't new docs out there that are yet to be processed. As in if ember 3.3 is released but isn't indexed yet you should wait for this app to finish processing it via the cron job we have on heroku before you can proceed to modify any of the existing docs from your machine.
+- First run `yarn start` so that you have all the docs from s3 on your local. This is important so that we don't loose other versions of docs that are already out there when the index files are generated
+- Then go to `./tmp/s3-docs/<the_version_you_want_to_replace>` and override the file there with the yuidoc file that you want to be processed. Ensure that the file name is same as the one that's already there.
+- Then run `yarn start --project=ember-data --version=3.2.0 --ignorePreviouslyIndexedDoc`. Make sure you enter the entire version(including patch version).
 
 ## Generating API Documentation and Testing API Docs Locally
 
 These steps are only necessary if you are trying to run the ember-api-docs
 app with documentation pulled from a local copy of ember.js.
 
-1. Clone the following 4 repositories into a single parent directory. Install dependencies for each app as described in their respective `README` files.
-   - [ember.js](https://github.com/emberjs/ember.js)
-   - [data (ember data)](https://github.com/emberjs/data)
-   - [ember-jsonapi-docs](https://github.com/ember-learn/ember-jsonapi-docs)
-   - [ember-api-docs](https://github.com/ember-learn/ember-api-docs)
-1. Set up the project according to the instructions above in `Running the app`.
-1. From the `ember-jsonapi-docs` directory, run `./generate-local.sh yui ember 2.18.0`. This command runs the Ember documentation build, generates jsonapi output, and copies it to the `ember-api-docs` directory. To build ember data documentation, run `./generate-local.sh yui ember-data 2.17.2`.
-   - _If you encounter an error like `ember-2.18.0 has already been indexed in json-docs`, then use a new unique version number like `2.18.1`, or whatever is appropriate. 
-   - If your `rev-index/ember-X.X.X.json` file fails to generate, make sure you have all dependencies installed for the ember.js repo 
-   - If you are debugging failed builds, periodically clear out the contents of the `tmp` directory, and run the script again. Past failed runs can cause subsequent runs to fail in unexpected ways._ 
-1. Run the API app with the newly generated local data by running `API_HOST=http://localhost:4200 ember s` in the `ember-api-docs` directory.
-
+1.  Clone the following 4 repositories into a single parent directory. Install dependencies for each app as described in their respective `README` files.
+    - [ember.js](https://github.com/emberjs/ember.js)
+    - [data (ember data)](https://github.com/emberjs/data)
+    - [ember-jsonapi-docs](https://github.com/ember-learn/ember-jsonapi-docs)
+    - [ember-api-docs](https://github.com/ember-learn/ember-api-docs)
+1.  Set up the project according to the instructions above in `Running the app`.
+1.  From the `ember-jsonapi-docs` directory, run `./generate-local.sh yui ember 2.18.0`. This command runs the Ember documentation build, generates jsonapi output, and copies it to the `ember-api-docs` directory. To build ember data documentation, run `./generate-local.sh yui ember-data 2.17.2`.
+    - \_If you encounter an error like `ember-2.18.0 has already been indexed in json-docs`, then use a new unique version number like `2.18.1`, or whatever is appropriate.
+    - If your `rev-index/ember-X.X.X.json` file fails to generate, make sure you have all dependencies installed for the ember.js repo
+    - If you are debugging failed builds, periodically clear out the contents of the `tmp` directory, and run the script again. Past failed runs can cause subsequent runs to fail in unexpected ways.\_
+1.  Run the API app with the newly generated local data by running `API_HOST=http://localhost:4200 ember s` in the `ember-api-docs` directory.

index.js

@@ -1,4 +1,14 @@
 // eslint-disable-next-line
 require = require('esm')(module /*, options*/)
+const argv = require('minimist')(process.argv.slice(2))
+
+let possibleProjects = ['ember', 'ember-data']
+let projects =
+	argv.project && possibleProjects.includes(argv.project) ? [argv.project] : possibleProjects
+let specificDocsVersion = argv.version ? argv.version : ''
+
+let ignorePreviouslyIndexedDoc =
+	projects.length !== 0 && specificDocsVersion !== '' && argv.ignorePreviouslyIndexedDoc
+
 const { apiDocsProcessor } = require('./main.js')
-apiDocsProcessor()
+apiDocsProcessor(projects, specificDocsVersion, ignorePreviouslyIndexedDoc)

lib/fetch-yui-docs.js

@@ -38,11 +38,14 @@ function getObjects() {
 	})
 }
 
-function downloadFile({ Key }) {
+const getSrcFilePath = Key => {
 	let name = path.basename(Key)
 	let dir = path.basename(path.dirname(Key))
+	return path.join('tmp', 's3-docs', dir, name)
+}
 
-	let finalFile = path.join('tmp', 's3-docs', dir, name)
+function downloadFile({ Key }) {
+	let finalFile = getSrcFilePath(Key)
 
 	return mkdirp(path.dirname(finalFile)).then(() => {
 		return new RSVP.Promise((resolve, reject) => {
@@ -77,7 +80,7 @@ function filterReleaseDocs({ Key }) {
 	return versionRegex.test(tag) && /-docs\.json/.test(key)
 }
 
-export default function fetchYuiDocs(projects, specificDocsVersion) {
+export default function fetchYuiDocs(projects, specificDocsVersion, ignorePreviouslyIndexedDoc) {
 	return getObjects().then(docs => {
 		let projectFiles = projects.map(p => `${p}-docs.json`)
 
@@ -89,7 +92,16 @@ export default function fetchYuiDocs(projects, specificDocsVersion) {
 			)
 
 		if (!isEmpty(trim(specificDocsVersion))) {
-			return RSVP.map(filteredDocs, downloadFile)
+			if (ignorePreviouslyIndexedDoc) {
+				return RSVP.map(filteredDocs, ({ Key }) => {
+					console.log(
+						`Skipping download of ${Key} so that you can use your local file. We hope you know what you're doing 😅`
+					)
+					return getSrcFilePath(Key)
+				})
+			} else {
+				return RSVP.map(filteredDocs, downloadFile)
+			}
 		}
 
 		let projectDocs = groupBy(filteredDocs, ({ Key }) => Key.split('/')[2].replace('.json', ''))

lib/read-docs.js

@@ -4,7 +4,11 @@ import path from 'path'
 import RSVP from 'rsvp'
 import { dasherize } from 'inflected'
 
-export default function readDocs(projects, specificVersion = '') {
+export default function readDocs(
+	projects,
+	specificVersion = '',
+	ignorePreviouslyIndexedDoc = false
+) {
 	console.log('Reading project files')
 
 	let docs = {}
@@ -22,7 +26,7 @@ export default function readDocs(projects, specificVersion = '') {
 		}
 
 		folders = folders.filter(f => {
-			if (!prevIndexedVersions.includes(f)) {
+			if (!prevIndexedVersions.includes(f) || ignorePreviouslyIndexedDoc) {
 				return f
 			} else {
 				let version = f.replace('tmp/s3-docs/v', '').replace(`/${projectName}-docs.json`, '')

lib/s3-sync.js

@@ -8,7 +8,7 @@ import humanSize from 'human-size'
 // To increase s3's download & upload dir perf
 http.globalAgent.maxSockets = https.globalAgent.maxSockets = 30
 
-const { AWS_ACCESS_KEY, AWS_SECRET_KEY, AWS_SHOULD_PUBLISH, SKIP_S3_SYNC } = process.env
+const { AWS_ACCESS_KEY, AWS_SECRET_KEY, AWS_SHOULD_PUBLISH } = process.env
 
 const client = S3.createClient({
 	s3Options: {
@@ -75,17 +75,13 @@ const syncDir = (operation, options) => {
 }
 
 export function downloadExistingDocsToLocal() {
-	if (!SKIP_S3_SYNC) {
-		let revDocsDirDownloadOptions = revDocsDirUploadOptions
-		delete revDocsDirDownloadOptions.s3Params.GrantRead
+	let revDocsDirDownloadOptions = revDocsDirUploadOptions
+	delete revDocsDirDownloadOptions.s3Params.GrantRead
 
-		return waitForAllPromises([
-			syncDir('download', jsonDocsDirDownloadOptions),
-			syncDir('download', revDocsDirDownloadOptions),
-		])
-	}
-	console.log('Skipping download of pre-cached docs')
-	return Promise.resolve()
+	return waitForAllPromises([
+		syncDir('download', jsonDocsDirDownloadOptions),
+		syncDir('download', revDocsDirDownloadOptions),
+	])
 }
 
 export function uploadToS3() {

main.js

@@ -1,6 +1,5 @@
 import RSVP from 'rsvp'
 import fs from 'fs-extra'
-const argv = require('minimist')(process.argv.slice(2))
 
 import markup from './lib/markup'
 import readDocs from './lib/read-docs'
@@ -13,23 +12,18 @@ import saveDoc from './lib/save-document'
 import revProjVersionFiles from './lib/rev-docs'
 import { downloadExistingDocsToLocal, uploadToS3 } from './lib/s3-sync'
 
-export function apiDocsProcessor() {
+export function apiDocsProcessor(projects, specificDocsVersion, ignorePreviouslyIndexedDoc) {
 	RSVP.on('error', reason => {
 		console.log(reason)
 		process.exit(1)
 	})
 
-	let possibleProjects = ['ember', 'ember-data']
-	let projects =
-		argv.project && possibleProjects.includes(argv.project) ? [argv.project] : possibleProjects
-	let specificDocsVersion = argv.version ? argv.version : ''
-
 	let docsVersionMsg = specificDocsVersion !== '' ? `. For version ${specificDocsVersion}` : ''
 	console.log(`Downloading docs for ${projects.join(' & ')}${docsVersionMsg}`)
 
 	downloadExistingDocsToLocal()
-		.then(() => fetchYuiDocs(projects, specificDocsVersion))
-		.then(() => readDocs(projects, specificDocsVersion))
+		.then(() => fetchYuiDocs(projects, specificDocsVersion, ignorePreviouslyIndexedDoc))
+		.then(() => readDocs(projects, specificDocsVersion, ignorePreviouslyIndexedDoc))
 		.then(docs => {
 			return RSVP.map(projects, projectName => {
 				return RSVP.map(docs[projectName], doc => {