A solution for continuous deployment

[hosseinzoda]

Why It's needed?

1. Netlify has request rate limit. https://www.netlify.com/docs/api/#rate-limiting
2. Making it reliable and scalable

My suggestion

By setting up deploy window we can fix this issue.

What's good about it

1. We can determine the next deploy time and show it to the user.
2. The current setup is compatible with it.

Core: Depending on few variables and the state of deployment determine next deploy or git push trigger

[hosseinzoda]

This also helps with handling concurrent add requests.

[hosseinzoda]

To clarify.
I mean we should manage when to actually do add project. https://github.com/openassistive/OpenATBackend/blob/master/js/savejson/savejson.controller.js#L43

So first thing comes to mind is to create a queue of add projects until deploy window finishes. And then push to git

[hosseinzoda]

Anyway this is not an issue for now.

[willwade]

There are problems creating our own CI system:

1. Time and effort
2. Where is it hosted. We currently run the api off heroku off a low-priced tier. Its poor for hosting content.
3. We could use github pages - however its difficult to get SSL on this. We want this to be as low-cost and as flexible to re-host as possible. Using cloudflare, etc to get round these issues is a headache..

So the real issue is:

1. Netlify has rate limiting. In the first 6 months we may have a lot of adds' causing us to break the limit. thats not good

And on a side-note:

1. Im worried about spam/poorly written entries. i.e. there is no moderation. Equally though I'm concerned that in a moderated system we won't get round to moderating stuff quick enough..

Hmm. How about:

• On /save we commit to gitHub as is
• We run the new caching service service to cache all items locally.  #7
• We only deploy once a hour IF the cache is different from the last deploy time. (This would require a new service that is run on a timed basis.. cron?)

AND

• we tag all entries as "to-be-moderated"
• if an item has "to-be-moderated" tag then we colour it or label it different on the frontend

[hosseinzoda]

we tag all entries as "to-be-moderated"
if an item has "to-be-moderated" tag then we colour it or label it different on the frontend

Correct, This helps a lot on filtering contents. Also we can associate an ip address of issuer to it. So we can easily remove spams.

There are problems creating our own CI system:

I did not suggest to create another CI. My suggestion was close your instructions. By limiting amount of deploy request this will get fixed. (Also con-current requests will get fixed)
Here's the instruction for my suggestion

1. On /save add the save content to the queue. And set a cron-job on the server for next check for processing deploy.
2. On deploy commit to github.

Now next time to process a deploy may vary depending on few parameters and state of deployment.

[willwade]

Ah ok. Makes sense. I think we are agreeing then!

One small problem though with a queue.. What if someone goes and adds something directly to GitHub inbetween. Wouldn't this create a conflict

(Maybe so unlikely we ignore this edgecase)

Instead - commit everything to GitHub as we do. And only deploy once an hour. To reduce over hitting gitHub however keep the local queue to 1 minute.

(Just leaving this here: https://devcenter.heroku.com/articles/scheduler#installing-the-add-on. Basically we just need a service e.g. /deploy that is accessible via the web. Would be useful to restrict this to an IP)

[hosseinzoda]

/save does know what to do on conflict. It will replace it.

(Just leaving this here: https://devcenter.heroku.com/articles/scheduler#installing-the-add-on. Basically we just need a service e.g. /deploy that is accessible via the web. Would be useful to restrict this to an IP)

Correct.

[willwade]

/save does know what to do on conflict. It will replace it.

Yep. Whats your thoughts on:

Instead - commit everything to GitHub as we do. And only deploy once an hour. To reduce over hitting gitHub however keep the local queue to 1 minute.

?

[hosseinzoda]

Instead - commit everything to GitHub as we do. And only deploy once an hour. To reduce over hitting gitHub however keep the local queue to 1 minute.

I think It's dependent to build time (time it takes to complete a deploy process). Like if It's 10 seconds then margin between deploy triggers can be one minute.

Github does not have much to do on changes occur, I think It's inexpensive task.

Anyway one or few build(s) every minute will not exceed rate limit.

[hosseinzoda]

We can write some code to figure this out. depending on previous build time.

[willwade]

Its quick right now. My guess under 2 minutes. but yeah - we can work out exact timing as we go