INFOGLOBO CHALLENGE

This is a project to crawler RSS Feed and transpile to json format.

Architecture schema

                   ----- Account (django) - \
       (node)    /                            -> database (postgres)
---> Api Gateway   ----- Auth (django) ---- /
                 \
                   ----- RSS (flask) -> (Feed URL)

• Api Gateway domain

  • Api Gateway manager micro services routers
  • Api Gateway guarantee authetication if has JWT token
  • Api Gateway generate idLog and idReq for each request (this is used to tracker user requests and tracker request paths)
  • Api Gateway logger all input and output data and request status code

• Account Domain

  • Account manager user in database (create, change, delete)
  • Account change and delete need authentication

• Auth Domain

  • Auth manager token (create and verify)
  • Generate token if user already registered

• RSS Domain

  • RSS transpile RSS Feed to JSON
  • RSS transpile need authentication

• All Micro Services

  • All microservice logger in /logs/requests.log

Main technologies used in the project

1. Languages

   Python 
   Javascript (node)
   

2. HTTP Frameworks

   Django (python)
   Flask (python)
   express (node)
   

3. DevOps Domain

   docker (development)
   docker-compose (development)
   
   zappa (deploy django aws lambda)
   serverless (deploy node aws lambda)
   
   bitbucket-pipeline (CI/CD)
   

How to use project

• Configure settings file
  Inside each project has a settings.sample.ini copy this and rename to settings.ini

  cp ./settings.sample.ini ./settings.ini
  

• Start the project

  docker-compose up
  

• Check if all is ok

  GET in http://localhost:80
  

Project URL's

• if you have Postman
  Open the Postman.
  In the top left of app has a import next to new, select tab Import for link and past this:

  https://www.getpostman.com/collections/274374b88a94fff38349
  

  after this, create environment for a project, and add these key/value: gateway = http://localhost:80

• if not

  • Api Gateway
    • GET /
  • Auth
    • POST /v1/auth/login
    • POST /v1/auth/verify
  • Account
    • POST /v1/accounts
    • GET /v1/accounts/my-account (Auth)
    • PUT /v1/accounts/my-account (Auth)
    • PATCH /v1/accounts/my-account (Auth)
    • DELETE /v1/accounts/my-account (Auth)
  • RSS
    • GET /v1/rss/json (Auth)

OBS: All urls have a body validator, because this i don't specify body params here, after your hit a url you're receive a body params.
In the postman url, all body params is present in saved urls.

OBS: to change password in Account use PUT or PATCH url and pass password key/value in the body.

How to auth in the project ?

• if you have Postman using my project url

  • Create account in POST /v1/accounts
  • Get a token in POST /v1/auth/login
  • and be happy (i created a script in postman url to set auth_token every time you create a token ;) )
  • don't forget to try to acess a authenticated url

• if not

  • First step is cry
  • Create account in POST /v1/accounts
  • Get a token in POST /v1/auth/login
  • Set access token in header Authorization, remenber to add JWT before the token
  • try to acess a authenticated url
  • OBS: All request need Authorization header

Have you looked at the logs?

After run in the first time, in the root of project folders it is created a ./logs/requests.log look to this.

SPOILER ALERT

all apps logger in this fille, and all logger contains idLog and idReq, idLog is unique browser id and idReq is generated every time wich user get a new url (however this last id is unique in all steps of request, is possible to track start point and steps in the middle of the requests)

• How this work ?
  • idLog is generated by api gateway and save in cookie of browser and set in header['idLog'] of each request
  • idReq is generated by api gateway for each received request, and set in header['idReq'] of each request
  • for each request from a external micro service this headers is sended to keep request tracker

How to run project test

1. use docker-compose

   docker-compose run (account,auth,gateway or rss) run-test
   

2. inside docker-compose (no need to download all dependency every time)

   docker-compose run (account,auth,gateway or rss) /bin/sh
   run-test
   

3. manually

   # create a virtulenv
   virtualenv -p python venv
   ./scripts/test.sh # this install requirements
   

Obs: auth micro services has test suit configurated, but no has test case, because no has developed code, just third part apps.

All micro services has /script/ folder this is a script to start, run test and deploy applications. all scripts is executable.

Project structure explanation

• /config -> all app configuration files (not exist in node app)
• /app or /src is a folder to develop your code here, named /app in python and /src in node.
  • /shared is a code shared between all apps of aplication (reusable codes, middleware, interceptors)
  • /app-name is a self contained app, no has dependecy of another apps to work
    • api_v* is a folder to versioning api of app, used to more organization in the future when app has many versions URLs
• /scripts/ each script of application, facilitators scripts, test scripts, deploy script and more.

OBS: In python structures, we try to keep django syntax, because it's the best known python framework, and it's simple to teach another framework to others people if the syntax and organization is similar.

ISN'T READY FOR PRODUCTION

all apps are running using the framework webserver, for this reason not use this in production.
configure apps correctly for production, has many tutorials in the web

Another OBS

• TODO Attention -> if use vscode, i suggest to install Todo Tree in your IDE to see all todo's in the code.
• Python Linter -> use pylint and pep8 and set line length to 100
• Node Linter -> use prettier and set to use single quote
• Linters -> I suggest to enable format on save and dissable auto save in your IDE

Future improvements

• SWAGGER -> ADD swagger in all micro services for mapper apis
• CACHE -> ADD cache on RSS URL, because it has static content throughout the day
• API GATEWAY -> Improve jwt validation (see a TODO inside /src/shared/middleware/auth)
• API GATEWAY -> Improve inputd data logger (see a TODO inside /src/shared/middleware/logger)