AWS Fargate Support (#244)

* AWS Fargate Agent, Option & Recorder

* Break Collector into platform parts

* Initial Fargate tests; AWSFargateAgent/Collector/Options

* Fix Fargate Collector instantiation

* Include package file

* Fargate snapshot collection for Task, Containers, Docker & Process

* Centralized environment detection

* Manual check to avoid circular dependency

* Restore print_function

* Better background thread; docs & tests

* Check existence before referencing

* No unicode characters for py2

* Rename method

* Remove more unicode chars

* Test: Show extra spans if any

* Dump the full span

* Remove debug

* Move span dump into its own helper method

* Add debug checks

* Dont test log spans here

* better way to filter log spans

* Fargate checks

* Temporarily disable lambda instrumentation

* Logger vs print

* Fix egid + euid calcs

* Less debug and more safeties

* No root path in root url

* Fix payload & add tests

* Convert JSON responses

* Remake headers each time for now

* Log detected environment

* Breakout data collection into helpers for simplicity

* Set INSTANA_TEST for test runs

* Migrate Python metric collection to dedicated helper

* Fix entityId

* Cleanup Options; Add all supported env vars

* Normalize and standardize on extra_http_header handling

* Uncommong lambda inst

* Same service_name handling regardless of environment

* Skip asynqp tests which are breaking in Python 3.8

* Use a default agent if not specified

* Late import to avoid circular import

* Remove unecessary logging

* Py27 compatibility

* Add support for INSTANA_ZONE

* Global INSTANA_SECRETS support

* Apply secrets check to procces env vars; cleanup

* INSTANA_TAGS support; Helper stability improvements

* Fix test stragglers

* Add support for INSTANA_ENDPOINT_PROXY

* Endpoint proxy tests

* Reset proxy var in tests

* Add Lambda & Fargate boot messages

* Version path for fargate

* Pylint fixes

* Docker metrics support

* Refactor, normalization and all the other cool words

* PyLint told me to do it

* Refactor and respect snapshot reporting flag

* Subpackages not needed

* Linter fixes

* Update imports with new path

* Without snapshot flag, do nothing

* Make sure file exists b4 trying to read it

* Fix test class names

* Minor cleanup

* Update python containers to latest

* Update zone and tags handling

* Package collection cleanup

* Add to_pretty_json helper

* Debug helpers

* Round CPU floats and rootbeer floats too

* Fetch ECMU metadata only on interval

* Lock cleanup

* Better lock syncronization

* Fix lock check

* Set lock acquire to blocking

* Consider with_snapshot in delta reporting

* Fix reporting of mandatory fields

* Fix data payload init

* Simplify things and remove a thread

* Change test container image to make cassandra happy

* Py27 tests use stretch

* Add Python 2.7 compatibility division

* pylint all the things

* Updated hierarchy of Option classes; INSTANA_TIMEOUT in ms

* Improved log level handling & tests

* Refactor Host agent metric collection

* Dot the Is

* Remove debug

* Docker blkio metrics are accumalative

Co-authored-by: Peter Giacomo Lombardo <[email protected]>

Author: Peter Giacomo Lombardo

.circleci/config.yml

@@ -6,7 +6,7 @@ version: 2
 jobs:
   python27:
     docker:
-      - image: circleci/python:2.7.15
+      - image: circleci/python:2.7.16-stretch
       - image: circleci/postgres:9.6.5-alpine-ram
       - image: circleci/mariadb:10.1-ram
       - image: circleci/redis:5.0.4
@@ -35,13 +35,15 @@ jobs:
             pip install -e '.[test]'
       - run:
           name: run tests
+          environment:
+            INSTANA_TEST: true
           command: |
             . venv/bin/activate
             pytest -v
 
   python38:
     docker:
-      - image: circleci/python:3.7.7-stretch
+      - image: circleci/python:3.7.8-stretch
       - image: circleci/postgres:9.6.5-alpine-ram
       - image: circleci/mariadb:10-ram
       - image: circleci/redis:5.0.4
@@ -67,13 +69,15 @@ jobs:
             pip install -e '.[test]'
       - run:
           name: run tests
+          environment:
+            INSTANA_TEST: true
           command: |
             . venv/bin/activate
             pytest -v
 
   py27cassandra:
     docker:
-      - image: circleci/python:2.7.15
+      - image: circleci/python:2.7.16-stretch
       - image: circleci/cassandra:3.10
         environment:
           MAX_HEAP_SIZE: 2048m
@@ -94,13 +98,16 @@ jobs:
             pip install -e '.[test-cassandra]'
       - run:
           name: run tests
+          environment:
+            INSTANA_TEST: true
+            CASSANDRA_TEST: true
           command: |
             . venv/bin/activate
-            CASSANDRA_TEST=1 pytest -v tests/clients/test_cassandra-driver.py
+            pytest -v tests/clients/test_cassandra-driver.py
 
   py36cassandra:
     docker:
-      - image: circleci/python:3.6.8
+      - image: circleci/python:3.6.11
       - image: circleci/cassandra:3.10
         environment:
           MAX_HEAP_SIZE: 2048m
@@ -118,13 +125,16 @@ jobs:
             pip install -e '.[test-cassandra]'
       - run:
           name: run tests
+          environment:
+            INSTANA_TEST: true
+            CASSANDRA_TEST: true
           command: |
             . venv/bin/activate
-            CASSANDRA_TEST=1 pytest -v tests/clients/test_cassandra-driver.py
+            pytest -v tests/clients/test_cassandra-driver.py
 
   gevent38:
     docker:
-      - image: circleci/python:3.8.2
+      - image: circleci/python:3.8.5
     working_directory: ~/repo
     steps:
       - checkout
@@ -138,9 +148,12 @@ jobs:
             pip install -e '.[test-gevent]'
       - run:
           name: run tests
+          environment:
+            INSTANA_TEST: true
+            GEVENT_TEST: true
           command: |
             . venv/bin/activate
-            GEVENT_TEST=1 pytest -v tests/frameworks/test_gevent.py
+            pytest -v tests/frameworks/test_gevent.py
 workflows:
   version: 2
   build: