Code Level Metrics (CodeStream) (#513)

* Initial experiment

Co-authored-by: Lalleh Rafeei <[email protected]>

* Expand source code testing

* Move source code context to traces

* Add source context to external traces

* First pass at finalizing attributes

* Inspection tweaking

* Attribute cacheing

Co-authored-by: Lalleh Rafeei <[email protected]>

* Add initial source context setting.

* Revert "Add initial source context setting."

This reverts commit 1512e19.

* Move source code context init

Co-authored-by: Uma Annamalai <[email protected]>
Co-authored-by: Lalleh Rafeei <[email protected]>

* Add setting to gate attribute collection.

* Add source code node

* Add database trace source context

* Add error trace attributes

* Add source code context to transaction types

* Fix settings in timetrace enter

* Fix function traces in flask

Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>

* Remove attributes from transactions

* Instrument django with source context

* Update all traces to include source context kwargs

* Wire source context into wsgi applications

* Source context instrumentation updates

* Wire source context into sentinels

* Add framework instrumentation updates.

* Swap FunctionTrace for FunctionTraceWrapper (or modify FunctionTrace to include "source")

* Finish framework instrumentation updates

* Add source to MessageTrace.

* Add setting override for source context tests

Co-authored-by: Lalleh Rafeei <[email protected]>

* Fix validate_database_trace_inputs validator

Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>

* Fix redis wrappers

* Add source context to pika instrumentation.

* Fix failing tests

* Fix asyncpg wrappers

* Add source context to SolrTrace.

* Add source context to strawberry

* Fix strawberry versions

* Update Twisted framework instrumentation with source context.

* Clean up code

* More code cleanup

* Enable codestream attributes by default

* Fix Django CBV dispatch wrapper.

* TEMP: update agent version guessing logic.

* Fix django rest CBV dispatch wrapper.

* Add better discovery and testing for builtins

Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>

* Update agent version.

* Fix gRPC web transaction source issue.

* Rename to code level metrics

* Add testing for code level metrics

* Rename extract function

* Fix extraction of objects and class types

* Add testing for tornado

Co-authored-by: Uma Annamalai <[email protected]>

* Add flask testing

* Add grpc testing

* Fix celery instrumentation

* Add starlette testing

* Fastapi tests

* Falcon tests

* Django rest testing

* Aiohttp tests

* Starting django tests

* Add CodeStream spans for GraphQL

* Fix pyramid golden signals

* Add cherrypy testing

* Correct flask rest views

Co-authored-by: Uma Annamalai <[email protected]>

* Fix sanic golden signals

* Fix partials inspection for code level metrics

* Add pika codestream tests

* Add django codestream tests

* Remove beta versioning logic.

* Patch pika tests for py2

* Fix Py2 issues

* Pin grpc of agent_streaming py27

* Standardize all python workflows

* Fix graphql 2 middleware double wrapping

* Fix graphql span counts

* Fix flask dependencies

* More exact pypy versions

* Fix tox running in pypy

* Rearrange python versions

* Move pypy3 to 3.6

* Update cross agent tests for AWS

* Fix PyPy code level metrics tests

* Fix flask blueprints tests

* Fix graphqlserver tests

* Patch py27 django rest tests

* Fix py2 syntax

* Fix py2 metrics for flaskrest

Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>
Co-authored-by: Lalleh Rafeei <[email protected]>

Author: 3 people

.github/workflows/tests.yml

@@ -215,7 +215,7 @@ async def send_inject_browser_agent(self, message):
 
 
 class ASGIWebTransaction(WebTransaction):
-    def __init__(self, application, scope, receive, send):
+    def __init__(self, application, scope, receive, send, source=None):
         self.receive = receive
         self._send = send
         scheme = scope.get("scheme", "http")
@@ -237,6 +237,7 @@ def __init__(self, application, scope, receive, send):
             request_path=request_path,
             query_string=query_string,
             headers=headers,
+            source=source,
         )
 
         if self._settings:
@@ -314,6 +315,7 @@ async def nr_async_asgi(receive, send):
                 scope=scope,
                 receive=receive,
                 send=send,
+                source=wrapped,
             ) as transaction:
 
                 # Record details of framework against the transaction for later

newrelic/api/asgi_application.py

@@ -24,11 +24,11 @@
 
 class BackgroundTask(Transaction):
 
-    def __init__(self, application, name, group=None):
+    def __init__(self, application, name, group=None, source=None):
 
         # Initialise the common transaction base class.
 
-        super(BackgroundTask, self).__init__(application)
+        super(BackgroundTask, self).__init__(application, source=source)
 
         # Mark this as a background task even if disabled.
 
@@ -97,7 +97,7 @@ def create_transaction(transaction):
 
                 return None
 
-            return BackgroundTask(_application, _name, _group)
+            return BackgroundTask(_application, _name, _group, source=wrapped)
 
         proxy = async_proxy(wrapped)
 

newrelic/api/background_task.py

@@ -68,12 +68,12 @@ def __init__(
         database_name=None,
         **kwargs
     ):
-        parent = None
+        parent = kwargs.pop("parent", None)
+        source = kwargs.pop("source", None)
         if kwargs:
-            if len(kwargs) > 1:
-                raise TypeError("Invalid keyword arguments:", kwargs)
-            parent = kwargs["parent"]
-        super(DatabaseTrace, self).__init__(parent)
+            raise TypeError("Invalid keyword arguments:", kwargs)
+
+        super(DatabaseTrace, self).__init__(parent=parent, source=source)
 
         self.sql = sql
 
@@ -260,7 +260,7 @@ def _nr_database_trace_wrapper_(wrapped, instance, args, kwargs):
         else:
             _sql = sql
 
-        trace = DatabaseTrace(_sql, dbapi2_module, parent=parent)
+        trace = DatabaseTrace(_sql, dbapi2_module, parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)

newrelic/api/database_trace.py

@@ -56,12 +56,12 @@ class DatastoreTrace(TimeTrace):
     """
 
     def __init__(self, product, target, operation, host=None, port_path_or_id=None, database_name=None, **kwargs):
-        parent = None
+        parent = kwargs.pop("parent", None)
+        source = kwargs.pop("source", None)
         if kwargs:
-            if len(kwargs) > 1:
-                raise TypeError("Invalid keyword arguments:", kwargs)
-            parent = kwargs["parent"]
-        super(DatastoreTrace, self).__init__(parent)
+            raise TypeError("Invalid keyword arguments:", kwargs)
+
+        super(DatastoreTrace, self).__init__(parent=parent, source=source)
 
         self.instance_reporting_enabled = False
         self.database_name_enabled = False
@@ -187,7 +187,7 @@ def _nr_datastore_trace_wrapper_(wrapped, instance, args, kwargs):
         else:
             _operation = operation
 
-        trace = DatastoreTrace(_product, _target, _operation, parent=parent)
+        trace = DatastoreTrace(_product, _target, _operation, parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)

newrelic/api/datastore_trace.py

@@ -23,12 +23,12 @@
 
 class ExternalTrace(CatHeaderMixin, TimeTrace):
     def __init__(self, library, url, method=None, **kwargs):
-        parent = None
+        parent = kwargs.pop("parent", None)
+        source = kwargs.pop("source", None)
         if kwargs:
-            if len(kwargs) > 1:
-                raise TypeError("Invalid keyword arguments:", kwargs)
-            parent = kwargs["parent"]
-        super(ExternalTrace, self).__init__(parent)
+            raise TypeError("Invalid keyword arguments:", kwargs)
+
+        super(ExternalTrace, self).__init__(parent=parent, source=source)
 
         self.library = library
         self.url = url
@@ -94,7 +94,7 @@ def dynamic_wrapper(wrapped, instance, args, kwargs):
         else:
             _method = method
 
-        trace = ExternalTrace(library, _url, _method, parent=parent)
+        trace = ExternalTrace(library, _url, _method, parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)
@@ -111,7 +111,7 @@ def literal_wrapper(wrapped, instance, args, kwargs):
         else:
             parent = None
 
-        trace = ExternalTrace(library, url, method, parent=parent)
+        trace = ExternalTrace(library, url, method, parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)

newrelic/api/external_trace.py

@@ -23,12 +23,12 @@
 
 class FunctionTrace(TimeTrace):
     def __init__(self, name, group=None, label=None, params=None, terminal=False, rollup=None, **kwargs):
-        parent = None
+        parent = kwargs.pop("parent", None)
+        source = kwargs.pop("source", None)
         if kwargs:
-            if len(kwargs) > 1:
-                raise TypeError("Invalid keyword arguments:", kwargs)
-            parent = kwargs["parent"]
-        super(FunctionTrace, self).__init__(parent)
+            raise TypeError("Invalid keyword arguments:", kwargs)
+
+        super(FunctionTrace, self).__init__(parent=parent, source=source)
 
         # Handle incorrect groupings and leading slashes. This will
         # cause an empty segment which we want to avoid. In that case
@@ -138,7 +138,7 @@ def dynamic_wrapper(wrapped, instance, args, kwargs):
         else:
             _params = params
 
-        trace = FunctionTrace(_name, _group, _label, _params, terminal, rollup, parent=parent)
+        trace = FunctionTrace(_name, _group, _label, _params, terminal, rollup, parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)
@@ -157,7 +157,7 @@ def literal_wrapper(wrapped, instance, args, kwargs):
 
         _name = name or callable_name(wrapped)
 
-        trace = FunctionTrace(_name, group, label, params, terminal, rollup, parent=parent)
+        trace = FunctionTrace(_name, group, label, params, terminal, rollup, parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)

newrelic/api/function_trace.py

@@ -23,12 +23,12 @@
 
 class GraphQLOperationTrace(TimeTrace):
     def __init__(self, **kwargs):
-        parent = None
+        parent = kwargs.pop("parent", None)
+        source = kwargs.pop("source", None)
         if kwargs:
-            if len(kwargs) > 1:
-                raise TypeError("Invalid keyword arguments:", kwargs)
-            parent = kwargs["parent"]
-        super(GraphQLOperationTrace, self).__init__(parent)
+            raise TypeError("Invalid keyword arguments:", kwargs)
+
+        super(GraphQLOperationTrace, self).__init__(parent=parent, source=source)
 
         self.operation_name = "<anonymous>"
         self.operation_type = "<unknown>"
@@ -114,7 +114,7 @@ def _nr_graphql_trace_wrapper_(wrapped, instance, args, kwargs):
         else:
             parent = None
 
-        trace = GraphQLOperationTrace(parent=parent)
+        trace = GraphQLOperationTrace(parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)
@@ -135,7 +135,13 @@ def wrap_graphql_operation_trace(module, object_path):
 
 class GraphQLResolverTrace(TimeTrace):
     def __init__(self, field_name=None, **kwargs):
-        super(GraphQLResolverTrace, self).__init__(**kwargs)
+        parent = kwargs.pop("parent", None)
+        source = kwargs.pop("source", None)
+        if kwargs:
+            raise TypeError("Invalid keyword arguments:", kwargs)
+
+        super(GraphQLResolverTrace, self).__init__(parent=parent, source=source)
+
         self.field_name = field_name
         self._product = None
 
@@ -192,7 +198,7 @@ def _nr_graphql_trace_wrapper_(wrapped, instance, args, kwargs):
         else:
             parent = None
 
-        trace = GraphQLResolverTrace(parent=parent)
+        trace = GraphQLResolverTrace(parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)

newrelic/api/graphql_trace.py

@@ -22,12 +22,12 @@
 
 class MemcacheTrace(TimeTrace):
     def __init__(self, command, **kwargs):
-        parent = None
+        parent = kwargs.pop("parent", None)
+        source = kwargs.pop("source", None)
         if kwargs:
-            if len(kwargs) > 1:
-                raise TypeError("Invalid keyword arguments:", kwargs)
-            parent = kwargs["parent"]
-        super(MemcacheTrace, self).__init__(parent)
+            raise TypeError("Invalid keyword arguments:", kwargs)
+
+        super(MemcacheTrace, self).__init__(parent=parent, source=source)
 
         self.command = command
 
@@ -69,7 +69,7 @@ def _nr_wrapper_memcache_trace_(wrapped, instance, args, kwargs):
         else:
             _command = command
 
-        trace = MemcacheTrace(_command, parent=parent)
+        trace = MemcacheTrace(_command, parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)

newrelic/api/memcache_trace.py

@@ -29,13 +29,12 @@ class MessageTrace(CatHeaderMixin, TimeTrace):
     cat_synthetics_key = "NewRelicSynthetics"
 
     def __init__(self, library, operation, destination_type, destination_name, params=None, **kwargs):
-
-        parent = None
+        parent = kwargs.pop("parent", None)
+        source = kwargs.pop("source", None)
         if kwargs:
-            if len(kwargs) > 1:
-                raise TypeError("Invalid keyword arguments:", kwargs)
-            parent = kwargs["parent"]
-        super(MessageTrace, self).__init__(parent)
+            raise TypeError("Invalid keyword arguments:", kwargs)
+
+        super(MessageTrace, self).__init__(parent=parent, source=source)
 
         self.library = library
         self.operation = operation
@@ -132,7 +131,7 @@ def _nr_message_trace_wrapper_(wrapped, instance, args, kwargs):
         else:
             _destination_name = destination_name
 
-        trace = MessageTrace(_library, _operation, _destination_type, _destination_name, params={}, parent=parent)
+        trace = MessageTrace(_library, _operation, _destination_type, _destination_name, params={}, parent=parent, source=wrapped)
 
         if wrapper:  # pylint: disable=W0125,W0126
             return wrapper(wrapped, trace)(*args, **kwargs)