Refactor Logging for Improved Debugging (#115)

This pull request introduces several improvements and refactors across the codebase, focusing on enhanced logging, memory tracking, type checking, and developer usability. The most significant changes include replacing print statements with structured logging, adding detailed memory usage tracking for buffers, improving type and level checking for data types, and providing more informative string representations and debugging messages for core classes.

All runners and the generation scripts support different verbosity levels:
- `-v`: Show errors, warning and info messages
- `-vv`: Show errors, warnings, info and debug messages
- `-vvv`: Maximum verbosity with information about where the message was emitted. Also enable verbose compilation during the build step for the test runners.

## Added
- Logging and Error Handling Improvements: Replaced all `print` statements with structured logging using `DEFAULT_LOGGER` throughout the codebase, improving consistency and enabling better log management. 
- Added debug-level logs to type checking and parsing routines to indicate success/failure and binding exhaustion, aiding troubleshooting. 
- Implemented `__repr__` methods for core classes (`NodeTemplate`, `NodeBinding`, `NodeMapper`) to provide more informative string representations for debugging and logging
- Added `checkNumLevels` methods to relevant data type classes to validate that the number of levels assigned to buffers is supported by their type, and integrated warning logs if mismatches are detected during type inference.
- Introduced a `sizeInBytes` method to `VariableBuffer`, `TransientBuffer`, and related classes to standardize buffer size calculation. 

- Report the maximum static and dynamic memory usage for all memory level:
  - Added dynamic and maximum memory usage tracking to `NetworkContext` via `_dynamicSize` and `_maxDynamicSize` dictionaries, and updated memory allocation/deallocation logic in `MemoryAllocation.py` to maintain these statistics per memory level. (This is only used for untitled platforms to track the maximum buffer size.)
  - Exposed new summary methods (`_printMemorySummary`, `_printInputOutputSummary`) in deployer wrappers and implemented input/output summary logging in `SignPropDeployer`. 
  
## Changed
- Deployer workflow now uses `prepare(...)` instead of `generateFunction(...)`.

Author: Xeratec

CHANGELOG.md

@@ -4,6 +4,7 @@ This file contains the changelog for the Deeploy project. The changelog is divid
 ## Unreleased (Planned Release Target: v0.2.1)
 
 ### List of Pull Requests
+- Refactor Logging for Improved Debugging [#115](https://github.com/pulp-platform/Deeploy/pull/115)
 - Add reuse-tool as an SPDX license header linter [#113](https://github.com/pulp-platform/Deeploy/pull/113)
 - Bug fixes, API Cleanup and Reduce Compiler Warning on PULP [#112](https://github.com/pulp-platform/Deeploy/pull/112)
 - Fix PULP GEMM `batch` serialization [#109](https://github.com/pulp-platform/Deeploy/pull/109)
@@ -39,6 +40,11 @@ This file contains the changelog for the Deeploy project. The changelog is divid
 - Reshape operator support for PULP (`ReshapeTemplate` in bindings)
 - Missing class attributes in `Closure.py`
 - reuse_skip_wrapper.py to manually skip files
+- Centralized logging with `DEFAULT_LOGGER`, replacing `print` statements
+- Debug logs for type checking/parsing; `__repr__` for core classes
+- Buffer utilities: `checkNumLevels` validation and `sizeInBytes` method
+- Per–memory-level usage tracking and worst-case reporting in `NetworkContext`
+- Memory/I/O summaries and input/output logging in deployers
 
 ### Changed
 - Replaced platform-specific tags (`*-amd64`, `*-arm64`) with direct digest references in `Noelware/docker-manifest-action`.
@@ -65,6 +71,7 @@ This file contains the changelog for the Deeploy project. The changelog is divid
 - Removed unnecessary includes from the PULP platform header list, such as `DeeployBasicMath.h`, for cleaner code generation
 - Changed types and added correct casts to fix many compiler warnings in the PULP target library
 - Use [reuse-tool](https://github.com/fsfe/reuse-tool) in pre-commit, CI, and Makefile for SPDX license header linting
+- Deployer workflow now uses `prepare(...)` instead of `generateFunction(...)`.
 
 ### Fixed
 - Prevent node duplication for graphs generated via GraphSurgeon

Deeploy/AbstractDataTypes.py

@@ -21,6 +21,8 @@
 _DeeployType = TypeVar("_DeeployType", _PointerType, _ImmediateType, _StructType)
 _PythonType = TypeVar("_PythonType", str, int, float, Dict[str, "_PythonType"], Iterable["_PythonType"])
 
+from Deeploy.Logging import DEFAULT_LOGGER as log
+
 
 class _ClassPropertyDescriptor(object):
 
@@ -188,6 +190,10 @@ def typeMin(cls) -> int:
         else:
             return 0
 
+    @_classproperty
+    def nLevels(cls) -> int:
+        return cls.typeMax - cls.typeMin + 1
+
     @classmethod
     def partialOrderUpcast(cls, otherCls: Type[Immediate]) -> bool:
         if issubclass(otherCls, IntegerImmediate):
@@ -213,6 +219,10 @@ def checkValue(cls, value: Union[int, Iterable[int], np.ndarray], ctxt: Optional
             return False
         return True
 
+    @classmethod
+    def fitsNumLevels(cls, nLevels: int) -> bool:
+        return nLevels <= cls.nLevels
+
 
 class FloatImmediate(Immediate[Union[float, Iterable[float]], _ImmediateType]):
     typeMantissa: int  #: int: Represents the number of bits reserved for the mantissa part
@@ -308,7 +318,7 @@ def checkValue(cls, value: Optional[str], ctxt: Optional[_NetworkContext] = None
             return False
 
         if value is None or value == "NULL":
-            print("WARNING: Setting pointer value to NULL - Referenced data is invalid!")
+            log.warning("Setting pointer value to NULL - Referenced data is invalid!")
             return True
 
         reference = ctxt.lookup(value)
@@ -332,6 +342,10 @@ def checkPromotion(cls, _value: Union[Optional[str], Pointer], ctxt: Optional[_N
             value = _value
         return cls.checkValue(value, ctxt)
 
+    @classmethod
+    def fitsNumLevels(cls, nLevels: int) -> bool:
+        return cls.referencedType.fitsNumLevels(nLevels)
+
     def __init__(self, _value: Union[Optional[str], Pointer], ctxt: Optional[_NetworkContext] = None):
         """Initializes a pointer to a registered object in the NetworkContext
 

Deeploy/CommonExtensions/CodeTransformationPasses/MemoryAllocation.py

@@ -122,13 +122,31 @@ def apply(self,
         for buffer in reversed(self.topologicallySortBuffers(outputs + transients)):
             assert buffer._live == False, f"Tried to allocate already live buffer {buffer.name}"
             buffer._live = True
+
+            memoryLevel = "None" if not hasattr(buffer, "_memoryLevel") else buffer._memoryLevel
+            if memoryLevel not in ctxt._dynamicSize:
+                ctxt._dynamicSize[memoryLevel] = int(buffer.sizeInBytes())
+            else:
+                ctxt._dynamicSize[memoryLevel] += int(buffer.sizeInBytes())
+
             executionBlock.addLeft(buffer.allocTemplate, buffer._bufferRepresentation())
 
+        for levels in ctxt._dynamicSize.keys():
+            if levels not in ctxt._maxDynamicSize:
+                ctxt._maxDynamicSize[levels] = max(0, ctxt._dynamicSize[levels])
+            else:
+                ctxt._maxDynamicSize[levels] = max(ctxt._maxDynamicSize.get(levels, 0), ctxt._dynamicSize[levels])
+
         for buffer in inputs + transients:
             assert buffer._live == True, f"Tried to deallocate already dead buffer {buffer.name}"
             buffer._live = False
             # Don't deallocate if it's an alias of a live buffer
             if not buffer.has_live_ancestors(ctxt = ctxt):
+                memoryLevel = "None" if not hasattr(buffer, "_memoryLevel") else buffer._memoryLevel
+                if memoryLevel not in ctxt._dynamicSize:
+                    ctxt._dynamicSize[memoryLevel] = 0
+                else:
+                    ctxt._dynamicSize[memoryLevel] -= int(buffer.sizeInBytes())
                 executionBlock.addRight(buffer.deallocTemplate, buffer._bufferRepresentation())
 
         return ctxt, executionBlock
@@ -157,10 +175,30 @@ def apply(self,
 
         for buffer in outputs + transients:
             assert buffer._live == False, f"Tried to allocate already live buffer {buffer.name}"
+
+            memoryLevel = "None" if not hasattr(buffer, "_memoryLevel") else buffer._memoryLevel
+            if memoryLevel not in ctxt._dynamicSize:
+                ctxt._dynamicSize[memoryLevel] = int(buffer.sizeInBytes())
+            else:
+                ctxt._dynamicSize[memoryLevel] += int(buffer.sizeInBytes())
+
             buffer._live = True
 
+        for levels in ctxt._dynamicSize.keys():
+            if levels not in ctxt._maxDynamicSize:
+                ctxt._maxDynamicSize[levels] = max(0, ctxt._dynamicSize[levels])
+            else:
+                ctxt._maxDynamicSize[levels] = max(ctxt._maxDynamicSize.get(levels, 0), ctxt._dynamicSize[levels])
+
         for buffer in inputs + transients:
             assert buffer._live == True, f"Tried to deallocate already dead buffer {buffer.name}"
+
+            memoryLevel = "None" if not hasattr(buffer, "_memoryLevel") else buffer._memoryLevel
+            if memoryLevel not in ctxt._dynamicSize:
+                ctxt._dynamicSize[memoryLevel] = 0
+            else:
+                ctxt._dynamicSize[memoryLevel] -= int(buffer.sizeInBytes())
+
             buffer._live = False
 
         return ctxt, executionBlock

Deeploy/CommonExtensions/NetworkDeployers/NetworkDeployerWrapper.py

@@ -75,3 +75,9 @@ def generateBufferAllocationCode(self) -> str:
     # MultiEngineDeployer augment
     def _mapNode(self, node: gs.Node) -> Union[ONNXLayer, Any]:
         return self._innerObject._mapNode(node)
+
+    def _printMemorySummary(self):
+        return self._innerObject._printMemorySummary()
+
+    def _printInputOutputSummary(self):
+        return self._innerObject._printInputOutputSummary()

Deeploy/CommonExtensions/NetworkDeployers/SignPropDeployer.py

@@ -8,6 +8,7 @@
 
 from Deeploy.AbstractDataTypes import Pointer
 from Deeploy.DeeployTypes import DeploymentPlatform, NetworkDeployer, TopologyOptimizer
+from Deeploy.Logging import DEFAULT_LOGGER as log
 
 
 class SignPropDeployer(NetworkDeployer):
@@ -41,3 +42,16 @@ def _createIOBindings(self, ctxt, graph):
             nb.nLevels = (2**data_type.referencedType.typeWidth)
 
         return ctxt
+
+    def _printInputOutputSummary(self):
+        log.info('Input:')
+        for buf in self.inputs():
+            log.info(
+                f" - '{buf.name}': Type: {buf._type.referencedType.typeName}, nLevels: {buf.nLevels}, Signed: {buf._signed}, Offset: {self.inputOffsets[buf.name]}"
+            )
+
+        log.info('Output:')
+        for buf in self.outputs():
+            log.info(
+                f" - '{buf.name}': Type: {buf._type.referencedType.typeName}, nLevels: {buf.nLevels}, Signed: {buf._signed}"
+            )

Deeploy/CommonExtensions/OptimizationPasses/PassClasses.py

@@ -7,6 +7,7 @@
 import onnx_graphsurgeon as gs
 
 from Deeploy.DeeployTypes import NetworkContext
+from Deeploy.Logging import DEFAULT_LOGGER as log
 
 from .Matchers import Match, NonBranchingMatcher, SubgraphMatcher
 
@@ -129,7 +130,7 @@ def remove_subpass(self, name):
         try:
             del self._subpasses[name]
         except KeyError:
-            print(f"No subpass with name {name}, cannot remove!")
+            log.error(f"No subpass with name {name}, cannot remove!")
         except AttributeError:
             raise AttributeError("Cannot remove sub-pass before calling Pass.__init__!")
 

Deeploy/CommonExtensions/TypeCheckers/SignPropTypeChecker.py

@@ -6,7 +6,9 @@
 
 import onnx_graphsurgeon as gs
 
+from Deeploy.AbstractDataTypes import IntegerImmediate
 from Deeploy.DeeployTypes import ConstantBuffer, NetworkContext, NodeTypeChecker, OperatorRepresentation, VariableBuffer
+from Deeploy.Logging import DEFAULT_LOGGER as log
 
 
 class SignPropTypeChecker(NodeTypeChecker):
@@ -46,8 +48,15 @@ def typeInferOutput(self, ctxt: NetworkContext, node: gs.Node,
             nLevels = self._inferNumLevels(inputs, operatorRepresentation)
             signedness = self._inferSignedness(inputs, operatorRepresentation)
 
-            for obj, nLevels, sign in zip(outputs, nLevels, signedness):
-                obj.nLevels = nLevels
+            if nLevels is None or signedness is None:
+                return ctxt
+            for obj, nLevel, sign in zip(outputs, nLevels, signedness):
+                obj.nLevels = nLevel
                 obj._signed = sign
 
+                if issubclass(obj._type.referencedType, IntegerImmediate) and not obj._type.fitsNumLevels(nLevel):
+                    log.warning(
+                        f"{obj.name} has {nLevel} levels, but {obj._type.referencedType.typeName} only supports {obj._type.referencedType.nLevels} levels."
+                    )
+
         return ctxt