Merge pull request #129 from networktocode/develop

Release v2.0.8

Author: glennmatthews

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## v2.0.8 - 2021-12-09
+
+### Fixed
+
+- #115 - Add default `status` and `sequence` values for iCal notifications missing these fields
+- #124 - Handle encoded non-ASCII characters in email subjects.
+- #126 - Ignore a class of non-maintenance-notification emails from Telia.
+- #127 - Improve handling of Equinix and Lumen notifications.
+- #128 - Add capability to set `RE-SCHEDULED` status for Verizon rescheduled notifications.
+
 ## v2.0.7 - 2021-12-01
 
 ### Fixed

README.md

@@ -19,19 +19,26 @@ during a NANOG meeting that aimed to promote the usage of the iCalendar format.
 proposed iCalendar format, the parser is straight-forward and there is no need to define custom logic, but this library
 enables supporting other providers that are not using this proposed practice, getting the same outcome.
 
-You can leverage this library in your automation framework to process circuit maintenance notifications, and use the standardized [`Maintenance`](https://github.com/networktocode/circuit-maintenance-parser/blob/develop/circuit_maintenance_parser/output.py) to handle your received circuit maintenance notifications in a simple way. Every `maintenance` object contains, at least, the following attributes:
+You can leverage this library in your automation framework to process circuit maintenance notifications, and use the standardized [`Maintenance`](https://github.com/networktocode/circuit-maintenance-parser/blob/develop/circuit_maintenance_parser/output.py) model to handle your received circuit maintenance notifications in a simple way. Every `Maintenance` object contains the following attributes:
 
 - **provider**: identifies the provider of the service that is the subject of the maintenance notification.
 - **account**: identifies an account associated with the service that is the subject of the maintenance notification.
-- **maintenance_id**: contains text that uniquely identifies the maintenance that is the subject of the notification.
+- **maintenance_id**: contains text that uniquely identifies (at least within the context of a specific provider) the maintenance that is the subject of the notification.
 - **circuits**: list of circuits affected by the maintenance notification and their specific impact.
-- **status**: defines the overall status or confirmation for the maintenance.
-- **start**: timestamp that defines the start date of the maintenance in GMT.
-- **end**: timestamp that defines the end date of the maintenance in GMT.
-- **stamp**: timestamp that defines the update date of the maintenance in GMT.
+- **start**: timestamp that defines the starting date/time of the maintenance in GMT.
+- **end**: timestamp that defines the ending date/time of the maintenance in GMT.
+- **stamp**: timestamp that defines the update date/time of the maintenance in GMT.
 - **organizer**: defines the contact information included in the original notification.
+- **status**: defines the overall status or confirmation for the maintenance.¹
+- **summary**: human-readable details about this maintenance notification. May be an empty string.
+- **sequence**: a sequence number for notifications involving this maintenance window. In practice this is generally redundant with the **stamp** field, and will be defaulted to `1` for most non-iCalendar parsed notifications.²
+- **uid**: a unique (?) identifer for a thread of related notifications. In practice this is generally redundant with the **maintenance_id** field, and will be defaulted to `0` for most non-iCalendar parsed notifications.
 
-> Please, refer to the [BCOP](https://github.com/jda/maintnote-std/blob/master/standard.md) to more details about these attributes.
+> Please, refer to the [BCOP](https://github.com/jda/maintnote-std/blob/master/standard.md) to more details about the standardized meaning of these attributes.
+
+¹ Per the BCOP, **status** (`X-MAINTNOTE_STATUS`) is an optional field in iCalendar notifications. However, a `Maintenance` object will always contain a `status` value; in the case where an iCalendar notification omits this field, the `status` will be set to `"NO-CHANGE"`, and it's up to the consumer of this library to determine how to appropriately handle this case. Parsers of other notification formats are responsible for setting an appropriate value for this field based on the notification contents, and may or may not include `"NO-CHANGE"` as one of the possible reported values.
+
+² Per the BCOP, **sequence** is a mandatory field in iCalendar notifications. However, some NSPs have been seen to send notifications which, while otherwise consistent with the BCOP, omit the `SEQUENCE` field; in such cases, this library will report a sequence number of `-1`.
 
 ## Workflow
 

circuit_maintenance_parser/data.py

@@ -75,7 +75,16 @@ def init_from_emailmessage(cls: Type["NotificationData"], email_message) -> Opti
             cls.walk_email(email_message, data_parts)
 
             # Adding extra headers that are interesting to be parsed
-            data_parts.add(DataPart(EMAIL_HEADER_SUBJECT, email_message["Subject"].encode()))
+            data_parts.add(
+                DataPart(
+                    EMAIL_HEADER_SUBJECT,
+                    # decode_header() handles conversion from RFC2047 ASCII representation of non-ASCII content to
+                    #   a list of (string, charset) tuples.
+                    # make_header() merges these back into a single Header object containing this text
+                    # str() gets the simple Unicode representation of the Header.
+                    str(email.header.make_header(email.header.decode_header(email_message["Subject"]))).encode(),
+                )
+            )
             data_parts.add(DataPart(EMAIL_HEADER_DATE, email_message["Date"].encode()))
             # Ensure the data parts are processed in a consistent order
             return cls(data_parts=sorted(data_parts, key=lambda part: part.type))

circuit_maintenance_parser/output.py

@@ -37,8 +37,9 @@ class Status(str, Enum):
     - "CONFIRMED": Indicates maintenance event is definite.
     - "CANCELLED": Indicates maintenance event was cancelled.
     - "IN-PROCESS": Indicates maintenance event is in process (e.g. open).
-    - "COMPLETED":Indicates maintenance event completed (e.g. closed).
+    - "COMPLETED": Indicates maintenance event completed (e.g. closed).
     - "RE-SCHEDULED": Indicates maintenance event was re-scheduled.
+    - "NO-CHANGE": Indicates status is unchanged from a previous notification (dummy value)
     """
 
     TENTATIVE = "TENTATIVE"
@@ -48,6 +49,8 @@ class Status(str, Enum):
     COMPLETED = "COMPLETED"
     RE_SCHEDULED = "RE-SCHEDULED"
 
+    NO_CHANGE = "NO-CHANGE"
+
 
 class CircuitImpact(BaseModel, extra=Extra.forbid):
     """CircuitImpact class.
@@ -96,13 +99,13 @@ class Maintenance(BaseModel, extra=Extra.forbid):
         account:  identifies an account associated with the service that is the subject of the maintenance notification
         maintenance_id:  contains text that uniquely identifies the maintenance that is the subject of the notification
         circuits: list of circuits affected by the maintenance notification and their specific impact
-        status: defines the overall status or confirmation for the maintenance
         start: timestamp that defines the start date of the maintenance in GMT
         end: timestamp that defines the end date of the maintenance in GMT
         stamp: timestamp that defines the update date of the maintenance in GMT
         organizer: defines the contact information included in the original notification
 
     Optional attributes:
+        status: defines the overall status or confirmation for the maintenance
         summary: description of the maintenace notification
         uid: specific unique identifier for each notification
         sequence: sequence number - initially zero - to serialize updates in case they are received or processed out of
@@ -123,18 +126,18 @@ class Maintenance(BaseModel, extra=Extra.forbid):
         ...     summary="This is a maintenance notification",
         ...     uid="1111",
         ... )
-        Maintenance(provider='A random NSP', account='12345000', maintenance_id='VNOC-1-99999999999', circuits=[CircuitImpact(circuit_id='123', impact=<Impact.NO_IMPACT: 'NO-IMPACT'>), CircuitImpact(circuit_id='456', impact=<Impact.OUTAGE: 'OUTAGE'>)], status=<Status.COMPLETED: 'COMPLETED'>, start=1533704400, end=1533712380, stamp=1533595768, organizer='[email protected]', uid='1111', sequence=1, summary='This is a maintenance notification')
+        Maintenance(provider='A random NSP', account='12345000', maintenance_id='VNOC-1-99999999999', circuits=[CircuitImpact(circuit_id='123', impact=<Impact.NO_IMPACT: 'NO-IMPACT'>), CircuitImpact(circuit_id='456', impact=<Impact.OUTAGE: 'OUTAGE'>)], start=1533704400, end=1533712380, stamp=1533595768, organizer='[email protected]', status=<Status.COMPLETED: 'COMPLETED'>, uid='1111', sequence=1, summary='This is a maintenance notification')
     """
 
     provider: StrictStr
     account: StrictStr
     maintenance_id: StrictStr
     circuits: List[CircuitImpact]
-    status: Status
     start: StrictInt
     end: StrictInt
     stamp: StrictInt
     organizer: StrictStr
+    status: Status
 
     # Non mandatory attributes
     uid: StrictStr = "0"

circuit_maintenance_parser/parser.py

@@ -110,14 +110,18 @@ def parse_ical(gcal: Calendar) -> List[Dict]:
                     "provider": str(component.get("X-MAINTNOTE-PROVIDER")),
                     "account": str(component.get("X-MAINTNOTE-ACCOUNT")),
                     "maintenance_id": str(component.get("X-MAINTNOTE-MAINTENANCE-ID")),
-                    "status": Status(component.get("X-MAINTNOTE-STATUS")),
+                    # status may be omitted, per the BCOP
+                    "status": Status(component.get("X-MAINTNOTE-STATUS", "NO-CHANGE")),
                     "start": round(component.get("DTSTART").dt.timestamp()),
                     "end": round(component.get("DTEND").dt.timestamp()),
                     "stamp": round(component.get("DTSTAMP").dt.timestamp()),
                     "summary": str(component.get("SUMMARY")),
                     "organizer": str(component.get("ORGANIZER")),
                     "uid": str(component.get("UID")),
-                    "sequence": int(component.get("SEQUENCE")),
+                    # per the BCOP sequence is mandatory, but we have real examples where it's omitted,
+                    # usually in combination with an omitted status. In that case let's treat the sequence as -1,
+                    # i.e. older than all known updates.
+                    "sequence": int(component.get("SEQUENCE", -1)),
                 }
 
                 data = {key: value for key, value in data.items() if value != "None"}

circuit_maintenance_parser/parsers/equinix.py

@@ -112,8 +112,15 @@ def parse_subject(self, subject: str) -> List[Dict]:
         if maintenance_id:
             data["maintenance_id"] = maintenance_id[1]
         data["summary"] = subject.strip().replace("\n", "")
-        if "COMPLETED" in subject:
+        if "completed" in subject.lower():
             data["status"] = Status.COMPLETED
-        if "SCHEDULED" in subject or "REMINDER" in subject:
+        elif "rescheduled" in subject.lower():
+            data["status"] = Status.RE_SCHEDULED
+        elif "scheduled" in subject.lower() or "reminder" in subject.lower():
             data["status"] = Status.CONFIRMED
+        else:
+            # Some Equinix notifications don't clearly state a status in their subject.
+            # From inspection of examples, it looks like "Confirmed" would be the most appropriate in this case.
+            data["status"] = Status.CONFIRMED
+
         return [data]

circuit_maintenance_parser/parsers/lumen.py

@@ -93,7 +93,7 @@ def parse_tables(self, tables: ResultSet, data: Dict):
                             data["status"] = Status("COMPLETED")
                         elif status_string == "Postponed":
                             data["status"] = Status("RE-SCHEDULED")
-                        elif status_string == "Not Completed":
+                        elif status_string in ["Not Completed", "Cancelled"]:
                             data["status"] = Status("CANCELLED")
                         elif status_string == "Alternate Night":
                             data["status"] = Status("RE-SCHEDULED")

circuit_maintenance_parser/parsers/verizon.py

@@ -42,12 +42,14 @@ def parse_tables(self, tables: ResultSet, data: Dict):  # pylint: disable=too-ma
             if not cells_text:
                 continue
             if cells_text[0].startswith("Description of Maintenance"):
-                data["summary"] = cells_text[1]
+                data["summary"] = cells_text[1].replace(" ", "")
             elif cells_text[0].startswith("Verizon MASTARS Request number:"):
                 data["maintenance_id"] = cells_text[1]
             elif cells_text[0].startswith("Attention:"):
                 if "maintenance was not completed" in cells_text[0]:
                     data["status"] = Status("CANCELLED")
+                elif "request has been rescheduled" in cells_text[0]:
+                    data["status"] = Status("RE-SCHEDULED")
             elif cells_text[0].startswith("Maintenance Date/Time (GMT):"):
                 maintenance_time = cells_text[1].split("-")
                 start = parser.parse(maintenance_time[0].strip())

circuit_maintenance_parser/provider.py

@@ -301,6 +301,8 @@ class Sparkle(GenericProvider):
 class Telia(GenericProvider):
     """Telia provider custom class."""
 
+    _exclude_filter = {EMAIL_HEADER_SUBJECT: ["Disturbance Information"]}
+
     _default_organizer = "[email protected]"