Skip to content

Commit 8aeddef

Browse files
author
Christian Adell
authored
Merge pull request #107 from networktocode/release-v2.0.4
Release v2.0.4
2 parents 2f89d32 + d50f3c0 commit 8aeddef

34 files changed

+43026
-150
lines changed

.github/CODEOWNERS

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
# Default owner(s) of all files in this repository
2-
* @chadell @glennmatthews @pke11y @carbonarok
2+
* @chadell @glennmatthews @pke11y

CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
# Changelog
22

3+
## v2.0.4 - 2021-11-04
4+
5+
### Fixed
6+
7+
- #94 - Improve Geo service error handling.
8+
- #97 - Fix Readme image URLs.
9+
- #98 - Add handling for `Lumen` notification with Alt Circuit ID.
10+
- #99 - Extend `Zayo` Html parser to handle different table headers.
11+
- #102 - Add `Equinix` provider.
12+
- #104 - Use a local locations DB to map city to timezone as first option, keeping API as fallback option.
13+
- #105 - Extend `Colt` parser to support multiple `Maintenance` statuses.
14+
315
## v2.0.3 - 2021-10-01
416

517
### Added

README.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ You can leverage this library in your automation framework to process circuit ma
4242
5. Each `Parser` class supports one or a set of related data types, and implements the `Parser.parse()` method used to retrieve a `Dict` with the relevant keys/values.
4343

4444
<p align="center">
45-
<img src="https://raw.githubusercontent.com/nautobot/nautobot-plugin-circuit-maintenance/develop/docs/images/new_workflow.png" width="800" class="center">
45+
<img src="https://raw.githubusercontent.com/networktocode/circuit-maintenance-parser/develop/docs/images/new_workflow.png" width="800" class="center">
4646
</p>
4747

4848
By default, there is a `GenericProvider` that support a `SimpleProcessor` using the standard `ICal` `Parser`, being the easiest path to start using the library in case the provider uses the reference iCalendar standard.
@@ -63,6 +63,7 @@ By default, there is a `GenericProvider` that support a `SimpleProcessor` using
6363
- AquaComms
6464
- Cogent
6565
- Colt
66+
- Equinix
6667
- GTT
6768
- HGC
6869
- Lumen
@@ -306,3 +307,7 @@ The project is following Network to Code software development guidelines and is
306307

307308
For any questions or comments, please check the [FAQ](FAQ.md) first and feel free to swing by the [Network to Code slack channel](https://networktocode.slack.com/) (channel #networktocode).
308309
Sign up [here](http://slack.networktocode.com/)
310+
311+
## License notes
312+
313+
This library uses a Basic World Cities Database by Pareto Software, LLC, the owner of Simplemaps.com: The Provider offers a Basic World Cities Database free of charge. This database is licensed under the Creative Commons Attribution 4.0 license as described at: https://creativecommons.org/licenses/by/4.0/.

circuit_maintenance_parser/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
AWS,
1111
Cogent,
1212
Colt,
13+
Equinix,
1314
EUNetworks,
1415
GTT,
1516
HGC,
@@ -33,6 +34,7 @@
3334
AWS,
3435
Cogent,
3536
Colt,
37+
Equinix,
3638
EUNetworks,
3739
GTT,
3840
HGC,

circuit_maintenance_parser/data/worldcities.csv

Lines changed: 41002 additions & 0 deletions
Large diffs are not rendered by default.

circuit_maintenance_parser/parser.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
from circuit_maintenance_parser.errors import ParserError
1717
from circuit_maintenance_parser.output import Status, Impact, CircuitImpact
1818
from circuit_maintenance_parser.constants import EMAIL_HEADER_SUBJECT, EMAIL_HEADER_DATE
19+
from circuit_maintenance_parser.utils import Geolocator
1920

2021
# pylint: disable=no-member
2122

@@ -33,6 +34,8 @@ class Parser(BaseModel, extra=Extra.forbid):
3334
# _data_types are used to match the Parser to to each type of DataPart
3435
_data_types = ["text/plain", "plain"]
3536

37+
_geolocator = Geolocator()
38+
3639
@classmethod
3740
def get_data_types(cls) -> List[str]:
3841
"""Return the expected data type."""

circuit_maintenance_parser/parsers/cogent.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@
77
from bs4.element import ResultSet # type: ignore
88

99
from circuit_maintenance_parser.parser import Html, Impact, CircuitImpact, Status
10-
from circuit_maintenance_parser.utils import city_timezone
1110

1211
logger = logging.getLogger(__name__)
1312

@@ -48,7 +47,7 @@ def parse_div(self, divs: ResultSet, data: Dict): # pylint: disable=too-many-lo
4847
elif line.startswith("Cogent customers receiving service"):
4948
match = re.search(r"[^Cogent].*?((\b[A-Z][a-z\s-]+)+, ([A-Za-z-]+[\s-]))", line)
5049
if match:
51-
parsed_timezone = city_timezone(match.group(1).strip())
50+
parsed_timezone = self._geolocator.city_timezone(match.group(1).strip())
5251
local_timezone = timezone(parsed_timezone)
5352
# set start time using the local city timezone
5453
start = datetime.strptime(start_str, "%I:%M %p %d/%m/%Y")

circuit_maintenance_parser/parsers/colt.py

Lines changed: 36 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -4,66 +4,18 @@
44
import re
55
import csv
66
import io
7-
import base64
87

9-
from icalendar import Calendar # type: ignore
10-
11-
from circuit_maintenance_parser.parser import Csv
12-
from circuit_maintenance_parser.errors import ParserError
8+
from dateutil import parser
139
from circuit_maintenance_parser.output import Status, Impact, CircuitImpact
14-
from circuit_maintenance_parser.parser import ICal
10+
from circuit_maintenance_parser.parser import EmailSubjectParser, Csv
1511

1612
logger = logging.getLogger(__name__)
1713

1814
# pylint: disable=too-many-branches
1915

2016

21-
class ICalParserColt1(ICal):
22-
"""Colt Notifications Parser based on ICal notifications."""
23-
24-
def parse(self, raw: bytes):
25-
"""Method that returns a list of Maintenance objects."""
26-
result = []
27-
28-
# iCalendar data sometimes comes encoded with base64
29-
# TODO: add a test case
30-
try:
31-
gcal = Calendar.from_ical(base64.b64decode(raw))
32-
except ValueError:
33-
gcal = Calendar.from_ical(raw)
34-
35-
if not gcal:
36-
raise ParserError("Not a valid iCalendar data received")
37-
38-
for component in gcal.walk():
39-
if component.name == "VEVENT":
40-
maintenance_id = ""
41-
account = ""
42-
43-
summary_match = re.search(
44-
r"^.*?[-]\s(?P<maintenance_id>CRQ[\S]+).*?,\s*(?P<account>\d+)$", str(component.get("SUMMARY"))
45-
)
46-
if summary_match:
47-
maintenance_id = summary_match.group("maintenance_id")
48-
account = summary_match.group("account")
49-
50-
data = {
51-
"account": account,
52-
"maintenance_id": maintenance_id,
53-
"status": Status("CONFIRMED"),
54-
"start": round(component.get("DTSTART").dt.timestamp()),
55-
"end": round(component.get("DTEND").dt.timestamp()),
56-
"stamp": round(component.get("DTSTAMP").dt.timestamp()),
57-
"summary": str(component.get("SUMMARY")),
58-
"sequence": int(component.get("SEQUENCE")),
59-
}
60-
result.append(data)
61-
62-
return result
63-
64-
6517
class CsvParserColt1(Csv):
66-
"""Colt Notifications partial parser for circuit-ID's in CSV notifications."""
18+
"""Colt Notifications partial parser in CSV notifications."""
6719

6820
@staticmethod
6921
def parse_csv(raw):
@@ -73,4 +25,37 @@ def parse_csv(raw):
7325
parsed_csv = csv.DictReader(csv_data, dialect=csv.excel_tab)
7426
for row in parsed_csv:
7527
data["circuits"].append(CircuitImpact(impact=Impact("OUTAGE"), circuit_id=row["Circuit ID"].strip()))
28+
if not data.get("account"):
29+
search = re.search(r"\d+", row["OCN"].strip())
30+
if search:
31+
data["account"] = search.group()
32+
return [data]
33+
34+
35+
class SubjectParserColt1(EmailSubjectParser):
36+
"""Subject parser for Colt notifications."""
37+
38+
def parse_subject(self, subject):
39+
"""Parse subject.
40+
41+
Example:
42+
- [ EXTERNAL ] MAINTENANCE ALERT: CRQ1-12345678 24/10/2021 04:00:00 GMT - 24/10/2021 11:00:00 GMT is about to START
43+
- [ EXTERNAL ] MAINTENANCE ALERT: CRQ1-12345678 31/10/2021 00:00:00 GMT - 31/10/2021 07:30:00 GMT - COMPLETED
44+
"""
45+
data = {}
46+
search = re.search(
47+
r"\[.+\](.+).+?(CRQ\w+-\w+).+?(\d+/\d+/\d+\s\d+:\d+:\d+\s[A-Z]+).+?(\d+/\d+/\d+\s\d+:\d+:\d+\s[A-Z]+).+?([A-Z]+)",
48+
subject,
49+
)
50+
if search:
51+
data["maintenance_id"] = search.group(2)
52+
data["start"] = self.dt2ts(parser.parse(search.group(3)))
53+
data["end"] = self.dt2ts(parser.parse(search.group(4)))
54+
if search.group(5) == "START":
55+
data["status"] = Status("IN-PROCESS")
56+
elif search.group(5) == "COMPLETED":
57+
data["status"] = Status("COMPLETED")
58+
else:
59+
data["status"] = Status("CONFIRMED")
60+
data["summary"] = subject
7661
return [data]
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
"""Circuit Maintenance Parser for Equinix Email Notifications."""
2+
from typing import Any, Dict, List
3+
import re
4+
5+
from bs4.element import ResultSet # type: ignore
6+
from dateutil import parser
7+
8+
from circuit_maintenance_parser.output import Impact
9+
from circuit_maintenance_parser.parser import Html, EmailSubjectParser, Status
10+
11+
12+
class HtmlParserEquinix(Html):
13+
"""Custom Parser for HTML portion of Equinix circuit maintenance notifications."""
14+
15+
def parse_html(self, soup: ResultSet) -> List[Dict]:
16+
"""Parse an equinix circuit maintenance email.
17+
18+
Args:
19+
soup (ResultSet): beautiful soup object containing the html portion of an email.
20+
21+
Returns:
22+
Dict: The data dict containing circuit maintenance data.
23+
"""
24+
data: Dict[str, Any] = {"circuits": list()}
25+
26+
impact = self._parse_b(soup.find_all("b"), data)
27+
self._parse_table(soup.find_all("th"), data, impact)
28+
return [data]
29+
30+
@staticmethod
31+
def _isascii(string):
32+
"""Python 3.6 compatible way to determine if string is only english characters.
33+
34+
Args:
35+
string (str): string to test if only ascii chars.
36+
37+
Returns:
38+
bool: Returns True if string is ascii only, returns false if the string contains extended unicode characters.
39+
"""
40+
try:
41+
string.encode("ascii")
42+
return True
43+
except UnicodeEncodeError:
44+
return False
45+
46+
def _parse_b(self, b_elements, data):
47+
"""Parse the <b> elements from the notification to capture start and end times, description, and impact.
48+
49+
Args:
50+
b_elements (): resulting soup object with all <b> elements
51+
data (Dict): data from the circuit maintenance
52+
53+
Returns:
54+
impact (Status object): impact of the maintenance notification (used in the parse table function to assign an impact for each circuit).
55+
"""
56+
for b_elem in b_elements:
57+
if "UTC:" in b_elem:
58+
raw_time = b_elem.next_sibling
59+
# for non english equinix notifications
60+
# english section is usually at the bottom
61+
# this skips the non english line at the top
62+
if not self._isascii(raw_time):
63+
continue
64+
start_end_time = raw_time.split("-")
65+
if len(start_end_time) == 2:
66+
data["start"] = self.dt2ts(parser.parse(raw_time.split("-")[0].strip()))
67+
data["end"] = self.dt2ts(parser.parse(raw_time.split("-")[1].strip()))
68+
# all circuits in the notification share the same impact
69+
if "IMPACT:" in b_elem:
70+
impact_line = b_elem.next_sibling
71+
if "No impact to your service" in impact_line:
72+
impact = Impact.NO_IMPACT
73+
elif "There will be service interruptions" in impact_line.next_sibling.text:
74+
impact = Impact.OUTAGE
75+
return impact
76+
77+
def _parse_table(self, theader_elements, data, impact): # pylint: disable=no-self-use
78+
for th_elem in theader_elements:
79+
if "Account #" in th_elem:
80+
circuit_table = th_elem.find_parent("table")
81+
for tr_elem in circuit_table.find_all("tr"):
82+
if tr_elem.find(th_elem):
83+
continue
84+
circuit_info = list(tr_elem.find_all("td"))
85+
if circuit_info:
86+
account, _, circuit = circuit_info # pylint: disable=unused-variable
87+
data["circuits"].append(
88+
{"circuit_id": circuit.text, "impact": impact,}
89+
)
90+
data["account"] = account.text
91+
92+
93+
class SubjectParserEquinix(EmailSubjectParser):
94+
"""Parse the subject of an equinix circuit maintenance email. The subject contains the maintenance ID and status."""
95+
96+
def parse_subject(self, subject: str) -> List[Dict]:
97+
"""Parse the Equinix Email subject for summary and status.
98+
99+
Args:
100+
subject (str): subject of email
101+
e.g. 'Scheduled software upgrade in metro connect platform-SG Metro Area Network Maintenance -19-OCT-2021 [5-212760022356]'.
102+
103+
104+
Returns:
105+
List[Dict]: Returns the data object with summary and status fields.
106+
"""
107+
data = {}
108+
maintenance_id = re.search(r"\[(.*)\]$", subject)
109+
if maintenance_id:
110+
data["maintenance_id"] = maintenance_id[1]
111+
data["summary"] = subject.strip().replace("\n", "")
112+
if "COMPLETED" in subject:
113+
data["status"] = Status.COMPLETED
114+
if "SCHEDULED" in subject or "REMINDER" in subject:
115+
data["status"] = Status.CONFIRMED
116+
return [data]

circuit_maintenance_parser/parsers/lumen.py

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,15 @@ def parse_tables(self, tables: ResultSet, data: Dict):
100100
data["status"] = "CONFIRMED"
101101

102102
data_circuit = {}
103-
data_circuit["circuit_id"] = cells[idx + 1].string
103+
104+
# The table can include "Circuit ID" or "Alt Circuit ID" as columns +1 and +2.
105+
# Use the Circuit ID if available, else the Alt Circuit ID if available
106+
circuit_id = cells[idx + 1].string
107+
if circuit_id in ("_", "N/A"):
108+
circuit_id = cells[idx + 2].string
109+
if circuit_id not in ("_", "N/A"):
110+
data_circuit["circuit_id"] = circuit_id
111+
104112
impact = cells[idx + 6].string
105113
if "outage" in impact.lower():
106114
data_circuit["impact"] = Impact("OUTAGE")

0 commit comments

Comments
 (0)