Skip to content

Commit 6dcdae8

Browse files
rbiseck3Unstructured-DevOps
rbiseck3
and
Unstructured-DevOps
authored
feat/otel support (#78)
* use otel console exporter for timing information * Add docker compose setup with otel * fix child spans being emitted in multiprocessing fanout * Add opentelemetry as base dep * Add changelog entry * Bump version of unstructured * use better name for span * Update ingest test fixtures (#80) Co-authored-by: rbiseck3 <[email protected]> * Add missing kdbai dep --------- Co-authored-by: Unstructured-DevOps <[email protected]> Co-authored-by: rbiseck3 <[email protected]>
1 parent 85c165f commit 6dcdae8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

94 files changed

+1974
-421
lines changed

‎CHANGELOG.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
1-
## 0.0.7-dev2
1+
## 0.0.7-dev3
22

33
### Enhancements
44

55
* **support sharing parent multiprocessing for uploaders** If an uploader needs to fan out it's process using multiprocessing, support that using the parent pipeline approach rather than handling it explicitly by the connector logic.
6+
* **OTEL support** If endpoint supplied, publish all traces to an otel collector.
67

78
### Fixes
89

‎requirements/common/base.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ pydantic
66
dataclasses_json
77
tqdm
88
click
9+
opentelemetry-sdk

‎requirements/common/base.txt‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,30 @@ click==8.1.7
1010
# via -r ./requirements/common/base.in
1111
dataclasses-json==0.6.7
1212
# via -r ./requirements/common/base.in
13-
marshmallow==3.21.3
13+
deprecated==1.2.14
14+
# via
15+
# opentelemetry-api
16+
# opentelemetry-semantic-conventions
17+
importlib-metadata==7.1.0
18+
# via
19+
# -c ./requirements/common/constraints.txt
20+
# opentelemetry-api
21+
marshmallow==3.22.0
1422
# via dataclasses-json
1523
mypy-extensions==1.0.0
1624
# via typing-inspect
1725
numpy==1.26.4
1826
# via
1927
# -c ./requirements/common/constraints.txt
2028
# pandas
29+
opentelemetry-api==1.26.0
30+
# via
31+
# opentelemetry-sdk
32+
# opentelemetry-semantic-conventions
33+
opentelemetry-sdk==1.26.0
34+
# via -r ./requirements/common/base.in
35+
opentelemetry-semantic-conventions==0.47b0
36+
# via opentelemetry-sdk
2137
packaging==23.2
2238
# via
2339
# -c ./requirements/common/constraints.txt
@@ -40,10 +56,17 @@ tqdm==4.66.5
4056
# via -r ./requirements/common/base.in
4157
typing-extensions==4.12.2
4258
# via
59+
# opentelemetry-sdk
4360
# pydantic
4461
# pydantic-core
4562
# typing-inspect
4663
typing-inspect==0.9.0
4764
# via dataclasses-json
4865
tzdata==2024.1
4966
# via pandas
67+
wrapt==1.16.0
68+
# via
69+
# -c ./requirements/common/constraints.txt
70+
# deprecated
71+
zipp==3.20.1
72+
# via importlib-metadata

‎requirements/common/constraints.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,5 +62,5 @@ langchain-community>=0.2.5
6262
# instead of 7.2.0. Need to investigate
6363
importlib-metadata==7.1.0
6464

65-
unstructured==0.15.5
65+
unstructured==0.15.8
6666
numpy<2

‎requirements/connectors/airtable.txt‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,18 +16,34 @@ click==8.1.7
1616
# via -r ./requirements/connectors/../common/base.in
1717
dataclasses-json==0.6.7
1818
# via -r ./requirements/connectors/../common/base.in
19-
idna==3.7
19+
deprecated==1.2.14
20+
# via
21+
# opentelemetry-api
22+
# opentelemetry-semantic-conventions
23+
idna==3.8
2024
# via requests
25+
importlib-metadata==7.1.0
26+
# via
27+
# -c ./requirements/connectors/../common/constraints.txt
28+
# opentelemetry-api
2129
inflection==0.5.1
2230
# via pyairtable
23-
marshmallow==3.21.3
31+
marshmallow==3.22.0
2432
# via dataclasses-json
2533
mypy-extensions==1.0.0
2634
# via typing-inspect
2735
numpy==1.26.4
2836
# via
2937
# -c ./requirements/connectors/../common/constraints.txt
3038
# pandas
39+
opentelemetry-api==1.26.0
40+
# via
41+
# opentelemetry-sdk
42+
# opentelemetry-semantic-conventions
43+
opentelemetry-sdk==1.26.0
44+
# via -r ./requirements/connectors/../common/base.in
45+
opentelemetry-semantic-conventions==0.47b0
46+
# via opentelemetry-sdk
3147
packaging==23.2
3248
# via
3349
# -c ./requirements/connectors/../common/constraints.txt
@@ -56,6 +72,7 @@ tqdm==4.66.5
5672
# via -r ./requirements/connectors/../common/base.in
5773
typing-extensions==4.12.2
5874
# via
75+
# opentelemetry-sdk
5976
# pyairtable
6077
# pydantic
6178
# pydantic-core
@@ -69,3 +86,9 @@ urllib3==1.26.19
6986
# -c ./requirements/connectors/../common/constraints.txt
7087
# pyairtable
7188
# requests
89+
wrapt==1.16.0
90+
# via
91+
# -c ./requirements/connectors/../common/constraints.txt
92+
# deprecated
93+
zipp==3.20.1
94+
# via importlib-metadata

‎requirements/connectors/astradb.txt‎

Lines changed: 28 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# This file is autogenerated by pip-compile with Python 3.9
33
# by the following command:
44
#
5-
# pip-compile ./requirements/connectors/astra.in
5+
# pip-compile ./requirements/connectors/astradb.in
66
#
77
annotated-types==0.7.0
88
# via pydantic
@@ -11,7 +11,7 @@ anyio==3.7.1
1111
# -c ./requirements/connectors/../common/constraints.txt
1212
# httpx
1313
astrapy==1.4.1
14-
# via -r ./requirements/connectors/astra.in
14+
# via -r ./requirements/connectors/astradb.in
1515
cassandra-driver==3.29.1
1616
# via cassio
1717
cassio==0.1.8
@@ -30,6 +30,10 @@ click==8.1.7
3030
# geomet
3131
dataclasses-json==0.6.7
3232
# via -r ./requirements/connectors/../common/base.in
33+
deprecated==1.2.14
34+
# via
35+
# opentelemetry-api
36+
# opentelemetry-semantic-conventions
3337
deprecation==2.1.0
3438
# via astrapy
3539
dnspython==2.6.1
@@ -46,16 +50,20 @@ hpack==4.0.0
4650
# via h2
4751
httpcore==1.0.5
4852
# via httpx
49-
httpx[http2]==0.27.0
53+
httpx[http2]==0.27.2
5054
# via astrapy
5155
hyperframe==6.0.1
5256
# via h2
53-
idna==3.7
57+
idna==3.8
5458
# via
5559
# anyio
5660
# httpx
5761
# requests
58-
marshmallow==3.21.3
62+
importlib-metadata==7.1.0
63+
# via
64+
# -c ./requirements/connectors/../common/constraints.txt
65+
# opentelemetry-api
66+
marshmallow==3.22.0
5967
# via dataclasses-json
6068
mypy-extensions==1.0.0
6169
# via typing-inspect
@@ -64,6 +72,14 @@ numpy==1.26.4
6472
# -c ./requirements/connectors/../common/constraints.txt
6573
# cassio
6674
# pandas
75+
opentelemetry-api==1.26.0
76+
# via
77+
# opentelemetry-sdk
78+
# opentelemetry-semantic-conventions
79+
opentelemetry-sdk==1.26.0
80+
# via -r ./requirements/connectors/../common/base.in
81+
opentelemetry-semantic-conventions==0.47b0
82+
# via opentelemetry-sdk
6783
packaging==23.2
6884
# via
6985
# -c ./requirements/connectors/../common/constraints.txt
@@ -99,6 +115,7 @@ tqdm==4.66.5
99115
# via -r ./requirements/connectors/../common/base.in
100116
typing-extensions==4.12.2
101117
# via
118+
# opentelemetry-sdk
102119
# pydantic
103120
# pydantic-core
104121
# typing-inspect
@@ -112,3 +129,9 @@ urllib3==1.26.19
112129
# requests
113130
uuid6==2024.1.12
114131
# via astrapy
132+
wrapt==1.16.0
133+
# via
134+
# -c ./requirements/connectors/../common/constraints.txt
135+
# deprecated
136+
zipp==3.20.1
137+
# via importlib-metadata

‎requirements/connectors/azure-cognitive-search.txt‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,18 +22,34 @@ click==8.1.7
2222
# via -r ./requirements/connectors/../common/base.in
2323
dataclasses-json==0.6.7
2424
# via -r ./requirements/connectors/../common/base.in
25-
idna==3.7
25+
deprecated==1.2.14
26+
# via
27+
# opentelemetry-api
28+
# opentelemetry-semantic-conventions
29+
idna==3.8
2630
# via requests
31+
importlib-metadata==7.1.0
32+
# via
33+
# -c ./requirements/connectors/../common/constraints.txt
34+
# opentelemetry-api
2735
isodate==0.6.1
2836
# via azure-search-documents
29-
marshmallow==3.21.3
37+
marshmallow==3.22.0
3038
# via dataclasses-json
3139
mypy-extensions==1.0.0
3240
# via typing-inspect
3341
numpy==1.26.4
3442
# via
3543
# -c ./requirements/connectors/../common/constraints.txt
3644
# pandas
45+
opentelemetry-api==1.26.0
46+
# via
47+
# opentelemetry-sdk
48+
# opentelemetry-semantic-conventions
49+
opentelemetry-sdk==1.26.0
50+
# via -r ./requirements/connectors/../common/base.in
51+
opentelemetry-semantic-conventions==0.47b0
52+
# via opentelemetry-sdk
3753
packaging==23.2
3854
# via
3955
# -c ./requirements/connectors/../common/constraints.txt
@@ -63,6 +79,7 @@ typing-extensions==4.12.2
6379
# via
6480
# azure-core
6581
# azure-search-documents
82+
# opentelemetry-sdk
6683
# pydantic
6784
# pydantic-core
6885
# typing-inspect
@@ -74,3 +91,9 @@ urllib3==1.26.19
7491
# via
7592
# -c ./requirements/connectors/../common/constraints.txt
7693
# requests
94+
wrapt==1.16.0
95+
# via
96+
# -c ./requirements/connectors/../common/constraints.txt
97+
# deprecated
98+
zipp==3.20.1
99+
# via importlib-metadata

‎requirements/connectors/azure.txt‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@
66
#
77
adlfs==2024.7.0
88
# via -r ./requirements/connectors/azure.in
9-
aiohappyeyeballs==2.3.6
9+
aiohappyeyeballs==2.4.0
1010
# via aiohttp
11-
aiohttp==3.10.3
11+
aiohttp==3.10.5
1212
# via adlfs
1313
aiosignal==1.3.1
1414
# via aiohttp
@@ -49,6 +49,10 @@ cryptography==43.0.0
4949
# pyjwt
5050
dataclasses-json==0.6.7
5151
# via -r ./requirements/connectors/../common/base.in
52+
deprecated==1.2.14
53+
# via
54+
# opentelemetry-api
55+
# opentelemetry-semantic-conventions
5256
frozenlist==1.4.1
5357
# via
5458
# aiohttp
@@ -58,13 +62,17 @@ fsspec==2024.5.0
5862
# -c ./requirements/connectors/../common/constraints.txt
5963
# -r ./requirements/connectors/azure.in
6064
# adlfs
61-
idna==3.7
65+
idna==3.8
6266
# via
6367
# requests
6468
# yarl
69+
importlib-metadata==7.1.0
70+
# via
71+
# -c ./requirements/connectors/../common/constraints.txt
72+
# opentelemetry-api
6573
isodate==0.6.1
6674
# via azure-storage-blob
67-
marshmallow==3.21.3
75+
marshmallow==3.22.0
6876
# via dataclasses-json
6977
msal==1.30.0
7078
# via
@@ -83,6 +91,14 @@ numpy==1.26.4
8391
# via
8492
# -c ./requirements/connectors/../common/constraints.txt
8593
# pandas
94+
opentelemetry-api==1.26.0
95+
# via
96+
# opentelemetry-sdk
97+
# opentelemetry-semantic-conventions
98+
opentelemetry-sdk==1.26.0
99+
# via -r ./requirements/connectors/../common/base.in
100+
opentelemetry-semantic-conventions==0.47b0
101+
# via opentelemetry-sdk
86102
packaging==23.2
87103
# via
88104
# -c ./requirements/connectors/../common/constraints.txt
@@ -122,6 +138,7 @@ typing-extensions==4.12.2
122138
# azure-core
123139
# azure-identity
124140
# azure-storage-blob
141+
# opentelemetry-sdk
125142
# pydantic
126143
# pydantic-core
127144
# typing-inspect
@@ -133,5 +150,11 @@ urllib3==1.26.19
133150
# via
134151
# -c ./requirements/connectors/../common/constraints.txt
135152
# requests
153+
wrapt==1.16.0
154+
# via
155+
# -c ./requirements/connectors/../common/constraints.txt
156+
# deprecated
136157
yarl==1.9.4
137158
# via aiohttp
159+
zipp==3.20.1
160+
# via importlib-metadata

‎requirements/connectors/biomed.txt‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,16 +20,32 @@ click==8.1.7
2020
# via -r ./requirements/connectors/../common/base.in
2121
dataclasses-json==0.6.7
2222
# via -r ./requirements/connectors/../common/base.in
23-
idna==3.7
23+
deprecated==1.2.14
24+
# via
25+
# opentelemetry-api
26+
# opentelemetry-semantic-conventions
27+
idna==3.8
2428
# via requests
25-
marshmallow==3.21.3
29+
importlib-metadata==7.1.0
30+
# via
31+
# -c ./requirements/connectors/../common/constraints.txt
32+
# opentelemetry-api
33+
marshmallow==3.22.0
2634
# via dataclasses-json
2735
mypy-extensions==1.0.0
2836
# via typing-inspect
2937
numpy==1.26.4
3038
# via
3139
# -c ./requirements/connectors/../common/constraints.txt
3240
# pandas
41+
opentelemetry-api==1.26.0
42+
# via
43+
# opentelemetry-sdk
44+
# opentelemetry-semantic-conventions
45+
opentelemetry-sdk==1.26.0
46+
# via -r ./requirements/connectors/../common/base.in
47+
opentelemetry-semantic-conventions==0.47b0
48+
# via opentelemetry-sdk
3349
packaging==23.2
3450
# via
3551
# -c ./requirements/connectors/../common/constraints.txt
@@ -56,6 +72,7 @@ tqdm==4.66.5
5672
# via -r ./requirements/connectors/../common/base.in
5773
typing-extensions==4.12.2
5874
# via
75+
# opentelemetry-sdk
5976
# pydantic
6077
# pydantic-core
6178
# typing-inspect
@@ -67,3 +84,9 @@ urllib3==1.26.19
6784
# via
6885
# -c ./requirements/connectors/../common/constraints.txt
6986
# requests
87+
wrapt==1.16.0
88+
# via
89+
# -c ./requirements/connectors/../common/constraints.txt
90+
# deprecated
91+
zipp==3.20.1
92+
# via importlib-metadata

0 commit comments

Comments
 (0)