Skip to content

Commit 3695d6e

Browse files
committed
add some debugging msg.
1 parent 8ff2c94 commit 3695d6e

File tree

3 files changed

+20
-16
lines changed

3 files changed

+20
-16
lines changed

sdks/python/apache_beam/pipeline.py

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -561,9 +561,11 @@ def run(self, test_runner_api: Union[bool, str] = 'AUTO') -> 'PipelineResult':
561561
"""Runs the pipeline. Returns whatever our runner returns after running."""
562562
# All pipeline options are finalized at this point.
563563
# Call get_all_options to print warnings on invalid options.
564-
self.options.get_all_options(
564+
opts = self.options.get_all_options(
565565
retain_unknown_options=True, display_warnings=True)
566566

567+
logging.exception("all_options:" + str(opts))
568+
logging.exception('runner class:' + str(self.runner))
567569
for error_handler in self._error_handlers:
568570
error_handler.verify_closed()
569571

@@ -867,7 +869,7 @@ def _infer_result_type(
867869
inputs: Sequence[Union[pvalue.PBegin, pvalue.PCollection]],
868870
result_pcollection: Union[pvalue.PValue, pvalue.DoOutputsTuple]) -> None:
869871
"""Infer and set the output element type for a PCollection.
870-
872+
871873
This function determines the output types of transforms by combining:
872874
1. Concrete input types from previous transforms
873875
2. Type hints declared on the current transform
@@ -878,43 +880,43 @@ def _infer_result_type(
878880
Type variables (K, V, T, etc.) act as placeholders that get bound to
879881
concrete types through pattern matching. This requires both an input
880882
pattern and an output template:
881-
883+
882884
Input Pattern (from .with_input_types()):
883885
Defines where in the input to find each type variable
884886
Example: Tuple[K, V] means "K is the first element, V is the second"
885-
887+
886888
Output Template (from .with_output_types()):
887889
Defines how to use the bound variables in the output
888890
Example: Tuple[V, K] means "swap the positions"
889-
891+
890892
CONCRETE TYPES VS TYPE VARIABLES
891893
---------------------------------
892894
The system handles these differently:
893-
895+
894896
Concrete Types (e.g., str, int, Tuple[str, int]):
895897
- Used as-is without any binding
896898
- Do not fall back to Any
897899
- Example: .with_output_types(Tuple[str, int]) → Tuple[str, int]
898-
900+
899901
Type Variables (e.g., K, V, T):
900902
- Must be bound through pattern matching
901903
- Require .with_input_types() to provide the pattern
902904
- Fall back to Any if not bound
903905
- Example without pattern: Tuple[K, V] → Tuple[Any, Any]
904906
- Example with pattern: Tuple[K, V] → Tuple[str, int]
905-
907+
906908
BINDING ALGORITHM
907909
-----------------
908910
1. Match: Compare input pattern to concrete input
909911
Pattern: Tuple[K, V]
910912
Concrete: Tuple[str, int]
911913
Result: {K: str, V: int} ← Bindings created
912-
914+
913915
2. Substitute: Apply bindings to output template
914916
Template: Tuple[V, K] ← Note: swapped!
915917
Bindings: {K: str, V: int}
916918
Result: Tuple[int, str] ← Swapped concrete types
917-
919+
918920
Each transform operates in its own type inference scope. Type variables
919921
declared in a parent composite transform do NOT automatically propagate
920922
to child transforms.
@@ -925,36 +927,36 @@ class MyComposite(PTransform):
925927
def expand(self, pcoll):
926928
# Child scope - parent's K, V are NOT available
927929
return pcoll | ChildTransform()
928-
930+
929931
Type variables that remain unbound after inference fall back to Any:
930-
932+
931933
EXAMPLES
932934
--------
933935
Example 1: Concrete types (no variables)
934936
Input: Tuple[str, int]
935937
Transform: .with_output_types(Tuple[str, int])
936938
Output: Tuple[str, int] ← Used as-is
937-
939+
938940
Example 2: Type variables with pattern (correct)
939941
Input: Tuple[str, int]
940942
Transform: .with_input_types(Tuple[K, V])
941943
.with_output_types(Tuple[V, K])
942944
Binding: {K: str, V: int}
943945
Output: Tuple[int, str] ← Swapped!
944-
946+
945947
Example 3: Type variables without pattern (falls back to Any)
946948
Input: Tuple[str, int]
947949
Transform: .with_output_types(Tuple[K, V]) ← No input pattern!
948950
Binding: None (can't match)
949951
Output: Tuple[Any, Any] ← Fallback
950-
952+
951953
Example 4: Mixed concrete and variables
952954
Input: Tuple[str, int]
953955
Transform: .with_input_types(Tuple[str, V])
954956
.with_output_types(Tuple[str, V])
955957
Binding: {V: int} ← Only V needs binding
956958
Output: Tuple[str, int] ← str passed through, V bound to int
957-
959+
958960
Args:
959961
transform: The PTransform being applied
960962
inputs: Input PCollections (provides concrete types)

sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -475,6 +475,7 @@ def __init__(
475475
("grpc.http2.min_ping_interval_without_data_ms", 20_000),
476476
]
477477

478+
_LOGGER.exception("Starting a few grpc server with options:" + str(options))
478479
self.state = state
479480
self.provision_info = provision_info
480481
self.control_server = grpc.server(

sdks/python/setup.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
# See the License for the specific language governing permissions and
1515
# limitations under the License.
1616
#
17+
# Modify setup.py to trigger more tests.
1718

1819
"""Apache Beam SDK for Python setup file."""
1920
import glob

0 commit comments

Comments
 (0)