Merge branch 'main' into bewithgaurav/update_pr_template

Author: subrata-ms

.github/agents/release-manager.agent.md

@@ -48,6 +48,16 @@ build/
 
 # wheel files
 *.whl
+
+# Coverage reports and artifacts
+.coverage
+coverage.json
+coverage*.xml
+htmlcov/
+unified-coverage/
+*.profraw
+*.profdata
+*.info
 *.tar.gz
 *.zip
 
@@ -69,3 +79,11 @@ mssql_py_core/
 
 # learning files
 learnings/
+
+# Local development and experimental scripts (not part of the PR)
+add_platform_exclusions.py
+add_lcov_exclusions.py
+fix_multiline_log_exclusions.py
+test_pyodbc_decimal.py
+run_coverage_docker.ps1
+TRIAGE_REPORT_*.md

.gitignore

@@ -35,21 +35,20 @@ PyBind11 provides:
 - Memory-safe bindings
 - Clean and Pythonic API, while performance-critical logic remains in robust, maintainable C++.
 
-## What's new in v1.6.0
+## What's new in v1.8.0
 
 ### Enhancements
 
-- **Connection String Sanitization** - Connection string sanitization has been migrated from regex-based to parser-based logic, making it more robust and consistent with connection string parsing rules.
+- **ActiveDirectoryMSI Support for Bulk Copy** - Adds `Authentication=ActiveDirectoryMSI` support to bulk copy, enabling both system-assigned and user-assigned managed identity authentication for Azure-hosted services (#573).
+- **Row String-Key Indexing** - Row objects now support accessing values by column name as a string key (e.g., `row["col"]`), in addition to integer index and attribute access. Case-insensitive lookup is supported when the cursor's `lowercase` attribute is enabled (#589).
+- **Bundled ODBC Driver Upgrade** - Updated the bundled Microsoft ODBC Driver for SQL Server from 18.5.1.1 to 18.6.2.1 (#569).
 
 ### Bug Fixes
 
-- **GIL Release During ODBC Connect/Disconnect** - The driver now releases the GIL during blocking ODBC connect and disconnect calls, improving concurrency for multi-threaded applications.
-- **setinputsizes() SQL_DECIMAL Crash Fix** - Fixed a crash in `cursor.setinputsizes()` when specifying `SQL_DECIMAL` type hints.
-- **ODBC Catalog fetchone() Fix** - Fixed an issue where `fetchone()` on ODBC catalog method results returned incorrect data.
-- **cursor.execute() Invalid Cursor State Fix** - Fixed `cursor.execute()` raising an Invalid cursor state error when called with `reset_cursor=False`.
-- **executemany Type Annotation Fix** - Corrected the type annotation for `executemany` `seq_of_parameters` parameter to accept `Mapping` types.
-- **setup_logging Path Traversal Guard** - Added path canonicalization and traversal guard to `setup_logging`'s `log_file_path` parameter to prevent path traversal issues.
- 
+- **Deferred Connect-Attribute Use-After-Free** - Fixed a use-after-free in `Connection.setAttribute` for deferred ODBC attributes (e.g., `SQL_COPT_SS_ACCESS_TOKEN`) that caused SIGBUS on macOS arm64 and authentication failures on Windows and Azure SQL (#596).
+- **Connection String Parsed Multiple Times in Auth Path** - Refactored authentication handling to use dictionary-based parameter processing instead of repeated string parsing, improving reliability and performance (#590).
+- **executemany Type Annotation Regression** - Fixed a typing regression where `Cursor.executemany` rejected valid `list[tuple[...]]` arguments under mypy due to invariant `List` type. The parameter type now uses covariant `Sequence` matching PEP 249 (#586).
+
 For more information, please visit the project link on Github: https://github.com/microsoft/mssql-python
 
 If you have any feedback, questions or need support please mail us at mssql-python@microsoft.com.

PyPI_Description.md

@@ -340,10 +340,10 @@ jobs:
       } else {
         Write-Host "ODBC Driver 18 for SQL Server not found, installing..."
         
-        # Download ODBC Driver 18.5.2.1 (x64) from official Microsoft link
+        # Download ODBC Driver 18.6.2.1 (x64) from official Microsoft link
         $ProgressPreference = 'SilentlyContinue'
-        $installerUrl = "https://go.microsoft.com/fwlink/?linkid=2335671"
-        $installerPath = "$env:TEMP\msodbcsql_18.5.2.1_x64.msi"
+        $installerUrl = "https://go.microsoft.com/fwlink/?linkid=2358430"
+        $installerPath = "$env:TEMP\msodbcsql_18.6.2.1_x64.msi"
         
         Write-Host "Downloading ODBC Driver 18 (x64) from Microsoft..."
         Write-Host "  URL: $installerUrl"
@@ -808,7 +808,7 @@ jobs:
         odbcinst -u -d -n 'ODBC Driver 18 for SQL Server' || true
         echo 'Uninstalled ODBC Driver and cleaned up libraries'
         echo 'Verifying x86_64 debian_ubuntu driver library signatures:'
-        ldd mssql_python/libs/linux/debian_ubuntu/x86_64/lib/libmsodbcsql-18.5.so.1.1
+        ldd mssql_python/libs/linux/debian_ubuntu/x86_64/lib/libmsodbcsql-18.6.so.2.1
       "
     displayName: 'Uninstall ODBC Driver before running tests in $(distroName) container'
 
@@ -1133,7 +1133,7 @@ jobs:
         odbcinst -u -d -n 'ODBC Driver 11 for SQL Server' || true
         echo 'Uninstalled ODBC Driver and cleaned up libraries'
         echo 'Verifying arm64 debian_ubuntu driver library signatures:'
-        ldd mssql_python/libs/linux/debian_ubuntu/arm64/lib/libmsodbcsql-18.5.so.1.1
+        ldd mssql_python/libs/linux/debian_ubuntu/arm64/lib/libmsodbcsql-18.6.so.2.1
       "
     displayName: 'Uninstall ODBC Driver before running tests in $(distroName) ARM64 container'
 
@@ -1346,7 +1346,7 @@ jobs:
         odbcinst -u -d -n 'ODBC Driver 11 for SQL Server' || true
         echo 'Uninstalled ODBC Driver and cleaned up libraries'
         echo 'Verifying x86_64 rhel driver library signatures:'
-        ldd mssql_python/libs/linux/rhel/x86_64/lib/libmsodbcsql-18.5.so.1.1
+        ldd mssql_python/libs/linux/rhel/x86_64/lib/libmsodbcsql-18.6.so.2.1
       "
     displayName: 'Uninstall ODBC Driver before running tests in RHEL 9 container'
 
@@ -1571,7 +1571,7 @@ jobs:
         odbcinst -u -d -n 'ODBC Driver 18 for SQL Server' || true
         echo 'Uninstalled ODBC Driver and cleaned up libraries'
         echo 'Verifying arm64 rhel driver library signatures:'
-        ldd mssql_python/libs/linux/rhel/arm64/lib/libmsodbcsql-18.5.so.1.1
+        ldd mssql_python/libs/linux/rhel/arm64/lib/libmsodbcsql-18.6.so.2.1
       "
     displayName: 'Uninstall ODBC Driver before running tests in RHEL 9 ARM64 container'
 
@@ -1731,24 +1731,24 @@ jobs:
         echo 'Detected architecture: '\$architecture
         
         # Download the packages
-        curl -O https://download.microsoft.com/download/fae28b9a-d880-42fd-9b98-d779f0fdd77f/msodbcsql18_18.5.1.1-1_\$architecture.apk
+        curl -O https://download.microsoft.com/download/0b3d5518-b4a7-4a2b-afc7-7ee9e967f93c/msodbcsql18_18.6.2.1-1_\$architecture.apk
         curl -O https://download.microsoft.com/download/7/6d/76de322a-d860-4894-9945-f0cc5d6a45f8/mssql-tools18_18.4.1.1-1_\$architecture.apk
         
         # Download signatures for verification
-        curl -O https://download.microsoft.com/download/fae28b9a-d880-42fd-9b98-d779f0fdd77f/msodbcsql18_18.5.1.1-1_\$architecture.sig
+        curl -O https://download.microsoft.com/download/0b3d5518-b4a7-4a2b-afc7-7ee9e967f93c/msodbcsql18_18.6.2.1-1_\$architecture.sig
         curl -O https://download.microsoft.com/download/7/6d/76de322a-d860-4894-9945-f0cc5d6a45f8/mssql-tools18_18.4.1.1-1_\$architecture.sig
         
         # Import Microsoft GPG key and verify packages
         curl https://packages.microsoft.com/keys/microsoft.asc | gpg --import -
-        gpg --verify msodbcsql18_18.5.1.1-1_\$architecture.sig msodbcsql18_18.5.1.1-1_\$architecture.apk
+        gpg --verify msodbcsql18_18.6.2.1-1_\$architecture.sig msodbcsql18_18.6.2.1-1_\$architecture.apk
         gpg --verify mssql-tools18_18.4.1.1-1_\$architecture.sig mssql-tools18_18.4.1.1-1_\$architecture.apk
         
         # Install the packages
-        apk add --allow-untrusted msodbcsql18_18.5.1.1-1_\$architecture.apk
+        apk add --allow-untrusted msodbcsql18_18.6.2.1-1_\$architecture.apk
         apk add --allow-untrusted mssql-tools18_18.4.1.1-1_\$architecture.apk
         
         # Cleanup
-        rm -f msodbcsql18_18.5.1.1-1_\$architecture.* mssql-tools18_18.4.1.1-1_\$architecture.*
+        rm -f msodbcsql18_18.6.2.1-1_\$architecture.* mssql-tools18_18.4.1.1-1_\$architecture.*
         
         # Add mssql-tools to PATH
         export PATH=\"\$PATH:/opt/mssql-tools18/bin\"
@@ -1805,7 +1805,7 @@ jobs:
         odbcinst -u -d -n 'ODBC Driver 18 for SQL Server' || true
         echo 'Uninstalled system ODBC Driver and cleaned up libraries'
         echo 'Verifying x86_64 alpine driver library signatures:'
-        ldd mssql_python/libs/linux/alpine/x86_64/lib/libmsodbcsql-18.5.so.1.1 || echo 'Driver library not found'
+        ldd mssql_python/libs/linux/alpine/x86_64/lib/libmsodbcsql-18.6.so.2.1 || echo 'Driver library not found'
       "
     displayName: 'Uninstall system ODBC Driver before running tests in Alpine x86_64 container'
 
@@ -1981,24 +1981,24 @@ jobs:
         echo 'Detected architecture: '\$architecture
         
         # Download the packages
-        curl -O https://download.microsoft.com/download/fae28b9a-d880-42fd-9b98-d779f0fdd77f/msodbcsql18_18.5.1.1-1_\$architecture.apk
+        curl -O https://download.microsoft.com/download/0b3d5518-b4a7-4a2b-afc7-7ee9e967f93c/msodbcsql18_18.6.2.1-1_\$architecture.apk
         curl -O https://download.microsoft.com/download/7/6d/76de322a-d860-4894-9945-f0cc5d6a45f8/mssql-tools18_18.4.1.1-1_\$architecture.apk
         
         # Download signatures for verification
-        curl -O https://download.microsoft.com/download/fae28b9a-d880-42fd-9b98-d779f0fdd77f/msodbcsql18_18.5.1.1-1_\$architecture.sig
+        curl -O https://download.microsoft.com/download/0b3d5518-b4a7-4a2b-afc7-7ee9e967f93c/msodbcsql18_18.6.2.1-1_\$architecture.sig
         curl -O https://download.microsoft.com/download/7/6d/76de322a-d860-4894-9945-f0cc5d6a45f8/mssql-tools18_18.4.1.1-1_\$architecture.sig
         
         # Import Microsoft GPG key and verify packages
         curl https://packages.microsoft.com/keys/microsoft.asc | gpg --import -
-        gpg --verify msodbcsql18_18.5.1.1-1_\$architecture.sig msodbcsql18_18.5.1.1-1_\$architecture.apk
+        gpg --verify msodbcsql18_18.6.2.1-1_\$architecture.sig msodbcsql18_18.6.2.1-1_\$architecture.apk
         gpg --verify mssql-tools18_18.4.1.1-1_\$architecture.sig mssql-tools18_18.4.1.1-1_\$architecture.apk
         
         # Install the packages
-        apk add --allow-untrusted msodbcsql18_18.5.1.1-1_\$architecture.apk
+        apk add --allow-untrusted msodbcsql18_18.6.2.1-1_\$architecture.apk
         apk add --allow-untrusted mssql-tools18_18.4.1.1-1_\$architecture.apk
         
         # Cleanup
-        rm -f msodbcsql18_18.5.1.1-1_\$architecture.* mssql-tools18_18.4.1.1-1_\$architecture.*
+        rm -f msodbcsql18_18.6.2.1-1_\$architecture.* mssql-tools18_18.4.1.1-1_\$architecture.*
         
         # Add mssql-tools to PATH
         export PATH=\"\$PATH:/opt/mssql-tools18/bin\"
@@ -2056,7 +2056,7 @@ jobs:
         odbcinst -u -d -n 'ODBC Driver 18 for SQL Server' || true
         echo 'Uninstalled system ODBC Driver and cleaned up libraries'
         echo 'Verifying arm64 alpine driver library signatures:'
-        ldd mssql_python/libs/linux/alpine/arm64/lib/libmsodbcsql-18.5.so.1.1 || echo 'Driver library not found'
+        ldd mssql_python/libs/linux/alpine/arm64/lib/libmsodbcsql-18.6.so.2.1 || echo 'Driver library not found'
       "
     displayName: 'Uninstall system ODBC Driver before running tests in Alpine ARM64 container'
 

eng/pipelines/pr-validation-pipeline.yml

@@ -0,0 +1,201 @@
+#!/usr/bin/env python3
+"""
+Join multi-line LOG() calls onto single lines for LCOV coverage filtering.
+
+This script is used only during coverage builds to simplify LOG statement exclusion.
+It doesn't modify the original source files - it works on copies during the build.
+Adjacent string literals are concatenated at compile time, so runtime behavior is identical.
+
+Uses a proper C++ tokenizer to handle string literals, character literals, and comments
+correctly, avoiding issues with unbalanced parentheses or semicolons in strings.
+"""
+
+import re
+import sys
+from pathlib import Path
+
+
+_LOG_MACRO_PATTERN = re.compile(r'\bLOG[A-Z_]*\s*\(')
+
+
+def _find_log_macro_open(line: str):
+    """Return the index of the opening parenthesis for a LOG-like macro, if present."""
+    match = _LOG_MACRO_PATTERN.search(line)
+    if not match:
+        return None
+    return match.end() - 1
+
+
+def _find_log_statement_end(lines, start_line, open_paren_index):
+    """Find the line index where the LOG macro call closes, ignoring literals/comments.
+    
+    This properly handles:
+    - String literals: LOG("unbalanced (", x);
+    - Character literals: LOG(')', code);  
+    - Line comments: LOG("msg", x);  // comment with )
+    - Block comments: LOG("msg" /* comment ) */, x);
+    """
+    depth = 0
+    in_string = False
+    in_char = False
+    in_block_comment = False
+    escape = False
+    
+    for line_index in range(start_line, len(lines)):
+        line = lines[line_index]
+        i = open_paren_index if line_index == start_line else 0
+        in_line_comment = False
+        
+        while i < len(line):
+            ch = line[i]
+            nxt = line[i + 1] if i + 1 < len(line) else ''
+            
+            # Line comments consume rest of line
+            if in_line_comment:
+                break
+            
+            # Inside block comment - only look for */
+            if in_block_comment:
+                if ch == '*' and nxt == '/':
+                    in_block_comment = False
+                    i += 2
+                    continue
+                i += 1
+                continue
+            
+            # Inside string literal - handle escapes
+            if in_string:
+                if escape:
+                    escape = False
+                elif ch == '\\':
+                    escape = True
+                elif ch == '"':
+                    in_string = False
+                i += 1
+                continue
+            
+            # Inside character literal - handle escapes
+            if in_char:
+                if escape:
+                    escape = False
+                elif ch == '\\':
+                    escape = True
+                elif ch == "'":
+                    in_char = False
+                i += 1
+                continue
+            
+            # Check for comment starts
+            if ch == '/' and nxt == '/':
+                in_line_comment = True
+                break
+            if ch == '/' and nxt == '*':
+                in_block_comment = True
+                i += 2
+                continue
+            
+            # Check for literal starts
+            if ch == '"':
+                in_string = True
+                escape = False
+                i += 1
+                continue
+            if ch == "'":
+                in_char = True
+                escape = False
+                i += 1
+                continue
+            
+            # Count parentheses depth outside of literals/comments
+            if ch == '(':
+                depth += 1
+            elif ch == ')':
+                depth -= 1
+                if depth == 0:
+                    return line_index
+            
+            i += 1
+    
+    return None
+
+
+def join_log_statements(content: str) -> str:
+    """Join multi-line LOG macro calls onto a single line using proper C++ tokenization."""
+    lines = content.split('\n')
+    result = []
+    i = 0
+    
+    while i < len(lines):
+        line = lines[i]
+        
+        # Check if this line contains a LOG macro start
+        open_paren_index = _find_log_macro_open(line)
+        if open_paren_index is not None:
+            # Find where the LOG statement ends, respecting C++ syntax
+            end_index = _find_log_statement_end(lines, i, open_paren_index)
+            if end_index is not None and end_index > i:
+                # Multi-line LOG statement found - join it
+                full_statement = lines[i]
+                for join_index in range(i + 1, end_index + 1):
+                    full_statement += ' ' + lines[join_index].strip()
+                result.append(full_statement)
+                i = end_index + 1
+                continue
+        
+        # Not a LOG statement or single-line LOG - keep as is
+        result.append(line)
+        i += 1
+    
+    return '\n'.join(result)
+
+
+def process_file(filepath: Path) -> None:
+    """Process a single C++ source file."""
+    try:
+        with open(filepath, 'r', encoding='utf-8') as f:
+            content = f.read()
+        
+        modified = join_log_statements(content)
+        
+        with open(filepath, 'w', encoding='utf-8') as f:
+            f.write(modified)
+        
+        print(f"[INFO] Processed: {filepath}")
+    except Exception as e:
+        print(f"[ERROR] Failed to process {filepath}: {e}", file=sys.stderr)
+        sys.exit(1)
+
+
+def main():
+    """Process all .cpp and .hpp files in the pybind directory."""
+    if len(sys.argv) > 1:
+        # Process specific directory passed as argument
+        base_dir = Path(sys.argv[1])
+    else:
+        # Default to current directory
+        base_dir = Path.cwd()
+    
+    if not base_dir.exists():
+        print(f"[ERROR] Directory not found: {base_dir}", file=sys.stderr)
+        sys.exit(1)
+    
+    # Find all C++ source and header files (*.cpp, *.h, *.hpp)
+    # Note: using '*.h*' pattern to match both .h and .hpp extensions
+    cpp_files = list(base_dir.rglob('*.cpp')) + list(base_dir.rglob('*.h*'))
+    
+    if not cpp_files:
+        print(f"[WARNING] No .cpp or .hpp files found in {base_dir}")
+        return
+    
+    print(f"[INFO] Processing {len(cpp_files)} C++ files in {base_dir}")
+    for filepath in cpp_files:
+        # Skip cmake-generated files in build directory
+        if 'build' in filepath.parts:
+            continue
+        process_file(filepath)
+    
+    print(f"[SUCCESS] Joined LOG statements in {len(cpp_files)} files")
+
+
+if __name__ == '__main__':
+    main()