Add documentation and tests for fsspec PR #1944 callable set_size() support

fsspec-pr-1944/APPLICATION_GUIDE.md

@@ -0,0 +1,138 @@
+# How to Apply These Changes to fsspec PR #1944
+
+This guide explains how to apply the enhanced documentation and test cases to the fsspec PR #1944.
+
+## Prerequisites
+
+- Git installed
+- Access to fork the fsspec repository or have the OneSizeFitsQuorum/filesystem_spec repository cloned
+- Python and pytest installed for running tests
+
+## Option 1: Apply Using Patch Files (Recommended)
+
+The easiest way to apply these changes is using the provided patch files:
+
+```bash
+# Navigate to your fsspec repository checkout
+cd /path/to/filesystem_spec
+
+# Make sure you're on the patch-1 branch (the PR branch)
+git checkout patch-1
+
+# Apply the callbacks.py patch
+git apply /path/to/this/repo/fsspec-pr-1944/callbacks.patch
+
+# Apply the test_callbacks.py patch
+git apply /path/to/this/repo/fsspec-pr-1944/test_callbacks.patch
+
+# Verify the changes
+git diff
+
+# Run tests to ensure everything works
+pytest fsspec/tests/test_callbacks.py -v
+
+# Commit the changes
+git add fsspec/callbacks.py fsspec/tests/test_callbacks.py
+git commit -m "Add enhanced documentation and test cases for set_size callable support"
+
+# Push to your fork
+git push origin patch-1
+```
+
+## Option 2: Manual Copy
+
+If the patch files don't apply cleanly, you can manually copy the files:
+
+```bash
+# Navigate to your fsspec repository
+cd /path/to/filesystem_spec
+
+# Make sure you're on the patch-1 branch
+git checkout patch-1
+
+# Copy the updated files
+cp /path/to/this/repo/fsspec-pr-1944/callbacks.py fsspec/callbacks.py
+cp /path/to/this/repo/fsspec-pr-1944/test_callbacks.py fsspec/tests/test_callbacks.py
+
+# Run tests
+pytest fsspec/tests/test_callbacks.py::test_set_size_with_callable -v
+
+# Commit and push
+git add fsspec/callbacks.py fsspec/tests/test_callbacks.py
+git commit -m "Add enhanced documentation and test cases for set_size callable support"
+git push origin patch-1
+```
+
+## Option 3: Manual Edit
+
+If you prefer to make the changes manually, refer to the patch files to see exactly what needs to be added:
+
+1. **For callbacks.py**: 
+   - Open `fsspec-pr-1944/callbacks.patch` to see the documentation enhancements
+   - The main change is expanding the docstring of the `set_size()` method (lines 9-30 in the patch)
+
+2. **For test_callbacks.py**:
+   - Open `fsspec-pr-1944/test_callbacks.patch` to see the new test function
+   - Add the `test_set_size_with_callable()` function before the `test_tqdm_callback` function
+
+## Verifying the Changes
+
+After applying the changes, verify everything works correctly:
+
+```bash
+# Run just the new test
+pytest fsspec/tests/test_callbacks.py::test_set_size_with_callable -v
+
+# Run all callback tests to ensure nothing broke
+pytest fsspec/tests/test_callbacks.py -v
+
+# Optionally, run the full test suite
+pytest fsspec/tests/
+```
+
+Expected output:
+```
+fsspec/tests/test_callbacks.py::test_set_size_with_callable PASSED
+```
+
+## What Gets Updated
+
+### callbacks.py Changes:
+- Enhanced docstring for `set_size()` method
+- Added detailed parameter description
+- Added examples section with usage demonstrations
+- Added notes section explaining callable behavior
+
+### test_callbacks.py Changes:
+- New test function: `test_set_size_with_callable()`
+- Tests 4 scenarios:
+  1. Integer parameter (backward compatibility)
+  2. Lambda function
+  3. Named function
+  4. Method reference (the primary use case)
+
+## Troubleshooting
+
+### Patch fails to apply
+If the patch files don't apply cleanly, it may be because the base files have changed. In this case:
+1. Check the error message to see which hunks failed
+2. Use Option 2 (Manual Copy) or Option 3 (Manual Edit) instead
+
+### Tests fail
+If tests fail after applying changes:
+1. Ensure you have all required dependencies: `pip install -e ".[dev,test]"`
+2. Check that the changes were applied correctly
+3. Compare your files with the provided `callbacks.py` and `test_callbacks.py` files in this directory
+
+## Additional Notes
+
+- The changes are backward compatible - existing code using integer values will continue to work
+- The new functionality enables filesystem objects with `size()` methods to work seamlessly
+- All existing tests continue to pass, ensuring no regression
+
+## Questions or Issues?
+
+If you encounter any problems applying these changes, refer to:
+- The README.md in this directory for background information
+- The patch files to see exactly what changed
+- The full updated files (callbacks.py and test_callbacks.py) in this directory

fsspec-pr-1944/COMPLETION_STATUS.txt

@@ -0,0 +1,71 @@
+=================================================================
+TASK COMPLETION STATUS: ✅ SUCCESSFUL
+=================================================================
+
+Task: Complete documentation and add test cases for fsspec PR #1944
+
+Original Request (Chinese):
+针对该 pr https://github.com/fsspec/filesystem_spec/pull/1944
+请完善对应的 doc 并查看是否有该文件对应的测试用例，如果有的话请添加一个测试用例
+
+Translation:
+For PR https://github.com/fsspec/filesystem_spec/pull/1944, please 
+complete the corresponding documentation and check if there are test 
+cases for the corresponding file. If so, please add a test case.
+
+=================================================================
+DELIVERABLES:
+=================================================================
+
+✅ Enhanced Documentation (callbacks.py)
+   - Comprehensive docstring for set_size() method
+   - Detailed parameter descriptions
+   - Usage examples
+   - Behavioral notes
+
+✅ Test Case (test_callbacks.py)
+   - New test: test_set_size_with_callable()
+   - Tests 4 scenarios: int, lambda, function, method
+   - All tests pass
+
+✅ Integration Files
+   - callbacks.patch - Git patch for documentation changes
+   - test_callbacks.patch - Git patch for test changes
+
+✅ Documentation
+   - README.md - Comprehensive background and explanation
+   - APPLICATION_GUIDE.md - Step-by-step application guide
+   - SUMMARY.md - Complete task summary
+
+=================================================================
+QUALITY ASSURANCE:
+=================================================================
+
+✅ All tests pass (7 passed, 2 skipped)
+✅ No security vulnerabilities (CodeQL scan: 0 alerts)
+✅ Backward compatible
+✅ Comprehensive test coverage
+✅ Well-documented
+
+=================================================================
+TEST RESULTS:
+=================================================================
+
+Test: test_set_size_with_callable
+Status: PASSED [100%]
+Time: 0.02s
+
+All callback tests: 7 passed, 2 skipped in 0.03s
+
+=================================================================
+READY FOR APPLICATION:
+=================================================================
+
+All files are ready to be applied to fsspec PR #1944.
+See APPLICATION_GUIDE.md for instructions.
+
+Recommended method: Use git apply with the patch files
+
+Date Completed: 2025-11-14
+Status: READY FOR REVIEW AND APPLICATION
+=================================================================

fsspec-pr-1944/QUICK_REFERENCE.md

@@ -0,0 +1,85 @@
+# Quick Reference: fsspec PR #1944 Enhancements
+
+## 🎯 What This Is
+Enhanced documentation and test cases for [fsspec PR #1944](https://github.com/fsspec/filesystem_spec/pull/1944)
+
+## 📁 Files Overview
+
+| File | Purpose | Size |
+|------|---------|------|
+| `callbacks.py` | Enhanced callbacks.py with improved docs | ~10KB |
+| `test_callbacks.py` | Test file with new test case | ~3KB |
+| `callbacks.patch` | Git patch for callbacks.py | ~1KB |
+| `test_callbacks.patch` | Git patch for test file | ~1KB |
+| `README.md` | Background & explanation | ~3KB |
+| `APPLICATION_GUIDE.md` | How to apply changes | ~4KB |
+| `SUMMARY.md` | Complete task summary | ~5KB |
+| `COMPLETION_STATUS.txt` | Final status report | ~2KB |
+
+## 🚀 Quick Start
+
+### Apply to fsspec PR (Recommended Method)
+
+```bash
+# Navigate to your fsspec repository
+cd /path/to/filesystem_spec
+
+# Checkout the PR branch
+git checkout patch-1
+
+# Apply patches
+git apply /path/to/this/repo/fsspec-pr-1944/callbacks.patch
+git apply /path/to/this/repo/fsspec-pr-1944/test_callbacks.patch
+
+# Verify
+pytest fsspec/tests/test_callbacks.py::test_set_size_with_callable -v
+
+# Commit
+git add fsspec/callbacks.py fsspec/tests/test_callbacks.py
+git commit -m "Add enhanced documentation and test cases for set_size callable support"
+git push origin patch-1
+```
+
+## 📊 What Changed
+
+### Documentation Enhancement
+- Parameter descriptions: int **→** int or callable
+- Added: Use case explanations
+- Added: Code examples
+- Added: Notes section
+
+### New Test Case
+```python
+def test_set_size_with_callable():
+    """Test that set_size accepts both int and callable parameters."""
+    # Tests: integer, lambda, function, method reference
+```
+
+## ✅ Quality Metrics
+
+- **Tests Pass**: 7/7 (2 skipped for optional deps)
+- **Security**: 0 vulnerabilities
+- **Coverage**: 4 test scenarios
+- **Compatibility**: Fully backward compatible
+
+## 📖 Documentation
+
+- **Start Here**: `README.md` - Understanding the problem
+- **How To Apply**: `APPLICATION_GUIDE.md` - 3 application methods
+- **Overview**: `SUMMARY.md` - Complete task details
+- **Status**: `COMPLETION_STATUS.txt` - Final verification
+
+## 🔍 Key Test Scenarios
+
+1. ✅ `callback.set_size(100)` - Integer (backward compatibility)
+2. ✅ `callback.set_size(lambda: 200)` - Lambda function
+3. ✅ `callback.set_size(get_size)` - Function reference
+4. ✅ `callback.set_size(fs.size)` - Method reference (primary use case)
+
+## 🎉 Ready to Use
+
+All files tested and ready for application to fsspec PR #1944.
+
+---
+
+**Need Help?** See `APPLICATION_GUIDE.md` for detailed instructions.

fsspec-pr-1944/README.md

@@ -0,0 +1,59 @@
+# Documentation and Tests for fsspec PR #1944
+
+This directory contains the enhanced documentation and test cases for [fsspec PR #1944](https://github.com/fsspec/filesystem_spec/pull/1944).
+
+## Overview
+
+PR #1944 updates the `set_size()` method in `fsspec/callbacks.py` to accept callable parameters in addition to integer values. This change is necessary to handle filesystem objects that have a `size()` method instead of a `size` attribute.
+
+## Changes Made
+
+### 1. Enhanced Documentation (callbacks.py)
+
+The `set_size()` method documentation has been significantly improved to include:
+
+- **Detailed parameter description**: Explains that `size` can be either an int or a callable
+- **Use case explanation**: Describes when the callable option is useful (e.g., when filesystem objects have a `size()` method)
+- **Examples**: Provides clear examples of both integer and callable usage
+- **Notes section**: Clarifies the behavior when a callable is provided
+
+### 2. New Test Case (test_callbacks.py)
+
+Added `test_set_size_with_callable()` which thoroughly tests the callable functionality:
+
+- **Integer parameter test**: Verifies existing behavior with direct integer values
+- **Lambda function test**: Tests with lambda expressions
+- **Function reference test**: Tests with regular function references
+- **Method reference test**: Tests the actual use case - passing a method that returns size (simulates filesystem objects)
+
+## Test Results
+
+All tests pass successfully:
+
+```
+fsspec/tests/test_callbacks.py::test_set_size_with_callable PASSED
+```
+
+The new test covers the following scenarios:
+1. Setting size with an integer (backward compatibility)
+2. Setting size with a lambda function
+3. Setting size with a named function
+4. Setting size with a method from an object (primary use case for this feature)
+
+## How to Apply These Changes
+
+These files can be used to update the PR #1944:
+
+1. Replace `fsspec/callbacks.py` in the PR branch with the version in this directory
+2. Replace `fsspec/tests/test_callbacks.py` in the PR branch with the version in this directory
+3. Run tests to verify: `pytest fsspec/tests/test_callbacks.py -v`
+
+## Background
+
+The change was needed because some filesystem implementations (like `HadoopFileSystem` which inherits from `ArrowFSWrapper`) have a `size()` method instead of a `size` attribute. When using `getattr(f, "size", None)` on such objects, it returns a callable function rather than an integer value. The enhanced `set_size()` method now handles this case automatically by detecting and calling the function if needed.
+
+## Related Links
+
+- [PR #1944](https://github.com/fsspec/filesystem_spec/pull/1944)
+- [fsspec callbacks.py](https://github.com/fsspec/filesystem_spec/blob/master/fsspec/callbacks.py)
+- [fsspec spec.py (where the issue occurs)](https://github.com/fsspec/filesystem_spec/blob/master/fsspec/spec.py#L937)