feat: Add start and end line numbers to CodeChunker chunks

[AlanPonnachan]

This pull request resolves issue #326 by adding line number metadata to the Chunk objects generated by the CodeChunker.

Previously, chunks only had start_index and end_index which correspond to character offsets in the source text. This made it difficult to map a chunk back to its specific location (i.e., line numbers) in the original source code file, which is a common requirement for static analysis tools, code intelligence features, and displaying contextual search results.

This PR introduces the start_line and end_line attributes to each Chunk, providing this crucial context to the user.

Changes Implemented

1. Modified Chunk Dataclass (src/chonkie/types/base.py):

   • Added two new optional fields, start_line: Optional[int] and end_line: Optional[int], to the Chunk dataclass.
   • Updated the __repr__, from_dict, and to_dict methods to be aware of these new fields, ensuring they are correctly displayed and serialized.

2. Updated CodeChunker (src/chonkie/chunker/code.py):

   • The _create_chunks method now extracts the line number information from the tree-sitter nodes associated with each chunk.
   • It uses the node.start_point[0] and node.end_point[0] attributes to get the 0-indexed row numbers and converts them to 1-indexed line numbers for user-friendliness.
   • The chunk method was slightly refactored to ensure the node_groups data is available when creating chunks, while carefully preserving all existing logging statements.

3. Added New Test Case (tests/chunkers/test_code_chunker.py):

   • A new test, test_code_chunker_adds_line_numbers, has been added to verify the new functionality.
   • This test confirms that start_line and end_line are correctly populated, are valid integers, and accurately reflect the line numbers of the source code.

Fixes #326

[gemini-code-assist]

Summary of Changes

Hello @AlanPonnachan, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the CodeChunker functionality by introducing line number metadata to the Chunk objects. Previously, chunks only provided character offsets, which made it challenging to map them directly to specific lines in the original source code. By adding start_line and end_line attributes, this change resolves issue #326 and provides crucial contextual information, which is invaluable for static analysis tools, code intelligence features, and displaying more precise contextual search results.

Highlights

• Chunk Dataclass Enhancement: The Chunk dataclass in src/chonkie/types/base.py has been updated to include new optional fields: start_line and end_line. These fields provide 1-indexed line number information for each code chunk. The __repr__, from_dict, and to_dict methods were also adjusted to properly handle these new attributes.
• CodeChunker Line Number Extraction: The CodeChunker in src/chonkie/chunker/code.py now extracts line number information from tree-sitter nodes. The _create_chunks method populates the new start_line and end_line fields by converting the 0-indexed row numbers from tree-sitter to 1-indexed line numbers. The chunk method was refactored to ensure node_groups are available for this process.
• New Test Case for Line Numbers: A new test, test_code_chunker_adds_line_numbers, has been added to tests/chunkers/test_code_chunker.py. This test verifies that the CodeChunker correctly populates start_line and end_line for generated chunks, ensuring they are valid integers, 1-indexed, and accurately reflect the source code's line positions.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request successfully adds start_line and end_line numbers to the Chunk objects created by CodeChunker, which is a great enhancement for mapping chunks back to the source code. The changes in the Chunk dataclass and the CodeChunker logic to extract this information from tree-sitter nodes are well-implemented. The addition of a new test case to verify this functionality is also a good step.

My review focuses on two main points:

1. A potential UnboundLocalError in the refactored chunk method in src/chonkie/chunker/code.py due to a variable not being initialized before a try block.
2. Incorrect assertions in the new test test_code_chunker_adds_line_numbers in tests/chunkers/test_code_chunker.py, which seem to be based on a misunderstanding of the test fixture's content.

Addressing these points will make the implementation more robust and ensure the tests are correctly validating the new feature.