chunklet.base_chunker

Base Chunker Abstract Class

Defines the interface for chunkers.

Classes:

• BaseChunker –

  Abstract base class for chunkers.

BaseChunker

BaseChunker(verbose: bool = False)

Bases: ABC

Abstract base class for chunkers.

Defines the standard interface for chunking content into units.

Methods:

• batch_chunk –

  Process multiple items in parallel.

• chunk –

  Extract chunks.

• log_info –

  Log an info message if verbose is enabled.

Source code in src/chunklet/base_chunker.py

def __init__(self, verbose: bool = False):
    self.verbose = verbose

batch_chunk abstractmethod

batch_chunk(*args, **kwargs) -> Generator[Box, None, None]

Process multiple items in parallel.

Yields:

• Box ( Box ) –

  Box object, representing a chunk with its content and metadata.

Source code in src/chunklet/base_chunker.py

@abstractmethod
def batch_chunk(self, *args, **kwargs) -> Generator[Box, None, None]:
    """
    Process multiple items in parallel.

    Yields:
        Box: `Box` object, representing a chunk with its content and metadata.
    """
    pass

chunk abstractmethod

chunk(*args, **kwargs) -> list[Box]

Extract chunks.

Returns:

• list[Box] –

  list[Box]: List of chunks with content and metadata.

Source code in src/chunklet/base_chunker.py

@abstractmethod
def chunk(self, *args, **kwargs) -> list[Box]:
    """
    Extract chunks.

    Returns:
        list[Box]: List of chunks with content and metadata.
    """
    pass

log_info

log_info(*args, **kwargs) -> None

Log an info message if verbose is enabled.

Source code in src/chunklet/base_chunker.py

def log_info(self, *args, **kwargs) -> None:
    """Log an info message if verbose is enabled."""
    if self.verbose:
        logger.info(*args, **kwargs)