Customized text chunking

[sid8491]

How do I write my own logic for text chunking?
Which classes do I need to extend, and how do I return the final chunking output?

Any examples/documentation would be appreciated.

[logan-markewich]

You can create your own chunking logic by extending the text splitter.

Then the text splitter gets passed into the node parser, which also gets passed into service context

Text Splitter:
https://github.com/jerryjliu/llama_index/blob/main/llama_index/langchain_helpers/text_splitter.py

Node Parser:
https://github.com/jerryjliu/llama_index/blob/main/llama_index/node_parser/simple.py

[sid8491]

can you give example of how to do it?

You can create your own chunking logic by extending the text splitter.

Then the text splitter gets passed into the node parser, which also gets passed into service context

Text Splitter: https://github.com/jerryjliu/llama_index/blob/main/llama_index/langchain_helpers/text_splitter.py

Node Parser: https://github.com/jerryjliu/llama_index/blob/main/llama_index/node_parser/simple.py

can you give example of how to do it?

[logan-markewich]

The best example is definitely the source code.

Not sure what you have in mind though, it might be possible to achieve what you want with other methods

[dreamshit]

I don't know if it's what you want, that's how I achieved it:

class TxtParser(BaseParser):

    def _init_parser(self) -> Dict:
        return {}

    def parse_file(self, file: Path, errors: str = "ignore") -> str:
        pass

METIS_FILE_EXTRACTOR: Dict[str, BaseParser] = {
    ".csv": CSVParser(concat_rows=False),
    ".txt": TxtParser(),
}

documents = SimpleDirectoryReader(file_extractor=METIS_FILE_EXTRACTOR).load_data()