langchain_experimental.text_splitter
.SemanticChunker¶
- class langchain_experimental.text_splitter.SemanticChunker(embeddings: Embeddings, buffer_size: int = 1, add_start_index: bool = False, breakpoint_threshold_type: Literal['percentile', 'standard_deviation', 'interquartile'] = 'percentile', breakpoint_threshold_amount: Optional[float] = None, number_of_chunks: Optional[int] = None, sentence_split_regex: str = '(?<=[.?!])\\s+')[source]¶
Split the text based on semantic similarity.
Taken from Greg Kamradt’s wonderful notebook: https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb
All credits to him.
At a high level, this splits into sentences, then groups into groups of 3 sentences, and then merges one that are similar in the embedding space.
Methods
__init__
(embeddings[, buffer_size, ...])atransform_documents
(documents, **kwargs)Asynchronously transform a list of documents.
create_documents
(texts[, metadatas])Create documents from a list of texts.
split_documents
(documents)Split documents.
split_text
(text)transform_documents
(documents, **kwargs)Transform sequence of documents by splitting them.
- Parameters
embeddings (Embeddings) –
buffer_size (int) –
add_start_index (bool) –
breakpoint_threshold_type (Literal['percentile', 'standard_deviation', 'interquartile']) –
breakpoint_threshold_amount (Optional[float]) –
number_of_chunks (Optional[int]) –
sentence_split_regex (str) –
- __init__(embeddings: Embeddings, buffer_size: int = 1, add_start_index: bool = False, breakpoint_threshold_type: Literal['percentile', 'standard_deviation', 'interquartile'] = 'percentile', breakpoint_threshold_amount: Optional[float] = None, number_of_chunks: Optional[int] = None, sentence_split_regex: str = '(?<=[.?!])\\s+')[source]¶
- Parameters
embeddings (Embeddings) –
buffer_size (int) –
add_start_index (bool) –
breakpoint_threshold_type (Literal['percentile', 'standard_deviation', 'interquartile']) –
breakpoint_threshold_amount (Optional[float]) –
number_of_chunks (Optional[int]) –
sentence_split_regex (str) –
- async atransform_documents(documents: Sequence[Document], **kwargs: Any) Sequence[Document] ¶
Asynchronously transform a list of documents.
- create_documents(texts: List[str], metadatas: Optional[List[dict]] = None) List[Document] [source]¶
Create documents from a list of texts.
- Parameters
texts (List[str]) –
metadatas (Optional[List[dict]]) –
- Return type
List[Document]