langchain_text_splitters.character.CharacterTextSplitter

class langchain_text_splitters.character.CharacterTextSplitter(separator: str = '\n\n', is_separator_regex: bool = False, **kwargs: Any)

Splitting text that looks at characters.

Create a new TextSplitter.

Methods

__init__([separator, is_separator_regex])	Create a new TextSplitter.
atransform_documents(documents, **kwargs)	Asynchronously transform a list of documents.
create_documents(texts[, metadatas])	Create documents from a list of texts.
from_huggingface_tokenizer(tokenizer, **kwargs)	Text splitter that uses HuggingFace tokenizer to count length.
from_tiktoken_encoder([encoding_name, ...])	Text splitter that uses tiktoken encoder to count length.
split_documents(documents)	Split documents.
split_text(text)	Split incoming text and return chunks.
transform_documents(documents, **kwargs)	Transform sequence of documents by splitting them.

Parameters

• separator (str) –

• is_separator_regex (bool) –

• kwargs (Any) –

__init__(separator: str = '\n\n', is_separator_regex: bool = False, **kwargs: Any) → None

Create a new TextSplitter.

Parameters

• separator (str) –

• is_separator_regex (bool) –

• kwargs (Any) –

Return type

None

async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]

Asynchronously transform a list of documents.

Parameters

• documents (Sequence[Document]) – A sequence of Documents to be transformed.

• kwargs (Any) –

Returns

A list of transformed Documents.

Return type

Sequence[Document]

create_documents(texts: List[str], metadatas: Optional[List[dict]] = None) → List[Document]

Create documents from a list of texts.

Parameters

• texts (List[str]) –

• metadatas (Optional[List[dict]]) –

Return type

List[Document]

classmethod from_huggingface_tokenizer(tokenizer: Any, **kwargs: Any) → TextSplitter

Text splitter that uses HuggingFace tokenizer to count length.

Parameters

• tokenizer (Any) –

• kwargs (Any) –

Return type

TextSplitter

classmethod from_tiktoken_encoder(encoding_name: str = 'gpt2', model_name: Optional[str] = None, allowed_special: Union[Literal['all'], AbstractSet[str]] = {}, disallowed_special: Union[Literal['all'], Collection[str]] = 'all', **kwargs: Any) → TS

Text splitter that uses tiktoken encoder to count length.

Parameters

• encoding_name (str) –

• model_name (Optional[str]) –

• allowed_special (Union[Literal['all'], ~typing.AbstractSet[str]]) –

• disallowed_special (Union[Literal['all'], ~typing.Collection[str]]) –

• kwargs (Any) –

Return type

TS

split_documents(documents: Iterable[Document]) → List[Document]

Split documents.

Parameters

documents (Iterable[Document]) –

Return type

List[Document]

split_text(text: str) → List[str]

Split incoming text and return chunks.

Parameters

text (str) –

Return type

List[str]

transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]

Transform sequence of documents by splitting them.

Parameters

• documents (Sequence[Document]) –

• kwargs (Any) –

Return type

Sequence[Document]

Examples using CharacterTextSplitter

• # Automatically restart kernel after installs so that your environment can access the new packages

• # if you plan to use bson serialization, install also:

• %pip install –upgrade –quiet surrealdb langchain langchain-community

• Create a bes instance and index docs.

• Create collection if running for the first time. If the collection

• Dependencies {#dependencies}

• DocumentDB connection string

• Download the content

• Establishing a connection to the database is facilitated through the singlestoredb Python connector.

• Get an OpenAI token: https://platform.openai.com/account/api-keys

• Helper function for printing docs

• Here we useimport getpass

• Here we want to make sure the answer is minimally relevant

• If using the default Docker installation, use this instantiation instead:

• Load the document and split it into chunks

• Map reduce example

• Metadata {#metadata}

• OR

• OpenAI

• Option 1: use an OpenAI account

• Pip install necessary package

• Pip install necessary package {#pip-install-necessary-package}

• Pip install necessary packages

• Refresh is required for server use

• Retrievers

• Run tests with shell:

• Set env var OPENAI_API_KEY or load from a .env file

• Set up the OpenAI Environment Variables

• Setup {#setup}

• Swapping the `ByteStore` {#swapping-the-bytestore}

• This is a long document we can split up.

• This will only get documents for Ankush

• To make the caching really obvious, lets use a slower model.

• Uncomment this to install psychicapi if you don’t already have it installed

• Use Meilisearch vector store to store texts & associated embeddings as vector

• VDMS

• Vector stores

• Wait until the cluster is ready for use.

• You need to install sqlite-vss as a dependency.

• add texts

• adding_memory_chain_multiple_inputs.md

• analyticdb.md

• awadb.md

• baiduvectordb.md

• clickhouse.md

• conversational_retrieval_agents.md

• cosine: distance metric

• create cluster and add texts

• databricks_vector_search.md

• default metric is angular

• docs[0].metadata[“id”] == “id:testapp:testapp::32”

• document_comparison_toolkit.md

• drop first if index already exists

• duckdb.md

• ecloud_vector_search.md

• epsilla.md

• for example

• from langchain_community.embeddings.openai import OpenAIEmbeddings

• hologres.md

• import

• indexing.md

• initialize marqo

• install package

• lancedb.md

• openai

• or install latest:

• or shorter

• os.environ[“LANGCHAIN_TRACING_V2”] = “true”

• output length: 4

• pinecone.md

• qdrant.md

• replace

• scann.md

• tigris.md

• tiledb.md

• typesense.md

• use directly a `where_str` to delete

• usearch.md

• vectorstore.md

• with pip

• xata.md