langchain_community.document_loaders.url.UnstructuredURLLoader¶

class langchain_community.document_loaders.url.UnstructuredURLLoader(urls: List[str], continue_on_failure: bool = True, mode: str = 'single', show_progress_bar: bool = False, **unstructured_kwargs: Any)[source]¶

Load files from remote URLs using Unstructured.

Use the unstructured partition function to detect the MIME type and route the file to the appropriate partitioner.

You can run the loader in one of two modes: “single” and “elements”. If you use “single” mode, the document will be returned as a single langchain Document object. If you use “elements” mode, the unstructured library will split the document into elements such as Title and NarrativeText. You can pass in additional unstructured kwargs after mode to apply different unstructured settings.

Examples

from langchain_community.document_loaders import UnstructuredURLLoader

loader = UnstructuredURLLoader(

urls=[“<url-1>”, “<url-2>”], mode=”elements”, strategy=”fast”,

) docs = loader.load()

References

https://unstructured-io.github.io/unstructured/bricks.html#partition

Initialize with file path.

Methods

__init__(urls[, continue_on_failure, mode, ...])

Initialize with file path.

alazy_load()

A lazy loader for Documents.

aload()

Load data into Document objects.

lazy_load()

A lazy loader for Documents.

load()

Load file.

load_and_split([text_splitter])

Load Documents and split into chunks.
Parameters

  • urls (List[str]) –

  • continue_on_failure (bool) –

  • mode (str) –

  • show_progress_bar (bool) –

  • unstructured_kwargs (Any) –

__init__(urls: List[str], continue_on_failure: bool = True, mode: str = 'single', show_progress_bar: bool = False, **unstructured_kwargs: Any)[source]¶

Initialize with file path.

Parameters

  • urls (List[str]) –

  • continue_on_failure (bool) –

  • mode (str) –

  • show_progress_bar (bool) –

  • unstructured_kwargs (Any) –

async alazy_load() → AsyncIterator[Document]¶

A lazy loader for Documents.

Return type

AsyncIterator[Document]

async aload() → List[Document]¶

Load data into Document objects.

Return type

List[Document]

lazy_load() → Iterator[Document]¶

A lazy loader for Documents.

Return type

Iterator[Document]

load() → List[Document][source]¶

Load file.

Return type

List[Document]

load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶

Load Documents and split into chunks. Chunks are returned as Documents.

Do not override this method. It should be considered to be deprecated!

Parameters

text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

Return type

List[Document]

Examples using UnstructuredURLLoader¶