langchain.document_loaders.docusaurus.DocusaurusLoader

class langchain.document_loaders.docusaurus.DocusaurusLoader(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)[source]

Loader that leverages the SitemapLoader to loop through the generated pages of a Docusaurus Documentation website and extracts the content by looking for specific HTML tags. By default, the parser searches for the main content of the Docusaurus page, which is normally the <article>. You also have the option to define your own custom HTML tags by providing them as a list, for example: [“div”, “.main”, “a”].

Initialize DocusaurusLoader :param url: The base URL of the Docusaurus website. :param custom_html_tags: Optional custom html tags to extract content from pages. :param kwargs: Additional args to extend the underlying SitemapLoader, for example:

filter_urls, blocksize, meta_function, is_local, continue_on_failure

Attributes

web_path

Methods

__init__(url[, custom_html_tags])

Initialize DocusaurusLoader :param url: The base URL of the Docusaurus website. :param custom_html_tags: Optional custom html tags to extract content from pages. :param kwargs: Additional args to extend the underlying SitemapLoader, for example: filter_urls, blocksize, meta_function, is_local, continue_on_failure.

aload()

Load text from the urls in web_path async into Documents.

fetch_all(urls)

Fetch all urls concurrently with rate limiting.

lazy_load()

Lazy load text from the url(s) in web_path.

load()

Load sitemap.

load_and_split([text_splitter])

Load Documents and split into chunks.

parse_sitemap(soup)

Parse sitemap xml and load into a list of dicts.

scrape([parser])

Scrape data from webpage and return it in BeautifulSoup format.

scrape_all(urls[, parser])

Fetch all urls, then return soups for all results.

__init__(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)[source]

Initialize DocusaurusLoader :param url: The base URL of the Docusaurus website. :param custom_html_tags: Optional custom html tags to extract content from pages. :param kwargs: Additional args to extend the underlying SitemapLoader, for example:

filter_urls, blocksize, meta_function, is_local, continue_on_failure

aload() List[Document]

Load text from the urls in web_path async into Documents.

async fetch_all(urls: List[str]) Any

Fetch all urls concurrently with rate limiting.

lazy_load() Iterator[Document]

Lazy load text from the url(s) in web_path.

load() List[Document]

Load sitemap.

load_and_split(text_splitter: Optional[TextSplitter] = None) List[Document]

Load Documents and split into chunks. Chunks are returned as Documents.

Parameters

text_splitter – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

parse_sitemap(soup: Any) List[dict]

Parse sitemap xml and load into a list of dicts.

Parameters

soup – BeautifulSoup object.

Returns

List of dicts.

scrape(parser: Optional[str] = None) Any

Scrape data from webpage and return it in BeautifulSoup format.

scrape_all(urls: List[str], parser: Optional[str] = None) List[Any]

Fetch all urls, then return soups for all results.