langchain_community.document_loaders.confluence.ConfluenceLoader¶

class langchain_community.document_loaders.confluence.ConfluenceLoader(url: str, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None, cloud: Optional[bool] = True, number_of_retries: Optional[int] = 3, min_retry_seconds: Optional[int] = 2, max_retry_seconds: Optional[int] = 10, confluence_kwargs: Optional[dict] = None, *, space_key: Optional[str] = None, page_ids: Optional[List[str]] = None, label: Optional[str] = None, cql: Optional[str] = None, include_restricted_content: bool = False, include_archived_content: bool = False, include_attachments: bool = False, include_comments: bool = False, content_format: ContentFormat = ContentFormat.STORAGE, limit: Optional[int] = 50, max_pages: Optional[int] = 1000, ocr_languages: Optional[str] = None, keep_markdown_format: bool = False, keep_newlines: bool = False)[source]¶

Load Confluence pages.

Port of https://llamahub.ai/l/confluence This currently supports username/api_key, Oauth2 login or personal access token authentication.

Specify a list page_ids and/or space_key to load in the corresponding pages into Document objects, if both are specified the union of both sets will be returned.

You can also specify a boolean include_attachments to include attachments, this is set to False by default, if set to True all attachments will be downloaded and ConfluenceReader will extract the text from the attachments and add it to the Document object. Currently supported attachment types are: PDF, PNG, JPEG/JPG, SVG, Word and Excel.

Confluence API supports difference format of page content. The storage format is the raw XML representation for storage. The view format is the HTML representation for viewing with macros are rendered as though it is viewed by users. You can pass a enum content_format argument to specify the content format, this is set to ContentFormat.STORAGE by default, the supported values are: ContentFormat.EDITOR, ContentFormat.EXPORT_VIEW, ContentFormat.ANONYMOUS_EXPORT_VIEW, ContentFormat.STORAGE, and ContentFormat.VIEW.

Hint: space_key and page_id can both be found in the URL of a page in Confluence - https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>

Example

from langchain_community.document_loaders import ConfluenceLoader

loader = ConfluenceLoader(
    url="https://yoursite.atlassian.com/wiki",
    username="me",
    api_key="12345",
    space_key="SPACE",
    limit=50,
)
documents = loader.load()

# Server on perm
loader = ConfluenceLoader(
    url="https://confluence.yoursite.com/",
    username="me",
    api_key="your_password",
    cloud=False,
    space_key="SPACE",
    limit=50,
)
documents = loader.load()
Parameters

  • url (str) – _description_

  • api_key (str, optional) – _description_, defaults to None

  • username (str, optional) – _description_, defaults to None

  • oauth2 (dict, optional) – _description_, defaults to {}

  • token (str, optional) – _description_, defaults to None

  • cloud (bool, optional) – _description_, defaults to True

  • number_of_retries (Optional[int], optional) – How many times to retry, defaults to 3

  • min_retry_seconds (Optional[int], optional) – defaults to 2

  • max_retry_seconds (Optional[int], optional) – defaults to 10

  • confluence_kwargs (dict, optional) – additional kwargs to initialize confluence with

  • space_key (Optional[str], optional) – Space key retrieved from a confluence URL, defaults to None

  • page_ids (Optional[List[str]], optional) – List of specific page IDs to load, defaults to None

  • label (Optional[str], optional) – Get all pages with this label, defaults to None

  • cql (Optional[str], optional) – CQL Expression, defaults to None

  • include_restricted_content (bool, optional) – defaults to False

  • include_archived_content (bool, optional) – Whether to include archived content, defaults to False

  • include_attachments (bool, optional) – defaults to False

  • include_comments (bool, optional) – defaults to False

  • content_format (ContentFormat) – Specify content format, defaults to ContentFormat.STORAGE, the supported values are: ContentFormat.EDITOR, ContentFormat.EXPORT_VIEW, ContentFormat.ANONYMOUS_EXPORT_VIEW, ContentFormat.STORAGE, and ContentFormat.VIEW.

  • limit (int, optional) – Maximum number of pages to retrieve per request, defaults to 50

  • max_pages (int, optional) – Maximum number of pages to retrieve in total, defaults 1000

  • ocr_languages (str, optional) – The languages to use for the Tesseract agent. To use a language, you’ll first need to install the appropriate Tesseract language pack.

  • keep_markdown_format (bool) – Whether to keep the markdown format, defaults to False

  • keep_newlines (bool) – Whether to keep the newlines format, defaults to False

  • session (Optional[Session]) –

Raises

  • ValueError – Errors while validating input

  • ImportError – Required dependencies not installed.

Methods

__init__(url[, api_key, username, session, ...])

alazy_load()

A lazy loader for Documents.

aload()

Load data into Document objects.

is_public_page(page)

Check if a page is publicly accessible.

lazy_load()

A lazy loader for Documents.

load(**kwargs)

Load data into Document objects.

load_and_split([text_splitter])

Load Documents and split into chunks.

paginate_request(retrieval_method, **kwargs)

Paginate the various methods to retrieve groups of pages.

process_attachment(page_id[, ocr_languages])

process_doc(link)

process_image(link[, ocr_languages])

process_page(page, include_attachments, ...)

process_pages(pages, ...[, ocr_languages, ...])

Process a list of pages into a list of documents.

process_pdf(link[, ocr_languages])

process_svg(link[, ocr_languages])

process_xls(link)

validate_init_args([url, api_key, username, ...])

Validates proper combinations of init arguments
__init__(url: str, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None, cloud: Optional[bool] = True, number_of_retries: Optional[int] = 3, min_retry_seconds: Optional[int] = 2, max_retry_seconds: Optional[int] = 10, confluence_kwargs: Optional[dict] = None, *, space_key: Optional[str] = None, page_ids: Optional[List[str]] = None, label: Optional[str] = None, cql: Optional[str] = None, include_restricted_content: bool = False, include_archived_content: bool = False, include_attachments: bool = False, include_comments: bool = False, content_format: ContentFormat = ContentFormat.STORAGE, limit: Optional[int] = 50, max_pages: Optional[int] = 1000, ocr_languages: Optional[str] = None, keep_markdown_format: bool = False, keep_newlines: bool = False)[source]¶
Parameters

  • url (str) –

  • api_key (Optional[str]) –

  • username (Optional[str]) –

  • session (Optional[Session]) –

  • oauth2 (Optional[dict]) –

  • token (Optional[str]) –

  • cloud (Optional[bool]) –

  • number_of_retries (Optional[int]) –

  • min_retry_seconds (Optional[int]) –

  • max_retry_seconds (Optional[int]) –

  • confluence_kwargs (Optional[dict]) –

  • space_key (Optional[str]) –

  • page_ids (Optional[List[str]]) –

  • label (Optional[str]) –

  • cql (Optional[str]) –

  • include_restricted_content (bool) –

  • include_archived_content (bool) –

  • include_attachments (bool) –

  • include_comments (bool) –

  • content_format (ContentFormat) –

  • limit (Optional[int]) –

  • max_pages (Optional[int]) –

  • ocr_languages (Optional[str]) –

  • keep_markdown_format (bool) –

  • keep_newlines (bool) –

async alazy_load() → AsyncIterator[Document]¶

A lazy loader for Documents.

Return type

AsyncIterator[Document]

async aload() → List[Document]¶

Load data into Document objects.

Return type

List[Document]

is_public_page(page: dict) → bool[source]¶

Check if a page is publicly accessible.

Parameters

page (dict) –

Return type

bool

lazy_load() → Iterator[Document][source]¶

A lazy loader for Documents.

Return type

Iterator[Document]

load(**kwargs: Any) → List[Document][source]¶

Load data into Document objects.

Parameters

kwargs (Any) –

Return type

List[Document]

load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶

Load Documents and split into chunks. Chunks are returned as Documents.

Do not override this method. It should be considered to be deprecated!

Parameters

text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

Return type

List[Document]

paginate_request(retrieval_method: Callable, **kwargs: Any) → List[source]¶

Paginate the various methods to retrieve groups of pages.

Unfortunately, due to page size, sometimes the Confluence API doesn’t match the limit value. If limit is >100 confluence seems to cap the response to 100. Also, due to the Atlassian Python package, we don’t get the “next” values from the “_links” key because they only return the value from the result key. So here, the pagination starts from 0 and goes until the max_pages, getting the limit number of pages with each request. We have to manually check if there are more docs based on the length of the returned list of pages, rather than just checking for the presence of a next key in the response like this page would have you do: https://developer.atlassian.com/server/confluence/pagination-in-the-rest-api/

Parameters

  • retrieval_method (callable) – Function used to retrieve docs

  • kwargs (Any) –

Returns

List of documents

Return type

List

process_attachment(page_id: str, ocr_languages: Optional[str] = None) → List[str][source]¶
Parameters

  • page_id (str) –

  • ocr_languages (Optional[str]) –

Return type

List[str]

process_doc(link: str) → str[source]¶
Parameters

link (str) –

Return type

str

process_image(link: str, ocr_languages: Optional[str] = None) → str[source]¶
Parameters

  • link (str) –

  • ocr_languages (Optional[str]) –

Return type

str

process_page(page: dict, include_attachments: bool, include_comments: bool, content_format: ContentFormat, ocr_languages: Optional[str] = None, keep_markdown_format: Optional[bool] = False, keep_newlines: bool = False) → Document[source]¶
Parameters

  • page (dict) –

  • include_attachments (bool) –

  • include_comments (bool) –

  • content_format (ContentFormat) –

  • ocr_languages (Optional[str]) –

  • keep_markdown_format (Optional[bool]) –

  • keep_newlines (bool) –

Return type

Document

process_pages(pages: List[dict], include_restricted_content: bool, include_attachments: bool, include_comments: bool, content_format: ContentFormat, ocr_languages: Optional[str] = None, keep_markdown_format: Optional[bool] = False, keep_newlines: bool = False) → Iterator[Document][source]¶

Process a list of pages into a list of documents.

Parameters

  • pages (List[dict]) –

  • include_restricted_content (bool) –

  • include_attachments (bool) –

  • include_comments (bool) –

  • content_format (ContentFormat) –

  • ocr_languages (Optional[str]) –

  • keep_markdown_format (Optional[bool]) –

  • keep_newlines (bool) –

Return type

Iterator[Document]

process_pdf(link: str, ocr_languages: Optional[str] = None) → str[source]¶
Parameters

  • link (str) –

  • ocr_languages (Optional[str]) –

Return type

str

process_svg(link: str, ocr_languages: Optional[str] = None) → str[source]¶
Parameters

  • link (str) –

  • ocr_languages (Optional[str]) –

Return type

str

process_xls(link: str) → str[source]¶
Parameters

link (str) –

Return type

str

static validate_init_args(url: Optional[str] = None, api_key: Optional[str] = None, username: Optional[str] = None, session: Optional[Session] = None, oauth2: Optional[dict] = None, token: Optional[str] = None) → Optional[List][source]¶

Validates proper combinations of init arguments

Parameters

  • url (Optional[str]) –

  • api_key (Optional[str]) –

  • username (Optional[str]) –

  • session (Optional[Session]) –

  • oauth2 (Optional[dict]) –

  • token (Optional[str]) –

Return type

Optional[List]

Examples using ConfluenceLoader¶