langchain_community.document_loaders.json_loader
.JSONLoader¶
- class langchain_community.document_loaders.json_loader.JSONLoader(file_path: Union[str, Path], jq_schema: str, content_key: Optional[str] = None, is_content_key_jq_parsable: Optional[bool] = False, metadata_func: Optional[Callable[[Dict, Dict], Dict]] = None, text_content: bool = True, json_lines: bool = False)[source]¶
Load a JSON file using a jq schema.
- Setup:
pip install -U jq
- Instantiate:
from langchain_community.document_loaders import JSONLoader import json from pathlib import Path file_path='./sample_quiz.json' data = json.loads(Path(file_path).read_text()) loader = JSONLoader( file_path=file_path, jq_schema='.quiz', text_content=False)
- Load:
docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata)
{"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1}
- Async load:
docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata)
{"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quizg .json', 'seq_num': 1}
- Lazy load:
docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata)
{"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1}
Initialize the JSONLoader.
- Parameters
file_path (Union[str, Path]) – The path to the JSON or JSON Lines file.
jq_schema (str) – The jq schema to use to extract the data or text from the JSON.
content_key (str) – The key to use to extract the content from the JSON if the jq_schema results to a list of objects (dict). If is_content_key_jq_parsable is True, this has to be a jq compatible schema. If is_content_key_jq_parsable is False, this should be a simple string key.
is_content_key_jq_parsable (bool) – A flag to determine if content_key is parsable by jq or not. If True, content_key is treated as a jq schema and compiled accordingly. If False or if content_key is None, content_key is used as a simple string. Default is False.
metadata_func (Callable[Dict, Dict]) – A function that takes in the JSON object extracted by the jq_schema and the default metadata and returns a dict of the updated metadata.
text_content (bool) – Boolean flag to indicate whether the content is in string format, default to True.
json_lines (bool) – Boolean flag to indicate whether the input is in JSON Lines format.
Methods
__init__
(file_path, jq_schema[, ...])Initialize the JSONLoader.
A lazy loader for Documents.
aload
()Load data into Document objects.
Load and return documents from the JSON file.
load
()Load data into Document objects.
load_and_split
([text_splitter])Load Documents and split into chunks.
- __init__(file_path: Union[str, Path], jq_schema: str, content_key: Optional[str] = None, is_content_key_jq_parsable: Optional[bool] = False, metadata_func: Optional[Callable[[Dict, Dict], Dict]] = None, text_content: bool = True, json_lines: bool = False)[source]¶
Initialize the JSONLoader.
- Parameters
file_path (Union[str, Path]) – The path to the JSON or JSON Lines file.
jq_schema (str) – The jq schema to use to extract the data or text from the JSON.
content_key (str) – The key to use to extract the content from the JSON if the jq_schema results to a list of objects (dict). If is_content_key_jq_parsable is True, this has to be a jq compatible schema. If is_content_key_jq_parsable is False, this should be a simple string key.
is_content_key_jq_parsable (bool) – A flag to determine if content_key is parsable by jq or not. If True, content_key is treated as a jq schema and compiled accordingly. If False or if content_key is None, content_key is used as a simple string. Default is False.
metadata_func (Callable[Dict, Dict]) – A function that takes in the JSON object extracted by the jq_schema and the default metadata and returns a dict of the updated metadata.
text_content (bool) – Boolean flag to indicate whether the content is in string format, default to True.
json_lines (bool) – Boolean flag to indicate whether the input is in JSON Lines format.
- async alazy_load() AsyncIterator[Document] ¶
A lazy loader for Documents.
- Return type
AsyncIterator[Document]
- lazy_load() Iterator[Document] [source]¶
Load and return documents from the JSON file.
- Return type
Iterator[Document]
- load_and_split(text_splitter: Optional[TextSplitter] = None) List[Document] ¶
Load Documents and split into chunks. Chunks are returned as Documents.
Do not override this method. It should be considered to be deprecated!
- Parameters
text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.
- Returns
List of Documents.
- Return type
List[Document]