langchain_experimental.agents.agent_toolkits.spark.base.create_spark_dataframe_agent

langchain_experimental.agents.agent_toolkits.spark.base.create_spark_dataframe_agent(llm: BaseLLM, df: Any, callback_manager: Optional[BaseCallbackManager] = None, prefix: str = '\nYou are working with a spark dataframe in Python. The name of the dataframe is `df`.\nYou should use the tools below to answer the question posed of you:', suffix: str = '\nThis is the result of `print(df.first())`:\n{df}\n\nBegin!\nQuestion: {input}\n{agent_scratchpad}', input_variables: Optional[List[str]] = None, verbose: bool = False, return_intermediate_steps: bool = False, max_iterations: Optional[int] = 15, max_execution_time: Optional[float] = None, early_stopping_method: str = 'force', agent_executor_kwargs: Optional[Dict[str, Any]] = None, **kwargs: Any) → AgentExecutor

Construct a Spark agent from an LLM and dataframe.

Parameters

• llm (BaseLLM) –

• df (Any) –

• callback_manager (Optional[BaseCallbackManager]) –

• prefix (str) –

• suffix (str) –

• input_variables (Optional[List[str]]) –

• verbose (bool) –

• return_intermediate_steps (bool) –

• max_iterations (Optional[int]) –

• max_execution_time (Optional[float]) –

• early_stopping_method (str) –

• agent_executor_kwargs (Optional[Dict[str, Any]]) –

• kwargs (Any) –

Return type

AgentExecutor

Examples using create_spark_dataframe_agent

• in apache-spark root directory. (tested here with “spark-3.4.0-bin-hadoop3 and later”)