ghe6(dZddlZddlmZddlmZmZmZmZm Z ddl m Z ddl m Z mZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlm Z m!Z!ddl"m#Z#m$Z$GddeZ%GddeZ&Gdde#e$Z'dS)zA chain for evaluating ReAct style agents. This chain is used to evaluate ReAct style agents by reasoning about the sequence of actions taken and their outcomes. It uses a language model chain (LLMChain) to generate the reasoning and scores. N)Sequence)AnyOptional TypedDictUnioncast) AgentAction)AsyncCallbackManagerForChainRunCallbackManagerForChainRun Callbacks)OutputParserException)BaseLanguageModel) BaseChatModel)BaseOutputParser)BaseTool) ConfigDictField)LLMChain)EVAL_CHAT_PROMPTTOOL_FREE_EVAL_CHAT_PROMPT)AgentTrajectoryEvaluator LLMEvalChainc*eZdZUdZeed< eed<dS)TrajectoryEvalzBA named tuple containing the score and reasoning for a trajectory.score reasoningN)__name__ __module__ __qualname____doc__float__annotations__strn/var/www/FlaskApp/flask-venv/lib/python3.11/site-packages/langchain/evaluation/agents/trajectory_eval_chain.pyrr's-LL LLL?NNN&&r%rc>eZdZdZedefdZdedefdZdS)TrajectoryOutputParserzTrajectory output parser.returncdS)Nagent_trajectoryr$selfs r&_typezTrajectoryOutputParser._type3s!!r%textcd|vrtd||dd\}}||}}tjd|}|d|dvrtd |t |d}d|cxkrd ksntd ||dz d z }t|| S)aParse the output text and extract the score and reasoning. Args: text (str): The output text to parse. Returns: TrajectoryEval: A named tuple containing the normalized score and reasoning. Raises: OutputParserException: If the score is not found in the output text or if the LLM's score is not a digit in the range 1-5. zScore:z+Could not find score in model eval output: zScore: )maxsplitz (\d+(\.\d+)?)N.z0Score is not an integer digit in the range 1-5: z'Score is not a digit in the range 1-5: )rr)r splitstripresearchgroupintr)r-r/r score_str_scorernormalized_scores r&parsezTrajectoryOutputParser.parse7s  4  'DdDD  $zz)az@@ 9(00)//2C2C9 +Y77 >SFLLOO33'I4II FLLOO$$EQ'@$@@ "AI?$4 JJJJr%N) rrrr propertyr#r.rr?r$r%r&r(r(0si## "s"""X"(K#(K.(K(K(K(K(K(Kr%r(ceZdZUdZdZeeeed< e ed< e e Z e ed< dZ eed< ed Zed efd Zed efd Zedeeeeeeffd efdZedeed efdZe d&dedeeedee ded df dZed eefdZ ed eefdZ!dee"eefefd e"eefffd Z# d'de"eefdee$d e"eeffdZ% d'de"eefdee&d e"eeffdZ'dddddddededeeeefdeed e(d!eeed"ee"eefd#eded e"fd$Z)dddddddededeeeefdeed e(d!eeed"ee"eefd#eded e"fd%Z*xZ+S)(TrajectoryEvalChaina-A chain for evaluating ReAct style agents. This chain is used to evaluate ReAct style agents by reasoning about the sequence of actions taken and their outcomes. Based on the paper "ReAct: Synergizing Reasoning and Acting in Language Models" (https://arxiv.org/abs/2210.03629) Example: .. code-block:: python from langchain.agents import AgentType, initialize_agent from langchain_community.chat_models import ChatOpenAI from langchain.evaluation import TrajectoryEvalChain from langchain.tools import tool @tool def geography_answers(country: str, question: str) -> str: """Very helpful answers to geography questions.""" return f"{country}? IDK - We may never know {question}." llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0) agent = initialize_agent( tools=[geography_answers], llm=llm, agent=AgentType.OPENAI_FUNCTIONS, return_intermediate_steps=True, ) question = "How many dwell in the largest minor region in Argentina?" response = agent(question) eval_chain = TrajectoryEvalChain.from_llm( llm=llm, agent_tools=[geography_answers], return_reasoning=True ) result = eval_chain.evaluate_agent_trajectory( input=question, agent_trajectory=response["intermediate_steps"], prediction=response["output"], reference="Paris", ) print(result["score"]) # noqa: T201 # 0 N agent_tools eval_chain)default_factory output_parserFreturn_reasoningignore)extrar)cdS)z2Whether this evaluator requires a reference label.Fr$r,s r&requires_referencez&TrajectoryEvalChain.requires_references ur%cx|jdSddt|jdDS)zwGet the description of the agent tools. Returns: str: The description of the agent tools. N c>g|]\}}d|d|jd|jS)zTool z: z Description: )name description).0itools r& z:TrajectoryEvalChain._tools_description..sX   At#!##ty####   r%r1)rCjoin enumerater,s r&_tools_descriptionz&TrajectoryEvalChain._tools_descriptionsS   #2{{   ))91==      r%stepsct|tr|Sddt|dDS)zGet the agent trajectory as a formatted string. Args: steps (Union[str, List[Tuple[AgentAction, str]]]): The agent trajectory. Returns: str: The formatted agent trajectory. rNc Jg|] \}\}}d|d|jd|jd|!S)zStep z : Tool used: z Tool input: z Tool output: )rT tool_input)rRrSactionoutputs r&rUz.sm    (A' ! K     r%r1) isinstancer#rVrW)rYs r&get_agent_trajectoryz(TrajectoryEvalChain.get_agent_trajectorysY eS ! ! L{{   ,5UA+>+>       r% referencec|sdSd|dS)zFormat the reference text. Args: reference (str): The reference text. Returns: str: The formatted reference text. rMzX The following is the expected answer. Use this to measure correctness: [GROUND_TRUTH] z [END_GROUND_TRUTH] r$)ras r&_format_referencez%TrajectoryEvalChain._format_references/ 2  r%llmkwargsc t|tstd|rt}nt}t ||}|d|||p t d|S)aCreate a TrajectoryEvalChain object from a language model chain. Args: llm (BaseChatModel): The language model chain. agent_tools (Optional[Sequence[BaseTool]]): A list of tools available to the agent. output_parser (Optional[TrajectoryOutputParser]): The output parser used to parse the chain output into a score. Returns: TrajectoryEvalChain: The TrajectoryEvalChain object. z9Only chat models supported by the current trajectory eval)rdprompt)rCrDrFr$)r_rNotImplementedErrorrrrr()clsrdrCrFrergrDs r&from_llmzTrajectoryEvalChain.from_llms&#}-- %K   0%FF/F#f555 s #!'C+A+C+C      r%c gdS)zcGet the input keys for the chain. Returns: List[str]: The input keys. questionr+answerrar$r,s r& input_keyszTrajectoryEvalChain.input_keyssGFFFr%c ddgS)zeGet the output keys for the chain. Returns: List[str]: The output keys. rrr$r,s r& output_keyszTrajectoryEvalChain.output_keyss%%r%inputsc||d|d<t|S)zValidate and prep inputs.ra)rcgetsuper prep_inputs)r-rr __class__s r&rvzTrajectoryEvalChain.prep_inputss?"44VZZ 5L5LMM{ww""6***r% run_managerc i|}|jr |j|d<|ptj}|j||}tt|j |S)IRun the chain and generate the output. Args: inputs (Dict[str, str]): The input values for the chain. run_manager (Optional[CallbackManagerForChainRun]): The callback manager for the chain run. Returns: Dict[str, Any]: The output values of the chain. tool_descriptions callbacks) rCrXr get_noop_managerrDrun get_childrdictrFr?r-rrrx chain_input _run_manager raw_outputs r&_callzTrajectoryEvalChain._calls!j   G/3/FK+ ,"S&@&Q&S&S _(( <#9#9#;#;)  D$,22:>>???r%cKi|}|jr |j|d<|ptj}|j||d{V}tt|j |S)rzr{r|N) rCrXr r~rDarunrrrrFr?rs r&_acallzTrajectoryEvalChain._acall5s!j   G/3/FK+ ,"X&E&V&X&X ?// <#9#9#;#;0        D$,22:>>???r%)rar}tagsmetadatainclude_run_info predictioninputr+r}rrrc l|||||d} || ||||dS)a7Evaluate a trajectory. Args: prediction (str): The final predicted response. input (str): The input to the agent. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. reference (Optional[str]): The reference answer. callbacks (Callbacks): Callbacks to use for this chain run. Returns: dict: The evaluation result, which includes the score and optionally the reasoning for reaching that. rlTrrr}rrrreturn_only_outputs)r`__call__ r-rrr+rar}rrrrerrs r&_evaluate_agent_trajectoryz.TrajectoryEvalChain._evaluate_agent_trajectoryMsZ8 $ 9 9:J K K "    }}- $    r%c|K|||||d} || ||||dd{VS)aFAsynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. input (str): The input to the agent. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. reference (Optional[str]): The reference answer. callbacks (Callbacks): Callbacks to use for this chain run. Returns: dict: The evaluation result, which includes the score and optionally the reasoning for reaching that. rlTrN)r`acallrs r&_aevaluate_agent_trajectoryz/TrajectoryEvalChain._aevaluate_agent_trajectoryws|8 $ 9 9:J K K "    ZZ- $           r%)NN)N),rrrr rCrlistrr"rrr(rFrGboolr model_configr@rKr#rX staticmethodrrtupler r`rc classmethodrrrjrorqrrvr rr rr rr __classcell__)rws@r&rBrBbs,,\-1K$x.)00017,1E.---M)6"d"""0:LDX C   X  S(5c)9#:;;<    \ 0Xc]s\&59:>     hx01   67        [  DGDIGGGXG&T#Y&&&X&+%S#X(;"<+c3h++++++=A@@S#X@89@ c3h @@@@6BF@@S#X@=>@ c3h @@@@<$(#$(-1!&( ( ( (  ( #5c)9#:; ( C= ( ( tCy!( 4S>*( ( (  ( ( ( ( `$(#$(-1!&( ( ( (  ( #5c)9#:; ( C= ( ( tCy!( 4S>*( ( (  ( ( ( ( ( ( ( ( r%rB)(r r8collections.abcrtypingrrrrrlangchain_core.agentsr langchain_core.callbacks.managerr r r langchain_core.exceptionsr langchain_core.language_modelsr*langchain_core.language_models.chat_modelsrlangchain_core.output_parsersrlangchain_core.toolsrpydanticrrlangchain.chains.llmr2langchain.evaluation.agents.trajectory_eval_promptrrlangchain.evaluation.schemarrrr(rBr$r%r&rs  $$$$$$.----- <;;;;;<<<<<<DDDDDD::::::))))))&&&&&&&&))))))ONNNNNNN'''''Y'''/K/K/K/K/K-/K/K/Kd} } } } } 2L} } } } } r%