logs doc

documents/docs/advanced_usage/control_filtering.md

@@ -0,0 +1,2 @@
+# Control Filtering
+

documents/docs/advanced_usage/customization.md

@@ -0,0 +1,24 @@
+# Customization
+
+Sometimes, UFO may need additional context or information to complete a task. These information are important and customized for each user. UFO can ask the user for additional information and save it in the local memory for future reference. This customization feature allows UFO to provide a more personalized experience to the user.
+
+## Scenario
+
+Let's consider a scenario where UFO needs additional information to complete a task. UFO is tasked with booking a cab for the user. To book a cab, UFO needs to know the exact address of the user. UFO will ask the user for the address and save it in the local memory for future reference. Next time, when UFO is asked to complete a task that requires the user's address, UFO will use the saved address to complete the task, without asking the user again.
+
+
+## Implementation
+We currently implement the customization feature in the `HostAgent` class. When the `HostAgent` needs additional information, it will transit to the `PENDING` state and ask the user for the information. The user will provide the information, and the `HostAgent` will save it in the local memory base for future reference. The saved information is stored in the `blackboard` and can be accessed by all agents in the session.
+
+!!! note
+    The customization memory base is only saved in a **local file**. These information will **not** upload to the cloud or any other storage to protect the user's privacy.
+
+## Configuration
+
+You can configure the customization feature by setting the following field in the `config_dev.yaml` file.
+
+| Configuration Option   | Description                                  | Type    | Default Value                         |
+|------------------------|----------------------------------------------|---------|---------------------------------------|
+| `USE_CUSTOMIZATION`    | Whether to enable the customization.         | Boolean | True                                  |
+| `QA_PAIR_FILE`         | The path for the historical QA pairs.        | String  | "customization/historical_qa.txt"     |
+| `QA_PAIR_NUM`          | The number of QA pairs for the customization.| Integer | 20                                    |

documents/docs/advanced_usage/follower_mode.md

@@ -0,0 +1,83 @@
+# Follower Mode
+
+The Follower mode is a feature of UFO that the agent follows a list of pre-defined steps in natural language to take actions on applications. Different from the normal mode, this mode creates an `FollowerAgent` that follows the plan list provided by the user to interact with the application, instead of generating the plan itself. This mode is useful for debugging and software testing or verification.
+
+## Quick Start
+
+### Step 1: Create a Plan file
+
+Before starting the Follower mode, you need to create a plan file that contains the list of steps for the agent to follow. The plan file is a JSON file that contains the following fields:
+
+| Field | Description | Type |
+| --- | --- | --- |
+| task | The task description. | String |
+| steps | The list of steps for the agent to follow. | List of Strings |
+| object | The application or file to interact with. | String |
+
+Below is an example of a plan file:
+
+```json
+{
+    "task": "Type in a text of 'Test For Fun' with heading 1 level",
+    "steps": 
+    [
+        "1.type in 'Test For Fun'", 
+        "2.Select the 'Test For Fun' text",
+        "3.Click 'Home' tab to show the 'Styles' ribbon tab",
+        "4.Click 'Styles' ribbon tab to show the style 'Heading 1'",
+        "5.Click 'Heading 1' style to apply the style to the selected text"
+    ],
+    "object": "draft.docx"
+}
+```
+
+!!! note
+    The `object` field is the application or file that the agent will interact with. The object **must be active** (can be minimized) when starting the Follower mode.
+
+
+### Step 2: Start the Follower Mode
+To start the Follower mode, run the following command:
+
+```bash
+# assume you are in the cloned UFO folder
+python ufo.py --task_name {task_name} --mode follower --plan {plan_file}
+```
+
+!!! tip
+    Replace `{task_name}` with the name of the task and `{plan_file}` with the path to the plan file.
+
+
+### Step 3: Run in Batch (Optional)
+
+You can also run the Follower mode in batch mode by providing a folder containing multiple plan files. The agent will follow the plans in the folder one by one. To run in batch mode, run the following command:
+
+```bash
+# assume you are in the cloned UFO folder
+python ufo.py --task_name {task_name} --mode follower --plan {plan_folder}
+``` 
+
+UFO will automatically detect the plan files in the folder and run them one by one.
+
+!!! tip
+    Replace `{task_name}` with the name of the task and `{plan_folder}` with the path to the folder containing plan files.
+
+
+## Evaluation
+You may want to evaluate the `task` is completed successfully or not by following the plan. UFO will call the `EvaluationAgent` to evaluate the task if `EVA_SESSION` is set to `True` in the `config_dev.yaml` file.
+
+You can check the evaluation log in the `logs/{task_name}/evaluation.log` file. 
+
+# References
+The follower mode employs a `PlanReader` to parse the plan file and create a `FollowerSession` to follow the plan. 
+
+## PlanReader
+The `PlanReader` is located in the `ufo/module/sessions/plan_reader.py` file.
+
+:::module.sessions.plan_reader.PlanReader
+
+<br>
+## FollowerSession
+
+The `FollowerSession` is also located in the `ufo/module/sessions/session.py` file.
+
+:::module.sessions.session.FollowerSession

documents/docs/agents/evaluation_agent.md

@@ -26,10 +26,25 @@ The `EvaluationAgent` generates the following outputs after evaluation:
 
 | Output | Description | Type |
 | --- | --- | --- |
-| Reason | The detailed reason for your judgment, by observing the screenshot differences and the <Execution Trajectory>. | String |
-| Sub-score | The sub-score of the evaluation in decomposing the evaluation into multiple sub-goals. | List of Dictionaries |
-| Complete | The completion status of the evaluation, can be `yes`, `no`, or `unsure`. | String |
-
+| reason | The detailed reason for your judgment, by observing the screenshot differences and the <Execution Trajectory>. | String |
+| sub_scores | The sub-score of the evaluation in decomposing the evaluation into multiple sub-goals. | List of Dictionaries |
+| complete | The completion status of the evaluation, can be `yes`, `no`, or `unsure`. | String |
+
+Below is an example of the evaluation output:
+
+```json
+{
+    "reason": "The agent successfully completed the task of sending 'hello' to Zac on Microsoft Teams. 
+    The initial screenshot shows the Microsoft Teams application with the chat window of Chaoyun Zhang open. 
+    The agent then focused on the chat window, input the message 'hello', and clicked the Send button. 
+    The final screenshot confirms that the message 'hello' was sent to Zac.", 
+    "sub_scores": {
+        "correct application focus": "yes", 
+        "correct message input": "yes", 
+        "message sent successfully": "yes"
+        }, 
+    "complete": "yes"}
+```
 
 !!!info
     The log of the evaluation results will be saved in the `logs/{task_name}/evaluation.log` file.

ufo/module/sessions/session.py

@@ -176,7 +176,7 @@ def __init__(
         """
         Initialize a session.
         :param task: The name of current task.
-        :param plan_dir: The path of the plan file to follow.
+        :param plan_file: The path of the plan file to follow.
         :param should_evaluate: Whether to evaluate the session.
         :param id: The id of the session.
         """