Sample Web Application Built with AutoGen (microsoft#695)

* Adding research assistant code

* Adding research assistant code

* checking in RA files

* Remove used text file

* Update README.md to include Saleema's name to the Contributors list.

* remove extraneous files

* update gitignore

* improve structure on global skills

* fix linting error

* readme update

* readme update

* fix wrong function bug

* readme update

* update ui build

* cleanup, remove unused modules

* readme and docs updates

* set default user

* ui build update

* add screenshot to improve instructions

* remove logout behaviour, replace with note to developers to add their own logout logic

* Create blog and edit ARA README

* Added the stock prices example in the readme for ARA

* Include edits from review with Saleema

* fix format issues

* Cosmetic changes for betting debug messages

* edit authors

* remove references to request_timeout to support autogen v0.0.2

* update bg color for UI

* readme update

* update research assistant blog post

* omit samples folder from codecov

* ui build update + precommit refactor

* formattiing updates  fromo pre-commit

* readme update

* remove compiled source files

* update gitignore

* refactor, file removals

* refactor for improved structure - datamodel, chat and db helper

* update gitignore

* refactor, file removals

* refactor for improved structure - datamodel, chat and db helper

* refactor skills view

* general refactor

* gitignore update and general refactor

* skills update

* general refactor

* ui folder structure refactor

* improve support for skills loading

* add fetch profile default skill

* refactor chat to autogenchat

* qol refactor

* improve metadata display

* early support for autogenflow in ui

* docs update general refactor

* general refactor

* readme update

* readme update

* readme and cli update

* pre-commit updates

* precommit update

* readme update

* add steup.py for older python build versions

* add manifest.in, update app icon

* in-progress changes to agent specification

* remove use_cache refs

* update datamodel, and fix for default serverurl

* request_timeout

* readme update, fix autogen values

* fix pyautogen version

* precommit formatting and other qol items

* update folder structure

* req update

* readme and docs update

* docs update

* remove duplicate in yaml file

* add support for explicit skills addition

* readme and documentation updates

* general refactor

* remove blog post, schedule for future PR

* readme update, add info on llmconfig

* make use_cache False by default unless set

* minor ui updates

* upgrade ui to use latest uatogen lib version 0.2.0b5

* Ui refactor, support for adding arbitrary model specifications

* formatting/precommit checks

* update readme, utils default skill

---------

Co-authored-by: Piali Choudhury <pialic@microsoft.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

Author: 3 people

.coveragerc

@@ -3,3 +3,4 @@ branch = True
 source = autogen
 omit =
   *test*
+  *samples*

samples/apps/autogen-assistant/.gitignore

@@ -0,0 +1,24 @@
+database.sqlite
+.cache/*
+autogenra/web/files/user/*
+autogenra/web/files/ui/*
+OAI_CONFIG_LIST
+scratch/
+autogenra/web/workdir/*
+autogenra/web/ui/*
+autogenra/web/skills/user/*
+.release.sh
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/

samples/apps/autogen-assistant/MANIFEST.in

@@ -0,0 +1,5 @@
+recursive-include  autogenra/web/ui *
+recursive-exclude  notebooks *
+recursive-exclude  frontend *
+recursive-exclude  docs *
+recursive-exclude  tests *

samples/apps/autogen-assistant/README.md

@@ -0,0 +1,119 @@
+# AutoGen Assistant
+
+![ARA](./docs/ara_stockprices.png)
+
+AutoGen Assistant is an Autogen-powered AI app (user interface) that can converse with you to help you conduct research, write and execute code, run saved skills, create new skills (explicitly and by demonstration), and adapt in response to your interactions.
+
+### Capabilities / Roadmap
+
+Some of the capabilities supported by the app frontend include the following:
+
+- [x] Select fron a list of agents (current support for two agent workflows - `UserProxyAgent` and `AssistantAgent`)
+- [x] Modify agent configuration (e.g. temperature, model, agent system message, model etc) and chat with updated agent configurations.
+- [x] View agent messages and output files in the UI from agent runs.
+- [ ] Support for more complex agent workflows (e.g. `GroupChat` workflows)
+- [ ] Improved user experience (e.g., streaming intermediate model output, better summarization of agent responses, etc)
+
+Project Structure:
+
+- _autogenra/_ code for the backend classes and web api (FastAPI)
+- _frontend/_ code for the webui, built with Gatsby and Tailwind
+
+## Getting Started
+
+AutoGen requires access to an LLM. Please see the [AutoGen docs](https://microsoft.github.io/autogen/docs/FAQ#set-your-api-endpoints) on how to configure access to your LLM provider. In this sample, We recommend setting up your `OPENAI_API_KEY` or `AZURE_OPENAI_API_KEY` environment variable and then specifying the exact model parameters to be used in the `llm_config` that is passed to each agent specification. See the `get_default_agent_config()` method in `utils.py` to see an example of setting up `llm_config`. The example below shows how to configure access to an Azure OPENAI LLM.
+
+```python
+llm_config = LLMConfig(
+        config_list=[{
+                "model": "gpt-4",
+                "api_key": "<azure_api_key>",
+                "api_base": "<azure api base>",
+                "api_type": "azure",
+                "api_version": "2023-06-01-preview"
+        }],
+        temperature=0,
+    )
+```
+
+```bash
+export OPENAI_API_KEY=<your_api_key>
+```
+
+### Install and Run
+
+To install a prebuilt version of the app from PyPi. We highly recommend using a virtual environment (e.g. miniconda) and **python 3.10+** to avoid dependency conflicts.
+
+```bash
+pip install autogenra
+autogenra ui --port 8081  # run the web ui on port 8081
+```
+
+### Install from Source
+
+To install the app from source, clone the repository and install the dependencies.
+
+```bash
+pip install -e .
+```
+
+You will also need to build the app front end. Note that your Gatsby requires node > 14.15.0 . You may need to [upgrade your node](https://stackoverflow.com/questions/10075990/upgrading-node-js-to-latest-version) version as needed.
+
+```bash
+npm install --global yarn
+cd frontend
+yarn install
+yarn build
+```
+
+The command above will build the frontend ui and copy the build artifacts to the `autogenra` web ui folder. Note that you may have to run `npm install  --force --legacy-peer-deps` to force resolve some peer dependencies.
+
+Run the web ui:
+
+```bash
+autogenra ui --port 8081 # run the web ui on port 8081
+```
+
+Navigate to <http://localhost:8081/> to view the web ui.
+
+To update the web ui, navigate to the frontend directory, make changes and rebuild the ui.
+
+## Capabilities
+
+This demo focuses on the research assistant use case with some generalizations:
+
+- **Skills**: The agent is provided with a list of skills that it can leverage while attempting to address a user's query. Each skill is a python function that may be in any file in a folder made availabe to the agents. We separate the concept of global skills available to all agents `backend/files/global_utlis_dir` and user level skills `backend/files/user/<user_hash>/utils_dir`, relevant in a multi user environment. Agents are aware skills as they are appended to the system message. A list of example skills is available in the `backend/global_utlis_dir` folder. Modify the file or create a new file with a function in the same directory to create new global skills.
+
+- **Conversation Persistence**: Conversation history is persisted in an sqlite database `database.sqlite`.
+
+- **Default Agent Workflow**: The default a sample workflow with two agents - a user proxy agent and an assistant agent.
+
+## Example Usage
+
+Let us use a simple query demonstrating the capabilities of the research assistant.
+
+```
+Plot a chart of NVDA and TESLA stock price YTD. Save the result to a file named nvda_tesla.png
+```
+
+The agents responds by _writing and executing code_ to create a python program to generate the chart with the stock prices.
+
+> Note than there could be multiple turns between the `AssistantAgent` and the `UserProxyAgent` to produce and execute the code in order to complete the task.
+
+![ARA](./docs/ara_stockprices.png)
+
+> Note: You can also view the debug console that generates useful information to see how the agents are interacting in the background.
+
+<!-- ![ARA](./docs/ara_console.png) -->
+
+## FAQ
+
+- How do I add more skills to the research assistant? This can be done by adding a new file with documented functions to `autogenra/web/skills/global` directory.
+- How do I specify the agent configuration (e.g. temperature, model, agent system message, model etc). You can do either from the UI interface or by modifying the default agent configuration in `utils.py` (`get_default_agent_config()` method)
+- How do I reset the conversation? You can reset the conversation by deleting the `database.sqlite` file. You can also delete user files by deleting the `autogenra/web/files/user/<user_id_md5hash>` folder.
+- How do I view messages generated by agents? You can view the messages generated by the agents in the debug console. You can also view the messages in the `database.sqlite` file.
+
+## Acknowledgements
+
+Based on the [AutoGen](https://microsoft.github.io/autogen) project.
+Adapted in October 2023 from a research prototype (original credits: Gagan Bansal, Adam Fourney, Victor Dibia, Piali Choudhury, Saleema Amershi, Ahmed Awadallah, Chi Wang)

samples/apps/autogen-assistant/autogenra/__init__.py

@@ -0,0 +1,64 @@
+import json
+import time
+from typing import List
+from .datamodel import FlowConfig, Message
+from .utils import extract_successful_code_blocks, get_default_agent_config, get_modified_files
+from .autogenflow import AutoGenFlow
+import os
+
+
+class ChatManager:
+    def __init__(self) -> None:
+        pass
+
+    def chat(self, message: Message, history: List, flow_config: FlowConfig = None, **kwargs) -> None:
+        work_dir = kwargs.get("work_dir", None)
+        scratch_dir = os.path.join(work_dir, "scratch")
+        skills_suffix = kwargs.get("skills_prompt", "")
+
+        # if no flow config is provided, use the default
+        if flow_config is None:
+            flow_config = get_default_agent_config(scratch_dir, skills_suffix=skills_suffix)
+
+        # print("Flow config: ", flow_config)
+        flow = AutoGenFlow(config=flow_config, history=history, work_dir=scratch_dir, asst_prompt=skills_suffix)
+        message_text = message.content.strip()
+
+        output = ""
+        start_time = time.time()
+
+        metadata = {}
+        flow.run(message=f"{message_text}", clear_history=False)
+
+        agent_chat_messages = flow.receiver.chat_messages[flow.sender][len(history) :]
+        metadata["messages"] = agent_chat_messages
+
+        successful_code_blocks = extract_successful_code_blocks(agent_chat_messages)
+        successful_code_blocks = "\n\n".join(successful_code_blocks)
+        output = (
+            (
+                flow.sender.last_message()["content"]
+                + "\n The following code snippets were used: \n"
+                + successful_code_blocks
+            )
+            if successful_code_blocks
+            else flow.sender.last_message()["content"]
+        )
+
+        metadata["code"] = ""
+        end_time = time.time()
+        metadata["time"] = end_time - start_time
+        modified_files = get_modified_files(start_time, end_time, scratch_dir, dest_dir=work_dir)
+        metadata["files"] = modified_files
+
+        print("Modified files: ", modified_files)
+
+        output_message = Message(
+            user_id=message.user_id,
+            root_msg_id=message.root_msg_id,
+            role="assistant",
+            content=output,
+            metadata=json.dumps(metadata),
+        )
+
+        return output_message

samples/apps/autogen-assistant/autogenra/autogenchat.py

@@ -0,0 +1,134 @@
+from typing import List, Optional
+from dataclasses import asdict
+import autogen
+from .datamodel import AgentFlowSpec, FlowConfig, Message
+
+
+class AutoGenFlow:
+    """
+    AutoGenFlow class to load agents from a provided configuration and run a chat between them
+    """
+
+    def __init__(
+        self, config: FlowConfig, history: Optional[List[Message]] = None, work_dir: str = None, asst_prompt: str = None
+    ) -> None:
+        """
+        Initializes the AutoGenFlow with agents specified in the config and optional
+        message history.
+
+        Args:
+            config: The configuration settings for the sender and receiver agents.
+            history: An optional list of previous messages to populate the agents' history.
+
+        """
+        self.work_dir = work_dir
+        self.asst_prompt = asst_prompt
+        self.sender = self.load(config.sender)
+        self.receiver = self.load(config.receiver)
+
+        if history:
+            self.populate_history(history)
+
+    def _sanitize_history_message(self, message: str) -> str:
+        """
+        Sanitizes the message e.g. remove references to execution completed
+
+        Args:
+            message: The message to be sanitized.
+
+        Returns:
+            The sanitized message.
+        """
+        to_replace = ["execution succeeded", "exitcode"]
+        for replace in to_replace:
+            message = message.replace(replace, "")
+        return message
+
+    def populate_history(self, history: List[Message]) -> None:
+        """
+        Populates the agent message history from the provided list of messages.
+
+        Args:
+            history: A list of messages to populate the agents' history.
+        """
+        for msg in history:
+            if isinstance(msg, dict):
+                msg = Message(**msg)
+            if msg.role == "user":
+                self.sender.send(
+                    msg.content,
+                    self.receiver,
+                    request_reply=False,
+                )
+            elif msg.role == "assistant":
+                self.receiver.send(
+                    msg.content,
+                    self.sender,
+                    request_reply=False,
+                )
+
+    def sanitize_agent_spec(self, agent_spec: AgentFlowSpec) -> AgentFlowSpec:
+        """
+        Sanitizes the agent spec by setting loading defaults
+
+        Args:
+            config: The agent configuration to be sanitized.
+            agent_type: The type of the agent.
+
+        Returns:
+            The sanitized agent configuration.
+        """
+
+        agent_spec.config.is_termination_msg = agent_spec.config.is_termination_msg or (
+            lambda x: "TERMINATE" in x.get("content", "").rstrip()
+        )
+
+        if agent_spec.type == "userproxy":
+            code_execution_config = agent_spec.config.code_execution_config or {}
+            code_execution_config["work_dir"] = self.work_dir
+            agent_spec.config.code_execution_config = code_execution_config
+        if agent_spec.type == "assistant":
+            agent_spec.config.system_message = (
+                autogen.AssistantAgent.DEFAULT_SYSTEM_MESSAGE
+                + "\n\n"
+                + agent_spec.config.system_message
+                + "\n\n"
+                + self.asst_prompt
+            )
+
+        return agent_spec
+
+    def load(self, agent_spec: AgentFlowSpec) -> autogen.Agent:
+        """
+        Loads an agent based on the provided agent specification.
+
+        Args:
+            agent_spec: The specification of the agent to be loaded.
+
+        Returns:
+            An instance of the loaded agent.
+        """
+        agent: autogen.Agent
+        agent_spec = self.sanitize_agent_spec(agent_spec)
+        if agent_spec.type == "assistant":
+            agent = autogen.AssistantAgent(**asdict(agent_spec.config))
+        elif agent_spec.type == "userproxy":
+            agent = autogen.UserProxyAgent(**asdict(agent_spec.config))
+        else:
+            raise ValueError(f"Unknown agent type: {agent_spec.type}")
+        return agent
+
+    def run(self, message: str, clear_history: bool = False) -> None:
+        """
+        Initiates a chat between the sender and receiver agents with an initial message
+        and an option to clear the history.
+
+        Args:
+            message: The initial message to start the chat.
+            clear_history: If set to True, clears the chat history before initiating.
+        """
+        self.sender.initiate_chat(
+            self.receiver,
+            message=message,
+            clear_history=clear_history,
+        )