diff --git a/.gemini/GEMINI.md b/.gemini/GEMINI.md index 9ed9201ea..30a11ed6b 100644 --- a/.gemini/GEMINI.md +++ b/.gemini/GEMINI.md @@ -3,41 +3,40 @@ A CatalogItem is an object which represents a widget that can be instantiated by an LLM. It centralizes the widgetBuilder function for the widget along with the data schema and name. ## Structure of a CatalogItem + - Only the variable that defines the CatalogItem should be public. Everything else should be private. -- The dataSchema must be defined as a Schema object. Look at pkgs/flutter_genui/lib/src/catalog/elevated_button.dart as a guide. Remember to use `Schema.object` and `Schema.enumString` etc. -- The Schema should *not* have a "props" member or an "id" member, as those will be injected at a higher level. The schema should just be an object that includes all the properties that are specific to this widget, e.g. content to display. +- The dataSchema must be defined as a Schema object. Look at `pkgs/flutter_genui/lib/src/catalog/core_widgets/elevated_button.dart` as a guide. +- Remember to use `Schema.object` and `Schema.enumString` etc. +- The Schema should _not_ have a "props" member or an "id" member, as those will be injected at a higher level. The schema should just be an object that includes all the properties that are specific to this widget, e.g. content to display. - The only imports should be 'package:firebase_ai/firebase_ai.dart', 'package:flutter/material.dart' and the import for CatalogItem - this will be '../../model/catalog_item.dart' for the SDK, or 'package:flutter_genui/flutter_genui.dart' for the example apps. Any other utilities etc that are needed should be inlined into the file. - The name of the CatalogItem should be in lower camel case. -- Widgets that require controllers or state in order to instantly reflect user inputs (e.g. Radio buttons, checkboxes, text fields) should be implemented as StatefulWidgets which maintain internal UI state. Look at libs/src/widgets/radio_group.dart as an example. There should be a private StatefulWidget class definition in the file. +- Widgets that require controllers or state in order to instantly reflect user inputs (e.g. Radio buttons, checkboxes, text fields) should be implemented as StatefulWidgets which maintain internal UI state. Look at `libs/src/widgets/radio_group.dart` as an example. There should be a private StatefulWidget class definition in the file. - The input schema for CatalogItems should not include parallel lists of data - instead use a single list of object where it makes sense. -- If you are implementing a CatalogItem that needs to compose other CatalogItems, include a "child" or "children" parameter of String type, which will contain the ID of the child to reference. When building the child or children, use the buildChild function and pass in the string. See pkgs/flutter_genui/lib/src/catalog/elevated_button.dart and pkgs/flutter_genui/lib/src/catalog/column.dart as examples. +- If you are implementing a CatalogItem that needs to compose other CatalogItems, include a "child" or "children" parameter of String type, which will contain the ID of the child to reference. When building the child or children, use the buildChild function and pass in the string. See `pkgs/flutter_genui/lib/src/catalog/elevated_button.dart` and `pkgs/flutter_genui/lib/src/catalog/column.dart` as examples. - All interactive UI elements where the user can input data e.g. by typing or selecting from options, or take action by clicking buttons etc, should handle those actions by calling the `dispatchEvent` callback, and including all the context needed for the LLM to understand what the user has done. E.g. if they have selected a specific option, the handler call should include the name of the selected option as the value. -- IMPORTANT: `required` is *not* a parameter of Schema.object! For this Schema class, all parameters are required by default. Instead, there is an `optionalProperties` which you specify if there are any optional parameters. +- IMPORTANT: `required` is _not_ a parameter of Schema.object! For this Schema class, all parameters are required by default. Instead, there is an `optionalProperties` which you specify if there are any optional parameters. +- When parsing a catalog item in the widgetBuilder, use extension types for the data required by the catalog item, similar to `examples/travel_app/lib/src/catalog/travel_carousel.dart`. ## How to create a new a CatalogItem -1. Understand the structure of a CatalogItem by looking at pkgs/flutter_genui/lib/src/model/catalog_item.dart for the type, and pkgs/flutter_genui/lib/src/catalog/elevated_button.dart as an example to follow. - +1. Understand the structure of a CatalogItem by looking at `pkgs/flutter_genui/lib/src/model/catalog_item.dart` for the type, and `pkgs/flutter_genui/lib/src/catalog/elevated_button.dart` as an example to follow. 2. Create a new file in the relevant app, which contains a declaration of a global variable with a CatalogItem. The location of the file should be: - - For the generic flutter_genui SDK: pkgs/flutter_genui/lib/src/catalog/ - - For an example app, at examples//lib/src/catalog/ e.g. for the travel_app: examples/travel_app/lib/src/catalog/ -3. Review your implementation and compare it to the examples, to ensure that the use of parameters matchces and the structure of the code is similar. Fix any mistakes. +- For the generic flutter_genui SDK: `pkgs/flutter_genui/lib/src/catalog/` +- For an example app, at `examples/[MY_APP]/lib/src/catalog/` e.g. for the travel_app: `examples/travel_app/lib/src/catalog/` +3. Review your implementation and compare it to the examples, to ensure that the use of parameters matches and the structure of the code is similar. Fix any mistakes. 4. Write a test for the CatalogItem and run it, fixing any mistakes. - 5. Update the Catalog definition for the app or SDK to include the new item. ## How to update the CatalogItem API When the CatalogItem API changes, it is necessary to update all the existing catalog items to match. -1. Understand the change that has been requested, and carefully update the code at pkgs/flutter_genui/lib/src/model/catalog_item.dart - -2. Update all places that create and use CatalogItems e.g. pkgs/flutter_genui/lib/src/model/catalog.dart. Consider all the code at pkgs/flutter_genui/lib/* when doing this. - -3. Find all CatalogItem implementations which will be at examples/travel_app/lib/src/catalog/*, pkgs/flutter_genui/lib/src/catalog/* etc. Update each of them to match the changes in the API. +1. Understand the change that has been requested, and carefully update the code at `pkgs/flutter_genui/lib/src/model/catalog_item.dart`. +2. Update all places that create and use CatalogItems e.g. `pkgs/flutter_genui/lib/src/model/catalog.dart`. Consider all the code at `pkgs/flutter_genui/lib/*` when doing this. +3. Find all CatalogItem implementations which will be at `examples/travel_app/lib/src/catalog/*`, `pkgs/flutter_genui/lib/src/catalog/*` etc. Update each of them to match the changes in the API. ## How to update every CatalogItem -1. Find all CatalogItem implementations which will be at examples/travel_app/lib/src/catalog/*, pkgs/flutter_genui/lib/src/catalog/* etc. Update each of them. +1. Find all CatalogItem implementations which will be at `examples/travel_app/lib/src/catalog/*`, `pkgs/flutter_genui/lib/src/catalog/*` etc. Update each of them. diff --git a/.gemini/README.md b/.gemini/README.md new file mode 100644 index 000000000..0e3d5f6ec --- /dev/null +++ b/.gemini/README.md @@ -0,0 +1,17 @@ +# Gemini Configuration + +This directory (`.gemini/`) contains configuration and context files for Google's Gemini tools, including: + +- **`gemini-cli`**: The command-line interface for Gemini. +- **Gemini Code Assist**: An AI-powered assistant that integrates with GitHub and IDEs and provides for automated code reviews, among other features. + +The files in this directory are used to customize the behavior of these tools for this specific repository. + +- **`GEMINI.md`**: Provides project-specific context, instructions, and guidelines that are included in the context when using Gemini CLI and Code Assist. This helps the AI understand the project's conventions and requirements. +- **`config.yaml`**: Configuration for the Gemini for GitHub tools, such as settings for code review. +- **`styleguide.md`**: Contains the project's style guide, which is used by the Gemini for Github tools to ensure that generated reviews adhere to the project's conventions. + +## Documentation + +- [Gemini CLI Documentation](https://github.com/google-gemini/gemini-cli) +- [Gemini Code Assist Documentation](https://cloud.google.com/products/gemini/code-assist) diff --git a/examples/travel_app/IMPLEMENTATION.md b/examples/travel_app/IMPLEMENTATION.md new file mode 100644 index 000000000..86f34336a --- /dev/null +++ b/examples/travel_app/IMPLEMENTATION.md @@ -0,0 +1,86 @@ +# Travel App Implementation + +This document provides a comprehensive overview of the architecture, purpose, and implementation of the Travel App example. + +## Purpose + +The Travel App is a demonstration of a generative UI application built using the `flutter_genui` package. It functions as a conversational travel agent assistant. Instead of a static, predefined user interface, the app's UI is dynamically generated by a large language model (LLM) in response to the user's prompts. This allows for a highly flexible and context-aware user experience, where the UI adapts to the flow of the conversation. + +## Architecture + +The application is structured into several distinct layers, each with a specific responsibility. + +### 1. UI Layer (`main.dart`) + +This is the main entry point and the visible part of the application. Its responsibilities are: + +- **Initialization**: It initializes Firebase and Firebase App Check. +- **UI Scaffolding**: It provides the basic structure of the app, which consists of an app bar, a text input field for user prompts, a send button, and a main content area where the dynamic UI is rendered. +- **State Management**: It holds and manages the lifecycle of the `ConversationManager`, which is the core of the app's logic. + +### 2. Conversation Management Layer ([`package:flutter_genui`](../../pkgs/flutter_genui/IMPLEMENTATION.md)) + +The `ConversationManager` from the `flutter_genui` package is the central orchestrator of the application. + +- **Conversation Flow**: It manages the back-and-forth between the user and the AI model. +- **Prompt Handling**: It takes the user's text prompt, combines it with the conversation history and the system prompt, and sends it to the AI Client. +- **UI Generation**: It receives tool-call requests from the model, which are essentially instructions to render specific UI widgets. +- **Widget Rendering**: It looks up the requested widget in the `Catalog`, validates the provided data against the widget's schema, and uses the widget's builder function to create and display the Flutter widget. +- **State Tracking**: It maintains the state of the rendered UI, including the history of the conversation. + +### 3. AI/Model Layer (`package:flutter_genui`) + +The `AiClient` class, also part of `flutter_genui`, abstracts the communication with the underlying generative AI model (e.g., a Google Gemini model via Firebase). It handles the API calls to the model, sending prompts and receiving the model's responses, including the tool calls that drive the UI generation. + +### 4. Widget Catalog (The "Tools") + +This is the collection of predefined UI components that the AI can use to construct the interface. It acts as the "API" that the AI targets. + +- **Definition**: The catalog is defined in `lib/src/catalog.dart` as a `Catalog` instance, which is a list of `CatalogItem`s. +- **Custom Components**: The travel app defines several custom `CatalogItem`s in the `lib/src/catalog/` directory, such as `travel_carousel`, `itinerary_with_details`, and `filter_chip_group`. +- **Standard Components**: It also uses standard, pre-built components from `flutter_genui` like `column`, `text`, `elevatedButton`, etc. + +## Implementation Details + +### The System Prompt + +The interaction with the model is heavily guided by a detailed **system prompt** defined in `main.dart`. This prompt is critical to the application's success. It instructs the model on: + +- **Persona**: To act as a helpful travel agent. +- **Conversation Flow**: To first ask clarifying questions and then present results. +- **Tool Usage**: Which widgets (tools) to use in different situations (e.g., use `travel_carousel` for options, `itinerary_with_details` for final results). +- **UI Style**: How to compose widgets for a good user experience (e.g., breaking up itineraries with other content). +- **Available Data**: It includes a JSON string (`imagesJson`) containing a list of available image URLs and descriptions, ensuring the model uses relevant and high-quality images. + +### The Widget Catalog + +The `catalog` is the cornerstone of the dynamic UI generation. It's a registry of all possible UI components the model can render. + +Each component in the catalog is a `CatalogItem` with three key parts: + +1. **`name`**: A unique string that the AI uses to identify the widget (e.g., `'travel_carousel'`). +2. **`dataSchema`**: A `Schema` object that defines the structure and types of data the widget expects. This schema is provided to the model so it knows what parameters to generate for the tool call. The descriptions within the schema are crucial for the model to understand the purpose of each parameter. +3. **`widgetBuilder`**: A Dart function that takes the data generated by the model (as a `Map`) and returns a Flutter `Widget`. + +### Catalog Item Definition + +Each custom widget in `lib/src/catalog/` follows a consistent pattern: + +1. **Schema Definition**: A `_schema` variable defines the data structure using `Schema.object`, `Schema.string`, `Schema.array`, etc. +2. **Data Accessor**: A Dart `extension type` is defined to provide type-safe access to the `Map` data received from the model. This avoids manual casting and error-prone map access. +3. **`CatalogItem` Instance**: A final variable creates the `CatalogItem`, passing the `name`, `dataSchema`, and `widgetBuilder`. +4. **Widget Implementation**: A standard Flutter `StatelessWidget` or `StatefulWidget` that contains the actual UI code for the component. + +### Dynamic UI Composition + +Widgets can be nested to create complex layouts. This is achieved by having a widget's schema accept the ID of another widget as a parameter (e.g., the `child` property of `itinerary_with_details`). + +The `widgetBuilder` for a container-like widget receives a `buildChild` function as an argument. It can call `buildChild(childId)` to recursively ask the `ConversationManager` to build and return the child widget, which is then placed in the parent's widget tree. + +### User Interaction and Events + +The UI is not just for display; it's interactive. Widgets like `optionsFilterChip` and `filterChipGroup` can capture user input. + +- **Event Dispatching**: The `widgetBuilder` receives a `dispatchEvent` function. This function can be called in response to user actions (like a button press or item selection). +- **Event Data**: `dispatchEvent` sends an event type and a value back to the `ConversationManager`. +- **Conversation Update**: The `ConversationManager` adds this event information to the conversation history and sends it to the model on the next turn. This informs the model of the user's actions, allowing it to respond accordingly (e.g., refining a search based on a selected filter). diff --git a/pkgs/flutter_genui/IMPLEMENTATION.md b/pkgs/flutter_genui/IMPLEMENTATION.md new file mode 100644 index 000000000..666360de4 --- /dev/null +++ b/pkgs/flutter_genui/IMPLEMENTATION.md @@ -0,0 +1,64 @@ +# `flutter_genui` Package Implementation + +This document provides a comprehensive overview of the architecture, purpose, and implementation of the `flutter_genui` package. + +## Purpose + +The `flutter_genui` package provides the core framework for building Flutter applications with dynamically generated user interfaces powered by large language models (LLMs). It enables developers to create conversational UIs where the interface is not static or predefined, but is instead constructed by an AI in real-time based on the user's prompts and the flow of the conversation. + +The package supplies the essential components for managing the conversation, interacting with the AI model, defining a vocabulary of UI widgets, and rendering the dynamic UI. + +## Architecture + +The package is designed with a layered architecture, separating concerns to create a flexible and extensible framework. + +### 1. AI Client Layer (`lib/src/ai_client/`) + +This layer is responsible for all communication with the generative AI model. + +- **`LlmConnection`**: An abstract interface defining the contract for a connection to an LLM. This allows for different LLM backends to be implemented. +- **`AiClient`**: The default implementation of `LlmConnection`. It handles the complexities of interacting with the Gemini API, including model configuration, retry logic with exponential backoff, and tool management. It uses a "forced tool calling" approach, where the model is required to call a function to produce its output. +- **`AiTool`**: An abstract class for defining tools that the AI can invoke. These tools are the bridge between the AI and the application's capabilities. The `DynamicAiTool` provides a convenient way to create tools from simple functions. + +### 2. Conversation Management Layer (`lib/src/core/`) + +This is the central nervous system of the package, orchestrating the entire generative UI process. + +- **`ConversationManager`**: The main orchestrator. It manages the conversation history, sends prompts to the `AiClient`, processes the AI's responses (which are UI manipulation actions), and updates the UI. It maintains the state of the conversation and the rendered UI "surfaces". +- **`ConversationWidget`**: A Flutter widget that renders the chat history, including user prompts and the dynamic UI responses generated by the AI. +- **`EventDebouncer`**: A utility to batch and debounce UI events (like text field changes) before sending them to the `ConversationManager`. This prevents excessive calls to the AI model during rapid user input. + +### 3. UI Model Layer (`lib/src/model/`) + +This layer defines the data structures that represent the dynamic UI and the conversation. + +- **`Catalog` and `CatalogItem`**: These classes define the registry of available UI components. The `Catalog` holds a list of `CatalogItem`s, and each `CatalogItem` defines a widget's name, its data schema, and a builder function to render it. +- **`UiDefinition` and `UiEvent`**: `UiDefinition` represents a complete UI tree to be rendered, including the root widget and a map of all widget definitions. `UiEvent` is a data object representing a user interaction (e.g., a button tap), which is sent back to the `ConversationManager`. +- **`ChatMessage`**: A sealed class representing the different types of messages in the conversation history: `UserPrompt`, `TextResponse`, `UiResponse`, and `SystemMessage`. +- **`DynamicUi`**: The Flutter widget responsible for recursively building the UI tree from a `UiDefinition`. It uses the provided `Catalog` to find the correct widget builder for each node in the tree. + +### 4. Widget Catalog Layer (`lib/src/catalog/`) + +This layer provides a set of core, general-purpose UI widgets that can be used out-of-the-box. + +- **`core_catalog.dart`**: Defines the `coreCatalog`, which includes fundamental widgets like `Column`, `Text`, `ElevatedButton`, `TextField`, `CheckboxGroup`, and `RadioGroup`. +- **Widget Implementation**: Each core widget follows the standard `CatalogItem` pattern: a schema definition, a type-safe data accessor using an `extension type`, the `CatalogItem` instance, and the Flutter widget implementation. + +## How It Works: The Generative UI Cycle + +1. **User Input**: The user enters a text prompt in the UI. +2. **Prompt Dispatch**: The UI calls `ConversationManager.sendUserPrompt()`. +3. **Conversation Update**: The `ConversationManager` adds the user's prompt to the conversation history. +4. **AI Invocation**: The `ConversationManager` calls `llmConnection.generateContent()`, passing the full conversation history, a system prompt, and an `outputSchema`. The `outputSchema` defines the structure of the expected response, which includes a list of UI actions (`add`, `update`, `delete`). +5. **Model Processing**: The `AiClient` sends the request to the LLM. The LLM, guided by the system prompt and the schemas of the available widgets (tools), decides how to respond. +6. **Tool Call Response**: The model generates a response that calls the internal "output" tool. The arguments to this tool call conform to the `outputSchema` and contain the instructions for manipulating the UI. +7. **Action Processing**: The `ConversationManager` receives the structured response. It iterates through the list of actions. + - For an **`add`** action, it creates a new `UiResponse` message and adds it to the chat history. It also stores the conversation context for this new UI "surface". + - For an **`update`** action, it finds the existing `UiResponse` with the matching `surfaceId` and replaces it with the new definition. + - For a **`delete`** action, it removes the corresponding `UiResponse` from the chat history. +8. **UI Rendering**: The `_uiDataStreamController` in the `ConversationManager` emits the updated chat history. The `ConversationWidget` rebuilds, and its `ListView` now includes the new or updated `UiResponse`. +9. **Dynamic UI Build**: The `DynamicUi` widget within the `ConversationWidget` receives the `UiDefinition` and recursively builds the Flutter widget tree using the `Catalog`. +10. **User Interaction**: The user interacts with the newly generated UI (e.g., clicks a button). +11. **Event Dispatch**: The widget's `onPressed` handler calls the `dispatchEvent` function, creating a `UiEvent`. +12. **Event Handling**: The `EventDebouncer` collects and sends the event to the `ConversationManager`. +13. **Cycle Repeats**: The `ConversationManager` adds the UI event to the conversation history for that surface and calls the AI again to get an updated UI, thus continuing the cycle.