improvements(ai): Improve AI streaming UI/UX interactions + better separation of AI provider responsibilities #2039

2025-04-01T04:28:43+08:00

zachgoll commented

2025-04-01 04:28:43 +08:00

(Migrated from github.com)

The initial implementation of the personal finance AI in #2022 scaffolded out a working domain for interacting with a generic, external "LLM" and providing chat responses to a user. This PR offers some improvements to the domain and stronger boundaries between the "LLM Provider" and the Maybe app.

Orchestrating a chat between an LLM and the Maybe app involves several core responsibilities:

Chat configuration
- Developer instructions
- Callable function definitions
- Model
- Chat history / previous provider response ID
Orchestration of assistant response with tool calling
- Streaming output text
- Executing function requests from LLM
- Issuing "follow up" requests to the LLM with tool call results
- Handling errors
Broadcasting chat updates
- Broadcasting new messages
- Broadcasting streaming message updates
- Broadcasting assistant "thinking" states
Generic LLM Provider interface
- Parsing concrete LLM provider responses and stream chunks
- Normalizing concrete data to "LLM Concept" data types
- Emitting normalized chunks to caller

Previously, we had a lot of the "orchestration" logic happening within the provider. This PR cleans up those domain boundaries and moves a lot of the "orchestration" of the chat to Assistant and leaving the provider solely responsible for parsing, normalizing, and emitting stream events to the Assistant

Provider::LlmConcept - defines a generic interface that all LLM providers must implement, including the chat_response method (streamable)
- Provider::Openai - our starting concrete implementation for now that streams the LLM responses
Assistant - listens to lifecycle events emitted from the responder, persists chat state, and broadcasts chat updates.
- Assistant::Configurable - controls the configuration of the assistant including developer instructions and callable functions
- Assistant::Responder - orchestrates a single chat response from the generic LLM, including follow up responses for tool calls
- Assistant::FunctionToolCaller - fulfills LLM function requests and normalizes them to ToolCall::Function domain objects
- Assistant::Function - the base class for a callable assistant function
- Assistant::Broadcastable - handles chat UI updates via turbo streams

The initial implementation of the personal finance AI in #2022 scaffolded out a working domain for interacting with a generic, external "LLM" and providing chat responses to a user. This PR offers some improvements to the domain and stronger boundaries between the "LLM Provider" and the Maybe app. Orchestrating a chat between an LLM and the Maybe app involves several core responsibilities: - **Chat configuration** - Developer instructions - Callable function definitions - Model - Chat history / previous provider response ID - **Orchestration of assistant response with tool calling** - Streaming output text - Executing function requests from LLM - Issuing "follow up" requests to the LLM with tool call results - Handling errors - **Broadcasting chat updates** - Broadcasting new messages - Broadcasting streaming message updates - Broadcasting assistant "thinking" states - **Generic LLM Provider interface** - Parsing concrete LLM provider responses and stream chunks - Normalizing concrete data to "LLM Concept" data types - Emitting normalized chunks to caller Previously, we had a lot of the "orchestration" logic happening within the provider. This PR cleans up those domain boundaries and moves a lot of the "orchestration" of the chat to `Assistant` and leaving the provider solely responsible for parsing, normalizing, and emitting stream events to the `Assistant` - `Provider::LlmConcept` - defines a generic interface that all LLM providers must implement, including the `chat_response` method (streamable) - `Provider::Openai` - our starting concrete implementation for now that streams the LLM responses - `Assistant` - listens to lifecycle events emitted from the responder, persists chat state, and broadcasts chat updates. - `Assistant::Configurable` - controls the configuration of the assistant including developer instructions and callable functions - `Assistant::Responder` - orchestrates a single chat response from the generic LLM, including follow up responses for tool calls - `Assistant::FunctionToolCaller` - fulfills LLM function requests and normalizes them to `ToolCall::Function` domain objects - `Assistant::Function` - the base class for a callable assistant function - `Assistant::Broadcastable` - handles chat UI updates via turbo streams

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: gavin/maybe#2039