Streaming
Real-time token, tool-call, and streaming-tool event processing with Conduit and StreamEvent.
Real-time token, incremental tool-call delta, and streaming tool execution processing using Conduit and StreamEvent.
Key Concepts
- Incremental Token & Tool-Call Delivery: Providers (
OpenAI,Gemini, andOllama) emitLLMChunk runId chunkText (Just toolCall)as tool calls complete or arrive over the wire alongside text deltas. - Structured Lifecycle Events:
StreamEventmodelsLLMStart,LLMChunk,LLMEnd,ToolStart,ToolChunk,ToolEnd,ToolErrorEvent,ChainStart, andChainEnd. - Streaming Tools (
StreamingTool m): Tools likestreamingShellToolemit liveToolChunkoutput lines while executing, wrapped safely withwithStreamingSecurity. - Lifting Streams (
liftChatStream): Bridge resource-safeChatStreams (EventStream (ExceptT LangchainError (ResourceT IO))) into arbitrary(MonadIO m, MonadError LangchainError m)pipelines.
Working Code Example
Compare local execution via Ollama and cloud API execution via OpenAI / OpenRouter. Use the toggle tabs or the global provider switcher in the header to switch:
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE OverloadedStrings #-}
module Ollama.Stream (runApp) where
import Control.Monad.Except (runExceptT)
import Control.Monad.IO.Class (liftIO)
import Control.Monad.Trans.Resource (runResourceT)
import Data.Conduit (runConduit, (.|))
import qualified Data.Conduit.List as CL
import qualified Data.Text.IO as T
import Langchain.Prelude
import System.IO (hFlush, stdout)
runApp :: IO ()
runApp = do
o <- newOllama "gemma3" defaultConfig
let msgs = [userMessage "What is the meaning of life"]
res <- runResourceT $ runExceptT $ runConduit $ stream o msgs Nothing .| CL.mapM_ onStreamEvent
case res of
Left err -> T.putStrLn $ "\nError: " <> errorMessage err
Right () -> T.putStrLn "\n--- Stream Finished ---"
where
onStreamEvent = \case
LLMChunk _ chunk _ -> liftIO $ do
T.putStr chunk
hFlush stdout
_ -> pure (){-# LANGUAGE LambdaCase #-}
{-# LANGUAGE OverloadedStrings #-}
module OpenAI.Stream (runApp) where
import Control.Monad.Except (runExceptT)
import Control.Monad.IO.Class (liftIO)
import Control.Monad.Trans.Resource (runResourceT)
import Data.Conduit (runConduit, (.|))
import qualified Data.Conduit.List as CL
import qualified Data.Text.IO as T
import Langchain.Prelude
import OpenAI.Common (defaultModelName, getOpenRouterModel)
import System.IO (hFlush, stdout)
runApp :: IO ()
runApp = do
o <- getOpenRouterModel defaultModelName
let msgs = [userMessage "What is the meaning of life"]
res <- runResourceT $ runExceptT $ runConduit $ stream o msgs Nothing .| CL.mapM_ onStreamEvent
case res of
Left err -> T.putStrLn $ "\nError: " <> errorMessage err
Right () -> T.putStrLn "\n--- Stream Finished ---"
where
onStreamEvent = \case
LLMChunk _ chunk _ -> liftIO $ do
T.putStr chunk
hFlush stdout
_ -> pure (){-# LANGUAGE LambdaCase #-}
{-# LANGUAGE OverloadedStrings #-}
module Gemini.Stream (runApp) where
import Control.Monad.Except (runExceptT)
import Control.Monad.IO.Class (liftIO)
import Control.Monad.Trans.Resource (runResourceT)
import Data.Conduit (runConduit, (.|))
import qualified Data.Conduit.List as CL
import qualified Data.Text.IO as T
import Gemini.Common (defaultGeminiModelName, getGeminiModel)
import Langchain.Prelude
import System.IO (hFlush, stdout)
runApp :: IO ()
runApp = do
T.putStrLn "--- Gemini Conduit Token Streaming ---"
g <- getGeminiModel defaultGeminiModelName
let msgs = [userMessage "Explain why Haskell has referential transparency in 2 short paragraphs"]
res <- runResourceT $ runExceptT $ runConduit $ stream g msgs Nothing .| CL.mapM_ onStreamEvent
case res of
Left err -> T.putStrLn $ "\nError: " <> errorMessage err
Right () -> T.putStrLn "\n--- Stream Finished ---"
where
onStreamEvent = \case
LLMChunk _ chunk _ -> liftIO $ do
T.putStr chunk
hFlush stdout
_ -> pure ()Core Types & Functions
-- | Streaming invocation on any ChatModel provider (OpenAI, Gemini, Ollama)
stream :: ChatModel model => model -> [Message] -> Maybe (ModelConfig model) -> ChatStream
-- | Stream events with incremental text and tool-call chunks
data StreamEvent
= LLMStart { runId :: Text, modelName :: Text, inputMessages :: [Message] }
| LLMChunk { runId :: Text, chunkText :: Text, toolCallDelta :: Maybe ToolCall }
| LLMEnd { runId :: Text, finalMessage :: Message, tokenUsage :: Maybe TokenUsage }
| ToolStart { runId :: Text, toolName :: Text, toolInput :: Value }
| ToolChunk { runId :: Text, toolName :: Text, toolChunkText :: Text }
| ToolEnd { runId :: Text, toolName :: Text, toolOutput :: Value }
| ToolErrorEvent { runId :: Text, toolName :: Text, toolErrorPayload :: LangchainError }
| ChainStart { runId :: Text, chainName :: Text, chainInput :: Value }
| ChainEnd { runId :: Text, chainName :: Text, chainOutput :: Value }
| NodeStart { runId :: Text, nodeId :: Text, nodeState :: Value }
| NodeEnd { runId :: Text, nodeId :: Text, nodeState :: Value }
-- | Lift a resource-safe ChatStream into an agent or application EventStream
liftChatStream :: (MonadIO m, MonadError LangchainError m) => ChatStream -> EventStream m
-- | Streaming tools yielding incremental Text chunks
data StreamingTool m = StreamingTool
{ streamingToolDef :: Tool m
, streamingToolExecute :: Value -> ConduitT () Text m (Either LangchainError Text)
}
createStreamingTool :: Monad m => Text -> Text -> Value -> (Value -> ConduitT () Text m (Either LangchainError Text)) -> StreamingTool m
fromTool :: Monad m => Tool m -> StreamingTool m
streamingShellTool :: MonadIO m => SecurityPolicy -> StreamingTool mTool-Call Streaming Across Providers
OpenAI, Gemini, and Ollama all emit incremental LLMChunk events during ChatModel.stream:
- OpenAI (Langchain.Provider.OpenAI): Accumulates streamed delta.tool_calls fragments across SSE frames and emits completed ToolCall values via LLMChunk "" (Just toolCall) prior to LLMEnd.
- Gemini (Langchain.Provider.Gemini): Emits ToolCall deltas incrementally from Interactions v1 function_call SSE events alongside text deltas.
- Ollama (Langchain.Provider.Ollama): Forwards streamed tool_calls directly from ollama-haskell chunks as LLMChunk "" (Just toolCall).
Running This Example
Local Ollama
Ensure your Ollama daemon is running locally with the target model:
ollama run gemma3 # or your desired model
stack run streamollamaOpenAI / OpenRouter
Ensure your OPENROUTER_API_KEY or OPENAI_API_KEY is exported:
export OPENROUTER_API_KEY="your-api-key"
stack run streamopenaiGoogle Gemini
Set your GEMINI_API_KEY in environment or store in ~/.gemini_api_key:
export GEMINI_API_KEY="AIzaSy..."
cabal run streamgemini