Feature request: Memory-efficient streaming API for embed()

[fede-kamel]

Problem

When embedding large datasets (thousands or millions of texts), the current embed() method accumulates all results in memory before returning. This causes:

• Out-of-memory errors for very large datasets
• Memory pressure when processing many texts sequentially
• No way to process results incrementally (e.g., save to database as embeddings arrive)

For enterprise workloads processing large document corpora, this is a significant limitation.

Proposed Solution

A new embed_stream() method that:

• Processes texts in configurable batches
• Yields embeddings one at a time via an iterator
• Keeps memory usage proportional to batch_size rather than total dataset size
• Works with both v1 and v2 clients

Usage Example

import cohere

client = cohere.Client()

# Process large dataset incrementally
for embedding in client.embed_stream(
    texts=large_text_list,  # Can be thousands of texts
    model="embed-english-v3.0",
    input_type="classification",
    batch_size=20
):
    save_to_database(embedding.index, embedding.embedding)
    # Only batch_size worth of embeddings in memory at a time

Memory Impact

Dataset Size	Current embed()	Proposed embed_stream()
1,000 texts	~4 MB	~20 KB
100,000 texts	~400 MB	~20 KB
1,000,000 texts	~4 GB+ (OOM)	~20 KB

Context

We are using the Cohere Python SDK at Oracle for processing large embedding workloads. We have a working implementation in PR #698 that has been tested with the real Cohere API, passes all unit tests, and is backward compatible (no changes to existing embed()).

Additional Details

• No breaking changes to existing APIs
• Optional dependency on ijson for more efficient incremental parsing (works without it)
• Supports both embeddings_floats and embeddings_by_type response formats

[mkozakov]

Thanks @fede-kamel we're not going to go ahead with this change to keep it consistent across our other SDKs

[fede-kamel]

Thanks for the response. I want to make sure I understand what “consistency across SDKs” means in practical terms. @mkozakov

Could you clarify:

1. Is the concern specifically that Go/Java/TypeScript do not yet expose an embed_stream-style API?
2. What concrete condition would make this acceptable (for example: parity plan across SDKs, shared design doc, experimental flag)?
3. Is there a recommended SDK-level pattern today for large embedding workloads that cannot hold all vectors in memory?

Why this is a concrete problem for us at Oracle:

• We process large corpora (hundreds of thousands to millions of texts).
• Current Python embed() batching still accumulates all response objects and merged embeddings before returning.
• For 1,000,000 texts with 1024-dim float vectors, the raw float payload alone is ~3.8 GiB (before Python object overhead), which is enough to cause memory pressure/OOM in real pipelines.
• We need incremental consumption so we can persist each embedding as it arrives.

If cross-SDK consistency is the blocker, sharing the acceptance criteria or roadmap would help us contribute in the format you want.