Support OpenAI/Anthropic compatible APIs for broader LLM choice

[lehuygiang28]

Is your feature request related to a problem?

Yes. The project is currently vendor-locked to Cloudflare Worker AI and Google Gemini. This rigidly restricts users from utilizing other LLM providers, open-source models, or self-hosted solutions.

Describe the solution you'd like

Support standard API formats (OpenAI-compatible, Anthropic, Gemini) as open options. Provide configuration fields allowing users to define:

1. Custom Base URL (Endpoint)
2. API Key
3. Custom Model Name

This decoupling enables users to connect any compatible service (e.g., Groq, DeepSeek, Together AI) or local setups (e.g., Ollama, vLLM) effortlessly.

Describe alternatives you've considered

• Hardcoding new providers: Manually adding support for Groq, Mistral, etc., one by one. This is unscalable and bloats the codebase.
• Requiring external proxies: Forcing users to set up middleware (like LiteLLM) to translate requests into the currently supported formats. This adds unnecessary infrastructure complexity.

Additional context

No response

[devarshishimpi]

Thanks for opening this. I agree with the direction here.

The current implementation is too tightly coupled to Cloudflare Workers AI and Gemini. The better long-term shape is not to hardcode more providers one by one, but to migrate model/provider configuration into a standard provider system.

I’m thinking this should support a small set of API formats:

• OpenAI-compatible
• Anthropic-compatible
• Gemini-compatible
• Cloudflare Workers AI as a native option

Users would configure providers from the dashboard with:

• Provider name
• API format
• Custom base URL / endpoint
• API key
• Custom model name
• Per-model rate/token limits

That means services like Groq, DeepSeek, Together AI, vLLM, LM Studio, and Ollama can work without adding a separate hardcoded integration for each vendor.

Rather than keeping the current Gemini/Cloudflare setup as a parallel legacy path, we should migrate the existing defaults into the new provider/model config format. Platform-level secrets would remain environment variables, but LLM provider credentials should live in the dashboard and be stored securely, likely encrypted with an app-level encryption key.

I’d be very open to a contribution here if you’re interested in working on it. A good first version would probably include:

• A provider/model config schema and migration
• An OpenAI-compatible adapter
• Dashboard provider setup fields
• A “test connection” action
• Migration of the existing Gemini and Cloudflare defaults into the new format

Happy to help review an approach before you start a PR.

[devarshishimpi]

Self assigning this, hopefully this should be live in a week.