[FEATURE]: Proposal for Modular Architecture, Async Worker Pipeline and Hybrid Search Foundation

[VanshKaushal]

Feature and its Use Cases

🚀 Feature Proposal

While exploring the SmartNotes repository, I noticed that the current project is in an early architectural stage and many advanced AI features such as hybrid search, agent automation, and scalable indexing will require stronger system boundaries.

I would like to propose introducing a modular architecture direction focused on enabling future scalability and contributor productivity.

🔧 What is the feature?

This proposal suggests introducing foundational system enhancements including:

• Monorepo-based modular package structure (apps + core packages)
• Vault abstraction layer with pluggable storage adapters
• Event-driven note lifecycle system
• Asynchronous background worker pipeline for AI indexing
• Hybrid search foundation (keyword + semantic readiness)
• Basic observability layer for indexing/search debugging
• Minimal frontend shell for demonstrating end-to-end system flow

👥 Who would benefit from it?

• Contributors — easier feature development due to clear module boundaries
• Users — improved performance due to async indexing and scalable search
• Maintainers — better system maintainability and extensibility
• Future GSoC contributors — clear architecture entry points

📌 What scenarios would this address?

• Prevent API request blocking during heavy AI processing
• Enable addition of new AI agents without core refactors
• Support future vector search integration and ranking improvements
• Allow introduction of distributed workers in future
• Improve debugging through structured logging and metrics

Additional Context

I have been experimenting locally with a production-oriented architectural prototype implementing some of these ideas (worker service, vault abstraction, hybrid search foundation).

The intention is NOT to introduce a large refactor immediately, but to understand:

1. Whether this architectural direction aligns with the project roadmap
2. Which subsystem would be most valuable to contribute first (worker / vault / search)
3. Preferred scope and size for initial pull requests

I would be happy to break this work into smaller focused contributions based on maintainer feedback.

Looking forward to guidance 🙂

Code of Conduct

• I have joined the Discord server and will post updates there
• I have searched existing issues to avoid duplicates