You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Enable chatbot to search bipolar research docs #345
I referenced this article and updated the prompt to provide the URLs of three reputable sources.
If the source came from Balancer's database, say "Balancer's sources:"
If the source came from the internet, say "External sources:"
@StubberG3 I'm working on enabling the chatbot to use Balancer's sources in #345 and @taichan03 closed the PR #247 associated with this ticket so I'm going to mark this ticket as "Done" and drop a link to it in #345 so we can use the work as needed
I do have some thoughts on the instructions you provided, Sahil. To be honest, I asked ChatGPT to look for holes in the logic and make suggestions. Chat had lots to say, which I'll add the output as code after this comment.
I am still reading the suggestions myself, so I'm not asking you to make any changes before I suggest updates. We need definitely need to be sure that the Balancer "chunking" follows Chat's advice.
Anyone have some time tomorrow night, Monday, Oct 11? I can talk around 5pm. If everyone is busy, then I'll just bring this up at the in-person meeting.
ChatGPT suggestions for improving Balancer instructions:
There are several subtle but important logic gaps and potential failure points you’ll want to tighten before deploying AskJJ. Here’s a structured review of the weaknesses and what could go wrong at runtime or in testing:
🧠 1. Ambiguity in “semantic search” and scope of query expansion
Issue:
The instruction says “use conceptually related terms and synonyms” but gives no rule for how far to expand.
This can easily lead to:
Over-broad retrieval (pulling unrelated psychiatric or neurological topics like schizophrenia or ADHD).
Over-narrow retrieval if the embedding model is weak on domain context.
Fix:
Add explicit boundaries:
“Expand to closely related terms (e.g., ‘mania,’ ‘hypomania,’ ‘mood stabilization,’ ‘antidepressant-induced switching’) but avoid tangential disorders (e.g., schizophrenia, ADHD) unless explicitly mentioned.”
🩺 2. Lack of hierarchy for conflicting or duplicate findings
Issue:
No rule for what to do when multiple articles give contradictory information (e.g., lithium vs. lamotrigine first-line treatment).
The AI might blend them or cherry-pick one arbitrarily.
Fix:
Add a tie-breaking or synthesis rule:
“If multiple sources disagree, summarize each perspective with its citation rather than merging them into a single answer.”
📚 3. Unclear document-retrieval fallback path
Issue:
“If no relevant information is found… state that it’s not available.”
But what if partial matches are found (e.g., tangentially related to the question)?
There’s no rule for thresholding or confidence.
Fix:
Specify tiers:
“If only tangential results are found, preface the answer with ‘No direct match was found; however, related findings include…’”
🧩 4. Citation formatting vs. page detection logic
Issue:
Citations assume every document has a {name} and {page_number}—which might not exist in JSON, TXT, or PDF-chunked text.
This can lead to null or malformed references.
Fix:
Add fallback logic:
“If page number metadata is unavailable, cite using document title or section heading only.”
⚖️ 5. No explicit instruction on how many references or search results to use
Issue:
AskJJ might:
Return one strong citation (under-supported)
Or dump 10 weakly related ones (overload)
Fix:
“Use the top 3–5 semantically closest passages unless one fully answers the question.”
🔍 6. No rule for multi-part questions
Issue:
If a user asks: “How does lamotrigine work and what are its side effects?” the model may search only one part.
There’s no instruction to decompose complex questions.
Fix:
“Decompose multi-clause questions into sub-queries and perform separate searches for each key topic before composing a final response.”
⚠️ 7. No handling for hallucination control
Issue:
It says “only provide answers based on information found,” but doesn’t explicitly forbid reasoning beyond those facts.
Without that reinforcement, the LLM might interpolate or infer medically unsafe information.
Fix:
“Do not infer or extrapolate beyond the retrieved text. If evidence is incomplete, state the limitation explicitly.”
Perfect — got it. Here’s the continuation in that same clear, structured tone and hierarchy:
---
🗂️ **8. Missing metadata normalization rule (continued)**
**Issue:**
Search results may contain inconsistent metadata (`"title"`, `"source"`, `"page_num"`, etc.). Without a normalization rule, the model may output mismatched or null fields, leading to broken citations or even hallucinated file names.
**Fix:**
Add a preprocessing step or explicit guidance:
“Normalize all retrieved document metadata before generating citations. If metadata fields are missing, replace them with consistent placeholders such as ‘[Unnamed Source]’ or ‘[No Page Number]’ rather than fabricating values.”
---
🧭 **9. No guidance for context carryover across turns**
**Issue:**
If a user asks follow-up questions (“What about side effects?” right after “How does lithium work?”), AskJJ may lose reference to the previous topic and perform an unrelated search.
**Fix:**
Include continuity logic:
“When the user’s new query appears contextually linked to the prior one, retain the previous search context (e.g., medication name or concept) and merge results unless the user explicitly changes topics.”
---
💬 **10. No differentiation between document-based and general questions**
**Issue:**
AskJJ is told to *always* use `search_documents`, even for unrelated prompts like “Who built this chatbot?” or “What does serotonin do in general?”
This can waste compute or return irrelevant text.
**Fix:**
Add a quick intent gate:
“Before calling `search_documents`, check if the question likely pertains to the uploaded corpus. If not, explain that only uploaded documents are searchable.”
---
🧮 **11. No scoring threshold or quality check for weak matches**
**Issue:**
A semantic search may return fragments with low similarity scores (e.g., 0.45 cosine similarity). The model may still treat them as evidence.
**Fix:**
“Use a confidence threshold (e.g., similarity ≥ 0.7) to filter weak matches. If no passages exceed the threshold, state clearly that no relevant evidence was found.”
---
📑 **12. Unclear synthesis hierarchy (micro vs. macro evidence)**
**Issue:**
Retrieved snippets can differ in granularity—some describe mechanisms (“lamotrigine inhibits glutamate release”), others summarize outcomes (“effective for bipolar depression”). AskJJ may merge them without signaling that they represent different levels of evidence.
**Fix:**
“Group evidence by type (mechanism / clinical / guideline) before synthesis. Present each group in its own paragraph or bullet section.”
---
💡 **13. No defined behavior for conflicting temporal evidence**
**Issue:**
Older sources may contradict newer ones (e.g., early vs. modern understanding of rapid cycling). There’s no instruction to prefer recent evidence.
**Fix:**
“Prefer the most recent publication dates when conflicts arise, and note when older findings differ.”
---
🧱 **14. Incomplete fallback behavior when no documents are uploaded**
**Issue:**
The model has no defined response if the user hasn’t uploaded any documents yet. That could cause silent failure or a null search.
**Fix:**
“If no documents are available for search, return: ‘No documents are uploaded. Please upload your research files before asking medical questions.’”
---
🚦 **15. No clarity about uncertainty or evidence limits**
**Issue:**
Without explicit uncertainty signaling, AskJJ might sound overconfident—especially dangerous for clinical content.
**Fix:**
“When evidence is partial or indirect, preface statements with qualifiers like ‘The uploaded documents suggest…’ or ‘Evidence in your corpus is limited regarding…’.”
---
**See table of suggestions below**
✅ **Overall takeaway:**
AskJJ’s base instructions are solid for a first iteration but need **guardrails around scope, confidence, context continuity, and evidence synthesis** to ensure reliability under real clinical-knowledge testing.
Would you like me to convert this list into a **clean “AskJJ Prompt v2” draft**—ready to paste back into your Python code block?
There is also a clean version of Python code instructions from CHAT I've downloaded. But I don't want to confuse things and share it early without reviewing the updates with humans first.
Sorry, Sahil and Sarah. My cold has now put me in the zombie stage. The meeting today at 1pm won/t be productive. BUT! I am getting more comfortable working in code again since I am reskinning my portfolio site. When my brain cells are back, I can pull down a copy of the repo and see if I can make it work. I can test it locally using the Python AI prompt changes and give you a report. I absolutely am not comfortable PUSHING to the repo -- I'll share my code conclusions with you, Sahil, to do the remote repo changes .
Copying this over to #325 in the KanBan so Sarah can know I can't make it.
I'm going to drop the ChatGPT produced improved rules (in Python) for the AI queries for Balancer.
Sahil, please do what is necessary to put this in a new separate issue? or branch where I can test the new rules against the old rules (using the questions Sarah created). I do not trust Chat enough to push its code to the final repo without testing.
Do I need to write a new KanBan issue for this request?
I'm also putting this note in Slack for speed of communication. Code snippet to follow in next comment.
`INSTRUCTIONS = """
You are AskJJ — an AI assistant that helps users find and understand information about bipolar disorder
from their uploaded research documents using semantic search.
🔎 SEMANTIC SEARCH STRATEGY
• When a user’s question likely relates to the uploaded documents, perform a semantic search using search_documents.
• Expand search terms with close synonyms and related concepts (e.g., “mania,” “hypomania,” “mood stabilization,” “antidepressant-induced switching”).
• Avoid tangential or unrelated disorders (e.g., schizophrenia, ADHD) unless explicitly mentioned.
• Handle both clinical and lay language (brand/generic names, abbreviations, symptoms, treatments).
• Use top 3–5 most semantically relevant passages (similarity ≥ 0.7). Filter out weak matches.
📂 FUNCTION USAGE & CONTEXT
• If no documents are uploaded, respond: “No documents are available. Please upload your research files before asking medical questions.”
• If the user’s question is unclear or multi-part, briefly clarify or split into sub-queries before searching.
• Retain prior topic context across consecutive related turns (e.g., continuing discussion about the same medication).
• Do not search for greetings, meta-questions (“How do you work?”), or off-topic queries.
🧠 EVIDENCE HANDLING & SYNTHESIS
• Base all answers ONLY on text retrieved from the documents. Never infer or extrapolate beyond retrieved content.
• When multiple sources disagree, summarize each perspective separately with its citation.
• Prefer newer or guideline-level evidence when conflicts occur. Note if older sources differ.
• Group evidence by type when relevant (mechanistic / clinical / guideline) for clarity.
• If no direct match is found but related material exists, preface: “No direct match was found; however, related findings include…”
• If no relevant results exist, clearly state: “This information does not appear in your uploaded documents.”
📑 CITATION RULES
• Include citations after each supported statement using this format: [Name {name}, Page {page_number}]
• If page metadata is missing, use a stable alternative such as: [Name {name}, Section {section}] or [Name {name}, Paragraph {para}]
• Never fabricate names or page numbers.
• Normalize all metadata fields before citing; if missing, use consistent placeholders (e.g., [Unnamed Source]).
💬 RESPONSE FORMAT
Structure responses in this consistent format:
Direct Answer — concise summary addressing the user’s question.
Evidence Summary — short paragraphs or bullets with supporting facts and citations.
Notes & Limitations — describe conflicts, uncertainty, or corpus limits.
If no relevant data is found:
→ “This information is not available in your uploaded documents.”
⚠️ SAFETY & SCOPE LIMITS
• Do not provide medical advice, dosing guidance, or individualized treatment recommendations.
• Provide educational information only.
• If the user appears to be in crisis, include a standard crisis resource response.
🧩 ERROR & QUALITY CONTROL
• If search_documents fails or returns empty, state that the information was not found and invite the user to refine or re-upload.
• Avoid hallucinated text, fabricated references, or excessive speculation.
• Maintain a professional, medically neutral tone at all times.
"""
`
Hi, Sue,
These generally are better. Problem is with the one on Abilify. This medication and other similar have a black box warning against use in the elderly (causes higher mortality.). That is apparently not in our documents.
This sort of solidifies the need for me to editing and offering up the documentation on side effects/contraindications.
Overall though, the responses were better than the originals!
Hi Sarah
Sahil and I have a meeting to do some testing with some new, enhanced Python instructions for Balancer tomorrow at 5pm. Hopefully the new instructions won't break the code (make the Balancer Repo on Sahil's laptop unusable) and will give us better results than the old instructions. Wish us luck!
Also if you want to attend let us know! But its going to be tedious even if everything goes right.
Hey Sahil, we can skip the idea for setting up the repo on my computer (as I suggested on Slack). I have an easier workflow.
When we meet online, I will train you to run the AskJJ prompts on your local repo and then add the responses to the Balancer Google Sheets.
(Side note: We have to make sure you have editing/writing permissions on the Balancer Google account.)
When you are done, I will then PASS/PARTIAL/FAIL the responses and discuss the results with Sarah and you.
Then we can decide if the new Python instructions are better than the old ones.
I'm rerunning all the prompts in the correct order for a baseline comparison to the new instructions.
That baseline will be ready when we have a chance to talk, hopefully this week on Tuesday.
Here is the copy of this Slack note for #345 kanban for reference.
Activity
[-]Reuse "Ask all papers" endpoint for the chatbot[/-][+]Enable chatbot to search bipolar research docs [/+]sahilds1 commentedon Aug 19, 2025
This ticket relates to
sahilds1 commentedon Aug 19, 2025
Cross posting form #211 for viz
@StubberG3 I'm working on enabling the chatbot to use Balancer's sources in #345 and @taichan03 closed the PR #247 associated with this ticket so I'm going to mark this ticket as "Done" and drop a link to it in #345 so we can use the work as needed
Merge pull request #364 from sahilds1/345-add-rag-to-the-chatbot
12 remaining items
sue-hntr commentedon Oct 12, 2025
@sahilds1 @SarahSorlien
I do have some thoughts on the instructions you provided, Sahil. To be honest, I asked ChatGPT to look for holes in the logic and make suggestions. Chat had lots to say, which I'll add the output as code after this comment.
I am still reading the suggestions myself, so I'm not asking you to make any changes before I suggest updates. We need definitely need to be sure that the Balancer "chunking" follows Chat's advice.
Anyone have some time tomorrow night, Monday, Oct 11? I can talk around 5pm. If everyone is busy, then I'll just bring this up at the in-person meeting.
sue-hntr commentedon Oct 12, 2025
ChatGPT suggestions for improving Balancer instructions:
There are several subtle but important logic gaps and potential failure points you’ll want to tighten before deploying AskJJ. Here’s a structured review of the weaknesses and what could go wrong at runtime or in testing:
sue-hntr commentedon Oct 12, 2025
sue-hntr commentedon Oct 12, 2025
There is also a clean version of Python code instructions from CHAT I've downloaded. But I don't want to confuse things and share it early without reviewing the updates with humans first.
sue-hntr commentedon Oct 17, 2025
@SarahSorlien @sahilds1 copied from Slack message
Sorry, Sahil and Sarah. My cold has now put me in the zombie stage. The meeting today at 1pm won/t be productive. BUT! I am getting more comfortable working in code again since I am reskinning my portfolio site. When my brain cells are back, I can pull down a copy of the repo and see if I can make it work. I can test it locally using the Python AI prompt changes and give you a report. I absolutely am not comfortable PUSHING to the repo -- I'll share my code conclusions with you, Sahil, to do the remote repo changes .
Copying this over to #325 in the KanBan so Sarah can know I can't make it.
sue-hntr commentedon Oct 19, 2025
@sahilds1 @SarahSorlien
Hey Sahil!
I'm going to drop the ChatGPT produced improved rules (in Python) for the AI queries for Balancer.
Sahil, please do what is necessary to put this in a new separate issue? or branch where I can test the new rules against the old rules (using the questions Sarah created). I do not trust Chat enough to push its code to the final repo without testing.
Do I need to write a new KanBan issue for this request?
I'm also putting this note in Slack for speed of communication. Code snippet to follow in next comment.
sue-hntr commentedon Oct 19, 2025
`INSTRUCTIONS = """
You are AskJJ — an AI assistant that helps users find and understand information about bipolar disorder
from their uploaded research documents using semantic search.
🔎 SEMANTIC SEARCH STRATEGY
• When a user’s question likely relates to the uploaded documents, perform a semantic search using search_documents.
• Expand search terms with close synonyms and related concepts (e.g., “mania,” “hypomania,” “mood stabilization,” “antidepressant-induced switching”).
• Avoid tangential or unrelated disorders (e.g., schizophrenia, ADHD) unless explicitly mentioned.
• Handle both clinical and lay language (brand/generic names, abbreviations, symptoms, treatments).
• Use top 3–5 most semantically relevant passages (similarity ≥ 0.7). Filter out weak matches.
📂 FUNCTION USAGE & CONTEXT
• If no documents are uploaded, respond: “No documents are available. Please upload your research files before asking medical questions.”
• If the user’s question is unclear or multi-part, briefly clarify or split into sub-queries before searching.
• Retain prior topic context across consecutive related turns (e.g., continuing discussion about the same medication).
• Do not search for greetings, meta-questions (“How do you work?”), or off-topic queries.
🧠 EVIDENCE HANDLING & SYNTHESIS
• Base all answers ONLY on text retrieved from the documents. Never infer or extrapolate beyond retrieved content.
• When multiple sources disagree, summarize each perspective separately with its citation.
• Prefer newer or guideline-level evidence when conflicts occur. Note if older sources differ.
• Group evidence by type when relevant (mechanistic / clinical / guideline) for clarity.
• If no direct match is found but related material exists, preface: “No direct match was found; however, related findings include…”
• If no relevant results exist, clearly state: “This information does not appear in your uploaded documents.”
📑 CITATION RULES
• Include citations after each supported statement using this format:
[Name {name}, Page {page_number}]
• If page metadata is missing, use a stable alternative such as:
[Name {name}, Section {section}] or [Name {name}, Paragraph {para}]
• Never fabricate names or page numbers.
• Normalize all metadata fields before citing; if missing, use consistent placeholders (e.g., [Unnamed Source]).
💬 RESPONSE FORMAT
Structure responses in this consistent format:
If no relevant data is found:
→ “This information is not available in your uploaded documents.”
• Do not provide medical advice, dosing guidance, or individualized treatment recommendations.
• Provide educational information only.
• If the user appears to be in crisis, include a standard crisis resource response.
🧩 ERROR & QUALITY CONTROL
• If search_documents fails or returns empty, state that the information was not found and invite the user to refine or re-upload.
• Avoid hallucinated text, fabricated references, or excessive speculation.
• Maintain a professional, medically neutral tone at all times.
"""
`
SarahSorlien commentedon Oct 22, 2025
Hi, Sue,
These generally are better. Problem is with the one on Abilify. This medication and other similar have a black box warning against use in the elderly (causes higher mortality.). That is apparently not in our documents.
This sort of solidifies the need for me to editing and offering up the documentation on side effects/contraindications.
Overall though, the responses were better than the originals!
sue-hntr commentedon Oct 22, 2025
@SarahSorlien @sahilds1
Hi Sarah
Sahil and I have a meeting to do some testing with some new, enhanced Python instructions for Balancer tomorrow at 5pm. Hopefully the new instructions won't break the code (make the Balancer Repo on Sahil's laptop unusable) and will give us better results than the old instructions. Wish us luck!
Also if you want to attend let us know! But its going to be tedious even if everything goes right.
SarahSorlien commentedon Oct 22, 2025
sue-hntr commentedon Oct 26, 2025
@sahilds1 @SarahSorlien
Hey Sahil, we can skip the idea for setting up the repo on my computer (as I suggested on Slack).
I have an easier workflow.
When we meet online, I will train you to run the AskJJ prompts on your local repo and then add the responses to the Balancer Google Sheets.
(Side note: We have to make sure you have editing/writing permissions on the Balancer Google account.)
When you are done, I will then PASS/PARTIAL/FAIL the responses and discuss the results with Sarah and you.
Then we can decide if the new Python instructions are better than the old ones.
I'm rerunning all the prompts in the correct order for a baseline comparison to the new instructions.
That baseline will be ready when we have a chance to talk, hopefully this week on Tuesday.
Here is the copy of this Slack note for #345 kanban for reference.
sahilds1 commentedon Oct 27, 2025
I'm marking this ticket as "Done" b/c the chatbot's responses are better after enabling it to search the bipolar research docs:
#345 (comment)