Beta Test Version 1.5
last updated December 19, 2023
Major themes for this release
- Expanded referenced content
- Improved chat experience
- Faster feedback mechanism
Expanded referenced content
In addition to all sources used in the construction of the underlying evidence graph, the knowledge base has been expanded to contain the great majority of the textual portions of the Merck Manuals, Professional Edition. (Tables, figures, images, and other multimedia are slated to be added in a subsequent release.) This brings the coverage of conditions that one can ask questions about to approximately 2,500.
Improved chat experience
In addition to being able to answer clinical questions directly from our knowledge base, the Copilot chat is now capable of answering questions relating to the clinical note and the differentials the clinical note generates. Answers relating to clinical questions are now more concise and and more often displayed in a list oriented format. This is especially true when questions regarding signs, symptoms, etiologies, diagnoses, and treatments are not referenced with patient information.
In the event that the knowledge base does not contain a reference that is specific to the question presented, the chat will fall back to a large language model (LLM) that has been specifically trained on medical information. This is indicated by the lack of a reference for the response.
Beta Test Version 1.5 uses Anthropic’s Claude LLMs for analyzing referenced documents and synthesizing responses to user questions. Please note that while Claude is advanced in processing and generating information, its outputs are AI-generated and may not always reflect the most current data or specialized expert insights, particularly when there are no references found.
The mouse-over highlights that used to appear within the answer text have been removed and the references for a given item have been coalesced to superscripts at the end of lines for easier identification. Mousing over these superscripts still brings up the reference text and appropriate links.
Each chat response now contains the provision for an easy to use feedback mechanism. More about this below.
The maximum conversation length has been significantly extended and can be extended further if needed. It is now far less likely to receive a “Please start a new chat session” message.
Faster feedback mechanism
In addition, the Copilot's standard feedback mechanism, located in the upper right hand corner of the page, we now provide a quick feedback on individual chat responses. This feedback plays a key role in the speed to which we can improve responses and address concerns.
Each response is followed by a thumbs up and a thumbs down icon . Clicking the thumbs up icon indicates that you agree with the response. Clicking the thumbs down icon indicates that you have an issue with the response. Selections are indicated by a circle around the icon. You may change a selection at any time. We log all responses for analysis.
Should you wish to add additional detail about why you responded in a particular manner, you may do so via the standard feedback mechanism.
While both positive and negative feedback help our analyses, negative feedback is more important for our ability to improve the responses.
Known issues with the chat feature
The following issues are known to our development team and we are in the process of addressing them:
- Speed - Chat responses may take a few seconds. As we scale up our processing infrastructure this will continue to improve.
- Differentials reported by the chat with respect to the patient note may not completely align with those derived analytically using the knowledge base (the DDxs shown in the Differentials Pane, the Copilot's right hand column).
- Non-clinical questions are not answered inline but are routed to our FAQs, which appear in a popup window.
- Occasionally, some clinical questions, especially those that are hard to interpret the meaning of, may refer to our FAQs - please try rephrasing these questions.
- In some instances, the reference links may point to content that is indirectly related to the response provided.
- A reference superscript may be presented but the popup for the underlying content and link may fail to appear.
- Radical shifting of question context, e.g. from Ectopic pregnancy to Meniscal tears, or ambiguity can lead to unexpected responses - in this event please try asking your question again, perhaps also rephrasing it.
- Giving the chat contradictory information can confuse it.