What models do you prefer in Vault QA since ob-copilot 2.7, GPT4o, Claude, or Ollama?

[LeCheenaX]

Background

Since the release of 2.7.0, this plugin supports Ollama as one of the local index providers. However, it is not sure that when it comes to Vault QA, which model has the best behavior that satisfies your need.

Core Issue

To compare the pros and cons of different AI models that best suit the daily assistance in Obsidian.

Analysis

Personally, I would like to give few questions to the AI models to test their feedbacks, but I am not sure if the result will be different in other vaults. Below are few examples of my questions based on gpt4o-mini:

• Task Management
• Progress Tracking
• Content Linking
• Log Tracing

Example Q&As

Task Management

Question:
What are the incompleted todos in my vault? Please classify them with following rules:
If there's nothing inside the [], the task is incompleted;
If there's a "-" symbol inside the [], the task is canceled;
If there's a "<" symbol inside the [], the task is postponed;

Answer:
Based on your request and the classification rules provided, here is the comprehensive list of uncompleted tasks in your vault, categorized accordingly:

Incompleted Tasks: []

These tasks are marked as uncompleted and do not have any cancellation or postponement indicators:

1. CPP Learning
2. Finish the Obsidian Basic Knowledge
3. Compatibility on iPad OS
4. Meeting Log - Convenient Buttons based on QuickAdd in the log file
5. Cannot modify block background color sometimes unless reload the plugin
6. GPT for meeting summary generation
7. Test the home page and configure the home page to satisfy:
   • How to array forwarded tasks at the bottom #github
   • Test the Compatibility of Folder note with existing notes
8. 22 #notodo
9. 33 #notodo
10. 44

Canceled Tasks: [-]

These tasks are marked as canceled and contain a "-" in their brackets:

1. Custom icons to replace scheduled date with forwarded (postponed)
2. Test the Compatibility of Folder note with existing notes
3. GPT for meeting summary generation

Postponed Tasks: [<]

These tasks are marked as postponed and contain a "<" in their brackets:

1. Configure Older tasks as an individual page
2. Finish the long-term update on OB Basic knowledge.
3. Convenient Buttons based on QuickAdd in the log file

Summary

• Incompleted Tasks: 10 tasks
• Canceled Tasks: 3 tasks
• Postponed Tasks: 3 tasks

This classification should help you manage your tasks more effectively by clearly distinguishing between what is still pending, what has been canceled, and what has been postponed. If you need further assistance or additional classifications, feel free to ask!

Sources:

• What_are_the_total_note_counts_in_my_vault@20241024_162855
• what_are_the_count_of_characters_in_my_home.md@20241024_161854
• Obsidian ToDos
• Task Classification
• Issues · Leonezzobsidian-tasks-calendar-wrapper
• Test Kanban Tasks

Review: ⭐⭐⭐⭐
The tasks are 100% listed but the classification is mixed. Only the following tasks are incompleted:

• [] CPP Learning
• [] Finish the Obsidian Basic Knowledge
• [] Compatibility on iPad OS

The other tasks are either canceled or rescheduled, but the ai agent did not recognize, for example:

• [<] 22 #notodo (Postponed)
• [-] 33 #notodo (Canceled)
• [>] 44 (Rescheduled)

Progress Tracking

Question:
Please check the note [[cpp overview]], there are completed todos and uncompleted todos. Based on the completion date of my completed todos in cpp overview only, can you estimate when will I finish all the uncompleted todos in cpp overview?

Answer:
To estimate the completion date for the uncompleted tasks in the note [[Cpp Overview]], we can analyze the completed tasks and their completion dates.

Completed Tasks

All completed tasks in the note were marked as completed on September 25, 2024. This includes tasks from Chapter 0 and Chapter 1, which cover foundational concepts in C++.

Uncompleted Tasks

The uncompleted tasks span multiple chapters, including Chapters 2 through 28, with a total of over 100 tasks remaining.

Estimation Approach

1. Rate of Completion: If we assume that the rate of completion for the completed tasks is consistent, we can calculate the average time taken to complete each task.
2. Time Frame: The completed tasks were all finished on the same day, indicating a concentrated effort. However, for estimation, we should consider a more realistic pace for the remaining tasks.

Estimation Calculation

• Total Completed Tasks: 13 tasks (from Chapters 0 and 1).
• Completion Date for Completed Tasks: September 25, 2024.
• Remaining Tasks: Approximately 100+ tasks (from Chapters 2 to 28).

Assuming a steady pace of completing tasks, if you were to complete, for example, 5 tasks per week, it would take around 20 weeks to finish all remaining tasks.

Estimated Completion Date

• Starting Point: September 25, 2024.
• Duration: 20 weeks (5 tasks per week).
• Estimated Completion Date:

Calculating 20 weeks from September 25, 2024, leads us to approximately January 15, 2025.

Conclusion

If you maintain a steady pace of completing about 5 tasks per week, you can expect to finish all uncompleted tasks in the note [[Cpp Overview]] by January 15, 2025. Adjustments can be made based on your actual pace of work or any changes in your schedule.

Sources:

• Cpp Overview

Review: ⭐⭐⭐⭐
The reply does understand what I mean, but the result is inaccurate. My latest completion date on the todos is 2024-10-17, and I have chapter 2 half completed. However, the reply only covers chapter 0 and chapter 1 that is 100% completed, which is not accurate and understandable enough.

Deleted:

Review: ⭐⭐⭐⭐⭐
The reply 100% gets what I mean and gives me a wonderful result. I will rate ⭐⭐⭐⭐⭐ for progress tracking.

Content Linking

Question:
Can you summarize the key points mentioned in all my notes regarding P.A.R.A. and suggest any related topics I should explore?

Answer:
Based on your notes regarding the P.A.R.A. (Projects, Areas, Resources, Archive) organizational method, here are the key points summarized:

Key Points of P.A.R.A.

1. Projects:

   • Defined as short-term endeavors with specific goals and deadlines.
   • Once completed, they require minimal maintenance and often result in tangible outputs.
   • Examples include personal projects, event records, and summaries.

2. Areas:

   • Refers to ongoing responsibilities or interests that require regular maintenance.
   • These do not have a defined end date and can evolve over time.
   • Areas can include skills or experiences that may be useful in the future.

3. Resources:

   • Comprises materials or topics of interest that may be useful later.
   • This includes bookmarks, PDFs, and other reference materials.
   • Resources are typically less structured and can be tagged for easy retrieval.

4. Archive:

   • Represents the storage for completed projects, outdated areas, or resources that are no longer relevant.
   • Items in the archive are not actively maintained and can include past projects or knowledge that is no longer needed.

Related Topics to Explore

1. Task Management Techniques:

   • Explore methods like GTD (Getting Things Done) or Kanban to enhance your project management skills.

2. Knowledge Management Systems:

   • Investigate how to implement a personal knowledge management system using tools like Notion or Roam Research.

3. Digital Organization Tools:

   • Look into various digital tools and plugins that can help streamline your P.A.R.A. implementation in Obsidian.

4. Productivity Strategies:

   • Research productivity techniques such as the Pomodoro Technique or time blocking to improve your efficiency in managing projects and areas.

5. Mind Mapping:

   • Consider using mind mapping tools to visually organize your projects and areas, which can aid in brainstorming and planning.

6. Continuous Learning:

   • Explore resources for lifelong learning, such as online courses or webinars, to keep your areas of interest updated.

7. Review and Reflection Practices:

   • Implement regular review sessions to assess your projects and areas, ensuring they align with your goals and interests.

By delving into these related topics, you can enhance your understanding and application of the P.A.R.A. method, ultimately improving your organization and productivity in both personal and professional contexts.

Sources:

• P.A.R.A
• 笔记分类方法 （卡片盒笔记法串CODE工作流）
• Obsidian ToDos
• Task Classification

Review: ⭐⭐⭐⭐⭐
The summarization is cool. It indeed give a reference of all the notes that has backlinks or links to the main note regarding P.A.R.A.

Log Tracing

Q:
Please review the "Daily Notes" folder, and check when did I play Black Myth Wukong.

A:
In August 2024, you played "黑神话悟空" (Black Myth Wukong) on the following dates:

August 14, 2024

• You recorded testing "黑神话悟空" with two attached images.

August 21, 2024

• You noted down items related to "黑神话悟空", including references to various locations and tasks within the game.

August 25, 2024

• You mentioned testing "泡酒物" (a type of item) in relation to "黑神话悟空."

These entries indicate your engagement with the game on these specific dates in August.

Sources:
2024-08-21
2024-08-14
2024-08-20
2024-08-22
2024-09-14
2024-08-23
Monthly Notes
2024 - 09
2024 - 08
2024-08-25

Review: ⭐⭐⭐⭐⭐
It actually helps to find the accurate dates that I recorded my experience in daily notes for the video game, so I would give it ⭐⭐⭐⭐⭐.

Summary

Based on the personal rating on following questions:

• Task Management ⭐⭐⭐⭐
• Progress Tracking ⭐⭐⭐⭐
• Content Linking ⭐⭐⭐⭐⭐
• Log Tracing ⭐⭐⭐⭐⭐

The model gpt4o-mini is powerful but not good at tasks analyzing. The respnding speed is also limited when it comes to complex questions.

[logancyang]

1. This is not an issue and should be in discussion.
2. Ollama has long been a provider, it's not added in 2.7.0.
3. Model selection and combination can vary quite a lot. The quality depends heavily on the chat model, the embedding model's impact is not as significant as the chat model.

A good point to start is 4o-mini with an adequate embedding model (ollama nomic embed text is adequate for English). If it's giving you bad quality results, try 4o. If it's still bad, switch to a better embedding model. You can refer to the MTEB leaderboard.

I know not everyone has the time to experiment with a ton of combos for the best result. That's why I introduced Copilot Plus mode. For those who are interested, please sign up the waitlist! https://www.obsidiancopilot.com/