Build a self-improving answer bot
Learn how to build a self-improving answer bot. Questions go in. Useful answers come out. Every miss becomes a clue about what the system should learn next.

Someone asks a question in Slack. You know the answer exists somewhere. Maybe it’s in a meeting note, an out-of-date wiki, or—worst case scenario—it lives in one person’s head.
Eventually someone finds the answer, shares it, and everyone moves on. Great for now, but the answer stays in the thread, the knowledge isn’t captured, and the question comes back again next week.
This guide turns that cursed loop into a Self-improving answer bot. Questions go in. Useful answers come out. Every miss becomes a clue about what the system should learn next.
You’ll build it in two parts:
(Part 1) The Q&A Agent — Use Notion’s guided setup to create your Q&A Agent. Once it’s ready, review the Q&A Log, Learning Log, and Playbook created behind the scenes.
(Part 2) The Self-Improving Agent — Follow this cookbook to build an agent that reviews the Q&A Log and trusted sources. The agent then drafts targeted updates and applies them only after human approval.
By the end, every question your team asks will make the next answer better.
Build time: About 90 minutes once prerequisites are confirmed. Allow additional time for testing and troubleshooting.
Reading time: About 45 minutes.
Q&A Log | Q&A Agent | Self-Improving Agent |
|---|---|---|
Captures new and missing knowledge, proposed changes, and update status. Links to changes in trusted sources. | Answers questions in Slack, using trusted sources. Records new, missing, or conflicting information in the Q&A Log. | Proposes focused updates to trusted sources based on Q&A Log entries. Applies approved changes after human review. |
Start with the questions your team is tired of answering. The ones that send someone digging through Slack and scattered tools or tagging in the same expert over and over again. That’s the loop you’ll fix.
Here are a few common examples:
IT
#it-help, #access-requests
Example questions
How do I connect to the VPN?
Which software is approved for screen recording?
How do I set up a new laptop?
Trusted sources
Setup guides
Troubleshooting pages
Supported-software policies
Known issues
Product
#product-questions, #roadmap-help
Example questions
What changed in the latest release?
What did we decide about this feature?
Is this feature available to Enterprise customers?
Trusted sources
PRDs and decision logs
Project databases
Meeting notes
Issue trackers
People
#people-help, #new-hire-questions
Example questions
How do I enroll in benefits?
What is our parental leave policy?
Where can I find the new-hire checklist?
Trusted sources
Employee handbook
Benefits and office guides
Onboarding documentation
Approved FAQs
Sales and enablement
#sales-questions, #sales-enablement
Example questions
What’s our approved answer when a prospect asks about data residency?
How do we position our product against a competitor?
Which customer story should I share with a healthcare prospect?
Trusted sources
Approved messaging
Sales playbooks
Competitive battlecards
Customer stories
A little prep before we start cooking. Make sure Custom Agents are enabled, Slack is connected, and you have the right permissions to build and test the entire system.
Custom Agent creation is enabled
A workspace owner has enabled creation in
Settings→Notion AI→Agents.There is no fixed credit number for this build. Credits vary with source reading, actions, model choice, and run frequency. Measure real test runs in Insights before turning on broad triggers or schedules.
Slack is connected
A Slack admin has connected Slack to Notion.
For Slack Enterprise Grid, Custom Agents can view and post messages is set to
Read and post messages.The Slack account used for the connection has the same email address as the Notion account. A mismatch can cause triggers to fail without an error.
Check that you have the right permissions
You are able to edit and share every trusted source.
The Q&A Agent will be able to read and view every trusted source.
The Self-Improving Agent is able to edit trusted sources.
A question lands in Slack. Instead of hunting for the answer across different tools, the Q&A Agent searches the sources your team trusts and brings the answer back to the conversation.
Notion's guided setup will walk you through the initial build and get your agent up and running.
To get started:
Open Agents in your sidebar and click
+ New Agent.Choose
Self-improving answer botto launch the guided setup.
Start by choosing what your agent should answer and where it should look for answers.
1.1 Choose the questions your agent will answer
Pick from common use cases like Product, IT, Sales, People, or Office.
Have a different use case in mind? Choose
Customto define your own.


1.2 Choose where your agent will answer questions
Pick an existing Slack channel or create a new one right from the guided setup.
1.3 Connect your trusted knowledge sources
By default, your agent can access public Notion pages and Slack channels.
Review which sources it can access. Keep it focused on sources your team trusts.

Once you've got the ingredients in place, click Create agent and let Notion take it from here.
The guided setup takes you to your agent’s setup page. Take a peek behind the scenes and make a few tweaks before testing.
2.1 Review your Q&A Log database
The Q&A Log is a database that keeps a history of questions, answers, and unresolved requests so you can track how your agent is performing.
Best practices:
The database comes with properties like request
status,question type,requester, andrelated Slack thread. Think about what else your team might want to track and adjust the properties as needed.Use question types to spot recurring requests and statuses to identify questions that still need attention.

2.2 Review your Learning Log database
The Learning Log is a database where your agent saves new knowledge and memories you share with it. Each entry includes details like its source, scope, and confidence.
Best practices:
Your log will be empty at first. Test your agent by asking a few questions your team is likely to ask.
Review new entries to make sure the information comes from reliable sources and applies to the right situations.
Correct outdated information or remove anything your agent should no longer rely on. Nobody wants stale ingredients in the mix!
2.3 Fine-tune your Playbook
Think of the Playbook as your agent's recipe for answering questions. It's a Notion page that guides your agent on how to find answers and when to ask for help.
Best practices:
Tell your agent which sources to check first. For example, you might want it to search your internal knowledge base before digging through Slack conversations.
Be clear about what your agent should never do and when to escalate a question rather than guess.
Give your agent guidance on how to respond and what to do when someone corrects an answer.

You can edit the Q&A Log, Learning Log, and Playbook directly. You can also ask your agent to update them.
Time for a taste test! The guided setup will first ask whether your agent has enough context to answer questions. If anything's missing, you can share more knowledge or connect additional sources.
3.1 Test your agent with real questions
Choose one of the suggested questions or write your own. Start with questions your team usually asks, then try a few trickier ones where information might be missing or conflicting.
As you review the answers, check that your agent:
Used the question and relevant thread context.
Found a source-supported answer.
Kept the response concise.
Linked to supporting sources.
Created a log record only when there was something worth capturing.
3.2 Try your agent in Slack and review the Q&A Log
Once you're happy with the answers, select Looks right - I'm ready to try it in Slack.
Ask a few questions in your chosen Slack channel and check that your Q&A Log captures each outcome correctly.
Supported answers should appear in the Answers view with the question, answer, and source Slack thread.
Missing information should produce an honest “I don't know” and a record in the Requests view, with the answer left empty.
Conflicting information should be acknowledged without guessing, with the unresolved question captured in the Requests view.
Corrections should prevent the agent from repeating unsupported information and create a Requests record for follow-up.
3.3 Troubleshoot common issues
Even the best recipes don't always turn out perfectly on the first try, and that's okay! If something isn't working as expected, here are a few things to check.
Answer failures
The agent provided an answer even though its trusted sources did not support a reliable response.
The agent said the answer was missing, even though it exists in a configured trusted source.
The answer was too long or did not link to supporting sources.
Q&A Log failures
A relevant question did not create a Q&A Log record.
A supported answer did not appear in the Answers view.
An unsupported, incomplete, or conflicting answer did not appear in the Requests view.
The record’s
Question,Answer, orSource Slack threadwas inaccurate or missing.In a direct-chat test, the agent invented a Slack thread, source, or other evidence that wasn’t provided.
Tell your agent what went wrong and ask it to make the necessary adjustments. Then, try the same question again to see if the fix worked.
Your Q&A Agent is ready to meet the team! Before rolling it out, a few finishing touches can help make the introduction smoother.
Publish your agent and confirm it's responding in the right Slack channel.
Share your Q&A Agent with other agent managers who can help maintain it and handle unresolved questions.
Let your team know how to flag incorrect answers or share feedback.
Review the Q&A Log regularly to spot recurring questions and opportunities to improve.
You now have a complete Q&A system.
Your Q&A Agent can answer questions in Slack, show the sources it used, and record every answer or unresolved request in the Q&A Log. The log gives your team a clear view of what people are asking and where the available knowledge falls short.
If you arrived from the in-product guided setup, you already have this Part 1 foundation. Continue below to add the Self-Improving Agent.
Part 1 helps with the question in front of you. Part 2 makes every miss useful.
The Q&A Log shows you exactly where the system gets stuck. The Self-Improving Agent picks up those loose ends, gathers evidence, and drafts the smallest useful fix. A person still decides what becomes trusted knowledge.
Now build the second half of the loop.
Part 1 needed only Question, Answer, and Source Slack thread. Before configuring the Self-Improving Agent, add the following properties to the Q&A Log database by pasting this prompt into the Notion Agent:
The proposal itself lives in the Q&A Log page body, not in a property. Use this structure:
Reviewers edit the draft directly or leave page comments. When the draft is ready to apply, they change Status to Approved.
1. Create a starter agent
Time to close the loop and turn those knowledge gaps into trusted knowledge. Start with a draft agent, then teach it where to look, what to change, and how to ask for approval.
Open Agents in the sidebar and click +, then paste in this prompt.
Alternatively, you can just click this link. Save it, then continue to the next step.
2. Finish the setup
2.1 Add Notion triggers
In Triggers, set three Notion triggers:
Page added to database: Run when any page is added to the Q&A Log. The Self-Improving Agent stops when Answer is present and reviews the entry when Answer is empty.
Property updated in database: Run when Status changes to Needs agent review so the Agent can reread the edited draft and page comments, then revise the same proposal.
Property updated in database: Run when Status changes to Approved so the Agent can apply the approved draft.


2.2 Add the Q&A Log and trusted sources
Add the Q&A Log with Can edit access.
In Tools & Access, add the same trusted sources as you did for the Q&A Agent.
Add Notion pages with Can edit access.
Add connections with Read and write access
2.3 Update the job and instructions
Now that you’ve added the trusted sources, Q&A Log, and Notion triggers, you can update the Self-Improving Agent’s instructions.
Identify the person who should review proposed changes. They’ll be notified in real-time by the Self-Improving Agent.

Replace the placeholder below with their actual Notion @mention, then paste this prompt into the Notion Agent from the Self-Improving Agent settings page:
Optional: If you’d like to group reviewer notifications, choose either Daily or Weekly, update the placeholder, then paste this prompt into Notion Agent from the Self-Improving Agent settings page.
2.4 Don’t forget to save
Click the blue Save button at the top right.

3. Test your Self Improving Agent
The Self-Improving Agent works through Requests in the Q&A Log, so that’s where you’ll test it.
3.1 Confirm setup is complete
The Self-Improving Agent has Can edit access to the Q&A Log and trusted Notion sources.
Connected trusted sources have Read and write access.
All three Notion triggers point to the Q&A Log, use the correct events, and remain toggled off:
A page is added.
Statuschanges toNeeds agent review.Statuschanges toApproved.
The default reviewer is an actual Notion @mention.
Instructions tell the Self-Improving Agent how to:
Draft or revise the proposal in the page body with supporting evidence.
Set
Update typeandProposed destination.Assign or retain the
Reviewerand mention them when review is needed.Read direct edits and comments before revising the same draft.
Consolidate clear duplicates without losing unique evidence.
Wait for human approval before changing a trusted source.
Apply only the approved draft, record
Applied at, and setStatustoPublished.
3.2 Test it out with real entries
Set up testing with Notion Agent. From the Self-Improving Agent settings page, paste the prompt below into Notion Agent.
When Notion Agent finishes, click Save. Turn on only the trigger you’re testing, then turn it off before testing the next path.
Confirm that:
Proposal: The Self-Improving Agent writes a focused draft and supporting evidence in the page body, sets
Update typeandProposed destination, assigns or retains theReviewer, setsStatustoNeeds human review, mentions the reviewer, and leaves the trusted source unchanged.Reviewer edits or comments: Edit the page-body draft or leave a comment, then set
StatustoNeeds agent review. Confirm the Agent revises the same draft, returns it toNeeds human review, and mentions the reviewer again without changing the trusted source.Duplicate: The Agent preserves unique evidence and archives only the clear duplicate.
Approved change: The Agent changes only the temporary source according to
Update typeandProposed destination, recordsApplied at, and setsStatustoPublished.
3.3 Troubleshoot common issues
Something will probably get weird, but that’s okay! That’s what testing is for.
Check only the failures you observed:
Trigger failures
Adding a record did not start the Self-Improving Agent.
Changing
StatustoNeeds agent reviewdid not start a revision.Changing
StatustoApproveddid not apply the change.
Proposal and review failures
The proposal was unsupported, too broad, or missing its evidence, destination, or reviewer.
The Self-Improving Agent changed a source before approval.
The reviewer was not mentioned.
Reviewer feedback did not produce a revised proposal.
Duplicate and approved-change failures
A clear duplicate was not archived, or unique evidence was lost.
A record that merely overlapped or conflicted was incorrectly archived.
An approved change modified unrelated content.
The completed record did not include
Applied atorStatus = Published.
Paste the checked failures into the Notion Agent prompt below, then paste the prompt into a new Notion Agent chat from the Self-Improving Agent settings page.
Save the changes and rerun only the failed paths with fresh test records.
4. Publish the Self-Improving Agent
The loop is ready. The Agent can turn Requests into drafts, wait for a person, then apply only what gets approved.
Publish it. Turn on all three Notion triggers and click Save.
Optional: Share the Self-Improving Agent. Share it with any other agent managers who will help maintain it.
Introduce the Self-Improving Agent to the knowledge owners. Let the people responsible for its trusted sources know how drafts, comments, and approvals work. You can use this draft message as a starting point.
Put the whole loop to work
This is where the pieces start behaving like a system. A question can become an answer, a Request, a reviewed draft, and better source material without disappearing between tools. The Q&A Log keeps the handoffs visible, and people still decide what becomes trusted knowledge.
1. Turn on the full system
Turn on the Q&A Agent’s Slack trigger.
Turn on all three Self-Improving Agent triggers:
A page is added to the Q&A Log.
Statuschanges toNeeds agent review.Statuschanges toApproved.
Save both agents.
Confirm both agents use the same Q&A Log and the reviewer can open every record assigned to them.
2. Review proposals in the Q&A Log
When a record reaches Needs human review, the Self-Improving Agent mentions the assigned Reviewer.
The reviewer should check:
QuestionandAnswerfor context.Source Slack threadandSupporting sourcesfor evidence.Proposed changefor the exact update.Source to updatefor the intended destination.
Then choose one path:
Approve the proposal: If the evidence, wording, and destination are correct, set
Status = Approved. TheSelf-Improving Agent will apply the change, record
Applied at, and setStatus = Published.Request changes: Add specific guidance to
Feedback from reviewer, then setStatus = Needs agent review. The agent will revise the proposal and return the record toNeeds human review.
Reviewers should not set Status = Published. That status confirms the approved change was actually applied.
3. If a run fails
Open the failed run and confirm that the expected Slack message or database status triggered it.
Check trusted-source, Q&A Log, Slack, and destination access before changing the instructions.
Confirm the Q&A Log record contains the required evidence,
Reviewer,Status, andSource to update.Fix one issue, then rerun the failed case.
4. Maintain and optimize
Keep the workflow moving: Check the Q&A Log for records stuck in
Needs agent review,Needs human review, orApproved. Make sure every proposal has a reviewer and every approved record reachesPublished.Improve run quality: Review failed, slow, or unnecessary runs. Fix trigger scope, access, or source quality before rewriting instructions.
Control credit use: Check Insights regularly. Reduce trigger frequency, source breadth, model cost, or unnecessary actions when usage rises without improving outcomes.
Protect reliability: Rerun the relevant test outcomes after changing instructions, tools, permissions, triggers, or sources.
Expand gradually: Add one source, Slack channel, reviewer group, or use case at a time. Expand only when quality and cost are stable.
Start with a template
Browse over 10,000 templates in our template gallery
Something we didn’t cover?