Guide: Designing and Testing GenAI Flows

---

Start with the Basics

What are you trying to do?
Be clear on the goal.
Example: For Dotly, the goal was to help people figure out what to learn next, based on their job, interests, and confidence level.

What does good look like?
Describe success.
For Dotly: the user gets a clear, encouraging recommendation within 3 messages. It makes sense, feels relevant, and they’re not overwhelmed.

---

Map the Flow

What could go wrong?
Think through worst-case scenarios.
Dotly sometimes got too enthusiastic and recommended learning Python to someone who just wanted help with Excel. Not helpful.

What’s the ideal journey?
Sketch it out.
User says “I work in marketing, not sure where to start with AI.” Dotly replies with one clarifying question, then offers a couple of solid learning options. Done.

Where might users mess it up?
Expect the unexpected.
We had people write “I hate maths and I’m scared of tech” or “Just tell me what to do” – so we trained Dotly to respond calmly and supportively.

---

Build and Test

Write the system prompt.
Tell the AI what it’s doing.
Dotly’s system prompt: “You are a friendly AI learning guide. You ask questions to understand the user’s role, goals, and confidence. You recommend one step at a time. You do not overexplain. Always check understanding.”

Set the temperature.
Lower = safer.
We started with 0.7, but Dotly got too chatty. Dropped to 0.3 to stay focused.

Add tools/functions if needed.
Dotly uses a retrieval tool to recommend from our learning content. We had to make sure the prompt told it when to use the tool and when to reply directly.

Run tests
Create batches of user inputs.

• We wrote 20 test prompts like:
• “I’m a product manager. I don’t know where to start.”
• “I want to learn AI but I only have 10 minutes.”
• “Just tell me what’s most useful.”
• “I already know SQL. What next?”

Look for failures.
One test had Dotly recommend something the user had already done. Another one missed their fear of AI completely. We logged these.

Score them.
Each response got a “Laura score” from 1–5. 5 meant we’d be proud to show it in a demo. 1 meant rewrite everything.

Log changes needed.
We spotted patterns. “Too vague” usually meant we needed to tweak the system prompt. “Too repetitive” meant temperature or tool use was off.

---

Keep Tweaking

Don’t be precious. Every change teaches you something.
We ran 5–6 full batches before Dotly was good enough for the pilot. Still tweaking it now based on real users.

---

Example Test Table

System Prompt	Functions	Temp	Prompt	Response / Conversation	Batch	Any Errors?	Details	Complete?	Laura’s Score (1–5)	Important Notes
“You are a friendly AI learning guide...”	Content retrieval from learning DB	0.3	“I work in HR and want to learn about AI”	Recommended prompting and a short article	Batch 1	No	Slightly generic, but good enough	Yes	4	Safe but a bit boring. Didn’t ask a follow-up.
Same as above	Retrieval + 'confidence check' fallback	0.3	“I hate maths. What should I learn?”	Reassured user, recommended spreadsheet task	Batch 1	No	Nice tone, built trust	Yes	5	Felt human. Hit the tone perfectly.
Same as above	Retrieval + fallback	0.5	“Just tell me what’s useful”	Gave 3 suggestions – a bit overwhelming	Batch 2	Yes	Broke the one-step-at-a-time rule	Yes	2	Needed firmer guidance in the system prompt.
Same as above	Retrieval + custom response formatting	0.3	“I’ve done the SQL journey already”	Suggested SQL again	Batch 2	Yes	Didn’t check progress properly	No	2	Needs memory/state added or user progress filter.
Refined version – added explicit “only recommend unseen content”	Retrieval only	0.2	“I work in product, 10 mins a day max”	Recommends bite-size task with estimated time	Batch 3	No	Concise and well-paced	Yes	5	Good use of time constraints. Copy/paste-worthy.