The Real Work Behind a "Simple" AI Tool

I only achieve simplicity with enormous effort.

— Clarice Lispector

What I wanted were PM tools that anyone could use. They needed to be quick to onboard, since most PMs don't have the time to spare for learning how to handle elaborate setup. They needed to be generic enough that they weren't constrained to a particular AI assistant, while still easily customizable to meet a team's particular needs. And most importantly, they needed to be free because a lot of product teams are working on shoestring budgets with nothing allocated for AI.​

​

Clearly, the answer was going to be writing my own. You'd think this would be easy. You'd be wrong. There was a surprisingly rigorous process needed to make something trustworthy, user-friendly, and maintainable. So here is the guidance I wish I had from the get-go.​​

Rule #1 of product management: iterate

My initial concept was straightforward. I wanted to use readily-available free AI assistants, like Claude and ChatGPT. I wanted something that was ready-to-go immediately and didn't require any special knowledge or coding. I wanted the instructions to be written in plain language, so users could alter them, if they wanted. But, I also needed memory of some sort, so the platform could remember what had happened in previous conversations; this allowed for user customizations and for historical analysis. These requirements led me into projects, which are built pretty much expressly for this sort of use case. 

​​​

The first few tools I made were basically just long text files intended to be loaded into the "project instructions" field, like a complex prompt that you could type into a chat. I told Claude (my AI assistant of choice) what I wanted the tools to do and say. It helped me write the instructions and kept track of the branching logic. Simple. Easy-peasy.

 

But then I gradually started realizing the hole I was digging for myself. Since I was developing an entire suite of tools, I needed similar elements every time. Unfortunately, sections were starting to get out of sync as I found better wording in one place but forgot to change it in other places. As the tools were getting more complex, the AI assistant was getting less reliable and less consistent. Proofing grew increasingly difficult because every error fixed by AI tended to compound downstream into 10 more errors due to the way LLMs tend to micro-focus on text instead of looking at the big picture.

I needed a new way of working. So I did what PMs always do. I iterated.

​​

AI can't be trusted

By now, we all know that AI hallucinates and creates details to fill in the gaps wherever they support the premise of the output. But an even bigger issue is that AI assistants often just don't follow their instructions. The more complex the instructions, the more likely they are to mess things up. And these tools I was writing were long and messy. I needed a more rigorous approach.

Through trial and error, I settled on some rules for how to write long instructions:

​

• Short sentences. Since I was using AI to write for AI, it was important to recognize that it has a tendency to write in the long, convoluted style that we know and hate. I had to re-edit every line to break these down to remove ambiguity.

• Stay positive. No, I'm not talking about you. I'm talking about language. It's much easier for AI assistants to consistently follow positive commands than negative ones. Tell it to do something, rather than tell it to not do something.

• Explicit rules. Claude was writing instructions that were extremely vague, because it already knew what to do. It was becoming almost a case where it was leaving little shortcut reminders for itself. Again, this meant going line-by-line to make sure that instructions were precise without room for ambiguity.

• Overviews. Because AI assistants have a tendency to skip entire sections, I found that writing overviews gave them a basic understanding of the sort of content each section contained and where to find the rules to handle particular circumstances.

• Table of Contents (ToC). Similarly, a table of contents at the beginning gives the assistant a map to the entire tool, which improves the likelihood of finding the right section.

• Routing. AI assistants have a hard time keeping track of multiple pieces of information at the same time. When writing something complicated, like "If this, do X. If that, do Y. Otherwise, do Z.", it might get a bit lost. But when you compound the problem by having branching logic inside branching logic, pieces get missed. And it just gets worse when those trigger conditions are complex, like "If this and that but not this other thing, but alternatively also in this completely other case, do X." Much like a table of contents for the sections, having some sort of router to explicitly list all of the branches can be a big help. You can include more detail later as needed, but if the assistant has a good idea of what the decision point is going to be, it seems to more reliably behave.

• Checklists. The best written instructions in the world still tend to be unreliable as they get longer. I've found that having checklists before critical workflows can help a lot. For example, before delivering a file, have a short bulleted section of important rules to make sure that it actually did all the things you told it to do.

• Follow up. AI tends to be very very lazy. It likes to meet the bare minimum requirements and then stop, though it will often claim that it's done. For example, say you are asking it to proof something for grammar. Once it finds 3 or 4 issues, it tends to stop and present those issues as if they are a complete list. So, prepare to spend a LOT of time asking, "Did you finish or did you stop when you found some examples?"

​

Building with blocks

There were some sections that needed to be in every single tool. For example, how to create a consistent visual design for reports across the entire suite. Or, walking the user in chat through the process of seeing what customizations are available and then setting them. Obviously, though, this led to drift. Some ways of phrasing instructions worked better than others, but then I'd have to remember to go back and make the exact same fix everywhere. Also, some of these tool files were getting very long, which made them harder to proof.

​

To solve this, I leaned on my background as a programmer for a popular coding technique known as modularity or code reuse. This is the idea of having shared logic live in a single place, and then pulling it in where needed. AI assistants don't quite have that same sort of environment available, so I had to come up with a simpler way to achieve the same thing.

​

First, I created a core text file for each tool. Initially, this held all of the instructions for each tool. Then, I identified where I had similar pieces of functionality across multiple files. These pieces were pulled out into their own individual text files called blocks. In the place of the original content in the core files, I put in anchor text like {{BLOCK: name of block}}. Finally, I wrote instructions for how to assemble the core file with its blocks, by replacing each anchor in core with the matching content from the block (sort of like how mail merges work). During assembly, the original core files and block files are left intact, while a new final file is created for the end users. I call this new file a release build, again in a throwback to my coder days.

​

The process of creating release builds is still just prompt-style instructions in Claude, and it requires no coding skill or knowledge. I do have a checklist (remember that from above?) to help make sure everything gets done in the right order. Does every anchor have a matching block file? Does the version number need to be updated? Are all the blocks written in a way that they can be used in any core file? Do all the required customization fields for a block exist in the core file? When the assistant finds anything amiss, it stops the build, and we work through the problem together in chat before trying again.

 

Proof like you mean it

Fixing those two issues just left me with the proofing problem. These release builds could be thousands of lines long with hundreds of branching decision points. Trying to catch everything by hand proved to be impossible. There had to be a better way!

And of course, there was. Again, drawing on engineering processes, the key was developing a process that combined manual QA with an automated test suite. The process ended up consisting of five steps:

​

1. Static read-through. After writing the initial tool with the help of Claude, I re-read everything one line at a time and make edits where needed. I focus particularly on my rules for writing long instructions from above, but also just trying to make sure that everything actually does what I want it to do.

2. Regression pre-flight. Then I do a pass to make sure that all the changes from the read-through didn't break something else. I think of this as the butterfly effect of coding. It never fails to amaze me that one little change in one section can completely derail or change something in a completely different section.

3. Full simulated runs. Then, I set up a file containing all the different kinds of scenarios I can think of to test, and pass that off to Claude so it can roleplay as both the AI assistant and the end user. As the end user, Claude is encouraged to say​​​ random things, be uncooperative, give the wrong answers, ask for things out of order, and do other similar behaviors so we aren't just testing the expected path. Watching Claude argue with itself is a strange experience, but it's also where most of the real problems surface. Then, Claude and I together audit the output for possible logic gaps and inconsistencies. We also look for user experience issues. Is the conversation with the AI assistant scripted to be usable, friendly, and pleasant while still allowing the assistant enough freedom to respond to unexpected situations? I keep up a cycle of running simulations, auditing the results, and fixing the problem spots until everything feels solid.

4. Incognito testing. This is the one that always causes the most surprises. I open an incognito window in Claude, give it nothing but the AI tool and the simulation instructions, and then see what happens. This allows me to make sure that Claude hasn't been "cheating" in the simulation runs by using its knowledge of what the tool is supposed to do and what it is supposed to output. It's the closest I can get to recreating what it's like when a new end user tries out the tool on their computer. The results of the first one of these tests are usually pretty devastating. It sends me back into the cycle of fixing things and running simulations.

5. Proof in ChatGPT. Finally, because all of my development is done in Claude, I need to verify that it also works in ChatGPT. I run at least one incognito pass with the simulation and at least one incognito manual pass where I am playing the end user, just in case ChatGPT interprets instructions differently.

So, what advice would I give to others who are interested in creating their own no-code AI tools?

 

First, you don't need any sort of technical or coding background. Instead, you need curiosity plus a desire to solve a problem. You need a lot of attention to detail, because you are going to have to pore over the instructions. You need a lot of patience; the process takes time. You need to do a lot of proofing, and even then know that it's not possible to get perfect results every time because AI assistants don't behave the same every time. 

 

Second, expect the first incognito test to humble you. No matter how thorough you thought you were, you missed things. Probably a lot of things. For example, the first time I tried an incognito test, every single output file came back in the wrong format because I had naively assumed that if I mentioned something was a Word file that it would produce a Word file. Spoiler: it did not. I had to explicitly write "if the expected output is a Word file, create a Word file as the output with an extension of .docx" in the tool.

 

And finally, remember that while AI can genuinely be a powerful assistant, it is still useless without you there for inspiration, guidance, and human insight. You are both the vision and what brings it to life. So go build something!