Fabrizio Ferri-Benedetti, who spent many years as a technical writer for Splunk and New Relic, joins Ben and Ryan for a conversation about the evolving role of documentation in software development. They explore how documentation can (and should) be integrated with code, the importance of quality control, and the hurdles to maintaining up-to-date documentation. Plus: Why technical writers shouldn’t be afraid of LLMs.
Fabrizio is now the lead documentation engineer Tinybird, a data platform for user-facing analytics. Get started with their docs or explore their blog.
Find Fabrizio’s blog here. Some reading suggestions:
Find Fabrizio on LinkedIn or GitHub.
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I'm your host, Ben Popper, Director of Content here at Stack Overflow, joined by my co-host, Editor of our blog, Ryan Donovan. Ryan, your voice is a little scratchy today?
Ryan Donovan It's a little rough today, yeah, but it's on the mend. I'm on the upswing.
BP Okay. Apologies to our listeners, Ryan's beautiful baritone is not quite with us, but he's on the mend, so rest assured, things will get better from here. So Ryan, I think both you and I have a lot of experience discussing, thinking about, and in some ways creating documentation. You were a technical writer for a long time, you wrote API docs, and ever since I got to Stack Overflow, I've been thinking about these ideas because Stack Overflow for Teams is kind of a way of doing documentation inside of a company. One of the first pieces I ever wrote for the company was called ‘Documentation Must Die,’ and it was about just how there's a better way to do it naturally in a question and answer format when a question arises, not asking somebody to write an article out of context. And at that time, Joel Spolsky was still with the company and I got to interview him for that, and the way he put it was to imagine you're building a big building, a big apartment complex. The better the blueprints are that you leave behind, the easier it will be for somebody to fix the plumbing in the future or address a structural issue, and the worse the blueprints are, the harder it's going to be, so think about documentation that way. But there was a great piece that was in our newsletter last week. Cassidy Williams put it in there, and it was all about documentation as a form of engineering, elevating our respect for this craft and doing it in ways that align with some of the workflow and the tools of engineering so that it can be kind of an integral part of just creating a company with a healthy codebase and a great developer experience. So I want to bring on Fabrizio Benedetti, the author of the piece and a documentation engineer, to discuss some of his thoughts on the topic. Fabrizio, welcome to the Stack Overflow Podcast.
Fabrizio Benedetti Hi there. Thank you for having me here.
BP So for folks who are listening, tell them a little bit about how you got into the world of software and development and what led you to this kind of unique title that you have today?
FB Well, I majored in psychology something like 20 years ago, but I was always drawn to tech and software. And I landed my first job as a content editor for a software download portal. That was 15 years ago, and since then I have been a writer in tech in the sense of handling different types of content deliverables, coming up with content strategies, and as of late I've been gravitating more and more towards developer documentation because it really brings together two things that I love– one is programming and the other is writing. So I get to do both things as a documentation engineer. I haven't seen this job title that much, though it's increasingly common, but really what I do is technical writing with a stronger technical accent.
RD I did a lot of developer docs as well, and it's been interesting to see the evolution of documentation as, “You put it in a help file included with the product. Done,” to where you're putting Markdown files in every Git repo with every service so it's part of the actual code. What do you think is driving that trend?
FB Oh, you mean docs as code in the sense of having the documentation together with the code?
RD Yeah.
FB I think it's a very practical sociological thing, really. The whole docs as code movement, or as an idea if you want, was born out of the need of getting the developers to participate more in the documentation process. So instead of asking developers to enter a CMS or using tools that weren’t of their liking, what happens is that technical writers, or documentation engineers as we fancy ourselves sometimes, we are getting into the repos where the code is living and we are kind of growing the documentation out of that code together with the code. And we are using the same tools as developers are using, the same methodology, and we are also asking them to apply the same quality checks and have the same care they dedicate to code when they craft documentation. And in that sense, technical writers are always few. I haven't seen a company where there are too many technical writers. So we end up acting as editors in chief of the documentation. And as I say in my post, documentation is infrastructure first and first and foremost, because it's what enables you to use the product without being the product like a server or any other infrastructure.
BP So you mentioned can you apply some of the same quality checks to documentation as you can to code? I guess the question would be, how do you write unit tests for documentation? What is the way to determine if the documentation is good if the analogy is to writing code?
FB So there's many things you can do there. The simplest of course is, if you're authoring docs using Markdown or any other markup language, you can lint the Markdown itself for things like broken links, formatting, maybe you're using heading levels the wrong way. That's the basics, the very minimum. Then there are tools out there called prose linters which is fascinating. It's a great idea, really, but essentially you can lint the prose of your documentation as if you were linting the code of your application. You define these rules, for example, I'm a big fan of the Vale linter, it's open source. You define these rules in a YAML file with endpoints that say something like, “If these regular expression patterns exist, suggest these other patterns or these other words,” and it uses natural language processing also to do that. This really helps with lowering the burden of technical writers and also the burden of engineers because they can even use those tools in the IDEs before the writer actually gets to edit anything. And lastly, I would say you can test code snippets. There are several interesting solutions out there, proof of concepts for testing the code snippets, see if they build against your code base, for example, or if they have any issue. So those are the three main dimensions I would say you could apply testing to docs.
RD When you say unit testing for docs, I always think of getting somebody to just go through it and follow every step and see if they get to the end result, but I wonder with the Gen AI, will people just feed the LLM the docs and say, “Go through this and build something? If you run into any problems, stop.”
FB You can do that. It's a bit expensive, I would say, because of the compute power required for that. I've been testing recently the computer use feature of Claude AI and it's fantastic. I was given this doc and I was telling Claude to pretend to be an engineer and see if they could find some Java documentation for something. And Claude AI just opened up, fired up this browser, Firefox, and it was taking screenshots along the way and trying to explain each step. So there's some interesting applications coming from that area in terms of artificial user testing.
BP So you also mentioned in the piece this idea that Git is really important within software and that it allows folks to ensure changes made in one place are reflected in another, that people come to consensus when making those changes, and that software that's updated doesn't have the tech debt that might come with versioning issues. How do you approach that for documentation? Because I do think a lot of people, when they think of documentation and they think of wikis, the first thing they think of is stale knowledge that's out of date and therefore isn't helping them and creating a frustration or a bad taste in their mouth when it comes to docs.
FB Well, there are several ways you can go about it. I think, first of all, you'll want to automate as much reference documentation as possible. It's those pieces that are trickier to follow up and detect and see if you need to update them, like the small parameters in an API endpoint documentation. So if you can automate part or all of those docs, that's already a win because it's documentation that will change together with the code. There are ways of doing that, like generating open API from the documentation, for example. I recently have seen some promising things like TypeSpec, where you can define an API's behavior in a simpler way rather than writing YAML files. And then I would say that there's really the agreement, and it's kind of an unwritten thing, but you can reach this work agreement within the team where documentation should really be part of the definition of done. And as you create, for example, the UI of the feature, or you tend to the infrastructure side of things when you release a new feature, the documentation should be within that stream from the very beginning, not as an end product.
RD I also like the idea of docs as ops that you had in there. As technical writers, we're sort of trying to anticipate the questions and problems that people can have, but that doesn't always happen. Somebody has an incident and there's nothing there for it and they find out the answer on the fly. Have you found that there's a way to sort of capture that information for the next time it happens?
FB What sort of situation, for example?
RD If somebody runs into a mystery 500 server error and they go and investigate the problem and they're like, “Oh, this setting was not set correctly,” or whatever, you deleted a file. Have you found that there's a way to make sure that information is in the next version of the docs?
FB Well, there's a bit of art there. For example, I'm a big fan of Slack notifications based on keywords, for example. For every channel here at work, I try to capture any mention of documentation, docs, errors, whatever, anything that matters to the docs. It's a lot of noise, of course, eventually. I've seen some interesting solutions that I would like to try out there. One is I think called Guru. Essentially, what they do is they scoop all the information from all the potential sources of knowledge within a company like Jira, wiki, Slack, etc, and they turn that into nuggets of knowledge that then you can organize and you can reuse. So I think it's still a problem to be solved, that one– and not just for documentation, but more in general for sharing knowledge within a company.
BP What you're talking about gets to a really interesting question that I think is top of mind for us at Stack Overflow, which is where are we headed with enterprise search and are we really moving towards a solution where you can ask a system to, like you said, ingest knowledge from Jira and Confluence and GitHub and all these other places and it's able to make sense of it to the degree that, when you then want to go ask a question, it can provide an answer. And so then it's like, “Well, then what is documentation then?” because all of this has been ingested, automated, and made into knowledge artifacts. I don't think we're at that point, obviously, but just hypothetically speaking, what is the role of the documentation engineer in a world where Gen AI is that capable?
FB Well, for one, there is the role of quality control. I think the documentation engineer wants to make sure that the content that comes out of any such system is valid, that it complies with whatever style guides or standards are out there. And in that sense, it's not that different from what a developer augmented through AI would do in the sense that you can ask an LLM to come up with a snippet or cook some boilerplate code for an issue, at least at the initial stage. But then it's the human, it's the developer, who is liable, who has to review and check that that code is going to work and it's safe, etc. So the human is also becoming an editor. You can have LLMs doing lots of menial tasks, lots of boring work even, but the human is coordinating and orchestrating the pieces there.
RD I think that was an interesting thing that you said there, the documentation person becoming an editor. And you mentioned all your Slack keyword things. I remember I did a survey of the engineering org at my last job to be like, “Where do you go for information?” and it was Slack. About 60% of the time it was Slack, and I have this feeling that we can create the best docs, but how do you get them there? Have you found a way that you can lead your engineers to the documentation?
FB You have to play with incentives a bit. So for example, one of course is that my existence means that I'm going to take some work off their shoulders. So for that, they are already grateful and it's a nice experience to have as a technical writer, of course, but it's not enough. So you have to see the documentation as something that you have to sell in a way internally, as something that will remove problems after launch, for example, or reduce the support burden, or help sell the feature so that the feature will perform better, which is usually also a metric that developers care about. So documentation might have several goals there. You really have to understand what piques the interest of the developer in each case, or as an organization or team level.
BP That's super interesting– the idea that you can lead a horse to water, but they may just ask in Slack because that's what they're used to, but if you tie it back to actual metrics and KPIs that maybe are associated with the work that they're doing, then they have this much bigger incentive to interact with the documentation and then you get this positive feedback loop of, “Oh, that did help me,” or, “This answer wasn't quite right,” or, “We've changed this and this needs to be updated.”
RD I think a lot of what you get to is sort of the shift of awareness of internal documentation. Most of the developer documentation I wrote in my career was for customers of that product, for people who bought an SDK or whatever. But now companies are thinking, “Oh, our code is complex. We don't necessarily have the ability to understand everything.” Do you think there's a sort of technical shift driving that or is it just an awareness filtering down to everybody?
FB It's interesting that you mentioned internal documentation because, to start, I think the divide between internal and external documentation is more virtual than real, and we should treat internal documentation with the same amount of care that we treat external documentation. First, because we want the organization to be productive, and second, because internal documentation might become external documentation, and it happens. Internal API endpoints suddenly get exposed and what you have is the internal docs so you better take care of those. But then the issue with complexity, that happens pretty much everywhere. It happened at the previous company I worked at where after 10 years you amass this amount of knowledge and it becomes increasingly unwieldy to update and maintain. There's no secret solution there, except that you cannot really reduce the amount of resources you dedicate to docs. Quite the opposite. I've seen many companies reducing their technical writing teams, their documentation teams, or the resources or the budget dedicated to documentation tooling, and I think that's a mistake. Of course, it's not about throwing money blindly at things, but one of the things I argue with my blog in general, is that I'm trying to advocate for more awareness that documentation matters and that it requires people and resources.
BP Just out of curiosity, you mentioned before, one way to get folks thinking about this might be to connect it back to a product being released faster or a product being more performant. Are there ways that you try to measure that, that you try to be able to say, “Look, I can show you the ROI of the documentation because when this happened, this happened.” Do you try to set up a system to do that?
FB There's this concept I came up with a couple months ago which is documentation observability. I've been documenting observability solutions for IT and software development so the concept was already familiar, and I was thinking, “Why don't we apply observability principles to documentation as well?” So at the current job at TinyBird, what we are setting up is product analytics throughout the product and documentation, and we are hoping– we're still in the early stages– but we are hoping to see the kind of interaction I'd like to see which is much more meaningful than web analytics. For example, picture a user using an API endpoint, getting an error, the error payload contains a URL to the docs, the user follows the link, you're tracking the session of the user, the user consumes the docs, goes back to the API, corrects the call, and he gets a 200. So there's already a journey there, and this might happen in CLI, it might happen in UIs, there's already a journey there, that path through the docs as if they were part of the product experience, and you can track success rates there, for example, or conversion or whatever, but it's considering the docs part of the product journey and part of the product. That way you're measuring it in the same trace, so to speak.
RD I know we touched on Gen AI a little bit. One of the things I hear about Gen AI being good for would be documentation, sort of generating automated documentation. Do you have any worries about that, either that it's going to push technical writers out either rightfully or unjustly, or that it's going to just create worse documentation?
FB Lots of technical writers are wondering about that. My answer is that, sure, let's start using Writer Bot. Let's start creating documentation using AI because some documentation is better than none. What companies relying solely on artificial intelligence will realize is that that documentation is not always secure– actually it's quite bad, usually– and will require editing, will require a human there to, again, do quality checks and improve the documentation after generation. So eventually it's going to happen that companies realize that just automated documentation is not the answer. What I recommend is that technical writers and documenters in general should not feel afraid of LLMs and generative docs, they should really think of the strategies to apply it at their jobs. So if you're talking, for example, of code snippets, that's something where Gen AI can be used with, I think, greater success than complete documentation. So it's a constant strategy. So the technical writers decide, “Okay, we have this pen here and it's enclosed and it's a code snippet. This is where I want to unleash Gen AI, but not the entire doc,” but that decision should come from whoever decides on the documentation strategy.
BP Great.
[music plays]
BP All right, everybody. It is that time of the show. Let's find someone on Stack Overflow who came on and shared a little knowledge or a little bit of curiosity, and in doing so, helped the community. “How to touch a file and mkdir if needed in one line.” Thanks to Sridhar. You were awarded this badge because your answer was so good that it got way more upvotes than the accepted answer. And we've helped over 25,000 people, so appreciate the knowledge. As always, I am Ben Popper, I'm the Director of Content here at Stack Overflow. Find me on X @BenPopper. If you want to come on the show or you want to hear us discuss something, email us, podcast@stackoverflow.com. And if you liked what you heard today, why don't you subscribe and get more in the future.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow. If you have hot takes that you want to share with us, please reach out to me on LinkedIn.
FB I'm Fabrizio Ferri-Benedetti, Documentation Engineer at TinyBird. My blog is passo.uno if you want to learn more about my documentation thoughts, and I strongly invite you to check out tinybird.co, our data platform solution. Thank you.
BP Awesome. All right, everybody. We'll put those links in the show notes for you to check out. Thanks for listening, and we will talk to you soon.
[outro music plays]