Terence Eden. He has a beard and is smiling.
Theme Switcher:

How to *actually* test your readme

· 24 comments · 150 words · Viewed ~5,732 times

🄯 CC BY-SA

If you've spent any time using Linux, you'll be used to installing software like this:

The README says to download from this link. Huh, I'm not sure how to unarchive .tar.xz files - guess I'll search for that. Right, it says run setup.sh hmm, that doesn't work. Oh, I need to set the permissions. What was the chmod command again? OK, that's working. Wait, it needs sudo. Let me run that again. Hang on, am I in the right directory? Here it goes. What, it crapped out. I don't have some random library - how the hell am I meant to install that? My distro has v21 but this requires <=19. Ah, I also need to upgrade something which isn't supplied by repo. Nearly there, just need to compile this obscure project from SourceForge which was inexplicably installed on the original dev's machine and then I'll be good to go. Nope. Better raise an issue on GitHub. Oh, look, it is tomorrow.

As a developer, you probably don't want to answer dozens of tickets complaining that users are frustrated with your work. You thought you made the README really clear and - hey! - it works on your machine.

There are various solutions to this problem - developers can release AppImages, or Snaps, or FlatPaks, or Docker or whatever. But that's a bit of stretch for a solo dev who is slinging out a little tool that they coded in their spare time. And, even those don't always work as seamlessly as you'd hope.

There's an easier solution:

  1. Follow the steps in your README
  2. See if they work.
  3. …
  4. That's it.

OK, that's a bit reductive! There are a million variables which go into a test - so I'm going to introduce you to a secret zeroth step.

  1. Spin up a fresh Virtual Machine with a recent-ish distro.

If you are a developer, your machine probably has a billion weird configurations and obscure libraries installed on it - things which definitely aren't on your users' machines. Having a box-fresh VM means than you are starting with a blank-slate. If, when following your README, you discover that the app doesn't install because of a missing dependency, you can adjust your README to include apt install whatever.

OK, but how?

Personally, I like Boxes as it gives you a simple choice of VMs - but there are plenty of other Virtual Machine managers out there.

List of Linux OSes.

Pick a standard OS that you like. I think the latest Ubuntu Server is pretty lightweight and is a good baseline for what people are likely to have. But feel free to pick something with a GUI or whatever suits your audience.

Once your VM is installed and set up for basic use, take a snapshot.

Pop up showing a snapshot of a virtual machine.

Every time you want to test or re-test a README, revert back to the original state of your box. That way you won't have odd half-installed packages laying about.

Your next step is to think about how much hand-holding do you want to do?

For example, the default Debian doesn't ship with git. Does your README need to tell people to sudo apt install git and then walk them through configuring it so that they can git clone your repo?

Possibly! Who is your audience? If you've created a tool which is likely to be used by newbies who are just getting started with their first Raspberry Pi then, yeah, you probably will need to include that. Why? Because it will save you from receiving a lot of repeated questions and frustrated emails.

OK, but most developers will have gcc installed, right? Maybe! But it doesn't do any harm to include it in a long list of apt get … anyway, does it? Similarly, does everyone know how to upgrade to the very latest npm?

If your software is designed for people who are experienced computer touchers, don't fall into the trap of thinking that they know everything you do. I find it best to assume people are intelligent but not experienced; it doesn't hurt to give slightly too much detail.

The best way to do this is to record everything you do after logging into the blank VM.

  1. Restore the snapshot.
  2. Log in.
  3. Run all the commands you need to get your software working.
  4. Once done, run history -w history.txt
    • That will print out every command you ran.
  5. Copy that text into your README.

Hey presto! You now have README instructions which have been tested to work. Even on the most bare-bones machine, you can say that your README will allow the user to get started with your software with the minimum amount of head-scratching.

Now, this isn't foolproof. Maybe the user has an ancient operating system running on obsolete hardware which is constantly bombarded by cosmic rays. But at least this way your issues won't be clogged up by people saying their install failed because lib-foobar wasn't available or that ./configure had fatal errors.

A great example is the Opus Codec README. I went into a fresh Ubuntu machine, followed the readme, ran the above history command, and got this:

Copied to 📋 
 sudo apt-get install git autoconf automake libtool gcc make
git clone https://gitlab.xiph.org/xiph/opus.git
cd opus
./autogen.sh
./configure
make
sudo make install

Everything worked! There was no missing step or having to dive into another README to figure out how to bind flarg 6.9 with schnorp-unstable.

So that's my plea to you, dear developer friend. Make sure your README contains both the necessary and sufficient information required to install your software. For your sake, as much as mine!

Wait! You didn't follow your own advice!

You're quite right. Feel free to send a pull request to correct this post - as I shall be doing with any unhelpful READMEs I find along the way.

Share this post on…

24 thoughts on “How to *actually* test your readme”

    1. @linc @blog My experience is that if the CI doesn't scream at me "your instructions no longer work, if they ever did", I won't do it.

      And if I tell myself that I should do it anyway, I'll probably stay away from that project because of the anxiety.

      So what I'm trying to say is that I really hope there's some clever way to automate this. Like rust doctests - it's documentation, but it's still tested.

      Reply | Reply to original comment on genserver.social

  4. Reading this whole post was worth it JUST to learn about history -w history.txt The rest was also smart and great advice but man, that one trick is definitely going to help me. Thanks!

    Reply

  7. I may be a minority audience as well, but there's a level of "I know how to do things in the terminal, but I don't usually compile my own programs" level of Linux user, where I'm trying my best to avoid being rm -rf'd by instructions but not a complete newb.

    Reply | Reply to original comment on bsky.app

  11. @Edent

    I also like to try it on various platforms…install it in a fresh FreeBSD VM/jail. Install it in an OpenBSD VM. Install it on Debian/Ubuntu. Install it on RedHat/CentOS. Install it on Minix. The more places I can test it, the more thorough my README can be.

    (and another reason I like #awk and simple C code because versions of awk/gcc/clang/tcc almost never matter to me and they work almost everywhere; though make(1) is a little more temperamental with GNU make, BSD make, imake, cmake, …)

    awk

    Reply | Reply to original comment on mastodon.bsd.cafe

  12. The "fresh VM" instinct is the whole game, most READMEs pass only because the author's machine has undeclared state (a global dep, an env var, a cached login), and a clean environment is the only thing that surfaces it. What's worked for me is going one step further: capture the steps by actually running them once, then replay them clean-room in a brand-new container and keep only the steps that pass independently: so the published quickstart is provably reproducible instead of "worked when I wrote it." Disclosure: I maintain readme2demo (MIT), an open-source CLI that does exactly this, so I'm biased toward the fresh-container approach.

    Reply

  13. @Edent Oh yes please!

    For bonus marks: test on more than one distro flavour (e.g. one apt-based & one rpm-based): I'm generally OK googling to figure out that the package called "foolib" on your distro is called "libfoo" on mine, but it's nicer not to have to.

    But not being told I even need such a package at all, and having to figure it out from cryptic error messages, sucks. (And I am *good* at figuring such things out--many people will just assume it's your project that sucks and give up!)

    Reply | Reply to original comment on mastodon.smears.org

  15. @Edent
    Great reminder!

    My team and I shamelessly uses interns for this.

    We promise the interns they will complete their first open source commit while shadowing us, and they do!

    But to get that far enough to make and test a change, they end up test running nearly all of our documentation for us.

    If they find a problem early, sometimes their first commit ends up being a wording fix to make our ReadMe more accessible, with their own choice of wording (after we successfully explain what our ReadMe failed to.)

    Reply | Reply to original comment on infosec.exchange

  20. @Edent One of the things I always say to new people joining the org is, "Here are the team new starter docs, there will be errors or omissions, please fix any bugs you find!".

    (There's a window of, ooh, about two weeks when things that are obvious to people who have been here ages won't be to new joiners. So there's a golden window for improvement. It helps set some cultural values, too.)

    Reply | Reply to original comment on mastodon.social

  23. @Edent The best advice I ever heard for documentation of a procedure was that it was a three person job.
    The one who knows how to do it should not write the docs.
    They demonstrate it to a person to write the doc - so that 'obvious' things get included.
    Then a third person has to successfully do the thing from just the doc alone.

    Reply | Reply to original comment on infosec.exchange

What links here from around this blog?

  1. A watch displaying "Hello World" and "I know C++". The latter if off centre because I don't know C++.Una Watch - SDK and Writing Your First App
  2. A tiny lego Storm Trooper eats a chocolate coin.I paid people to try and follow my README

What are your reckons?

All comments are moderated and may not be published immediately. Your email address will not be published.

See allowed HTML elements: <a href="" title="">
<abbr title="">
<acronym title="">
<b>
<blockquote cite="">
<br>
<cite>
<code>
<del datetime="">
<em>
<i>
<img src="" alt="" title="" srcset="" width="" height="">
<p>
<pre>
<q cite="">
<s>
<strike>
<strong>

Your post must include a link to this post.

How does this work?
To respond on your own website, write a post which contains a link to this post - then enter the URl of your page here. Learn more about WebMentions.