I was given exactly the kind of test you’re looking to run, several years ago prior to joining my current organization.
“List the tools and technologies you’ve used,” prompted my interviewer. I rambled off tools I’d both used in my day job and played with on the side. “So you’ve never used Docker?” Nope.” “Ansible?” Nope. “Kafka?” Strike three, I thought. “Deploy Kafka inside of a Docker container using Ansible. Here’s a laptop with no software on it!” – as I’m handed a laptop welcoming me to my new Mac.
Over the next 45 minutes, I was convinced the interviewer thought I was dense. At the end, having gotten to the point where Ansible was provisioning a non-starting Kafka inside the container, he asked me to stop, thanked me for my time, and left. I wasn’t getting this job; it was the most intimidating interview I’d ever had. I started working there three weeks later.
A couple of months in, I mentioned the interview to the interviewer: “Oh, I thought you did pretty well?” Our colleague’s jaw drops – apparently, he never says that about anyone.
I learned from this that the interviewer wasn’t asking me to deploy Kafka inside a Docker container using Ansible – he was testing to see whether or not I could learn. I’ve worked with some incredibly bright folks at this org, and I’ll highly vouch for the strategy.