Author Archives: inkblot

Tales of the Meta: Testing in Stonehearth


Hi! We’re going to try something new on the blog.

Nearly all of Team Stonehearth is new to game development. Oh, we’ve built lots of software before, but this is our first game. As we work our way through this game, we’re learning all kinds of neat things about what it’s like to be a game developer, about the games industry in general, and about ourselves.

So, every now and then we figured we’d share some of this stuff on the blog. Introducing….”Tales of the Meta.” These are blog entries not about Stonehearth, but about the process of developing Stonehearth.

First up, testing!

Stonehearth Automated Test Framework

An automated test is a program that runs the game (or a piece of the game) to make sure that it still works. This is important for any substantial piece of software, but it’s pretty much the only way to stay sane when you’re a small team working on a big project with lots of moving parts.

Wayyy back in one of our very first blog posts, I described how we wrote tests back in Jan 2013.  Basically, we wrote little Lua programs to build tiny little worlds, and cause some game action to occur. Here’s the code of one such “test.”

local MicroWorld = require ‘radiant.tests.micro_world’

function HarvestTest:start()
self:place_citizen(12, 12)
local tree = self:place_tree(-12, -12)

self:at(10, function() self:place_stockpile_cmd(4, 12) end)
self:at(100, function() self:harvest_cmd(tree) end)

The actual blog post has videos of some tests running. Check it out if you want to see what Stonehearth looked like 18 months ago.

This was a good, basic start, but there was a lot of stuff missing. For starters, the program doesn’t know how to determine if the test “passed” or “failed.” To determine if the game is broken, someone needed to watch the program run and visually verify that the game didn’t freak out. Also, this program doesn’t properly mimic a player’s interactions with the game. The program chops down a tree by running the harvest_cmd function, whereas a player will harvest by clicking a button on-screen, then dragging out a rectangle.

A real automated testing framework will have at least the following characteristics

  • Each test expects a particular result. The test will PASS if the result is achieved, and will FAIL if the result is not achieved.
  • Some automated way to run lots of tests in one go
  • Reporting when a test fails

And that’s pretty much what we’ve got today. Somewhat hilariously, it’s built using the mod framework. To run tests, we run Stonehearth with a command-line option to run the autotest mod instead of the usual game mod. The autotest mod has a list of test programs that it runs through sequentially, until either a test fails or it runs out of tests to run. Here’s a video of the autotests in action.

Neat right? Here’s what that harvest test code looks like under our current system. It’s still fast and easy to write tests, but this test has the notion of pass vs. fail, and chops the tree by actually using the UI and simulating a mouse drag.

local Point3f = _radiant.csg.Point3f

local harvest_tests = {}

function harvest_tests.chop_tree(autotest)
autotest.env:create_person(10, 10, { profession = 'worker' })
local tree = autotest.env:create_entity(1, 1, 'stonehearth:small_oak_tree')


autotest.ui:click_dom_element('#startMenu #harvest_menu')
autotest.ui:set_next_designation_region(-4, -4, 12, 12)

autotest:fail('worker failed to chop tree')

return harvest_tests

Automated Performance Tests

We want Stonehearth to run well on as many different graphics card configurations as possible.  To catch performance problems early, we have a couple of different tests that stress different aspects of the game: the AI system, the pathfinding system, and the renderer.  And we test on a variety of machines: an old AMD setup, a couple of high-end nVidia and AMD machines, and a few Intel ones too.

The tests themselves are automated, but even so we don’t want to manually run them on all of these machines.  That would mean that every time someone wanted to test a change, they’d have to go and install that new version of Stonehearth on every machine, run the tests, and then make sure that nothing bad happened.  Ugh.

Instead, each of our testing machines is outfitted with a little daemon.  This little guy waits for a new version of Stonehearth to be built by a developer–it then runs our automated tests on that new build, collects the results, and sends them off to a central database.  This database takes all the data from every single test machine and bundles them together in some pretty graphs, so that a developer can then check to make sure that every test for every machine looks good.  It’s great for seeing very quickly when something has gone wrong (or for bragging when something gets fixed!)

Here’s a shot of one performance test in action. Little ones, cover your eyes.




Desktop Tuesday: Quick UI Demo & Alpha 4 Update


Hi guys. Now that Alpha 4 is out, we’re going to spend the next couple of weeks fixing the latest crop of bugs and polishing up some of the new features based on your feedback. For starters, we’re adding an option to play the game with no combat; no more goblin invasions!

This week’s video focuses on hotkeys and some of the new UI interactions we’ve added. Here you go!


Stonehearth Alpha 4 Official Release!



Hi. I’m pleased to announce that we’ve officially launched Stonehearth Alpha 4! It is available now on Steam, and will be available on Humble Bundle shortly. This is by far the most significant, feature-filled release for us, so congratulations to the whole team. It’s onward and upward from here guys. Here’s a super high-level summary of what’s in the Alpha 4 release:

New Classes: Footman, Weaver, and Trapper!

You can now upgrade your citizens to three new classes.

  • Footman Basic fighting unit, who will automatically patrol your town and defend you from invaders. Invaders?
  • Weaver A new crafter unit who crafts decorative items and basic armor for your units.
  • Trapper The trapper’s back, with a less micro-intensive style of play!

Basic Combat!

After a while, you can expect your town to receive “visitors” interested in taking your stuff, so you’ll need them. Combat right now is fairly rough, and will undergo EXTENSIVE tuning during the next few alphas.

Building Designer, Phase 1

You can now design and build your own custom buildings. The tool is just in its beginning phases, and is a bit difficult to use right now. We know. It will get better!


The GUI has received an extensive overhaul, with a centralized scoreboard and management screens for your citizens and crafters.

Caravan Notifications!

Every now and then, you will receive a notification that a caravan has swung by your settlement. Caravans can provide new settlers or offer to trade new things for existing or craftable goods.

Video Update, Speed Painting for Alpha 4


Hi everyone. We’re now officially going through our final testing pass for Alpha 4! Traditionally, when we hit this phase of an upcoming new release, I paint up a new cover image for the game’s title screen. Here’s a video of the process, sped up of course. 


And here’s a link to the actual image. Click the image to get a large version, then right click and save.


Video Update, More Furniture!


For this week’s video update, I take a closer look at that house in the previous post’s screenshot, along with the new recipes for the Weaver. The march to an Alpha 4 release continues…here’s the video!

Get Stonehearth!

The Stonehearth Alpha is now available through Humble Bundle.

Dev Roadmap

Connect with Us