[Side Track] Video Games with AI Angle Part 3

Today OpenAI released their amazing work on Learning to Play Minecraft with Video PreTraining (VPT) which is a great occasion to release the third post of my Video Games with AI Angle trilogy [and thus for now the last one, even though I might at some point make it a trilogy like Star Wars or Matrix with more entries].


FEZ is one of the most interesting games I have ever played. It is a jump and run on voxel levels and you can rotate the camera to look at it from four directions (front, back, left, right). What makes this interesting is that while your kinematics are 2D – you walk on lines like you would in Super Mario, the 2D objects like floors or climbable walls change with the view. So you can not only reveal additional ways to navigate through the world, but also use this mechanic in between jumps to access regions you would not otherwise be able to access which leads to spatial puzzle solving that is unique and a lot of fun. Humans adapt to this surprisingly well and it would be great to see an AI do the same thing.

Crazy Machines

Crazy Machines is similar to the The Incredible Machine game series where you need to complete Rube Goldberg machines in order solve a particular given task. This combination of symbolic puzzling with understanding the dynamics of an environment w.r.t. the individual game elements (and their physical properties) is challenging to both humans and AI.

Dark Signs

Dark Signs is a rather old hacking game and when I last played it over a decade ago it was quite bug-ridden. However, overall it was a great experience and it is one of the most unique games I have played up to the point that I implemented my own simple version as a kid. What set it apart from most other hacking simulator games for at least a decade is that it has a semi-serious scripting language [maybe Bitburner is now beginning to catch up to it, but I haven’t had time to look into it, yet] and this scripting language can be used in any way you like – it is literally a sandbox or even an immersive sim for hacking. For instance, you can write your own port scanner with it and since it has delays, scanning an entire IP range will take a while. The same goes for password crackers. And just like in real life, it might be a good idea to try a dictionary attack. Some puzzles require more social engineering and you actually try via active or passive reconnaissance to find password hints etc., but I fondly remember one of the more challenging puzzles where you would find a compromised large chunk of code in Dark Signs’ scripting language and had to reverse engineer it to solve your task which required reading the code and drawing your flow diagrams until you had the solution. An AI that could start to perform this kind of reasoning would be quite fascinating.

Lost Vikings

Lost Vikings is a jump and run game with three vikings who have unique skills that need to be combined to traverse a level. I like again that it combines understanding the dynamics of the environment and parcours which an RL agent should be easily capable of, but also require the more symbolic planning to solve puzzle elements while coordinating your agents, erm, vikings and keeping track of how they need to interact to make progress together which seems closer to automated planning. [You can actually download this game for free from Blizzard. Not sure about part 2.]


The final game in this theme and series I would like to mention is Gunpoint which is described as “a game of creative infiltration” in its launch trailer. Besides an interesting jump and climb mechanic that enables the necessary 2D spatial navigation, you can rewire things in the environment, so motion sensors, doors, electricity etc. work differently. You can use this to your advantage by improvising traps, concealing your presence, gaining access to restricted areas etc. An AI that could observe and understand a building like in Gunpoint to then draw its decision which elements to manipulate to accomplish its mission would be phenomenal.

I understand that the AI skills required to tackle many of the games I’ve presented are currently far beyond the SOTA, but I think this shouldn’t prevent us from investigating how to combine different AI paradigms via neuro-symbolic approaches, but also via hybrid AI systems that combine automated planning, reasoning, neural elements, RL, traditional ML etc. to converge towards agents capable of the fluid intelligence required to tackle such complex tasks. First I enjoyed playing these games myself and now I am having additional fun contemplating how to solve them with AI one day. Maybe a sufficiently advanced AI needs to read books and play video games just like we do to challenge itself and achieve fluidity in thought and action.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply