[Side Track] Video Games with AI Angle Part 4

As Alien has shown, a trilogy can have more than three parts, so let us revisit our video games with AI angle series in a fourth installment.


Superliminal is a game out of CMU that plays with perspective a lot. For instance, it turns the visual perception clue that objects are smaller when they are further away on its head: If you pickup an object and while holding it move it over a background far away, it will actually scale that object to be as big as it would have to be in the distance in order to appear as having the same size, e.g., if you pick up a tiny chess piece, hold it in front of your eye over a room of the wall far away and let go of it, a massive chess piece will drop down. That alone is interesting, but it really becomes intriguing when you scale something like a puppet house which you can actually enter, since then you will actually shrink or grow your own size as you walk through it. But there are also other vignettes on perspective like projected cubes which are smeared across surfaces and appear misscaled and broken unless you look at them from exactly the right spatial point at which point in time you can pick them up as normal 3D geometry and quite a few additional elements.

It should also be noted that the game has a predecessor called the Museum of Simulation Technology which is less polished, but equally worth a look:

Manifold Garden

Manifold Garden is an interesting puzzle platformer, since it takes place in a mathematical space with endlessly repeating geometry. This lets you jump into infinity to land on the same geometry further down, so if there is something in front of you, but too far up to jump across, you can just look down, see it repeated below you and jump on it there. Furthermore, it is another game that lets you switch gravity allowing you to walk on walls and ceilings and together with the mathematically abstract art style that very much dissolves our usual notion of what is up or down making it entirely depend on your frame of reference.

Tool Assisted Speedruns

This might be more a statement at a meta level, but I think TAS (tool-assisted speedruns) raise two interesting points regarding our topic: i) Humans cannot react as quickly as machines, but if you even the playing field by giving them tools which allow snapshotting and thus alleviate the real-time constraints, humans become much more impressive at fast, agility-driven games like jump-and-runs and ii) the exploration and exploitation of glitches we see in speedruns is another interesting aspect underexplored in AI research. While there are famous examples like the speedboat game where the agent learned to drive in circles to maximize its score rather than win the race, I still find that humans are much more strategic in searching for and cataloging glitches to then optimally align them into one coherent speedrun of a game.


While incredibly boring for humans and something I would never play, games like Draw Puzzle deserve mentioning since i) drawing is still an underexplored modality and ii) while finding the things to fill in is trivial for humans, it is a good stepping stone for AI. For instance, filling in parts like a missing shoe or a hole in a plane or donut are reminiscent of masked language training and there are some other fluid intelligence aspects as well – for instance, when you temporarily need to switch to mathematical reasoning. There are also physics drawing games where you draw bridges or objects that fall, but my impression is that there is a lot of untapped potential in this genre and current instantiations still rely on crudely handling position and shape, since semantically interpreting the drawn object is most likely AI-hard itself.

Building Machines, Plants and Environments

I don’t want to spend too much time on this, since it is well known, but the plethora of games in which you build machines or plants or construct things in your environment lend themselves well to AI optimization. For instance, in Instruments of Destruction one can optimize the construction of vehicles based on their destructive potential.

A good example of the creative process free-form construction enables is exhibited in the following video as well as many similar ones. Just given a variable-length plank the player explores the design space it enables and finds that besides building giant constructs like turning his raft into a huge tornado, he can use it to travel over the map and under water without oxygen limitation, avoid environment triggers, become invisible to enemies, glitch through walls and ceilings etc.

In other videos he experiments with accelerating components and plant machinery to massively craft and refine materials. This fluid reuse reminds me of human tool usage. Just like a knife is not only essential for food prep and eating [or maybe self-defense in the worst case], but can be used for batoning (i.e., fire making), digging, tightening screws, carving, opening boxes, cleaning fish and all kinds of other usages easily forgotten in industrialized living, an inconspicuous plank can be much more useful in a video game than merely bridging a gap.

Finally, I should refer to all the games with programmable elements like LogicBots which comes with a lots of sensors, logic gates, functional gates, actuators etc. to enable players to build their robots. I have to admit I haven’t played it, yet, since I have access to real robots, but it is conceptually interesting despite its mediocre reviews. I should also point out that logic gates have found their way into all kinds of construction games including Minecraft, Starbound and Terraria (afair).

Time Travel

Usually, I am not a big fan of time travel, but when done right reasoning across time rather than primarily spatially is a refreshing change of scenery. One of the earliest instances of reasoning over time periods I have experienced was Day of Tentacle, the successor of Maniac Mansion, in which you play with three different characters in the same house in the presence, past and future due to an experiment gone wrong. Since it is a point & click adventure there are lots of puzzles which incorporate the element. One character is stuck in a tree in the future, so when you get someone in the past (well, not “someone”- George Washington, but that is another story) to chop down that tree, it will vanish in the future thus freeing your other character. Or when you get a bottle of wine included in a time capsule and open that in the future, you obtain vinegar when you open that capsule in the future. And similarly when you mess with US history in major ways like changing the flag design and signing into law that every household is required to own a vacuum, you can solve puzzles in the future by having a tentacle-shaped flag and a vacuum at your disposal just to name a few examples. You also freeze a hamster for the future, send a contract via pony express in the past and dry a pullover for a very long time until it has shrunk to fit the unfrozen hamster in the future among others. In short: You constantly need to consider not only what you can do right now, but how you can impact the future and since you can send items between time zones the puzzles become quite interesting.

Another notable use of time travel as a puzzle element came out of the Portal community which as part of the Workshop extensions experimented with portals through time which culminated in the introduction of time portals in Portal Reloaded as explained below:

While this might seem trivial at first glance, it certainly adds to the puzzles and means that not only can you use decay in the future to your advantage, but also that you can leverage future cubes in the presence as shown below:

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply