Exploration into Image Based Procedural Generation (Unity)

I watched this cool talk by Anastasia Opara called Creativity of Rules and Patterns: Designing Procedural Systems and since I have a soft spot for anything procedural, I was immediately inspired to play around with the concept.

In the talk, Anastasia describes an approach to procedural composition that imagines every element as physically weighted, and the goal is to balance the system around the central point. She uses a modified version of Newtons Law of Universal Gravitation to mathematically calculate the next elements position and mass that maximizes the equilibrium of the system.

screenshot from Anastasia’s talk . Red is the original element, Green is the element we are trying to optimize. In the optimal state, the yellow vector will have a magnitude of 0.

I found it really interesting how she brought concepts from the physical world to solve an artistic composition problem — and decided to try it out for myself in Unity. In the GIF below, the red cubes position and scale(mass) changes as I change the position and scale of the two blue cubes, in order to maximize the equilibrium with respect to the central sphere.

This was fun to play with, but I came to a fundamental problem. The third cube puts the system in equilibrium, so there’s no need for a fourth cube! Of course there are ways around this ( moving the central point, etc. ), but I felt like taking the exploration in a new direction.

I started reading about basic principles of design and composition, just to get more familiar with the subject. Gestalt’s principles of visual perception make you question things you always knew but never paid attention to, such as why we perceive objects to be part of a group. I also stumbled upon some compositional techniques, such as having directional elements that promote flow by drawing your eyes from one element to the next…

Flow… it would be cool to have a generation pattern that flowed…

The Diamond Square algorithm is a fractal technique used for terrain generation and has a really beautiful flow… what if I could use this algorithm to drive the generation? I thought about coding up the algorithm and storing the resulting values into a texture that I could use realtime. But since I just needed a texture at the end of the day, I decided to make life easier and use a pre-made image of the diamond square algorithm —

I was excited by this idea because if the technique worked out, you should be able to get new results by swapping out the base image :)

infinite diamond square fractal by Jason Davies

The overview of the algorithm I had in my head went something like this :

• Start at arbitrary point on the image.
• Chose the next cube’s point to be on of the 8 neighboring points ( at a given step size ) that minimizes the following two values:
• Difference in color value.
• Dot product of the previous and current direction of motion. This is so that the cubes are inclined to move in a flowing pattern.

I had parameters to control the step size, randomness applied to choosing the neighboring point positions, and weights of the minimization steps —

cubes are generating by minimizing the color difference from reference image, and angle of movement

Clearly the overlapping cubes were a problem — so I made a world grid that had as many units as the texture (2048 x 1024 in this case), and made sure every new cube was not intersecting with the filled world grid positions.

fsafa

So at this point, the cube sizes look kind of neat, and they were just cycling through 1–5 units.

For further exploration, I thought I would apply a modified version of Anastasia’s compositional approach, where we solve for the next cubes size by maximizing the equilibrium of the position and sizes of the previous N cubes.

IMHO I feel like just cycling through predefined scales was more reliable, but the balancing approach definitely has something interesting about it.

Soo that sums up the exploration — in the future I would like to take this exploration and tune it for a specific goal, such as alien cities, forests, or caves. It would also be cool to optimize the problem for a first person view, instead of only thinking of the 2D top down scattering problem.

Check out the Unity source code @ https://github.com/sneha-belkhale/procedural-image-based-scattering

Other projects @ https://codercat.tk

Renders / screenshots @ https://www.instagram.com/snayss/

And some results with other reference images below ❤

blurry galaxy

its jupiter!!

without a background image