# First steps with InfoGAN

A while back, I talked about using InfoGAN networks to learn interpretable dimensions for the shape domain of a conceptual space. As this has already been a few months ago, I think it is now time for an update. Where do I stand with my research with respect to this topic?

Researchers have reported that the InfoGAN  neural network architecture is capable of extracting meaningful and interpretable dimensions from unlabeled data sets. You just have to specify the number of dimensions you are looking for, provide some data set, and the network will do the rest for you. This has been applied to various data sets, including MNIST (gray-scale images of hand-written digits) and CelebA (faces of celebrities). As we are interested in interpretable dimensions for defining a conceptual space, I thought: “Well, let’s apply this to a data set of shapes. Maybe we find interpretable dimensions that we can use for spanning the shape domain in a conceptual space.”

Last summer, I found two Master students at my institute that were also interested in this topic. So I asked them to create a simple data set of shapes (rectangles, triangles, and ellipses) and to apply the original InfoGAN code that the developers had made publicly available on GitHub. After having solved a few technical issues, they managed to train the network on their artificial data set (their code can be found here). After training the network for 500 epochs (which is ten times as long as reported in the InfoGAN paper for the MNIST example), the generator part of the InfoGAN was able to create the shapes shown in Figure 1.

As you can see, the generated shapes are not perfect (especially the leftmost one is neither an ellipse, nor a rectangle, nor a triangle), but at least we don’t get random noise. Now in order to check whether the two extracted dimensions are actually interpretable, one can vary the value on one of these dimensions and see whether the generated images change. This is shown in Figure 2 where each row corresponds to a different value on the first extracted dimension.

As you can see, the generated shapes indeed do change as you change the value on this dimension. However, it is not quite clear how to interpret this change: The leftmost shape changes from something weird into a triangle, whereas the rightmost shape changes from a rectangle to an ellipse. The rectangle in the middle in contrast does not change much at all, it only becomes a tiny bit smaller. So although the network is able to generate more or less reasonable shapes, we do not get interpretable dimensions in this case.

The problem at this point is that we did not have a clear expectation about how many dimensions we need (we simply tried with two) and what kinds of dimensions we expected to find. That makes it hard to “debug” the behavior of the network: After all, how do you decide whether changing a specific hyperparmeter makes your results “better” if you have no clear idea what a “good” result looks like?

So I decided to simplify the problem.

What I am looking at currently is a simplified case where we only have rectangles that are centered in the image. This means, that the only two parameters that change in the data set are the width and the height of the rectangle – position, rotation and type of shape stay constant. This seems to be a fairly trivial scenario, but at least we know exactly how many dimensions we need and what these dimensions should express (namely, width and height). So I re-ran the whole thing with this simplified data set.

Using the original InfoGAN implementation was however causing some problems: On the one hand, the training seemed to be quite instable (about 85% of the runs crashed before finishing training). On the other hand, because this implementation uses an outdated version of TensorFlow, I was not able to run the code on GPU, which significantly slowed down the training process. However, when the code ran through I got results like in Figure 3 and 4.