Residual Net / ResNet in TL

[geometrikal]

I have been successfully using the resnet example code here. It would be nice to have a more concise form.

What do you think about adding a ResNet layer type? Interface would be similar to Conv2D, with addition of some flags such as type - what topology (BN-C-R / C-BN-R etc), and subsample - do we subsample by 2, etc.

[zsdonghao]

I am thinking about it as well~
But I found there are many kinds of resnet, for me, I found it is easier to build residual blocks by using a for loop. This is a open topic, please let me know, if anyone have suggestion.

BTW, I think this resnet example is clear than the one you read.

    n = Conv2d(256, (1,1), (1,1), act=None, W_init=w_init, b_init=None)(n)
    n = BatchNorm2d(decay=0.9, act=tf.nn.relu, gamma_init=g_init)(n)

    # res blocks
    nn = Conv2d(256, (3,3), (1,1), act=None, W_init=w_init, b_init=None)(n)
    nn = BatchNorm2d(decay=0.9, act=tf.nn.relu, gamma_init=g_init)(nn)
    nn = Conv2d(256, (3,3), (1,1), act=None, W_init=w_init, b_init=None)(nn)
    nn = BatchNorm2d(decay=0.9, act=None, gamma_init=g_init)(nn)
    n = Elementwise(tf.add)([n, nn])

    nn = Conv2d(256, (3,3), (1,1), act=None, W_init=w_init, b_init=None)(n)
    nn = BatchNorm2d(decay=0.9, act=tf.nn.relu, gamma_init=g_init)(nn)
    nn = Conv2d(256, (3,3), (1,1), act=None, W_init=w_init, b_init=None)(nn)
    nn = BatchNorm2d(decay=0.9, act=None, gamma_init=g_init)(nn)
    n = Elementwise(tf.add)([n, nn])

    nn = Conv2d(256, (3,3), (1,1), act=None, W_init=w_init, b_init=None)(n)
    nn = BatchNorm2d(decay=0.9, act=tf.nn.relu, gamma_init=g_init)(nn)
    nn = Conv2d(256, (3,3), (1,1), act=None, W_init=w_init, b_init=None)(nn)
    nn = BatchNorm2d(decay=0.9, act=None, gamma_init=g_init)(nn)
    n = Elementwise(tf.add)([n, nn])

    nn = Conv2d(256, (3,3), (1,1), act=None, W_init=w_init, b_init=None)(n)
    nn = BatchNorm2d(decay=0.9, act=tf.nn.relu, gamma_init=g_init)(nn)
    nn = Conv2d(256, (3,3), (1,1), act=None, W_init=w_init, b_init=None)(nn)
    nn = BatchNorm2d(decay=0.9, act=None, gamma_init=g_init)(nn)
    n = Elementwise(tf.add)([n, nn])

More info :

• any wise way to implement a residual network with 1000 layers ? #85
• https://github.com/wagamamaz/tensorlayer-tricks

[geometrikal]

Yes, that is nicely compact and clear, thanks! Maybe it is simpler to just add ResNet example (one that includes the downsampling and padding)? Off topic, but have you found that different ResNet block topologies are more suited to different types of data?

[zsdonghao]

Yes, I will summarise the examples.
I usually use the traditional one, I didn't find any big performance differences on different topologies on image applications.