Consider using Dagre.NET to visualize and perhaps edit/export the transformer architecture

[GeorgeS2019]

Is your feature request related to a problem? Please describe.
We need a way to visualize/edit/export (onnx) transformer architecture for .NET community.

Describe the solution you'd like
Dagre.NET and Dendrite could evolve to support this need. Seq2SeqSharp could export the architecture in Json format for import into Dendrite or use Dagre.NET to export images.

[zhongkaifu]

Thanks @GeorgeS2019 . It's a great idea.
Did you know if Dagre.NET project has any example of Json format for models ? If so, I can take a look. I'm also thinking about getting rid of BinaryFormatter for model serialization and de-serialization. :)

Thanks
Zhongkai Fu

[GeorgeS2019]

@zhongkaifu

It adhere as close as possible to the JavaScript library version

if there is anything you need, please feedback directly here.

Csharp versoin

DagreInputGraph dg = new DagreInputGraph();
var nd1 = dg.AddNode();
var nd2 = dg.AddNode();
dg.AddEdge(nd1, nd2);
dg.Layout(); 
Console.WriteLine($"node1 : {nd1.X} {nd1.Y}");
Console.WriteLine($"node2 : {nd2.X} {nd2.Y}");

JavaScript version

// Create a new directed graph 
var g = new dagre.graphlib.Graph();

// Set an object for the graph label
g.setGraph({});

// Default to assigning a new object as a label for each new edge.
g.setDefaultEdgeLabel(function() { return {}; });

// Add nodes to the graph. The first argument is the node id. The second is
// metadata about the node. In this case we're going to add labels to each of
// our nodes.
g.setNode("kspacey",    { label: "Kevin Spacey",  width: 144, height: 100 });
g.setNode("swilliams",  { label: "Saul Williams", width: 160, height: 100 });
g.setNode("bpitt",      { label: "Brad Pitt",     width: 108, height: 100 });
g.setNode("hford",      { label: "Harrison Ford", width: 168, height: 100 });
g.setNode("lwilson",    { label: "Luke Wilson",   width: 144, height: 100 });
g.setNode("kbacon",     { label: "Kevin Bacon",   width: 121, height: 100 });

// Add edges to the graph.
g.setEdge("kspacey",   "swilliams");
g.setEdge("swilliams", "kbacon");
g.setEdge("bpitt",     "kbacon");
g.setEdge("hford",     "lwilson");
g.setEdge("lwilson",   "kbacon");

dagre.layout(g);

[fel88]

Did you know if Dagre.NET project has any example of Json format for models ?

Dendrite supports only ONNX so far, but I can implement any Json format that you provide.

[zhongkaifu]

Thanks @GeorgeS2019 and @fel88 . It's really helpful.

In Seq2SeqSharp, I used to use Microsoft.Msagl.Drawing to draw operators/layers/networks, however, since it doesn't support .NET core and .NET 5.0, so I comment it out. But I still keep the empty of it. If you look at VisualizeNodes method in https://github.com/zhongkaifu/Seq2SeqSharp/blob/master/Seq2SeqSharp/Tools/ComputeGraphTensor.cs file, you will find these code.

So, one thing we could do it to replace those old code I comment out to your code in Dagre.NET project. In addition, methods in ComputeGraphTensor.cs are operator level, so they are good entry pointers for visualization and ONNX export.

I'm glad if you could do it and let me know if you have any further question on it.

Thanks
Zhongkai Fu

[GeorgeS2019]

@zhongkaifu @fel88

Is there a graph sub patterns or regex that can be used to search the complete e.g BERT.ONNX graph to create groups or nested graphs to create boundaries around e.g. Encoder, Decoder of a transformer architecture?

[zhongkaifu]

@GeorgeS2019 Yes, Seq2SeqSharp does have logic to create boundaries for sub-graphs. You can check method "IComputeGraph CreateSubGraph(string name)" in https://github.com/zhongkaifu/Seq2SeqSharp/blob/master/Seq2SeqSharp/Tools/ComputeGraphTensor.cs file as well. However, I already commented it out due to the same reason in above.

[GeorgeS2019]

@zhongkaifu thnx for the valuable tip :-)

[GeorgeS2019]

method CreateSubGraph(string name)

public IComputeGraph CreateSubGraph(string name)
{
    ComputeGraphTensor subGraph = new ComputeGraphTensor(m_weightTensorFactory, m_deviceId, m_needsBackprop, m_backprop, isSubGraph: true);

    if (m_visNeuralNetwork)
    {
        // Create parameters for neural network visualization
        subGraph.m_opsViz = m_opsViz;
        subGraph.m_setEdges = m_setEdges;
        subGraph.m_name2SubGraph = m_name2SubGraph;
        if (m_name2SubGraph.ContainsKey(name) == false)
        {
            int index = name.LastIndexOf(".");
            subGraph.m_subGraph = new Subgraph(name)
            {
                LabelText = name.Substring(index + 1)
            };

            m_name2SubGraph.Add(name, subGraph.m_subGraph);

            if (m_subGraph == null)
            {
                m_opsViz.RootSubgraph.AddSubgraph(subGraph.m_subGraph);
            }
            else
            {
                m_subGraph.AddSubgraph(subGraph.m_subGraph);
            }
        }
        else
        {
            subGraph.m_subGraph = m_name2SubGraph[name];
        }
    }

    return subGraph;
}

method VisualizeNodes(IEnumerable sourceNodes, IWeightTensor targetNode)

private void VisualizeNodes(IEnumerable<IWeightTensor> sourceNodes, IWeightTensor targetNode)
{
    if (!m_visNeuralNetwork || m_deviceId != 0)
    {
        return;
    }

    // Create node for target tensor
    int index = targetNode.Name.LastIndexOf('.');
    Microsoft.Msagl.Drawing.Node tgtNode = m_opsViz.AddNode(targetNode.Name);
    tgtNode.LabelText = targetNode.Name.Substring(index + 1);

    if (targetNode.IsTrainable)
    {
        tgtNode.Attr.FillColor = Microsoft.Msagl.Drawing.Color.LightSteelBlue;
    }

    if (m_subGraph != null)
    {
        // Current compute graph is a sub-graph
        m_subGraph.AddNode(tgtNode);
    }

    // Create edges for each source node and target node
    foreach (IWeightTensor sourceNode in sourceNodes)
    {
        if (!string.IsNullOrEmpty(sourceNode.Name) && !string.IsNullOrEmpty(targetNode.Name))
        {
            string key = $"{sourceNode.Name}->{targetNode.Name}";
            if (m_setEdges.Contains(key))
            {
                continue;
            }

            int srcIndex = sourceNode.Name.LastIndexOf('.');
            Microsoft.Msagl.Drawing.Node srcNode = m_opsViz.AddNode(sourceNode.Name);
            srcNode.LabelText = sourceNode.Name.Substring(srcIndex + 1);
            if (sourceNode.IsTrainable)
            {
                srcNode.Attr.FillColor = Microsoft.Msagl.Drawing.Color.LightSteelBlue;

                if (m_subGraph != null)
                {
                    m_subGraph.AddNode(srcNode);
                }
            }

            Edge edge = m_opsViz.AddEdge(sourceNode.Name, targetNode.Name);

            m_setEdges.Add(key);
        }
    }
}