Export file structure of to_python method

[info-rchitect]

Hi,

Can we clearly define the file structure created for a future to_python method? This info will also be helpful when also making the associated to_rust method. In O1 we only output pins, packages, sub_blocks, and registers. We had no concept of memory maps or address blocks IIRC.

Also, is there consensus on whether the export should be specific to the target or the application? vendor/falcon or vendor/example.

thx

[ginty]

Do we have an example IP-XACT we can refer to?
If you or @chrisnappi can provide something then it would be easier to define what should be produced from it.

[info-rchitect]

Do we have an example IP-XACT we can refer to?
If you or @chrisnappi can provide something then it would be easier to define what should be produced from it.

https://github.com/Origen-SDK/o2/blob/translator3/example/vendor/ip-xact/spirit1-4_ip-xact.xml

[ginty]

I was thinking of a more elaborate/real life example, but basically a component in IP-XACT is an Origen block.
So that example would just generate a single block (a sub-directory in <app>/blocks) containing a register file like https://github.com/Origen-SDK/o2/blob/master/example/example/blocks/dut/registers.py

[info-rchitect]

I was thinking of a more elaborate/real life example, but basically a component in IP-XACT is an Origen block.
So that example would just generate a single block (a sub-directory in /blocks) containing a register file like https://github.com/Origen-SDK/o2/blob/master/example/example/blocks/dut/registers.py

So it would be example/vendor/blocks/dut/registers.py?

[info-rchitect]

@ginty Also, do we need to have a sub-dir specifying python or would we potentially mix Python and Rust exported files in the same directory?

[info-rchitect]

@ginty Also, what will the API to connect sub-blocks to memory maps and address blocks look like? I believe we don't currently have a way to create a memory map and address block within a sub-block.

[ginty]

So it would be example/vendor/blocks/dut/registers.py?

Yeah, pretty much.

I think we might also add special handling for blocks which are DUTs (top-levels) vs. those which can be instantiated within a DUT, maybe the user will need to supply dut=true or similar at import-time.
We should follow what is done in example/blocks which is that all top-levels are below blocks/dut.
So in this example (with the block also called 'dut') it should be example/vendor/blocks/dut/derivatives/dut/registers.py, which would be less confusing were it to be called a device name, e.g. example/vendor/blocks/dut/derivatives/ip1234/registers.py

I think the Rust would go in the same dir, just with a different extension, .rs, .bin or whatever it is.

[ginty]

@ginty Also, what will the API to connect sub-blocks to memory maps and address blocks look like? I believe we don't currently have a way to create a memory map and address block within a sub-block.

This is creating memory maps and address blocks within a sub-block.
Say your example really contained two components, blk1 and blk2, then you would be creating:

• example/vendor/blocks/blk1/registers.py
• example/vendor/blocks/blk2/registers.py

Each containing their respective memory map and address blocks.

If there was a hierarchical relationship between them (and I'm kind of assuming IP-XACT can express such things which is why I was interested in a more complex example), say blk2 was a child of blk1, then you should also create example/vendor/blocks/blk1/sub_blocks.py containing:

SubBlock("blk2_instance_name", block_path="blk2")

[info-rchitect]

Thx @ginty for the explanations. Are the derivatives essentially a 1:1 map to an eventual target? Should the export also create target files?

[ginty]

Yes, and yes you could also create a corresponding target file.

[info-rchitect]

So I could choose not to make objects in memory and write directly to file and then load the files to make the objects in memory

[ginty]

Sure, and that's how I envisaged this would go originally.
However, I think the current path of implementing a to_python() has a significant advantage - which is that you take care of how to write out the files once, then all other importers from other formats can leverage that in future.
i.e. I think it is easier to build up the model in memory and then call to_python() on it, vs. writing out the files.

[info-rchitect]

Ok so to make everything modeled in memory first, don't we have to enhance the API to allow memory maps to connected to sub blocks or visa versa?

[ginty]

I think you might be right, I was thinking that #52 would give you the way to programmatically join blocks together, however I see now that this only provides a way to link up an existing blocks (i.e. those that already exist in the file system).
So yeah, something else is required, will try to add it soon.

[ginty]

Another thought I've been having on this is that maybe we should have a to_toml() output and this could replace both Python and Rust outputs.

I did some quick bench-marking the other day of how instantiating 20,000 registers compared in O2 vs O1 and O2 actually came out a bit slower (~3.5s vs. ~1.5s).
I suspect that this is not a reflection on Rust and that the bulk of the time is used up sending 20,000 regs worth of data over the Python -> Rust boundary.

So I'm wondering if we would be better off ditching Python for imported regs and use TOML instead. This is a format that ticks the box of being readable and easily editable by humans, but which is also well supported by Rust.
It would be interesting to generate a TOML file that defined some 20000 regs (format at your discretion) and then we can see how quickly that can be read in an instantiated in a purely Rust operation.

[info-rchitect]

@ginty I think I will complete all 3 formats so we can have proper benchmarking and be fully informed when we choose a direction.

[info-rchitect]

I think you might be right, I was thinking that #52 would give you the way to programmatically join blocks together, however I see now that this only provides a way to link up an existing blocks (i.e. those that already exist in the file system).
So yeah, something else is required, will try to add it soon.

Yeah the API does need some work IF one wants to use nested code blocks to add everything (which is the best way in the long haul IMO). However I think I could use this pattern for now.

owner = SubBlock("core0", block_path="core")
owner.with MemoryMap("test"):
    with AddressBlock("bank0"):
        SimpleReg("reg1", 0)
        with Reg("reg2", 0x0024, size=16) as reg:
            reg.bit([4,0], "adch", reset=0x1F)

[info-rchitect]

@ginty Not to throw a spanner in here (think that is correct usage) but could this make us reassess a Python only solution? Not trying to ditch Rust but would like to hear your thoughts on why Rust makes the most sense in the long haul. This is a question more for my edification rather than a serious proposal on my part.

[ginty]

I don't think the fact that register definition/instantiation is currently slower is a reflection on the overall performance we should expect. Also note that O1 was seriously optimized in this area, while O2 is barely a couple of months old, though it does already have lazy-loading of sub-blocks and register files which make this benchmark case of importing 20K regs at once not really realistic.
Also I'm pretty confident that anything we lose on the frontend we will get back on the backend when operations can be much more Rust-centric - e.g. pattern/flow optimization and rendering.
And finally the lib we are using for the Python/Rust binding is pretty new itself and already has performance improvements in mind, e.g. PyO3/pyo3#679

However, even if we ended up with O2 performance the same or a bit worse than O1 I would still support the current direction for the following non-performance-related reasons:

• I agree that Python makes sense for now, but I don't think it's a given that it will be the obvious choice for this type of application in the long term. I don't fancy going through another full re-write ever, so having our core in a C-compatible language is a good long term insurance policy against wanting to support a different application-level language in future.
• In recent years I have come to appreciate that compiled languages do provide a certain robustness that you don't get from dynamic languages - I guess there was a reason that most enterprise software is written in C++ or similar after all! And Rust forces you to write the most robust code of all. So O2 will feel much more solid to users than O1, hopefully never crashing within Origen itself and such issues always pointing back to user code.

Both of these are key reasons why I want to keep O2's Python footprint to a minimum.

[ginty]

Just to update this thread - I realised that I should probably try O2 compiled with Rust's --release option when benchmarking and with that change applied, O2 now instantiates the 20000 regs in ~0.5s vs O1's 1.5s.
That isn't even a like for like comparison - O1 just instantiates a placeholder and later materializes a register with bit instances on demand, whereas for O2, this is the time to fully instantiate all regs, bits and all.
So, it's fast!