Improve rule API and build dependency order

[waruqi]

Roadmap

• Insert custom rule before built-in rule
• Source file granularity dependency order control
• Unified DAG Implementation
• Insert and remove build tasks in real time
• Add a preparation phase before building to handle C++ module dependency scanning and other build preparation work
• Support for build.fence
• Support for job group for same rule
• ...

Related

• Add a "scan" compilation stage #6196
• Global batchjobs #6197
• PCH can be pre-compiled concurrently to the other targets #4167
• c++ modules 和 预编译头文件冲突问题 #4051
• bin2c构建优先级错误 #4801

[waruqi]

Unified DAG Implementation

If no additional dependency chains are added, xmake will add a default dependency order according to its internal build rules.

rule("foo")
    on_build_file(function (target, jobgraph, sourcefile, opt)
        jobgraph:add_job(target:name() .. "/" .. sourcefile, function(job, opt)
            -- TODO
        end)
    end, {jobgraph = true})

Source file granularity dependency order control

rule("foo")
    on_build_files(function (target, jobgraph, sourcebatch, opt)
        for _, sourcefile in sourcebatch.sourcefiles do
            local job_name = target:name() .. "/" .. sourcefile
            jobgraph:add_job(job_name, function(job, opt)
                -- TODO
            end)

            -- add dependency chain, job -> other target jobs
            jobgraph:add_deps(job_name, other_target:name() .. "/buildfiles")
        end
    end, {jobgraph = true})

Insert and remove build tasks in real time

rule("foo")
    on_build_file(function (target, jobgraph, sourcefile, opt)
        local job_name = target:name() .. "/" .. sourcefile
        jobgraph:add_job(job_name, function(job, opt)
            -- TODO

            jobgraph:add_job("zoo/build", function (job, opt)
            end)

            jobgraph:remove_job("bar/buildfiles")
        end)
    end, {jobgraph = true})

Add a preparation phase before building

on_prepare scripts are called before all build scripts, we can do some things, e.g. scan c++ module source files, autogen ...

rule("scan_module_files")
    on_prepare(function (target, opt)
        -- scan module files
    end)

or

rule("scan_module_files")
    on_prepare(function (target, jobgraph, opt)
        jobgraph:add_job(target:name() .. "/scanfiles", function (job, opt)
            -- scan module files
        end)
    end, {jobgraph = true})

or

rule("scan_module_files")
    set_extensions("*.mpp")
    on_prepare_files(function (target, jobgraph, sourcebatch, opt)
        jobgraph:add_job(target:name() .. "/scanfiles", function (job, opt)
            -- scan module files
        end)
    end, {jobgraph = true})

Insert custom rule before built-in rule

We can use add_parents to insert deps to the custom rules in the given builtin rules.

-- the builtin rule
rule("c++")
    -- ...

-- c++ depend on foo
rule("foo")
    add_parents("c++")
    -- ...

Support for job group for same rule

rule("foo")
    on_build_files(function (target, jobgraph, sourcebatch, opt)
        local group_name = target:name() .. "/buildfiles"
        jobgraph:add_group(group_name, function ()
            for _, sourcefile in sourcebatch.sourcefiles do
                local job_name = target:name() .. "/" .. sourcefile
                jobgraph:add_job(job_name, function(job, opt)
                    -- TODO
                end)
            end
        end)

        -- add dependency chain, other target jobs -> this build group
        jobgraph:add_deps(other_target:name() .. "/buildfiles", group_name)
    end, {jobgraph = true})

[waruqi]

This is an initial draft, any idea? @Arthapz @SirLynix @star-hengxing

[Arthapz]

It cover all my needs for module and the api is simple, I like it

[waruqi]

Ok, I will implement them in next version.