feat: allow to override parquet_prefix and include idle costs

[burahimu]

Hello there 👋🏼

This PR introduces several enhancements. The main changes are:

• Allow to override the full parquet prefix. We wanted to add a by hour partitioning
• Retrieve idle costs in order to know what we're wasting (we're not sure if the Glue table can handle idle costs)

Note

This PR is in draft until we confirm the latest point.

[lmello]

thanks for your contribution.
left some comments on it.

[lmello]

@burahimu We just merged a PR that implemented support for azure and removed the config from the code to external files to allow parametrization.

Can you have a look at the new code in v0.2.0-rc3 (which is in the main branch) and rebase your pr.

Thank you.

[burahimu]

@burahimu We just merged a PR that implemented support for azure and removed the config from the code to external files to allow parametrization.

Can you have a look at the new code in v0.2.0-rc3 (which is in the main branch) and rebase your pr.

Thank you.

Hi @lmello , it's rebased 👍🏼

[burahimu]

Hi 👋🏼

Just a little up about this PR 😇

[lmello]

@burahimu
can you please look at all the storage backends to see if they will respect the parameter, some of them might have different implementations. we would need to ensure they don't do any logic on topic of prefix and just write the file under the prefix for your change to work.

[lmello]

Also let me know which backend you are using and how you tested/validated your change. thank you!