Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

commit_dataset does not work with nested partitions #40

Closed
JoergRittinger opened this issue May 22, 2019 · 0 comments
Closed

commit_dataset does not work with nested partitions #40

JoergRittinger opened this issue May 22, 2019 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@JoergRittinger
Copy link
Contributor

The following code example

from kartothek.io.eager import commit_dataset, write_single_partition, create_empty_dataset_header
from kartothek.core.common_metadata import make_meta
import pandas as pd
import storefact


store = storefact.get_store_from_url("hfs:///tmp/some_directory")
for x in store.keys():
    store.delete(x)

df = pd.DataFrame({"a": [1, 1, 2, 2], "b": [3, 4, 5, 6]})

create_empty_dataset_header(
    store=store,
    dataset_uuid="uuid",
    table_meta={"table": make_meta(df, "table", ["a"])},
    partition_on=["a"],
)

partitions = []
for x in range(2):
    partitions.append(
        write_single_partition(
            store=store,
            dataset_uuid="uuid",
            data={"data": {"table": df}},
            partition_on=["a"],
        )
    )

commit_dataset(
    store=store,
    dataset_uuid="uuid",
    new_partitions=partitions,
    partition_on=["a"],
)

Will create the error message AttributeError: Accessing label attribute is not allowed while nested.

Additionally you can observe the behaviour that after commit_dataset the partitions list contains one less element. In my understanding commit_dataset should not manipulate input objects.

@fjetter fjetter added the bug Something isn't working label May 22, 2019
@lr4d lr4d mentioned this issue Jun 24, 2019
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Development

No branches or pull requests

3 participants