Cannot create a table from a pyarrow schema

[DavidEscott]

Apache Iceberg version

0.9.0

Please describe the bug 🐞

I can't figure out how one is actually expected to create a table from a pyarrow schema.

For this use case I have a pandas or polars table that I have directly created a pyarrow schema from. I would like to create a new empty table from this virgin pyarrow schema. It has no field ids and I would like pyiceberg to generate them.

io.pyarrow.pyarrow_to_schema would seem to be the way to convert the schema to an iceberg schema. It first calls the _HasIds() visitor, this is seemingly returning False because the schema doesn't have ids.

Next is to try and apply the name_mapping, but I don't want to provide a name_mapping.

It then fails warning me that the table doesn't have the Iceberg table does not have 'schema.name-mapping.default' defined, which is circular as I don't have a table yet.

---

It seems like pyarrow_to_schema is missing a case where one would apply the _ConvertToIcebergWithoutIDs visitor to the arrow schema as is done in _pyarrow_to_schema_without_ids

Is there some reason we are not supposed to build tables this way?

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[jayceslesar]

Does the following not work?

import pyarrow as pa

schema = pa.schema(
    [
        pa.field("foo", pa.string(), nullable=True),
        pa.field("bar", pa.int32(), nullable=False),
        pa.field("baz", pa.bool_(), nullable=True),
    ]
)

catalog.create_table(
    identifier="docs_example.bids",
    schema=schema,
)

via https://py.iceberg.apache.org/api/#create-a-table

[DavidEscott]

Not in my use-case, because I also want to specify the PartitionSpec for create_table. From what I can tell the signature of catalog.create_table doesn't work at present.

create_table suggests that it can accept a pa.Schema while simultaneously accepting a PartitionSpec and SortOrder. In order to construct a PartitionSpec you have to provide PartitionFields and for those you need the field_ids, which you don't have because it is a pa.schema and even if you did have ids, they wouldn't be guaranteed to match the ids that iceberg will assign.

It seems that one of the following should be done:

• A utility function needs to be provided to convert the pa.schema to an iceberg schema AND apply the field id numbering.
• Or PartitionSpec and SortOrder of create table need to be able to work with the names of columns instead of the field_ids.
• Or remove PartitionSpec/SortOrder from create table and instead direct users to create the table as an empty object, and then apply the partitioning and sorting rules to the table skeleton.

[DavidEscott]

There seem to be two existing issues related to this, but both are closed. [#1797] and [#1100]. I've adopted the approach from 1100, and it does work, but I certainly disagree with these tickets being closed.

[sungwy]

Hi @DavidEscott thanks for raising this issue! As you pointed out this has been a frequently discussed issue on PyIceberg, and there's still the original open issue that tracks this feature request: #1284 (comment)

Unfortunately the discussion thread ended with a disagreement on whether field_ids should be exposed as input parameters for the high level API calls, and the feature request you are raising was never implemented.

However, there is a way to create a table and add a partition spec to it from a pyarrow table. Could you check out the last example in https://py.iceberg.apache.org/api/#create-a-table that uses create_table_transaction in this section and let us know if that works for your use case?

[dannypage]

I've run into this as well, and appreciate the transaction method. However, I don't see a way to do this with sorting? Perhaps it's not possible to add sorting after the table is created and data is potentially added. In which case I'd probably prefer the idea that David suggested:

A utility function needs to be provided to convert the pa.schema to an iceberg schema AND apply the field id numbering.

I was doing a hacky version of this where I'd create a temp_table, grab the IDs created, purge it, and then use those ids with the creation of the final table. A function that could convert the schema would be ideal to skip that step.

Thanks!