Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partition by should not quote the items #201

Closed
Conq1 opened this issue Jul 11, 2023 · 5 comments · Fixed by #277
Closed

Partition by should not quote the items #201

Conq1 opened this issue Jul 11, 2023 · 5 comments · Fixed by #277
Labels
bug Something isn't working

Comments

@Conq1
Copy link

Conq1 commented Jul 11, 2023

Describe the enhancement requested

I would like to use the features of Iceberg partition transformation, but when adding them to "partition_by" in config they get quotes surrounding each item, meaning the create table will fail.

Justification for this enhancement

Partition transformation is a powerful Iceberg feature.

@Conq1
Copy link
Author

Conq1 commented Jul 11, 2023

For now I have created my own macro of config_cols.
Might be a better solution as config_cols is used multible other places.

New version without quoting:

{% macro config_cols(label, default_cols=none) %}
  {%- set cols = config.get(label | replace(" ", "_"), validator=validation.any[list, basestring]) or default_cols -%}
  {%- if cols is not none %}
    {%- if cols is string -%}
      {%- set cols = [cols] -%}
    {%- endif -%}
    {{ label }} ( {{ cols | join(', ') }}  )
  {%- endif %}
{%- endmacro -%}

@fabrice-etanchaud
Copy link

Hi @Conq1 , dremio usually accepts double quotes around object names (I just tested a partitioned table creation with partition by ("mycol")), are you sure the error is not coming from another problem, maybe a misnamed column ?

@Conq1
Copy link
Author

Conq1 commented Jul 13, 2023

Yes @fabrice-etanchaud . Problem is when you want to use partition-transformation like truncate(1, "mycol"):
https://docs.dremio.com/software/sql-reference/sql-commands/apache-iceberg-tables/apache-iceberg-create/
So quick fix for me was to remove the quotes, but a better fix maybe some other syntax to indicate when you want to use partition-transformation.

@fabrice-etanchaud
Copy link

Thank you @Conq1 , I understand. You are right, the solution may be to remove the systematic double quoting, and let the user decide.

@wabu
Copy link

wabu commented Nov 25, 2024

I'm facing the same issue. Any progress on implementing this?

@bcmeireles bcmeireles linked a pull request Dec 4, 2024 that will close this issue
1 task
@bcmeireles bcmeireles added the enhancement New feature or request label Dec 4, 2024
@bcmeireles bcmeireles added bug Something isn't working and removed enhancement New feature or request labels Feb 11, 2025
@bcmeireles bcmeireles linked a pull request Feb 11, 2025 that will close this issue
2 tasks
bcmeireles added a commit that referenced this issue Feb 12, 2025
### Summary

When using `partition_by`, dbt-dremio could be causing invalid
statements to partition columns, surrounding the way quotation was being
handled.

### Description

The sistematic double quoting has been removed from the connector's
logic, stopping such situations from happening and leaving quotation to
be decided by the user. An example of a table materialization under this
new logic is:

```
{{
    config(
        materialized = 'table',
        partition_by = ['forecast_effective_date', 'year("datetime_utc")', 'month("datetime_hour_ending_utc")']
    )
}}
```

### Test Results

n/a

### Changelog

-   [X] Added a summary of what this PR accomplishes to CHANGELOG.md

### Contributor License Agreement

<!--- Applicable for non Dremio employees and for first time
contributors -->

- [X] Please make sure you have signed our [Contributor License
Agreement](https://www.dremio.com/legal/contributor-agreement/), which
enables Dremio to distribute your contribution without restriction.

### Related Issue

#201
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
4 participants