HIVE-28956 : Use msdb.alterPartitions() API and implement batching for alter table add column cascade command #5814
Conversation
vikramahuja1001
force-pushed
the
DirectSqlAddColumn
branch
from
eb1c3b1 to
db2dd7b
Compare
asf-ci-hive
added
tests pending
tests failed
and removed
tests unstable
tests pending
labels
vikramahuja1001
force-pushed
the
DirectSqlAddColumn
branch
from
db2dd7b to
c754743
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests failed
tests pending
labels
| Table finalOldt = oldt; | ||
| int partitionBatchSize = MetastoreConf.getIntVar(handler.getConf(), | ||
| MetastoreConf.ConfVars.BATCH_RETRIEVE_MAX); | ||
| Batchable.runBatched(partitionBatchSize, parts, new Batchable<Partition, Void>() { |
There was a problem hiding this comment.
PR naming is confusing. Where is the direct sql here? This PR introduces batching for partition metadata update, or am I missing something?
There was a problem hiding this comment.
The previously used msdb.alterPartition was only using JDO. The newly added msdb.alterPartitions API is based on DirectSql.
So this PR is introducing Batching and also using direct sql based msdb.alterPartitions API.
There was a problem hiding this comment.
Where is the direct sql here?
There was a problem hiding this comment.
@deniskuzZ , PR naming is confusing, i will update the same from my end. Thanks for pointing it out in your earlier messages.
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
| tableName, part.getValues(), oldCols, finalOldt, part, null, null); | ||
| assert (colStats.isEmpty()); | ||
| oldPart.setParameters(part.getParameters()); | ||
| oldParts.add(oldPart); |
There was a problem hiding this comment.
why are you constructing oldParts here if it's only used when cascade is false?
vikramahuja1001
force-pushed
the
DirectSqlAddColumn
branch
from
926a4c7 to
2a5cc01
Compare
vikramahuja1001
changed the title
|
|
@deniskuzZ , i have made the necessary changes. Could you please check the PR once again? |
What changes were proposed in this pull request?
Currently Alter table add column cascade uses msdb.alterPartition API. This API only JDO based implementation. Instead of this use msdb.alterPartitions which supports both DirectSql and JDO Implementation and use batching as well.
Why are the changes needed?
DirectSql bein more optimized than JDO can improve performance for tables with very high number of partitions and very high number of columns.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Tested this on a 3 node cluster on a table with 5000+ partitions and 900+ columns.