Skip to content

Commit

Permalink
Merge pull request #45 from arenadata/6.7.0-sync
Browse files Browse the repository at this point in the history
ADBDEV-4011 6.7.0 sync
  • Loading branch information
xardazzzzzz authored Jul 31, 2023
2 parents 21efde5 + bcd3637 commit 8ad87e9
Show file tree
Hide file tree
Showing 484 changed files with 13,000 additions and 3,489 deletions.
22 changes: 14 additions & 8 deletions .github/workflows/create-release-on-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,17 +22,23 @@ jobs:
tag_name: ${{ github.ref }}
release_name: PXF Version ${{ github.ref }}
body: |
## 6.6.0 (04/06/2023)
## 6.7.0 (07/13/2023)
### Enhancements:
- [#949](https://github.com/greenplum-db/pxf/pull/949) Support for fixedwidth formatter with new `*:fixedwidth` PXF profiles
- [#954](https://github.com/greenplum-db/pxf/pull/954) Update table options names to not include dash character
- [#955](https://github.com/greenplum-db/pxf/pull/955) Bump jackson-databind from 2.13.4.1 to 2.13.4.2 in /automation
- [#956](https://github.com/greenplum-db/pxf/pull/956) Add pxfdelimited_import formatter to support multibyte delimiters for TEXT and CSV profiles
- [#960](https://github.com/greenplum-db/pxf/pull/960) Add support year with more than 4 digits in 'date' or 'timestamp'
- [#973](https://github.com/greenplum-db/pxf/pull/973) Enable write flow for FDW for non-text/csv formats
- [#976](https://github.com/greenplum-db/pxf/pull/976) Restrict PXF to listen to local requests only
- [#979](https://github.com/greenplum-db/pxf/pull/979) Add logging to the LineBreakAccessor for the write
- [#983](https://github.com/greenplum-db/pxf/pull/983) Bump Springboot to 2.7.12
- [#984](https://github.com/greenplum-db/pxf/pull/984) Enable writing data in JSON format using *:json profiles
- [#989](https://github.com/greenplum-db/pxf/pull/989) Bump snappy to 1.1.10.1
### Bug Fixes:
- [#940](https://github.com/greenplum-db/pxf/pull/940) Introduced options to handle decimal overflow when writing Parquet files
- [#967](https://github.com/greenplum-db/pxf/pull/967) FDW: Fix for skipping the dropped and correctly counting Projection Index
- [#978](https://github.com/greenplum-db/pxf/pull/978) Added erroring out logic for decimal overflow for ORC
draft: false
prerelease: false
Expand Down
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
# Changelog

## 6.7.0 (07/13/2023)

### Enhancements:

- [#956](https://github.com/greenplum-db/pxf/pull/956) Add pxfdelimited_import formatter to support multibyte delimiters for TEXT and CSV profiles
- [#960](https://github.com/greenplum-db/pxf/pull/960) Add support year with more than 4 digits in 'date' or 'timestamp'
- [#973](https://github.com/greenplum-db/pxf/pull/973) Enable write flow for FDW for non-text/csv formats
- [#976](https://github.com/greenplum-db/pxf/pull/976) Restrict PXF to listen to local requests only
- [#979](https://github.com/greenplum-db/pxf/pull/979) Add logging to the LineBreakAccessor for the write
- [#983](https://github.com/greenplum-db/pxf/pull/983) Bump Springboot to 2.7.12
- [#984](https://github.com/greenplum-db/pxf/pull/984) Enable writing data in JSON format using *:json profiles
- [#989](https://github.com/greenplum-db/pxf/pull/989) Bump snappy to 1.1.10.1

### Bug Fixes:

- [#967](https://github.com/greenplum-db/pxf/pull/967) FDW: Fix for skipping the dropped and correctly counting Projection Index
- [#978](https://github.com/greenplum-db/pxf/pull/978) Added erroring out logic for decimal overflow for ORC

## 6.6.0 (04/06/2023)

### Enhancements:
Expand Down
14 changes: 1 addition & 13 deletions automation/README.Linux.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Running Automation on Linux

**Note:** This information was documented based on the steps taken to get automation running on a Debian Bookworm (12) system.
They are intended to be used in tandem with the information in the main README file.

## Locale Setup

Expand All @@ -19,19 +20,6 @@ gpstop -a
gpstart -a
```

## SSH Setup

```sh
sudo tee /etc/ssh/sshd_config.d/pxf-automation.conf >/dev/null <EOF
# pxf automation uses an old SSH2 Java library that doesn't support newer KexAlgorithms
# this assumes that /etc/ssh/sshd_config contains "Include /etc/ssh/sshd_config.d/*.conf"
# if it doesn't, try adding this directly to /etc/ssh/sshd_config
KexAlgorithms +diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
HostKeyAlgorithms +ssh-rsa,ssh-dss
PubkeyAcceptedAlgorithms +ssh-rsa,ssh-dss
EOF
```

## Python 2 Setup

If you are following [the python instructions](README.md#python-2-setup), but do not have pip installed.
Expand Down
12 changes: 7 additions & 5 deletions automation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,18 +31,20 @@ The following algorithms _must_ be included:
- diffie-hellman-group14-sha1
- diffie-hellman-group1-sha1

We also need to ensure that the `rsa` and `dss` key algorithms are accepted for both the host and public key authentication.
If they are not, you can enable them with the following config file:

```bash
sudo tee -a /etc/ssh/sshd_config.d/pxf-automation-kex.conf <<EOF
sudo tee -a /etc/ssh/sshd_config.d/pxf-automation.conf >/dev/null <<EOF
# pxf automation uses an old SSH2 Java library that doesn't support newer KexAlgorithms
# this assumes that /etc/ssh/sshd_config contains "Include /etc/ssh/sshd_config.d/*.conf"
# if it doesn't, try adding this directly to /etc/ssh/sshd_config
KexAlgorithms +diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
HostKeyAlgorithms +ssh-rsa,ssh-dss
PubkeyAcceptedAlgorithms +ssh-rsa,ssh-dss
EOF
```


Then restart sshd based on your OS.
For MacOS, either run in terminal

Expand Down Expand Up @@ -135,7 +137,7 @@ If you wish to run with cache
make OFFLINE=true
```

Note: if you get an error saying that the jar does not exist, ensure that you have
Note: If you get an error saying that the jar does not exist, ensure that you have
a) installed the PXF server, and
b) only have 1 jar file inside `/usr/local/pxf/application/`

Expand Down Expand Up @@ -309,12 +311,12 @@ You can read more about TestNG here http://testng.org/doc/index.md
## Run reports and logs
Automation logs will generated into "automation-logs" directory and will be divided to directories according to each ran java classes.
Automation logs will be generated into "automation-logs" directory and will be divided to directories according to each ran java classes.
In every "class" directory will be files according to the following format: <time-stamp>_<ran method name>.log
<img src="images/72680961.png" class="confluence-embedded-image confluence-content-image-border" width="454" height="598" />
TestNg report will generated into target/surefire-reports
TestNg report will be generated into target/surefire-reports
<img src="images/68125531.png" class="confluence-embedded-image confluence-content-image-border" width="1084" height="612" />
# IDE Setup (IntelliJ) and Automation Debugging
Expand Down
2 changes: 1 addition & 1 deletion automation/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<hdp.hbase.version>1.3.2.1</hdp.hbase.version>
<hdp.hive.version>1.1.0</hdp.hive.version>
<orc.version>1.6.13</orc.version>
<snappy.java.version>1.1.1.7</snappy.java.version>
<snappy.java.version>1.1.10.1</snappy.java.version>
<powermock.version>1.6.4</powermock.version>
</properties>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import jsystem.framework.report.Reporter;

import org.greenplum.pxf.automation.structures.tables.basic.Table;
import org.greenplum.pxf.automation.structures.tables.pxf.ExternalTable;
import org.postgresql.util.PSQLException;

import org.greenplum.pxf.automation.utils.exception.ExceptionUtils;
Expand Down Expand Up @@ -158,7 +159,31 @@ public void insertData(Table source, Table target) throws Exception {
dataStringBuilder.append(",");
}
}
runQuery("INSERT INTO " + target.getName() + " VALUES " + dataStringBuilder);
insertData(dataStringBuilder.toString(), target);
}

/**
* Inserts data from the provided string into the target Table. The string is expected to contain data
* in SQL format that follows the 'INSERT INTO [table] VALUES ' clause.
*
* @param data string containing data to insert
* @param target table to insert data into, can be an internal, an external or a foreign table
* @throws Exception is operation fails
*/
public void insertData(String data, Table target) throws Exception {
if (!data.startsWith("(")) {
data = "(" + data;
}
if (!data.endsWith(")")) {
data = data + ")";
}

String query = "INSERT INTO " + target.getName() + " VALUES " + data;
if (target instanceof ExternalTable) {
runQueryInsertIntoExternalTable(query);
} else {
runQuery(query);
}
}

@Override
Expand Down Expand Up @@ -191,6 +216,16 @@ public long runQueryTiming(String query) throws Exception {
return System.currentTimeMillis() - startTimeInMillis;
}

/**
* Run query that inserts data into an external or a foreign table and ignores a warning about not being able to
* analyze a foreign table (if applicable) because PXF FDW does not yet support analyzing foreign tables.
* @param query query to run
* @throws Exception
*/
protected void runQueryInsertIntoExternalTable(String query) throws Exception {
runQueryWithExpectedWarning(query, ".* --- cannot analyze this foreign table", true, true);
}

/**
* Run query which expected to get warning in execution and match it to expected one.
*
Expand Down
Loading

0 comments on commit 8ad87e9

Please sign in to comment.