DLPX-89763 DLPX-86523 delphix-platform changes #477

justsanjeev · 2024-04-18T14:34:45Z

Problem

CIS is looking or a single home directory filesystem mounted at the /home location, currently we have the home dataset is mounted on /export/home

Due to that we see the below issues in the CIS Report

(1.45) 7402 Status of the '/home' partition in the '/etc/fstab' file
(1.46) 13248 Status of Mount Partition '/home' using mount command
(1.47) 7403 Status of the 'nodev' mount option setting for the '/home partition' defined in the '/etc/ fstab' file
(1.48) 14601 Status of the 'nodev' option for '/home' partition using 'mount' command

Solution

Mounting the home dataset to /home.
Post upgrade creating a softlink /export/home to /home so that the exiting tests and dependencies have minimal impact.

We have now implemented the delphix-legacy-link.service for the /export/home -> /home creation, we earlier tried to do the sae in the delphix-platform.services and tested to implement the change with ExecStart and ExecStartPost. It dod not work and then this change was implemented to achieve the same after the delphix-platform.services are up completely.

Testing Done

Build : appliance-build-orchestrator-pre-push/10952 - ☑️ [Partially]
Pre Checkin tests 172,051 - 🔴
- Reason: Failed because of the reason that the build tried to navigate to the directory /export/home before the softlink creation.
- we created a fix for this issue - https://github.com/delphix/qa-infra/pull/2270 for this successful run.
- We Re-Built the Pre Checkin Tests - 172175 - ✅
2025.2.0.0 VM Upgrade Test:
- Created a DE from 2025.2.0.0 branch - dlpx-qa-2025200-qar-171974-766f97e2.dlpxdc.co
- Added the source and target environments on the above engine.
- Created a dsource from an unstructured dataset created on source, and provisioned a vdb.
Took the upgrade Image s3://dev-de-images/builds/jenkins-selfservice/appliance-build/develop/pre-push/4295/upgrade-artifacts/internal-qa.upgrade.tar from above build.
Upgrade the 2025.2.0.0 Engine (Deferred).

After Upgrade, on above upgraded DE

/export/home -> /home symbolic link creation. - ✅

sanjeev.rohilla@APAC-SANJEEV ~ % ssh [email protected]
([email protected]) Password: 
delphix@ip-10-110-247-6:~$ ls -ltr /export/home
lrwxrwxrwx 1 root root 5 Apr 23 08:22 /export/home -> /home
delphix@ip-10-110-247-6:~$

From /var/log/syslog - The service execution is successful. -- ✅

Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Finished Delphix Appliance Platform Service.
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Create /export/home symlink to /home after platform setup...
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Delphix Nginx reverse proxy...
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Delphix default PostgreSQL database server...
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Delphix Perfomance Statistics...
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Started Delphix Telegraf Metric Collection Agent.
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Starting Docker Application Container Engine...
Apr 23 08:22:55 ip-10-110-247-6 bash[209243]: Ensuring /export/home is a symlink to /home...
Apr 23 08:22:55 ip-10-110-247-6 check_docker_start_deps[209248]: Waiting for /run/containerd/containerd.sock to listen
Apr 23 08:22:55 ip-10-110-247-6 check_docker_start_deps[209248]: num passes: 0
Apr 23 08:22:55 ip-10-110-247-6 bash[209243]: /export/home is a mountpoint, unmounting...
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: export-home.mount: Succeeded.
Apr 23 08:22:55 ip-10-110-247-6 check_docker_start_deps[209248]: /run/containerd/containerd.sock is listening
Apr 23 08:22:55 ip-10-110-247-6 bash[209243]: Removing existing /export/home directory...
Apr 23 08:22:55 ip-10-110-247-6 bash[209243]: Creating symlink: /export/home -> /home
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: delphix-legacy-link.service: Succeeded.
Apr 23 08:22:55 ip-10-110-247-6 systemd[1]: Finished Create /export/home symlink to /home after platform setup.

Full upgrade ✅

Apr 23 08:35:04 ip-10-110-247-6 systemd[1]: delphix-legacy-link.service: Succeeded.
Apr 23 08:35:04 ip-10-110-247-6 systemd[1]: Finished Create /export/home symlink to /home after platform setup.

Screenshot From Engine
Screenshot from Engine, The VDB created befor upgrade is still in Running state. - ✅
CIS Test: Uploading Compliance_Report_CISReport21Apr2025_dephx3sr1_20250421.pdf… - ✅

The output of sudo lsof +f -- /export/home

delphix@ip-10-110-247-6:~$ sudo lsof +f -- /export/home
 COMMAND     PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
 delphix-s  1819    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1821    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1824    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1826    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1827    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1828    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1834    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1835    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1836    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1837    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1838    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1839    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1847    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1858    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1861    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1862    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1880    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1881    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1882    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1883    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1884    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1885    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix-s  1886    root  cwd    DIR   0,53        8  129 /home/delphix
 java       4250    root  cwd    DIR   0,53        8  129 /home/delphix
 python3    5102    root  cwd    DIR   0,53        8  129 /home/delphix
 python3    5106    root  cwd    DIR   0,53        8  129 /home/delphix
 python3    5110    root  cwd    DIR   0,53        8  129 /home/delphix
 python3    5115    root  cwd    DIR   0,53        8  129 /home/delphix
 script    10313 delphix  cwd    DIR   0,53        8  129 /home/delphix
 supportlo 10321 delphix  cwd    DIR   0,53        8  129 /home/delphix
 bash      10322 delphix  cwd    DIR   0,53        8  129 /home/delphix
 timeout   13669    root  cwd    DIR   0,53        8  129 /home/delphix
 iostat    13670    root  cwd    DIR   0,53        8  129 /home/delphix
 timeout   13672    root  cwd    DIR   0,53        8  129 /home/delphix
 vmstat    13673    root  cwd    DIR   0,53        8  129 /home/delphix
 timeout   13675    root  cwd    DIR   0,53        8  129 /home/delphix
 zpool     13676    root  cwd    DIR   0,53        8  129 /home/delphix
 timeout   13678    root  cwd    DIR   0,53        8  129 /home/delphix
 zpool     13679    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     13760    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     13776    root  cwd    DIR   0,53        8  129 /home/delphix
 timeout   13812    root  cwd    DIR   0,53        8  129 /home/delphix
 mpstat    13813    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     13814    root  cwd    DIR   0,53        8  129 /home/delphix
 timeout   13816    root  cwd    DIR   0,53        8  129 /home/delphix
 arcstat   13817    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     13823    root  cwd    DIR   0,53        8  129 /home/delphix
 timeout   13825    root  cwd    DIR   0,53        8  129 /home/delphix
 arcstat   13826    root  cwd    DIR   0,53        8  129 /home/delphix
 timeout   13869    root  cwd    DIR   0,53        8  129 /home/delphix
 profile   13870    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     13874    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     13875    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     13986    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     14868    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     14872    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     14877    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     14880    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     14884    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     14888    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     14892    root  cwd    DIR   0,53        8  129 /home/delphix
 sleep     14970    root  cwd    DIR   0,53        8  129 /home/delphix
 sudo      14971    root  cwd    DIR   0,53        8  129 /home/delphix
 sudo      14973    root  cwd    DIR   0,53        8  129 /home/delphix
 lsof      14974    root  cwd    DIR   0,53        8  129 /home/delphix
 lsof      14975    root  cwd    DIR   0,53        8  129 /home/delphix
 delphix@ip-10-110-247-6:~$ sudo lsof +f -- /export/home^C
 delphix@ip-10-110-247-6:~$

We have also tested with the same logic of symbolic link creation in the delphix-platform.service and below is the setup details-

Test change PR: test changes duplicate with execStart #525
Description: This build implements the logic for the soft link in delphix-platform.service
Build :https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/appliance-build-orchestrator-pre-push/10904/console
Test VM : dlpx-qa-2025200-qar-171650-766f97e2.dlpxdc.co
Upgrade Image: s3://dev-de-images/builds/jenkins-selfservice/appliance-build/develop/pre-push/4265/upgrade-artifacts/internal-qa.upgrade.tar
After Upgrade
- The upgrade failed because the softlink creation was not successful since the export/home was busy.

...n/var/lib/delphix-platform/ansible/10-delphix-platform/roles/delphix-platform/tasks/main.yml

…rt/home DLPX-89763 DLPX-86523 delphix-platform changes PR URL: https://www.github.com/delphix/delphix-platform/pull/477

debian/preinst

...n/var/lib/delphix-platform/ansible/10-delphix-platform/roles/delphix-platform/tasks/main.yml

justsanjeev · 2025-04-16T15:53:23Z

We have made the latest changes here and added the below logic to delphix-platform.service to take care of the /export/home mount point and soft link /export/home -> /home

# Create /export/home symlink to /home if it doesn't already exist
ExecStartPost=/bin/sh -c '\
  echo "Ensuring /export/home is a symlink to /home..."; \
  if mountpoint -q /export/home; then \
    echo "/export/home is a mountpoint, unmounting..."; \
    umount /export/home || { echo "Failed to unmount /export/home"; exit 1; }; \
  fi; \
  if [ -e /export/home ] && [ ! -L /export/home ]; then \
    echo "Removing existing /export/home directory..."; \
    rm -rf /export/home; \
  fi; \
  if [ ! -d /export ]; then \
    mkdir /export; \
  fi; \
  if [ -d /export ] && [ ! -L /export/home ]; then \
    echo "Creating symlink: /export/home -> /home"; \
    ln -s /home /export/home; \
  fi;'

We this change we got a build : appliance-build-orchestrator-pre-push/10842.

Here we have the successful upgrade tests.
But to verify @sebroy 's comment, I again the same testing

-- Created an app data unstructured vdb on a target
-- Upgraded the engine
-- delphix-mgmt did not come up due to delphix-platform

Also my /var/log/syslog show

Apr 15 12:04:11 ip-10-110-212-88 dynamic-debug[156701]: init/main.c:1218 [main]initcall_blacklisted =p "initcall %s blacklisted\012"
Apr 15 12:04:11 ip-10-110-212-88 dynamic-debug[156701]: init/main.c:1179 [main]initcall_blacklist =p "blacklisting initcall %s\012"
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: Ensuring /export/home is a symlink to /home...
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: /export/home is a mountpoint, unmounting...
Apr 15 12:04:11 ip-10-110-212-88 sh[156704]: umount: /export/home: target is busy.
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: Failed to unmount /export/home
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: delphix-platform.service: Control process exited, code=exited, status=1/FAILURE
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: delphix-platform.service: Failed with result 'exit-code'.
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: Failed to start Delphix Appliance Platform Service.

Upon executing below

delphix@ip-10-110-212-88:~$ sudo systemctl enable delphix-platform.service
delphix@ip-10-110-212-88:~$ sudo systemctl restart delphix-mgmt

And my log shows that the services has started successfully

Apr 15 12:37:42 ip-10-110-212-88 dynamic-debug[165503]: init/main.c:1179 [main]initcall_blacklist =p "blacklisting initcall %s\012"
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Ensuring /export/home is a symlink to /home...
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: /export/home is a mountpoint, unmounting...
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: export-home.mount: Succeeded.
Apr 15 12:37:42 ip-10-110-212-88 systemd[164396]: export-home.mount: Succeeded.
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Removing existing /export/home directory...
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Creating symlink: /export/home -> /home
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: Finished Delphix Appliance Platform Service.
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: Starting Delphix management service...

Setup Present for the same setup: dlpx-qa-2025200-qar-170885-766f97e2.dlpxdc.co
Upgraded from: 2025.2.0.0
Upgrade Tar: s3://dev-de-images/builds/jenkins-selfservice/appliance-build/develop/pre-push/4235/upgrade-artifacts/internal-qa.upgrade.tar
An Engine retained in the Error State: dlpx-qa-2025200-qar-170888-766f97e2.dlpxdc.co

justsanjeev · 2025-04-16T15:57:56Z

Now here, we have created an alternative PR where we have added the logic to cleanup the /export/home mount point and creating the soft link /export/home -> /home.

We have added another service delphix-legacy-link.service

This Build is Validated : https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/appliance-build-orchestrator-pre-push/10846/
ManualUpgrade with an Unstructured database mounted: ✅
Manual Upgraded Engine, with deffered and Full Upgrade : dlpx-qa-2025200-qar-170959-766f97e2.dlpxdc.co
Failed Pre Checkin Test Rebuild: pre_checkin

…r validation

…/home

justsanjeev · 2025-04-18T10:08:00Z

Post changing my code as per above comments, I have Build appliance-build-orchestrator-pre-push/10897 running.

…xt reboot like deferred upgrade.

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from ec9dada to 317e58e Compare April 18, 2024 14:34

prakashsurya reviewed May 8, 2024

View reviewed changes

...n/var/lib/delphix-platform/ansible/10-delphix-platform/roles/delphix-platform/tasks/main.yml Show resolved Hide resolved

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 317e58e to 68118d0 Compare June 24, 2024 09:31

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 68118d0 to e56b7de Compare July 2, 2024 11:54

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 4 times, most recently from a4f876c to d356ef1 Compare July 11, 2024 10:29

justsanjeev mentioned this pull request Jul 12, 2024

DLPX-89763 DLPX-86523 delphix-platform changes #487

Closed

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 5 times, most recently from 38fba97 to 28cab6b Compare July 16, 2024 07:08

justsanjeev mentioned this pull request Jul 16, 2024

DLPX-89763 DLPX-86523 delphix-platform changes #473

Closed

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from b1d61e7 to 0c718f3 Compare September 25, 2024 17:25

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from b984a71 to f389cbe Compare October 7, 2024 10:17

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from 64091f7 to 5022542 Compare October 22, 2024 11:48

justsanjeev mentioned this pull request Oct 24, 2024

DLPX-86523 CIS: /home filesystem and mount options delphix/appliance-build#756

Open

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 5022542 to 744dc46 Compare November 4, 2024 07:54

justsanjeev marked this pull request as ready for review November 5, 2024 10:42

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 3 times, most recently from d39c466 to 52eee14 Compare November 25, 2024 07:26

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 3 times, most recently from b5743b5 to 3a9629e Compare December 3, 2024 08:11

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 5 times, most recently from 4e5c764 to d0ba9f9 Compare December 3, 2024 11:33

Adding logic to take case the /export/home mount before delting /expo…

43d194d

…rt/home DLPX-89763 DLPX-86523 delphix-platform changes PR URL: https://www.github.com/delphix/delphix-platform/pull/477

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from d0ba9f9 to 43d194d Compare January 16, 2025 06:30

sebroy requested changes Mar 26, 2025

View reviewed changes

debian/preinst Outdated Show resolved Hide resolved

debian/preinst Outdated Show resolved Hide resolved

...n/var/lib/delphix-platform/ansible/10-delphix-platform/roles/delphix-platform/tasks/main.yml Outdated Show resolved Hide resolved

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from d80aafb to 39573b8 Compare April 9, 2025 16:49

Changing as per Seb-s comment to alter the softlink logic.

2b194ff

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 39573b8 to 2b194ff Compare April 9, 2025 16:57

Changing the code to previous version to validate the comments.

78c7978

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from 2671611 to 24f6877 Compare April 14, 2025 16:03

Removing the softlink loginc from main.yml

5a17a79

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 24f6877 to 5a17a79 Compare April 15, 2025 09:26

justsanjeev added 2 commits April 18, 2025 15:15

Removing the autofs handling as per the concerns raised by Seb and ou…

55e6a6f

…r validation

Creating a different service for taking care of soft /export/home -> …

b1e8b82

…/home

Changing the delphix-legacy-link.service to perform the same steps ne…

8f0067f

…xt reboot like deferred upgrade.

justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 27a3f5e to 8f0067f Compare April 22, 2025 06:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DLPX-89763 DLPX-86523 delphix-platform changes #477

DLPX-89763 DLPX-86523 delphix-platform changes #477

justsanjeev commented Apr 18, 2024 •

edited

Loading

justsanjeev commented Apr 16, 2025 •

edited

Loading

justsanjeev commented Apr 16, 2025 •

edited

Loading

justsanjeev commented Apr 18, 2025

DLPX-89763 DLPX-86523 delphix-platform changes #477

Are you sure you want to change the base?

DLPX-89763 DLPX-86523 delphix-platform changes #477

Conversation

justsanjeev commented Apr 18, 2024 • edited Loading

Problem

Solution

Testing Done

justsanjeev commented Apr 16, 2025 • edited Loading

justsanjeev commented Apr 16, 2025 • edited Loading

justsanjeev commented Apr 18, 2025

justsanjeev commented Apr 18, 2024 •

edited

Loading

justsanjeev commented Apr 16, 2025 •

edited

Loading

justsanjeev commented Apr 16, 2025 •

edited

Loading