Skip to content
This repository was archived by the owner on Jan 1, 2024. It is now read-only.

Commit 5289f02

Browse files
hackallcodehackallcode
authored
Add restore step (#374)
Closes #148 Before this patch, it was possible to backup instances, but it wasn't possible to restore instances from these backups. Now it's possible to restore instance from the last or selected backup by the `restore` step. Backup should be in TGZ or directory format.
1 parent 52652f6 commit 5289f02

22 files changed

+886
-112
lines changed

‎.gitignore‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,12 @@
11
### Project ###
22

3+
# Dependencies
34
packages/
45
.rocks
56

7+
# Ignore backups fetched in molecule test
8+
molecule/*/backups/
9+
610
### Vagrant ###
711

812
# General
@@ -264,6 +268,3 @@ $RECYCLE.BIN/
264268
# Ignore all local history of files
265269
.history
266270
.ionide
267-
268-
# Ignore backups fetched in molecule test
269-
molecule/backup/backups/

‎defaults/main.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,11 @@ cartridge_eval_delay: 5
204204
cartridge_remote_backups_dir: /opt/tarantool/backups
205205
cartridge_fetch_backups: false
206206
cartridge_fetch_backups_dir: backups/
207+
cartridge_restore_backup_path: null
208+
cartridge_force_restore: false
209+
cartridge_allow_alien_backup: false
210+
cartridge_skip_cleanup_on_restore: false
211+
cartridge_paths_to_keep_before_restore: []
207212

208213
# Edit topology check
209214

‎doc/backups.md‎

Lines changed: 168 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,17 @@ The instance backup process contains three stages:
77
* [`box.snapshot()`](https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_snapshot/) is called;
88
* [`box.backup.start()`](https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_backup/start/) is called;
99
* `instance_backup_files` variable is set ([list of instance files](#instance-files) to back up);
10-
* `backup_files_from_machine` variable is set (list of files to back up for all instances on the same machine as a current one);
10+
* `backup_files_from_machine` variable is set (list of files to back up for
11+
all instances on the same machine as a current one);
1112
* Creating and fetching the backup archive:
1213
* all backup files are packed into the TGZ archive placed in [`cartridge_remote_backups_dir`](/doc/variables.md#backups-configuration);
1314
* all paths inside the archive are relative to `/`;
1415
* `backup_archive_path` variable is set (path to the instance backup archive on the remote machine);
1516
* fetching an archive from the remote machine to
1617
[`cartridge_fetch_backups_dir`](/doc/variables.md#backups-configuration)
1718
if [`cartridge_fetch_backups`](/doc/variables.md#backups-configuration) is `true`;
18-
* `fetched_backup_archive_path` is set if [`cartridge_fetch_backups`](/doc/variables.md#backups-configuration) is `true` (path to the fetched instance backup archive on the local machine);
19+
* `fetched_backup_archive_path` is set if [`cartridge_fetch_backups`](/doc/variables.md#backups-configuration)
20+
is `true` (path to the fetched instance backup archive on the local machine);
1921
* Stopping backup:
2022
* [`box.backup.stop()`](https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_backup/stop/) is called.
2123

@@ -31,7 +33,8 @@ There are three steps that allows to create a backup:
3133

3234
These files are added to an archive for each instance (excluding the stateboard):
3335

34-
* snapshot and vinyl files returned by [`box.backup.start()`](https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_backup/start/);
36+
* snapshot and vinyl files returned by
37+
[`box.backup.start()`](https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_backup/start/);
3538
* cluster-wide config and its backup;
3639
* instance configuration file;
3740
* application configuration file.
@@ -41,7 +44,52 @@ For a stateboard instance only snapshot files and instance configuration file wi
4144
## How to use backup steps?
4245

4346
* `backup` step can be used for performing all three backup stages, see [example](#using-backup-step);
44-
* `backup_start` + `backup_stop` steps can be used to perform some custom actions with instance files list, see [example](#using-backup_start-and-backup_stop).
47+
* `backup_start` + `backup_stop` steps can be used to perform some custom actions
48+
with instance files list, see [example](#using-backup_start-and-backup_stop).
49+
50+
## How to restore from saved backup?
51+
52+
To restore you can use the `restore` step. It contains three stages:
53+
54+
* find TGZ backup or directory with backup ([details here](#find-backup));
55+
* check for conflicting files ([details here](#check-for-conflicting-files));
56+
* replace current files with files from backup ([details here](#replace-current-files-with-files-from-backup)).
57+
58+
Examples of restoring from backup, you can see [below](#using-restore-step).
59+
60+
### Find backup
61+
62+
1. You can pass path to backup by `cartridge_restore_backup_path` variable.
63+
Path to a TGZ archive or a directory with backup can be specified.
64+
2. If `cartridge_restore_backup_path` variable is omitted, the role
65+
will look in the `cartridge_remote_backups_dir` directory
66+
(by default `/opt/tarantool/backups`) for the backup
67+
whose name contains the instance ID and is the latest alphabetically (with the latest date).
68+
3. If not found any file or directory backup that belongs to given instance, an error will occur.
69+
70+
You can see examples of using the described variables [below](#using-restore-step).
71+
72+
### Check for conflicting files
73+
74+
After selecting a backup, `restore` step checks the following:
75+
- files in the selected backup do not conflict with files in the system,
76+
taking into account that some files will be deleted
77+
(in order to overwrite such files, set the `cartridge_force_restore` flag);
78+
- probably the selected backup belongs to the instance that is currently being
79+
restored (in order to allow recovery from such backup, set the
80+
`cartridge_allow_alien_backup` flag).
81+
82+
You can see examples of using the described variables [below](#using-restore-step).
83+
84+
### Replace current files with files from backup
85+
86+
After all checks have been passed, step make two actions:
87+
1. If `cartridge_skip_cleanup_on_restore` is not set, step deletes the instance
88+
files that are in the system (except for `.tarantool.cookie` and files that
89+
match the templates from `cartridge_paths_to_keep_before_restore` list);
90+
2. Step unpacks files from selected backup.
91+
92+
You can see examples of using the described variables [below](#using-restore-step).
4593

4694
## Examples
4795

@@ -70,48 +118,52 @@ It can be used to create backup archive for each instance and fetch all archives
70118
- backup
71119
cartridge_fetch_backups: true
72120
cartridge_fetch_backups_dir: ./my_backups
73-
74121
```
75122
76123
After running this playbook, we have such archive on local machine:
77124
78125
```bash
79126
├── playbook.yml
80127
└── my_backups
81-
   ├── myapp-stateboard.2021-07-09-133438.tar.gz
128+
├── myapp-stateboard.2021-07-09-133438.tar.gz
82129
├── myapp.instance-1.2021-07-09-133438.tar.gz
83-
    └── myapp.instance-2.2021-07-09-133438.tar.gz
84-
130+
└── myapp.instance-2.2021-07-09-133438.tar.gz
85131
```
86132

87-
On the remote machines, backups are placed in `/opt/backups/` directory.
133+
On the remote machines, backups are placed in `/opt/tarantool/backups/` directory.
88134

89135
On each instance the following variables will be set:
90136

91-
```json
92-
instance-1: {
93-
"instance_backup_files": [
94-
"/opt/memtx/myapp.instance-1/00000000000000000008.snap",
95-
"/opt/vinyl/myapp.instance-1/00000000000000000000.vylog",
96-
"/opt/data/myapp.instance-1/config",
97-
"/opt/conf.d/myapp.instance-1.yml",
98-
"/opt/conf.d/myapp.yml"
99-
],
100-
"backup_archive_path": "/opt/backups/myapp.instance-1.2021-07-09-133438.tar.gz",
101-
"fetched_backup_archive_path": "/path/to/playbook/my_backups/myapp.instance-1.2021-07-09-133438.tar.gz"
102-
}
103-
104-
my-stateboard: {
105-
"instance_backup_files": [
106-
"/opt/memtx/myapp-stateboard/00000000000000000049.snap",
107-
"/opt/conf.d/myapp-stateboard.yml"
108-
],
109-
"backup_archive_path": "/opt/backups/myapp-stateboard.2021-07-09-133438.tar.gz",
110-
"fetched_backup_archive_path": "/path/to/playbook/my_backups/myapp-stateboard.2021-07-09-133438.tar.gz"
111-
}
112-
113-
...
114-
```
137+
- my-stateboard:
138+
139+
```json
140+
{
141+
"instance_backup_files": [
142+
"/opt/memtx/myapp-stateboard/00000000000000000049.snap",
143+
"/opt/conf.d/myapp-stateboard.yml"
144+
],
145+
"backup_archive_path": "/opt/tarantool/backups/myapp-stateboard.2021-07-09-133438.tar.gz",
146+
"fetched_backup_archive_path": "/path/to/playbook/my_backups/myapp-stateboard.2021-07-09-133438.tar.gz"
147+
}
148+
```
149+
150+
- instance-1:
151+
152+
```json
153+
{
154+
"instance_backup_files": [
155+
"/opt/memtx/myapp.instance-1/00000000000000000008.snap",
156+
"/opt/vinyl/myapp.instance-1/00000000000000000000.vylog",
157+
"/opt/data/myapp.instance-1/config",
158+
"/opt/conf.d/myapp.instance-1.yml",
159+
"/opt/conf.d/myapp.yml"
160+
],
161+
"backup_archive_path": "/opt/tarantool/backups/myapp.instance-1.2021-07-09-133438.tar.gz",
162+
"fetched_backup_archive_path": "/path/to/playbook/my_backups/myapp.instance-1.2021-07-09-133438.tar.gz"
163+
}
164+
```
165+
166+
- ...
115167

116168
### Using `backup_start` and `backup_stop`
117169

@@ -132,7 +184,7 @@ Let's imagine that we want to archive backup files not for each instance, but fo
132184
tasks:
133185
- name: 'Create backups dir'
134186
file:
135-
path: '/opt/backups'
187+
path: '/opt/tarantool/backups'
136188
owner: '{{ cartridge_app_user }}'
137189
group: '{{ cartridge_app_group }}'
138190
state: directory
@@ -147,7 +199,7 @@ Let's imagine that we want to archive backup files not for each instance, but fo
147199

148200
- name: 'Create archives with backups by machines'
149201
archive:
150-
dest: /opt/backups/{{ cartridge_app_name }}.{{ ansible_host }}.tar.gz
202+
dest: /opt/tarantool/backups/{{ cartridge_app_name }}.{{ ansible_host }}.tar.gz
151203
path: '{{ backup_files_from_machine }}'
152204
when: inventory_hostname in single_instances_for_each_machine
153205

@@ -162,7 +214,7 @@ Let's imagine that we want to archive backup files not for each instance, but fo
162214
I will produce such archives:
163215
164216
```
165-
[root@vm1 /]# tar -ztf /opt/backups/myapp.vm1.tar.gz
217+
[root@vm1 /]# tar -ztf /opt/tarantool/backups/myapp.vm1.tar.gz
166218
memtx/myapp.instance-1/00000000000000000004.snap
167219
data/myapp.instance-1/config/auth.yml
168220
data/myapp.instance-1/config/vshard_groups.yml
@@ -172,11 +224,90 @@ conf.d/myapp.yml
172224
memtx/myapp-stateboard/00000000000000000049.snap
173225
conf.d/myapp-stateboard.yml
174226

175-
[root@vm2 /]# tar -ztf /opt/backups/myapp.vm2.tar.gz
227+
[root@vm2 /]# tar -ztf /opt/tarantool/backups/myapp.vm2.tar.gz
176228
memtx/myapp.instance-2/00000000000000000004.snap
177229
data/myapp.instance-2/config/auth.yml
178230
data/myapp.instance-2/config/vshard_groups.yml
179231
data/myapp.instance-2/config/topology.yml
180232
conf.d/myapp.instance-2.yml
181233
conf.d/myapp.yml
182234
```
235+
236+
### Using `restore` step
237+
238+
If you want to restore an instance from the latest TGZ backup
239+
from `cartridge_remote_backups_dir`, just run restore step:
240+
241+
```yaml
242+
- name: 'Restore instance-1 from latest backup'
243+
hosts: instance-1
244+
become: true
245+
become_user: root
246+
gather_facts: false
247+
tasks:
248+
- name: 'Restore'
249+
import_role:
250+
name: tarantool.cartridge
251+
vars:
252+
cartridge_scenario:
253+
- restore
254+
```
255+
256+
You can select backup manually by `cartridge_restore_backup_path` variable:
257+
258+
```yaml
259+
- name: 'Restore instance-1 from selected backup'
260+
hosts: instance-1
261+
become: true
262+
become_user: root
263+
gather_facts: false
264+
tasks:
265+
- name: 'Restore'
266+
import_role:
267+
name: tarantool.cartridge
268+
vars:
269+
cartridge_restore_backup_path: '/opt/tarantool/backups/myapp.instance-1.2021-07-09-133438/'
270+
cartridge_scenario:
271+
- restore
272+
```
273+
274+
By default, only `.tarantool.cookie` will be kept in working directory.
275+
If you want to keep some more files, you can put in `cartridge_paths_to_keep_before_restore`
276+
variable some bash patterns:
277+
278+
```yaml
279+
- name: 'Restore instance-1, but keep clusterwide config and logs'
280+
hosts: instance-1
281+
become: true
282+
become_user: root
283+
gather_facts: false
284+
tasks:
285+
- name: 'Restore'
286+
import_role:
287+
name: tarantool.cartridge
288+
vars:
289+
cartridge_paths_to_keep_before_restore:
290+
- 'config'
291+
- '*.log'
292+
cartridge_scenario:
293+
- restore
294+
```
295+
296+
If application config was changed, but you want to replace it with config
297+
from backup, you should set `cartridge_force_restore` flag.
298+
299+
```yaml
300+
- name: 'Restore instance-1 with application config replacement'
301+
hosts: instance-1
302+
become: true
303+
become_user: root
304+
gather_facts: false
305+
tasks:
306+
- name: 'Restore'
307+
import_role:
308+
name: tarantool.cartridge
309+
vars:
310+
cartridge_force_restore: true
311+
cartridge_scenario:
312+
- restore
313+
```

‎doc/steps.md‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ but can be used in a custom one:
4343
- [backup](#step-backup)
4444
- [backup_start](#step-backup_start)
4545
- [backup_stop](#step-backup_stop)
46+
- [restore](#step-restore)
4647
- [check_new_topology](#step-check_new_topology)
4748

4849
## Role Variables Descriptions
@@ -532,13 +533,14 @@ If instance is running, an error will be returned.
532533

533534
Input variables from config:
534535

535-
- `cartridge_paths_to_keep_on_cleanup` - list of full paths or relative paths
536-
to work/memtx/vinyl/wal directory that should be kept on instance cleanup
537-
(it's possible to use bash patterns, e.g. `*.control`).
536+
- `cartridge_paths_to_keep_on_cleanup` - list of paths that are absolute or relative
537+
to `work/memtx/vinyl/wal` directory that should be kept on instance cleanup
538+
(`config` and `.tarantool.cookie` will be kept independently of this variable);
539+
it's possible to use bash patterns, e.g. `*.control`.
538540

539541
### Step `backup`
540542

541-
Create a [backup](/doc/backup.md) archive for each instance and fetch it on the local machine.
543+
Create a [backup](/doc/backups.md) archive for each instance and fetch it on the local machine.
542544

543545
Input variables from config:
544546

@@ -558,7 +560,7 @@ Output facts:
558560

559561
### Step `backup_start`
560562

561-
Start a [backup](/doc/backup.md) process on the instance.
563+
Start a [backup](/doc/backups.md) process on the instance.
562564

563565
Input variables from config:
564566

@@ -571,7 +573,23 @@ Output facts:
571573

572574
### Step `backup_stop`
573575

574-
Stop started [backup](/doc/backup.md) on the instance.
576+
Stop started [backup](/doc/backups.md) on the instance.
577+
578+
### Step `restore`
579+
580+
[Restore](/doc/backups.md#how-to-restore-from-saved-backup) each instance from a [backup](#step-backup) archive.
581+
582+
Input variables from config:
583+
584+
- `cartridge_paths_to_keep_before_restore` - list of paths that are absolute or relative
585+
to `work/memtx/vinyl/wal` directory that shouldn't be removed before instance restore
586+
(`.tarantool.cookie` will be kept independently of this variable);
587+
it's possible to use bash patterns, e.g. `*.control`.
588+
- `cartridge_restore_backup_path` - path to the instance backup archive on the remote machine;
589+
- `cartridge_remote_backups_dir` - directory with backups on the remote;
590+
- `cartridge_force_restore` - flag indicates that conflicting files should be overwritten;
591+
- `cartridge_allow_alien_backup` - flag indicates that backup of instance with another name can be used;
592+
- `cartridge_skip_cleanup_on_restore` - flag indicates that cleanup before restoring should be skipped.
575593

576594
## Step `check_new_topology`
577595

0 commit comments

Comments
 (0)