-
Notifications
You must be signed in to change notification settings - Fork 17
/
ddar.1.xml
218 lines (192 loc) · 10.1 KB
/
ddar.1.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
<?xml version="1.0" standalone='no'?>
<!DOCTYPE manpage SYSTEM "http://masqmail.cx/xmltoman/xmltoman.dtd">
<?xml-stylesheet type="text/xsl" href="http://masqmail.cx/xmltoman/xmltoman.xsl" ?>
<manpage name="ddar" section="1" desc="store multiple files efficiently in a de-duplicated archive">
<synopsis>
<cmd>ddar [-]c [-f] <arg>archive</arg> <arg>member</arg> [<arg>member</arg>...]</cmd>
<cmd>ddar [-]c [-f] [<arg>server</arg>:]<arg>archive</arg> [-N <arg>member-name</arg>] < <arg>member</arg></cmd>
<cmd>ddar [-]c [-f] [<arg>server</arg>:]<arg>archive</arg> [-N <arg>member-name</arg>] <arg>member</arg></cmd>
<cmd>ddar [-]c [-f] <arg>archive</arg> <arg>server</arg>:<arg>member</arg> [-N <arg>member-name</arg>]</cmd>
<cmd>ddar [-]c [-f] <arg>archive</arg> <arg>server</arg>:!<arg>cmd</arg> [-N <arg>member-name</arg>]</cmd>
<cmd>ddar [-]x [<arg>options</arg>] [-f] <arg>archive</arg> > <arg>member</arg></cmd>
<cmd>ddar [-]x [<arg>options</arg>] [-f] <arg>archive</arg> <arg>member-name</arg> > <arg>member</arg></cmd>
<cmd>ddar [-]t [-f] <arg>archive</arg></cmd>
<cmd>ddar [-]d [-f] <arg>archive</arg> <arg>member-name</arg> [<arg>member-name</arg>...]</cmd>
<cmd>ddar --fsck [-f] <arg>archive</arg></cmd>
<cmd>ddar --sha256sum [-f] <arg>archive</arg> [<arg>member</arg>...]</cmd>
</synopsis>
<description>
<p>ddar creates, modifies and extracts from archives. An archive is a directory
holding a collection of other files in a structure that makes it possible to
retrieve the original individual files (called members of the archive).</p>
<p>Members of an archive are stored efficiently: ddar finds regions that are
the same across members and stores them only once.</p>
<p>ddar follows the Unix philosophy, dealing only with the task of
de-duplication. It is intended to be used in conjunction with other standard
Unix tools such as <manref name="tar" section="1"/> and <manref name="gzip"
section="1"/>. For example, it can be used to augment the standard tar | gzip
-style pipeline, eliminating the need and complexities of incremental and
differential backups by making the storage of multiple large full backups
equally feasible.</p>
<p>ddar may be considered a companion to <manref name="tarsnap" section="1"/>.
Is useful for local archive storage in cases where the available upload
bandwidth is too low to deal with the size of the archive that needs to be
created.</p>
</description>
<section name="Mandatory arguments">
<p>One operation argument from <arg>c</arg>, <arg>x</arg>, <arg>t</arg>,
<arg>d</arg>, <arg>--fsck</arg> or <arg>--sha256sum</arg> must be specified. As
specifying an operation is mandatory, the prefix <arg>-</arg> is optional if
the operation is the first argument.</p>
<option>
<p><opt>c</opt></p>
<optdesc>Add <arg>member</arg> to <arg>archive</arg>. If <arg>archive</arg>
does not exist, it will be created. If <arg>member-name</arg> is not specified,
then the base name of <arg>member</arg> is used; if not specified, then a
suitable name is automatically generated. If <arg>server</arg> is specified
(where permitted), then the archive or member is accessed from
<arg>server</arg> as appropriate, with de-duplication taking place at the
source to save bandwidth. If <arg>cmd</arg> is specified, then it is called to
generate the member data on its stdout. All remote options require <manref
name="ssh" section="1"/> to be available and for ddar to also be installed at
the remote end.</optdesc>
</option>
<option>
<p><opt>x</opt></p>
<optdesc>Extract <arg>member-name</arg> from <arg>archive</arg>. The member is
written to stdout. If stdout is a terminal, then ddar will refuse unless
<arg>--force-stdout</arg> is used. If <arg>member-name</arg> is not specified,
then the last member to be added to the archive is used (based on addition
order, not time).</optdesc>
</option>
<option>
<p><opt>t</opt></p>
<optdesc>List the members of <arg>archive</arg>.</optdesc>
</option>
<option>
<p><opt>d</opt></p>
<optdesc>Delete <arg>member-name</arg> from <arg>archive</arg>.</optdesc>
</option>
<option>
<p><opt>--fsck</opt></p>
<optdesc>Check <arg>archive</arg> for internal consistency. This also verifies
that all members match the checksum computed when they were first stored. This
operation is extremely time consuming, requiring two passes over the
data.</optdesc>
</option>
<option>
<p><opt>--sha256sum</opt></p>
<optdesc>Print SHA-256 digests of members where available. If no members are
specified, then print the digests of all members in the archive.</optdesc>
</option>
<option>
<p><opt>[-f] <arg>archive</arg></opt></p>
<optdesc>Specify the <arg>archive</arg> upon which the operation will take
effect. <arg>-f</arg> is optional here as all operations require
<arg>archive</arg> and so this is not really an option. It exists for attempted
option parity with <manref name="tar" section="1"/> for ease of use.
</optdesc>
</option>
</section>
<options>
<option>
<p><opt>-N</opt> <arg>member-name</arg></p>
<optdesc>(create/append only) Override the <arg>member-name</arg> to be used
when adding a member to an archive.</optdesc>
</option>
<option>
<p><opt>--force-stdout</opt></p>
<optdesc>(extract only) Force ddar to extract a member to stdout even when
stdout is a terminal.</optdesc>
</option>
</options>
<section name="Notes">
<p>Unlike <manref name="tarsnap" section="1"/>, <manref name="tar"
section="1"/> includes atimes by default, causing many changes to its output,
thus making ddar ineffective at de-duplicating its archives. On Linux you can
use the relatime option to help avoid this, which has been the default since
2.6.30.</p>
<p>ddar will run more efficiently if the archive is stored on a filesystem that
does not degrade as directories become large, such as ext3 or ext4 with
dir_index enabled.</p>
<p>Normally, compression and encryption algorithms have the property that a
single bit flip in the input cause a complete apparant change in the output.
This works contrary to de-duplication, which looks for regions that are the
same to optimise storage. To avoid this issue, use <cmd>gzip --rsyncable</cmd>
for compression and if encryption is required then encrypt the entire archive,
for example with LUKS or Truecrypt.</p>
</section>
<section name="Examples">
<p>Back up your home directory to an external disk daily:</p>
<cmd>tar c ~|gzip --rsyncable|ddar cf /mnt/external_disk/home_backup</cmd>
<p>Back up your home directory to a remote server daily (using ssh, with ddar
also installed remotely):</p>
<cmd>tar c ~|gzip --rsyncable|ddar cf server:home_backup</cmd>
<p>Back up your home directory on a remote server to a local machine daily:</p>
<cmd>ddar cf server_home_backup server:\\!"'tar c ~|gzip --rsyncable'"</cmd>
<p>Restore your home directory from a local disk after a disaster:</p>
<cmd>ddar xf /mnt/external_disk/home_backup|tar xzC/</cmd>
<p>Restore your home directory from a remote server after a disaster:</p>
<cmd>ssh server ddar xf home_backup|tar xzC/</cmd>
</section>
<section name="Security">
<p>ddar relies on system security and does not encrypt data. For remote
archives, it relies on <manref name="ssh" section="1"/> to encrypt
communications. You may wish to keep your ddar archives in a directory to which
only you have access, adjust the permission bits on the ddar archive
directories themselves and/or encrypt the entire filesystem on which your ddar
archives are stored. For a cryptographically secure, hosted de-duplicating
backup solution, try <manref name="tarsnap" section="1"/> <<url
href="http://www.tarsnap.com/"/>>.</p>
</section>
<section name="Bugs">
<p>ddar currently assumes that SHA-256 never collides. Although no collisions
are known to date (2011), collisions must nevertheless exist due to the
pigeonhole principle. However, the probability of such a collision occurring is
so low that it is considered far more likely that the computer will perform a
random computation error, so save for a SHA-256 weakness being found ddar
should be safe from data corruption in this manner. A future version of ddar
may enhance the archive format to automatically detect and accomodate
collisions, although checking for collisions would impact performance.</p>
<p>As an extra precaution, ddar computes the SHA-256 of the entire member on
insertion into an archive, and verifies this on extraction. However, this means
that a failure would only be detected on extraction and not on insertion.</p>
</section>
<seealso>
<p><manref name="ar" section="1"/>, <manref name="tar" section="1"/>, <manref
name="gzip" section="1"/>, <manref name="tarsnap" section="1"/>.</p>
</seealso>
<section name="Author">
<p>ddar and this man page were both written by Robie Basak
<[email protected]>.</p>
</section>
<section name="Credits">
<p>ddar is sponsored by Synctus <<url href="http://www.synctus.com/"/>>, a
multi-master, conflict-free, real-time file replication system.</p>
<p>ddar was inspired by tarsnap <<url
href="http://www.tarsnap.com/"/>>, a cloud-based de-duplicating backup
tool.</p>
</section>
<section name="Copyright">
<p>Copyright 2010-2011 True Blue Logic Ltd.</p>
<p>This program is free software: you can redistribute it and/or modify it
under the terms of version 3 of the GNU General Public License as published by
the Free Software Foundation.</p>
<p>This program is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
details.</p>
<p>You should have received a copy of the GNU General Public License along with
this program. If not, see <<url
href="http://www.gnu.org/licenses/"/>>.</p>
<p>This manpage is in part derived from <manref name="ar" section="1"/>; thus
the following notices apply:</p>
<p>Copyright (c) 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000,
2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 Free Software
Foundation, Inc.</p>
<p>Permission is granted to copy, distribute and/or modify this document under
the terms of the GNU Free Documentation License, Version 1.3 or any later
version published by the Free Software Foundation; with no Invariant Sections,
with no Front-Cover Texts, and with no Back-Cover Texts.</p>
</section>
</manpage>