Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
d77752c
[maven-release-plugin] prepare release droid-6.9.6
tna-da-bot Sep 29, 2025
d9719c0
[maven-release-plugin] prepare for next development iteration
tna-da-bot Sep 29, 2025
3b625b7
Merge pull request #1469 from digital-preservation/version-bump-18098…
MancunianSam Sep 29, 2025
07f5483
Bump org.apache.maven.scm:maven-scm-provider-gitexe from 2.1.0 to 2.2.1
dependabot[bot] Sep 29, 2025
6a7913a
Bump org.apache.maven.scm:maven-scm-api from 2.1.0 to 2.2.1
dependabot[bot] Sep 29, 2025
db54bcd
Bump org.apache.commons:commons-lang3 from 3.18.0 to 3.19.0
dependabot[bot] Sep 29, 2025
edecaaa
Update docs
techncl Sep 30, 2025
ad1aa72
Bump junit.version from 5.13.4 to 6.0.0
dependabot[bot] Sep 30, 2025
199feda
Bump aws.version from 2.34.1 to 2.34.7
dependabot[bot] Oct 1, 2025
c7c3abf
Fix bug with DROID API.
MancunianSam Oct 1, 2025
9b3ba9b
Update to use new sig files
MancunianSam Oct 1, 2025
73b1a35
Merge pull request #1476 from digital-preservation/bug-with-droid-api
MancunianSam Oct 2, 2025
1f6988d
Merge branch 'develop' into update-to-use-new-sig-files
MancunianSam Oct 2, 2025
a8d67ff
Merge pull request #1465 from digital-preservation/dependabot/maven/o…
MancunianSam Oct 2, 2025
9236168
Merge pull request #1464 from digital-preservation/dependabot/maven/o…
MancunianSam Oct 2, 2025
3fb52aa
Merge pull request #1462 from digital-preservation/dependabot/maven/o…
MancunianSam Oct 2, 2025
23f3a0f
Merge branch 'dependabot/maven/aws.version-2.34.7' into dependabot/ma…
MancunianSam Oct 2, 2025
4f323ab
Merge pull request #1473 from digital-preservation/dependabot/maven/j…
MancunianSam Oct 2, 2025
2e03a6a
Merge branch 'develop' into dependabot/maven/aws.version-2.34.7
MancunianSam Oct 2, 2025
b13c982
Merge pull request #1474 from digital-preservation/dependabot/maven/a…
MancunianSam Oct 2, 2025
13a710f
Merge branch 'develop' into update-to-use-new-sig-files
MancunianSam Oct 2, 2025
2e192e9
Change 'Java 8 to 21' to 'Java 21'
techncl Oct 2, 2025
44c672b
Merge branch 'develop' into updateDocs
techncl Oct 2, 2025
38c5bb3
Merge pull request #1475 from digital-preservation/update-to-use-new-…
MancunianSam Oct 2, 2025
31cba1c
Merge branch 'develop' into updateDocs
MancunianSam Oct 2, 2025
b434fdf
Bump aws.version from 2.34.7 to 2.34.8
dependabot[bot] Oct 2, 2025
85c6bd4
Merge pull request #1471 from digital-preservation/updateDocs
techncl Oct 2, 2025
3acd286
Merge pull request #1477 from digital-preservation/dependabot/maven/a…
MancunianSam Oct 2, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# DROID (Digital Record Object Identification)

[![CI](https://github.com/digital-preservation/droid/workflows/CI/badge.svg)](https://github.com/digital-preservation/droid/actions?query=workflow%3ACI)
[![Coverage Status](https://coveralls.io/repos/github/digital-preservation/droid/badge.svg?branch=master)](https://coveralls.io/github/digital-preservation/droid?branch=master)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/uk.gov.nationalarchives/droid/badge.svg)](https://search.maven.org/search?q=g:uk.gov.nationalarchives)
[![Coverage Status](https://coveralls.io/repos/github/digital-preservation/droid/badge.svg?branch=main)](https://coveralls.io/github/digital-preservation/droid?branch=main)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/uk.gov.nationalarchives/droid/badge.svg)](https://central.sonatype.com/search?q=g%3Auk.gov.nationalarchives+droid)

## General Information

Expand All @@ -17,9 +17,9 @@ DROID is a platform-independent Java application. It can be invoked from two int

DROID allows files and folders to be selected from a file system for identification. This file list can be saved at any point. After the identification process had been run, the results can be output in various report formats, including CSV.

DROID is made available under the [New BSD License](https://raw.github.com/digital-preservation/droid/master/license.md).
DROID is made available under the [New BSD Licence](https://raw.github.com/digital-preservation/droid/master/license.md).

More information can be found on the [DROID github pages](https://digital-preservation.github.io/droid/).
More information can be found on the [DROID GitHub pages](https://digital-preservation.github.io/droid/).

## Installing DROID

Expand All @@ -42,23 +42,23 @@ $ droid -R directory
...
~~~

If you don't need every bit of information that DROID outputs by default you can make it print only a limited set of columns using the `-co`, or `--columns` option (note the `-a`, it's required in this case!):
If you don't need every bit of information that DROID outputs by default, you can make it print only a limited set of columns using the `-co`, or `--columns` option (note the `-a`; it's required in this case!):

~~~console
$ droid -co NAME PUID -a file.jpg
"NAME","PUID"
"file.jpg","fmt/1507"
~~~

To remove even more noise from the output you can use the `-qc`, or `--quote-commas` switch which tells DROID to use quotes only where necessary:
To remove even more noise from the output, you can use the `-qc`, or `--quote-commas` switch which tells DROID to use quotes only where necessary:

~~~console
$ droid -qc -co NAME PUID -a file.jpg
NAME,PUID
file.jpg,fmt/1507
~~~

Beyond these basic invocations there are many additional options for advanced tweaking of DROID's operation. To learn about them run:
Beyond these basic invocations, there are many additional options for advanced tweaking of DROID's operation. To learn about them run:

~~~console
$ droid --help
Expand Down Expand Up @@ -92,7 +92,7 @@ Type: Container Version: 20230822 File name: container-signature-20230822.xml
Type: Container Version: 20231127 File name: container-signature-20231127.xml
~~~

If for some reason you want to use another, possibly older signature file for identification you can change the default with the `-s`, or `--set-signature-file` option (remember to switch back to the latest version when you are done!):
If for some reason you want to use another, possibly older signature file for identification, you can change the default with the `-s`, or `--set-signature-file` option (remember to switch back to the latest version when you are done!):

~~~console
$ droid -s 114
Expand All @@ -104,21 +104,21 @@ Default signature file updated. Version: 20230822 File name: container-signatur

## Building DROID

DROID can be built from source using Maven. The source code can be obtained from the Github repository at [https://github.com/digital-preservation/droid](https://github.com/digital-preservation/droid)
DROID can be built from source using Maven. The source code can be obtained from the GitHub repository at [https://github.com/digital-preservation/droid](https://github.com/digital-preservation/droid)

Once the code is cloned into a folder (e.g. `droid`), executing `mvn clean install` inside it will build the code. After a successful build, two archives are provided inside the `droid-binary/target` folder.

### Linux / OSX users
### Linux / macOS users

You will need JAVA 8 to 17 installed to run DROID.
You will need JAVA 21 installed to run DROID.

Unpack the archive `droid-binary-${VERSION}-bin.zip`, then use the `droid.sh` script to run the application.

### Windows users
Archive `droid-binary-${VERSION}-bin-win64-with-jre.zip`


You will need JAVA 8 to 17 installed to run DROID. For Windows users who might not be able to install JAVA, the provided bundle includes JAVA 17.
You will need JAVA 21 installed to run DROID. For Windows users who might not be able to install JAVA, the provided bundle includes JAVA 21.

Unpack the archive `droid-binary-${VERSION}-bin-win64-with-jre.zip`, then use the `droid.bat` script to run the application.

Expand Down Expand Up @@ -166,7 +166,7 @@ DROID can now put a signature directly inside a `Sequence` attribute on the `<By

The full syntax can be used in either binary or container signature files.

## Developer related guidance
## Developer-related guidance

For any detais about contributing, testing, releasing, please check the [wiki](https://github.com/digital-preservation/droid/wiki)
For any details about contributing, testing, releasing, please check the [wiki](https://github.com/digital-preservation/droid/wiki)

22 changes: 11 additions & 11 deletions Signature syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,15 @@ The signature syntax compiled into XML by PRONOM is the original syntax defined

Container signatures support a wider syntax than binary signatures, for example allowing whitespace and ASCII strings. For the most part, this extended syntax just makes the signatures more human readable. In a few cases some new capabilities are supported that PRONOM can't currently compile for binary signatures.

All of the syntax can be used in either binary or container signatures in DROID itself, but PRONOM won't be able to compile container syntax into XML for binary signatures if you want to submit them to TNA.
All the syntax can be used in either binary or container signatures in DROID itself, but PRONOM won't be able to compile container syntax into XML for binary signatures if you want to submit them to TNA.

Container specific syntax is marked in the descriptions below.

### Value matching
The following syntax can be used to match bytes.

#### Bytes: 00 FF
Bytes to match are written as two digit, case insensitive hex values.
Bytes to match are written as two-digit, case-insensitive hex values.
```
ff Fe A1 00
```
Expand All @@ -42,7 +42,7 @@ A single character enclosed in single quotes can also be used as in place of a b
_Note:_ due to a bug in the byteseek library, only ASCII characters are currently supported by DROID. Characters over 127 may cause incorrect compilation, as it attempts to render the characters in the system default character set, which is probably not ISO-8859-1.

#### Byte ranges: \[n:m]
To match a byte within a range of values, we can use a byte range. This is written as two byte values separated with a colon, all enclosed in square brackets. Bytes are specified as 2 digit hex values.
To match a byte within a range of values, we can use a byte range. This is written as two byte values separated with a colon, all enclosed in square brackets. Bytes are specified as 2-digit hex values.
```
[30:39]
```
Expand Down Expand Up @@ -82,11 +82,11 @@ This will match all the bytes which have the 8th or 4th bit set (10001000):
_Note:_ the bitmask is not a standard part of the original PRONOM syntax; it originates in the byteseek matching library used by DROID. However, it is now being used in both binary and container signatures, although older versions of DROID won't be able to parse it in binary signatures.

### Alternatives: (00|F0|3C)
If there is a set of different values that can match, they can be specified inside round brackets, with the alternatives separated by the | character.
If there is a set of different values that can match, they can be specified inside round brackets, with the alternatives separated by the `|` character.
```
(00|F0|3C)
```
Alternatives can be longer than just a single byte. In this example, three byte sequences could be matched. Alternative sequences don't have to be the same length as each other:
Alternatives can be longer than just a single byte. In this example, three byte sequences could be matched. Alternative sequences don't have to be the same length as each other:
```
(00 01 | B0 B1 B2 | C0 C1 C2 C3)
```
Expand All @@ -97,13 +97,13 @@ Older binary signatures should only use hex bytes inside alternatives. Containe
```

#### Multi-byte sets: (00|C2|DE) or \[00 C2 DE]
The standard PRONOM binary syntax only supports multi-byte sets by specifying them as a set of alternative bytes. For example:
The standard PRONOM binary syntax only supports multi-byte sets by specifying them as a set of alternative bytes. For example:
```
(00|C2|DE)
```

**Container signature syntax**
Multi byte sets let you specify any set of bytes within square brackets. For example:
Multi-byte sets let you specify any set of bytes within square brackets. For example:
```
[00 C2 DE]
```
Expand All @@ -115,7 +115,7 @@ It can also include ranges or any other value matching syntax within the set:
```
[00 C2 DE 'A'-'Z' &01]
```
Strings can be used to specify particular byte values. Note that the string itself doesn't match - the set matches any of the characters in the string:
Strings can be used to specify particular byte values. Note that the string itself doesn't match - the set matches any of the characters in the string:
```
['A'-'Z' 'aeiou']
```
Expand All @@ -130,23 +130,23 @@ To match any byte, we can use two question marks:
```

#### Unlimited gaps *
To specify that there's a unlimited gap between two parts of an expression (subject to how far DROID is configured to actually scan or the end of the data, whichever comes first), we can write an asterisk.
To specify that there's an unlimited gap between two parts of an expression (subject to how far DROID is configured to actually scan or the end of the data, whichever comes first), we can add an asterisk.

The following expression will first match _30 31 23 33 4E_, and it will then search for _43 2A B1 D4 CC EF_ until the end of the data:
```
30 31 23 33 4E * 43 2A B1 D4 CC EF
```

#### Fixed gaps {n}
To specify that there's a gap of one or more bytes we don't care about, we can write the size of the gap as a decimal number, surrounded by curly brackets:
To specify that there's a gap of one or more bytes we don't care about, we can write the size of the gap as a decimal number, surrounded by curly braces:
```
01 02 03 {128} FF FE
```

Note that writing _{1}_ is equivalent to writing _??_.

#### Variable gaps {n-m}
To specify that there's a range of possible bytes we don't care about, we can write the range as two numbers separted by a hyphen, surrounded by curly brackets:
To specify that there's a range of possible bytes we don't care about, we can write the range as two numbers separated by a hyphen, surrounded by curly braces:
```
01 02 03 {128-256} FF FE
```
Expand Down
6 changes: 2 additions & 4 deletions cli-tests/test/cli-test.bats
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,12 @@ setup() {

@test "check for signature updates" {
run $DROID_BIN -c
assert_output -p 'Container signature update Version 20250925 is available'
assert_output -p 'Binary signature update Version 121 is available'
assert_output 'No signature updates are available'
}

@test "download signature updates" {
run $DROID_BIN -d
assert_output -p 'Signature update version 20,250,925 has been downloaded'
assert_output -p 'Signature update version 121 has been downloaded'
assert_output 'No signature updates are available'
}

@test "list signature files" {
Expand Down
4 changes: 2 additions & 2 deletions droid-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<artifactId>droid-parent</artifactId>
<groupId>uk.gov.nationalarchives</groupId>
<version>6.9.6-SNAPSHOT</version>
<version>6.9.7-SNAPSHOT</version>
<relativePath>../droid-parent</relativePath>
</parent>

Expand Down Expand Up @@ -181,7 +181,7 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.18.0</version>
<version>3.19.0</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,6 @@ public DroidAPI build() throws SignatureParseException {
droidCore.setSignatureFile(binarySignature.toAbsolutePath().toString());
droidCore.init();
droidCore.setMaxBytesToScan(this.maxBytesToScan);
droidCore.getSigFile().prepareForUse();
String containerVersion = StringUtils.substringAfterLast(containerSignature.getFileName().toString(), "-").split("\\.")[0];
String droidVersion = ResourceBundle.getBundle("options").getString("version_no");
ContainerApi containerApi = new ContainerApi(droidCore, containerSignature);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,45 @@ public class DroidAPISkeletonTest {

private static DroidAPI api;

/**
* Unlike the skeleton suite test in droid-core, the DROID Api initialises the containers which remove signatures
* if they're in the container file with no other signatures. There was a bug that made these work originally which
* has now been fixed so we need to ignore these.
*/
private static final List<String> FILES_WITH_REMOVED_PUIDS = List.of("fmt-137-signature-id-204.ods",
"fmt-39-signature-id-128.doc",
"fmt-292-signature-id-309.odp",
"fmt-294-signature-id-312.ods",
"fmt-295-signature-id-314.ods",
"fmt-296-signature-id-316.odg",
"fmt-483-signature-id-905.epub",
"fmt-430-signature-id-644.cdr",
"fmt-290-signature-id-306.odt",
"fmt-2042-signature-id-3408.aab",
"fmt-136-signature-id-203.odt",
"fmt-291-signature-id-308.odt",
"fmt-429-signature-id-642.cdr",
"fmt-2043-signature-id-3412.aar",
"fmt-126-signature-id-172.ppt",
"fmt-294-signature-id-313.ods",
"fmt-125-signature-id-171.ppt",
"fmt-296-signature-id-315.odg",
"fmt-161-signature-id-292.siard",
"fmt-140-signature-id-207.odb",
"fmt-482-signature-id-906.ibooks",
"fmt-292-signature-id-310.odp",
"fmt-297-signature-id-317.odg",
"fmt-290-signature-id-307.odt",
"fmt-61-signature-id-78.xls",
"fmt-293-signature-id-311.odp",
"fmt-40-signature-id-182.doc",
"fmt-139-signature-id-206.odg",
"fmt-138-signature-id-205.odp",
"x-fmt-412-signature-id-199.jar",
"x-fmt-88-signature-id-181.ppt",
"x-fmt-430-signature-id-208.msg");


@BeforeAll
static void setup() throws IOException, SignatureParseException {
s3Server = createS3Server();
Expand All @@ -85,7 +124,7 @@ public static Stream<SkeletonTest> data() throws IOException {
return Stream.concat(
Files.list(Paths.get("../droid-core/test-skeletons/fmt")),
Files.list(Paths.get("../droid-core/test-skeletons/x-fmt"))
).flatMap(path -> {
).filter(path -> !FILES_WITH_REMOVED_PUIDS.contains(path.getFileName().toString())).flatMap(path -> {
Matcher matcher = FILENAME.matcher(path.getFileName().toString());
if (!matcher.matches()) {
return null;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -357,9 +357,9 @@ public void should_return_multiple_results_if_a_directory_is_passed_for_file_or_

@Test
public void should_report_correct_version_for_the_binary_and_container_signature() {
assertThat(api.getContainerSignatureVersion(), is("20240715"));
assertThat(api.getContainerSignatureVersion(), is("20250925"));
assertThat(api.getDroidVersion(), is(ResourceBundle.getBundle("options").getString("version_no")));
assertThat(api.getBinarySignatureVersion(), is("120"));
assertThat(api.getBinarySignatureVersion(), is("121"));
}

static Stream<URI> emptyFileUris() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@
*/
public class DroidAPITestUtils {

static Path signaturePath = Paths.get("../droid-results/custom_home/signature_files/DROID_SignatureFile_V120.xml");
static Path containerPath = Paths.get("../droid-results/custom_home/container_sigs/container-signature-20240715.xml");
static Path signaturePath = Paths.get("../droid-results/custom_home/signature_files/DROID_SignatureFile_V121.xml");
static Path containerPath = Paths.get("../droid-results/custom_home/container_sigs/container-signature-20250925.xml");

public static DroidAPI createApi(URI endpointOverride) throws SignatureParseException {
return createApi(endpointOverride, signaturePath, containerPath, null);
Expand Down
2 changes: 1 addition & 1 deletion droid-binary/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<artifactId>droid-parent</artifactId>
<groupId>uk.gov.nationalarchives</groupId>
<version>6.9.6-SNAPSHOT</version>
<version>6.9.7-SNAPSHOT</version>
<relativePath>../droid-parent</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion droid-build-tools/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<artifactId>droid-parent</artifactId>
<groupId>uk.gov.nationalarchives</groupId>
<version>6.9.6-SNAPSHOT</version>
<version>6.9.7-SNAPSHOT</version>
<relativePath>../droid-parent</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion droid-command-line/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<artifactId>droid-parent</artifactId>
<groupId>uk.gov.nationalarchives</groupId>
<version>6.9.6-SNAPSHOT</version>
<version>6.9.7-SNAPSHOT</version>
<relativePath>../droid-parent</relativePath>
</parent>

Expand Down
4 changes: 2 additions & 2 deletions droid-command-line/src/main/resources/options.properties
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,8 @@ droid -Pr "profile.maxBytesToScan=32768" -a "C:\\Files\\A Folder" -p "C:\\Result
Properties which can be overridden are those prefixed by "profile" in the droid.properties file contained in the droid home folder, \
normally found in the .droid6 home folder under the user's home folder. These currently include the following properties:\n \
profile.defaultThrottle = 0 \n \
profile.defaultBinarySigFileVersion = DROID_SignatureFile_V120 \n \
profile.defaultContainerSigFileVersion = container-signature-20240715 \n \
profile.defaultBinarySigFileVersion = DROID_SignatureFile_V121 \n \
profile.defaultContainerSigFileVersion = container-signature-20250925 \n \
profile.processTar = true \n \
profile.processZip = true \n \
profile.processGzip = true \n \
Expand Down
Loading
Loading