Full recrawl and reindex in Solr

[OkkeKlein]

How would one go about doing a full recrawl of content (filesystemcrawler) and then only do a (hard) commit after all content has been indexed.

So basically do a fresh update on a live system.

[essiembre]

To perform a "clean" crawl (without sending only modifications, deletions, etc.) you simply have to delete your "workdir". More precisely, the crawlstore.

To not commit until you are done, you can set the "solrCommitDisabled" option to "true" in your Solr committer section. This means the committer will never send a Solr "commit" request, thus relying on your Solr configuration to decide when to commit, or your manual commit.

[OkkeKlein]

Thank you!

[OkkeKlein]

How to deal with deleted docs? A manual delete all? Or maybe parameter send with Solr committer?

[essiembre]

A simple approach would be to add two fields in your collection. One that identifies the source crawler, the second that identifies the crawl date. In your config, you can use the ConstantTagger to populate the first one, and for the second, you can use the CurrentDateTagger.

With this, you can use the "delete by query" approach on Solr. You would issue a query that deletes anything older than the date of your full recrawl, for the given crawler.