LabRAD upgrade issue: labrad-migrate-registry doesn't work with string

[xiaoyuejin]

I am a user of the old Delphi LabRAD manager for years. Now I am trying to upgrade to the new scalabrad manager, but backward compatibility is of my highest priority. For newer users who never touched the Delphi version, these issues might not be interesting…

Since I have multiple problems, I will split them to different issues.

Test platform: Windows 7 x64, Java 8 up to date, Python 2.7, pylabrad up to date, without footprint of previous versions of LabRAD.

I have a lot of old registry files (*.key) generated by the old Delphi manager. When the key contains a string, one MUST use the double quotation marks, otherwise the Delphi manager won't accept it. Now it has become a problem.

For example, a file named test.key contains:
"this is a test"

The latest registry does not accept this string. I get:

reg.dir()
Out[152]: ([], ['new_test', 'test'])

reg.get('test')
Traceback (most recent call last):

File "", line 1, in
reg.get('test')

File "c:\python27\lib\site-packages\labrad\client.py", line 65, in call
return f.result() if wait else f

File "c:\python27\lib\site-packages\concurrent\futures_base.py", line 429, in result
return self.__get_result()

File "c:\python27\lib\site-packages\concurrent\futures_base.py", line 381, in __get_result
raise exception_type, self._exception, self._traceback

Error: (nonArrayData | array):1:1 ...""this is a"

I must replace the double quotation marks with single quotation marks, then it works:

reg.get('new_test')
Out[154]: 'this is a test'

Now, there is this labrad-migrate-registry tool that is supposed to migrate old registry into the latest format. So I gave it a try. But there are a few problems...

1. The migration does not go to the sub-directories, but only read the .key files in the specific directory. This is bad if I have a registry with hundreds of sub-directories. Is there a switch to let the program go through all the directories and convert all the .key files? I didn't find the switch in the --help.

2. When the migration meets a .key file that contains strings with old double quotation mark format such as "xxx", it still fails. For example, the string "VNA" with double quotation mark with trigger this error:

Exception in thread "main" java.lang.RuntimeException: (nonArrayData | array):1:1 ...""VNA"
at scala.sys.package$.error(package.scala:27)
at org.labrad.util.Parsing$.parseOrThrow(Parsing.scala:33)
at org.labrad.registry.DelphiParsers$.parseData(DelphiStore.scala:75)
at org.labrad.registry.DelphiFormat$.stringToData(DelphiStore.scala:182)
at org.labrad.registry.DelphiFileStore.decodeData(DelphiStore.scala:62)
at org.labrad.registry.FileStore.getValue(FileStore.scala:89)
at org.labrad.registry.FileStore.getValue(FileStore.scala:17)
at org.labrad.registry.Migrate$LocalRegistry$$anonfun$get$2.apply(Migrate.scala:222)
at org.labrad.registry.Migrate$LocalRegistry$$anonfun$get$2.apply(Migrate.scala:221)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at org.labrad.registry.Migrate$LocalRegistry.get(Migrate.scala:221)
at org.labrad.registry.Migrate$.org$labrad$registry$Migrate$$traverse$1(Migrate.scala:100)
at org.labrad.registry.Migrate$.main(Migrate.scala:118)
at org.labrad.registry.Migrate.main(Migrate.scala)

Basically the labrad-migrate-registry tool fails exactly where the new registry itself would fail. I guess it's because the labrad-migrate-registry tool was not designed to migrate stone-age registry files like mine, but only can convert middle-age files? :)

Anyway, I am reporting it (as a bug?) and hopefully there is an easy fix available.

[maffoo]

We've used the migration tool to migrate registry data, including subfolders, so I'm not sure what the issue could be. If you have subdirectories that are not named with a .dir suffix (as in #95), then neither the registry server nor the migration tool wil recognize them, but the registry itself would never create such directories, so those must have been created in some other way?

As for the issues with string escaping, I think in fact the delphi manager always used single quotes for strings in registry files, so it's not surprising that it would fail on a key file containing double quotes (I think this was originally because pascal source code uses single quotes for string literals). Can you actually access the test.key file that you mentioned with a double-quoted string using the old delphi manager?

[xiaoyuejin]

Hi maffoo, yes, I actually can access the test.key file with double-quoted strings with my delphi manager (whereas single-quoted strings will return an error). :D

OK, now it is getting very clear what is going on. The "Main Stream" delphi manager always used single quotes for strings, as well as .dir suffix for directories. All the upgrade/migration of the new versions of LabRAD are designed around these formats.

But there exists a weird and forgotten branch in Boston since 2011 (version 2011.10b to be precise), where the manager does not do .dir suffix (it reads test.dir directory if you set the suffix manually though, so test and test.dir are TWO different directories), and ONLY takes double-quoted strings.

I have exclusively used the "Boston" version of manager in the past 6 years and never talked to the outside world so I never got a problem. Now that I finally decide to talk to the outside, I am like a caveman walking around in Manhattan... I guess nobody else is using the Boston version anymore. So end of the story.

As to my migration/upgrade, I think now I've understood the new version good enough, that I can just write up a short code, hook up with 1 old manager and 1 new manager at the same time, and literally read each and every data/registry using the old manager, and pass them to the new manager one by one to store them in new format. Might take a while (GBs of data) but it should not be difficult to do.

[maffoo]

I see. You should be able to use the labrad-migrate-registry tool to do the "online" migration where it connects to your running version of the delphi manager to read data, and sends it to a running version of scalabrad to write the data in the new format (or can write directly to disk). You just need to use a labrad:// url for the source registry when you run the migration tool. That way you shouldn't need to write any code yourself.

[xiaoyuejin]

But my old registry contains "XXX" (which makes the migration tool report an error, see above) and all the directories don't have .dir suffix (which means they are invisible for the migration tool). So, labrad-migrate-registry won't work for me. At least I must write the first part, the code that can go through all the files and directories of my registry. So I guess I might as well write the whole thing...

[maffoo]

What I'm saying is to run your old manager with the "boston" code and then tell labrad-migrate-registry to just talk to that manager to get the registry data. The migration code never needs to touch the actual files on disk.

[xiaoyuejin]

OK now I get it. Thanks.