Bitcoin 0.10 has been released - what should we do?

[zathras-crypto]

Hi all,

I'd like to get a discussion going with as much participation as possible. The issue is simply that Bitcoin 0.10 has been released, and this is not compatible with OmniCore 0.0.9.1 (due to headers first).

We're gearing up to release the UI, however we need a strategy to handle the issue. We can't have our first major end user release being a failure because we haven't planned for this appropriately.

Thoughts please?

Thanks
Z

[achamely]

How does the Omnicore process the omni transactions now. Are they triggered during the normal bitcoin tx processing? If so how about adding a middle man step.

High level thought process:
The new headers first method seems to grab multiple chains and determines which one has the most proof-of-work before downloading its actual blocks. Could we, Once the download blocks is triggered, side process and create a standalone list of Omni tx's with corresponding tx/block times. Then once the blockchain data is complete we trigger the full omni parsing of our new ordered list.

[dexX7]

Every time a new block is connected, it is forwarded to the meta layer at the moment.

For 0.10 instead of forwarding every new block, a block should be forwarded, when it's the very next to the last processed one, and the forwarding should continue, until no next block is available. But there are also "higher level" signals to subscribe to and it's not necessarily required to hook add handlers into ConnectTip (IIRC).

#263 worked, back in December, although I tested it basically only fully synchronized with new blocks added to the top.

The biggest hurdle is probably UI related, because it has changed significantly:

[msgilligan]

From my perspective (someone developing software to the RPC API) the 0.10.0 release is very desirable. Several of the improvements (faster blockchain downloading, RPC "warm-up" mode) address pain points that come up in testing/integrating.

There is also a fix for building on OS X 10.9.

And I almost certainly will update Bitcoin-Qt on my Mac.

[zathras-crypto]

Sorry, for clarification I mean strategically.

So yes absolutely we want to move to 0.10, but that will take time (an unknown quantity at this point).

Moreso things like:

• "What do we tell end users who want to use OmniCore but are already on Bitcoin 0.10?"
• "Do we wait to release the UI, or release based on the 0.9 bitcoin code?"
• "Is there any actual risk to end user funds/walletfiles from trying to use the wrong blockchain?"
• "Are there any 'frustrations' for end users from trying to use the wrong blockchain? (like it being downloaded from scratch, or having to reindex again and so on)
• "Etc etc"

The technical stuff is still important, but moreso end user facing issues such as:

• "What happens when a user with Bitcoin 0.10 and a v0.10 blockchain runs OmniCore 0.0.9.1?"
• "What happens when a user with OmniCore 0.0.9.1 and a v0.9 blockchain runs Bitcoin 0.10?"
• "Do locks still effectively prevent both being run together?"
• "Etc etc"

The actual questions surrounding a technical migration would be addressed as we actually worked towards OmniCore 0.0.10. I'm moreso concerned with the here and now - our ability to a) deliver this platform in a working fashion if we release now and b) our ability to support the platform in the context of the incompatibilities between the 9 and 10 branches of Bitcoin.

Hope that makes more sense :)

[msgilligan]

I guess I'm saying I support switching to 0.10.0 before making any new releases -- including the UI. At the very least we should take the time to quantify the effort before deciding.

[zathras-crypto]

So the decision was made to go ahead with a 0.0.9.1 based release, along with some code to force a shutdown during startup if we detect a blockchain from 0.10.

First question now to all - before I spin up the VM's to test, has anyone actually tried to run OmniCore 0.0.9 against a 0.10 blockchain? @msgilligan you mentioned you thought code was already there to prevent 0.9 bitcoin clients running when a 0.10 blockchain is present - have you tested this by any chance?

Thanks all
Z

[msgilligan]

I haven't tried it, no. I don't have any specific knowledge, just a hunch given what I know of the Bitcoin core team and the fact that this feature was on their roadmap for a long time.

The Downgrading section of the 0.10.0 release notes says "It is possible that the data from a completely synchronised 0.10 node may be usable in older versions as-is, but this is not supported and may break as soon as the older version attempts to reindex."

[zathras-crypto]

OK great, thanks for the info - that's kind of what I was concerned about - have a bitcoin 0.10 install and run Omnicore which may "break as soon as the older version attempts to reindex" - that kind of indicates to me that an older version will attempt to reindex, rather than cleanly shutting down.

I'll have a poke around and see what I can come up with :)

Thanks
Z

[dexX7]

Just playing devil's advocat here for the sake of being overly protective: the risk is that blocks are stored out of order, because they arrive out of order, and an old client may not make sense of it. I assume that this risk is also given, if the user is just a few blocks behind and catches up.

Blocks will be stored on disk out of order (in the order they are received, really), which makes it incompatible with some tools or other programs. Reindexing using earlier versions will also not work anymore as a result of this.

Reindexing and fetching the last blocks doesn't seem much different.

Edit: this was only a guess, but after thinking more about it: blocks are indexed, and a 0.9 client wouldn't necessarily locate blocks by scanning files, but rather looking into the blockdb, so the statement regarding reindexing gains more significance for me now and it is also a bit different than catching up.

@zathras-crypto: do you have an idea how to identify it?

[zathras-crypto]

Alrighty, testing this out myself ATM. Have taken a snapshot, deployed 0.10 and spun it up. Firstly no idea why it needed to write over a GB of delta when 'verifying blocks' - assuming it's doing more than just verification there.

It still appears to be syncing in sequence (days going down one by one, blocks processed going up sequentially about 50 blocks at per update).

[zathras-crypto]

OK. I performed the following test:

• Take a bitcoin core instance that's about 5 days behind
• Upgrade it to bitcoin 0.10
• Start it up, let it sync
• Run OmniCore 0.0.9.1

First thing to confirm, there is absolutely no detection or protection against using the bitcoin 0.10 blockchain with an earlier client (at least bitcoin 0.9.3 (our base) anyway). OmniCore fires up no errors or warnings.

[zathras-crypto]

Still nosing around looking for a way to detect if 0.10 has been used on the blockchain, but I did notice that even on 0.10 the blocks seemed to be processed in sequential order. Perhaps this is because peers aren't running 0.10?

http://pastebin.com/raw.php?i=2hRmGrp5

[dexX7]

You should try getblockchaininfo, but it looks like you're already fully synced now. It shows the number of headers as well blocks.

[dexX7]

So the decision was made to go ahead with a 0.0.9.1 based release, ...

Not that I want to hijack this decision, but I was curious about the block connection behavior and ended up porting Omni Core without UI to Bitcoin Core 0.10 in the last hours.

All regtests passed and at the moment I'm synchronizing on mainnet and dump every handler call. Looks good so far! And it appears that ConnectTip is triggered, once there is really a new, next, block:

I'm away for a few hours now, but I'll try to push everything wrapped into expressive commits later.

That being said, porting was so much easier than assumed, but only because I prepared most of it earlier, e.g. cleaning up the includes etc., ... :)

But key was also to start with 0.10 and add the components one by one, instead the other way around as last time, which came with like 100 merge conflicts. >_<

As heads up, those are requirements:

• Replace: CScript::mscore_getHex() and CScript::mscore_parse() #266 Replace: CScript::mscore_getHex() and CScript::mscore_parse()
• Fix and order includes, using directives, makefile #276 Fix and order includes, using directives, makefile
• (unpublished) compatibility patches

... and my current build also includes some of the other PRs, right now to ensure it also passes the other tests I have.

The incompatibility issues can be resolved in ~15-30 lines or so in total, so reviewing the PRs is probably most of the work for Omni Core based on Bitcoin Core 0.10 without UI modifications, but plain Bitcoin Core interface. Edit: to clarfiy: graphical interface. All the RPC interaction work fine.

@zathras-crypto: what do you think about porting the QT code? As very rough guess I'd assume the tricky part is to deal with 0.10's new interface, while the code blocks could probably be copied one to one. Will take a look at this later as well, if I find some time..

[zathras-crypto]

But key was also to start with 0.10 and add the components one by one, instead the other way around as last time, which came with like 100 merge conflicts. >_<

Hehe, that's also Michael's preferred methodology :)

All regtests passed

I think the primary issue is that I don't think anyone is comfortable changing our underlying bitcoin version in a minor release. It's GREAT to know this can be progressed in such a straight forward manner, and well done my friend :) but when looking at a risk angle - to play the safe card we need to change our base Bitcoin code in a major release not minor.

To be frank, the issue is driven by timing primarily. The powers that be want the UI released yesterday and putting whatever QT work is required aside for a moment, I would still insist on a whole bunch of testing for a new base bitcoin version. I really do appreciate the testing you've done so far, but for me I just don't feel comfortable enough yet to say "in 100% of potential circumstances there is absolutely no possible change to Omni data in bitcoin 0.10" which is needed to do a non-major release.

Sorry, not wanting to ramble on - long story short the main driver is to do 3 things:

1. Clean up the last criticals - signalling - potentially via your PR. Apologies for the delay there (sadly I'm in and out of hospital for various tests at the mo which is quite a time drain, but health has to come first)
2. Put in detection for a 0.10 blockchain and refuse to start (safety net)
3. Release the UI as 0.0.9.1

Once that's done, we could even skip 0.0.9.2 and move straight to a 0.0.10 with bitcoin 0.10.

what do you think about porting the QT code?

To be honest I don't think it'll be a major drama, I'll be applying similar logic to you & Michael with the copying files over - though there may need to be some time spent in QT designer adjusting some of the forms (that's always fun! /sarcasm)