Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge new sweeper branch to master #8667

Merged
merged 121 commits into from
Apr 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
e86843d
sweep: rename `NotifyPublishTx` to `StoreTx`
yyforyongyu Oct 25, 2023
1ace1fd
sweep: return fees from method `createSweepTx`
yyforyongyu Oct 25, 2023
b37444d
sweep: expand sweeper store to also save RBF-related info
yyforyongyu Oct 25, 2023
8b9d5e0
sweep: add new methods `GetTx` and `DeleteTx` to manage `TxRecord`
yyforyongyu Oct 25, 2023
f13a3a8
sweep: use `testify/mock` for `MockSweeperStore`
yyforyongyu Jan 10, 2024
ca0813b
docs: update release docs
yyforyongyu Jan 7, 2024
84a6fdc
sweep+contractcourt: track best height in `UtxoSweeper`
yyforyongyu Oct 23, 2023
18b06b7
sweep: replace `feeRateForPreference` with `Estimate`
yyforyongyu Oct 24, 2023
6ff6c86
multi: merge `DetermineFeePerKw` and `Estimate`
yyforyongyu Nov 3, 2023
530eed9
multi: rename `FeePreference` to `FeeEstimateInfo`
yyforyongyu Nov 3, 2023
3bcac31
sweep+lnrpc: add new interface `FeePreference`
yyforyongyu Nov 3, 2023
1870caf
sweep+lnd: introduce `UtxoAggregator` to handle clustering inputs
yyforyongyu Oct 24, 2023
519a874
lntest: reflect the updated fee estimator in tests
yyforyongyu Jan 25, 2024
61e9c11
rpcserver: skip calculating fee rate for PSBT funding
yyforyongyu Jan 26, 2024
1e6e443
docs: update release notes
yyforyongyu Jan 9, 2024
6a2e3fb
sweep: make pending input stateful
yyforyongyu Oct 24, 2023
4747871
multi: query mempool spend when a new input is received
yyforyongyu Oct 24, 2023
a263d68
sweep: delete pending inputs based on their states
yyforyongyu Oct 24, 2023
8b876be
sweep: add fee info for published inputs
yyforyongyu Oct 26, 2023
a8f5a09
sweep: don't give up an input based on number of attempts
yyforyongyu Oct 26, 2023
34b6a3d
sweep: add method `markInputFailed`
yyforyongyu Feb 27, 2024
fd92294
sweep: patch unit tests for `markInputsSwept` and `markInputsPendingP…
yyforyongyu Jan 10, 2024
124d176
sweep: remove unused param `testSpendChan`
yyforyongyu Jan 9, 2024
5496d02
sweep: fix logging
yyforyongyu Jan 9, 2024
b5e4384
lnd+sweep: remove unused `NextAttemptDeltaFunc`
yyforyongyu Jan 11, 2024
ebef367
btcdnotify: use `chain.NewRPCClientWithConfig` to init RPC client
yyforyongyu Feb 24, 2024
5fce91c
chainntnfs+sweep: add `LookupInputMempoolSpend` and use it in the
yyforyongyu Feb 24, 2024
536ac9f
docs: update release notes
yyforyongyu Jan 9, 2024
6f5b7a9
lntest+itest: fix `testOpenChannelLockedBalance`
yyforyongyu Mar 19, 2024
df4e51e
sweep: refactor `attachAvailableRBFInfo` to `decideStateAndRBFInfo`
yyforyongyu Mar 19, 2024
a7e9c08
sweep: make sweeper block-driven instead of time-driven
yyforyongyu Oct 26, 2023
9d5ddf2
sweep: add new interface `Cluster` to manage grouping inputs
yyforyongyu Oct 30, 2023
1530fee
sweep: add new inteface `InputSet` to manage inputs in a tx
yyforyongyu Oct 30, 2023
b536e9b
sweep: deepen the interface `Aggregator`
yyforyongyu Oct 30, 2023
0110a09
sweep: decrease coin selection lock scope
yyforyongyu Nov 1, 2023
ae43764
sweep: remove deprecated `rand.Seed`
yyforyongyu Jan 24, 2024
210b783
sweep: move mocks into one file
yyforyongyu Jan 30, 2024
c035093
sweep: add mocks and patch unit test for `sweepPendingInputs`
yyforyongyu Jan 30, 2024
465332f
multi: deprecate `batchwindowduration` config option
yyforyongyu Jan 31, 2024
d8f976f
trivial: fix comments and docs
yyforyongyu Mar 2, 2024
558d9bd
walletrpc: pass `Force` flag when sweeping via `BumpFee`
yyforyongyu Mar 14, 2024
521b1fc
itest+lntest: fix itest re the new block driven behavior
yyforyongyu Mar 14, 2024
d0a8f27
sweep: change `MaxInputsPerTx` from `int` to `uint32`
yyforyongyu Mar 19, 2024
db7eae9
sweep: expand `InputSet` with more interface methods
yyforyongyu Jan 16, 2024
6202c59
sweep: change `markInputsPublishFailed` to take outpoints
yyforyongyu Jan 16, 2024
bd5eec8
sweep: refactor `markInputsPendingPublish` to take `InputSet`
yyforyongyu Feb 21, 2024
e7400f6
sweep: introduce `BudgetInputSet` to manage budget-based inputs
yyforyongyu Feb 27, 2024
a088501
sweep: introduce `BudgetAggregator` to cluster inputs by deadlines
yyforyongyu Feb 27, 2024
1187b86
sweep: introduce `Bumper` interface to handle RBF
yyforyongyu Jan 17, 2024
21aff32
sweeper: fix existing sweeper tests
yyforyongyu Feb 28, 2024
6d6c544
sweep: remove RBF related tests
yyforyongyu Mar 15, 2024
f7bc82a
sweep: remove `FeeRate()` from `InputSet` interface
yyforyongyu Jan 17, 2024
cd5d074
sweep: add `FeeFunction` interface and a linear implementation
yyforyongyu Jan 24, 2024
f85661d
lnwallet+sweep: add new method `CheckMempoolAcceptance`
yyforyongyu Feb 28, 2024
ecd471a
lnwallet+sweep: calculate max allowed feerate on `BumpResult`
yyforyongyu Feb 29, 2024
11f7e45
lnwallet+sweep: introduce `TxPublisher` to handle fee bump
yyforyongyu Feb 29, 2024
90e727a
sweep: add monitor loop to `TxPublisher`
yyforyongyu Feb 29, 2024
23feef7
lnd: init publisher when creating new server
yyforyongyu Mar 17, 2024
7fb18bc
sweep: increase delta fee rate precision in fee function
yyforyongyu Mar 18, 2024
94390fc
itest: fix existing itests
yyforyongyu Mar 17, 2024
b76ccab
docs: update release notes for fee bumper
yyforyongyu Mar 17, 2024
19265ac
sweep: make sure non-fee related errors are notified
yyforyongyu Mar 26, 2024
0063770
sweep: remove the prefix used in `SweepState` types
yyforyongyu Mar 17, 2024
9e7d4b7
sweep: rename `pendingInputs` to `inputs` on sweeper
yyforyongyu Mar 17, 2024
28df2d7
lnrpc+sweep: make sure public interface takes public types as params
yyforyongyu Mar 17, 2024
1fa8ca7
sweep: fix `handlePendingSweepsReq` linter issue
yyforyongyu Mar 20, 2024
0fc5301
lnwallet+sweep: cap conf target used in fee estimator
yyforyongyu Mar 18, 2024
0527b2d
sweep: make sure exclusive inputs are not grouped
yyforyongyu Mar 17, 2024
658ba44
sweep: delay sweeping inputs with future locktimes
yyforyongyu Mar 20, 2024
0b30d4b
lnrpc+sweep: make `FeeEstimateInfo` an optional param
yyforyongyu Mar 18, 2024
370e4ce
lnd+sweep: remove unused config `MaxSweepAttempts`
yyforyongyu Mar 18, 2024
59fbcb1
sweep: rename `fee()` to `feeWithParent()` for clarity
yyforyongyu Mar 20, 2024
5952698
sweep: add a dedicated method to create sweeping txns
yyforyongyu Mar 21, 2024
106b97c
sweep: improve loggings
yyforyongyu Mar 21, 2024
fce86f9
sweep: cancel rebroadcasting of failed/replaced/confirmed txns
yyforyongyu Mar 21, 2024
e771993
multi: make `input.OutPoint` return `wire.OutPoint`
yyforyongyu Mar 27, 2024
1558835
sweep: exit when `handleNewInput` fails
yyforyongyu Mar 27, 2024
4c13ea1
sweep: pass default deadline height when clustering inputs
yyforyongyu Mar 27, 2024
54ade99
sweep: apply `fn.NewSet` and `fn.Map` in `validateInputs`
yyforyongyu Mar 27, 2024
c44b8a4
lnd: start using `BudgetAggregator`
yyforyongyu Mar 17, 2024
0a611aa
multi: add new config option `BudgetConfig` and `NoDeadlineConfTarget`
yyforyongyu Mar 18, 2024
aa44197
contractcourt: init `BudgetConfig` in unit tests
yyforyongyu Mar 18, 2024
01fd4e5
contractcourt: specify deadline and budget for commit sweep
yyforyongyu Mar 2, 2024
d1ad07f
contractcourt+lnwallet: specify deadline and budget for htlc success
yyforyongyu Mar 18, 2024
cab180a
contractcourt: specify deadline and budget for htlc timeout
yyforyongyu Mar 19, 2024
f4035ad
contractcourt: calculate value left when searching for commit deadline
yyforyongyu Mar 19, 2024
6f0c2b5
contractcourt: specify deadline and budget for anchor output
yyforyongyu Mar 19, 2024
33abbe1
contractcourt+lnd: make `IncubateOutputs` take fn.Option
yyforyongyu Mar 29, 2024
9c1e694
contractcourt: specify deadline and budget for nursery
yyforyongyu Mar 29, 2024
dc7d90c
contractcourt+sweep: offer direct-preimage spend via `SweepInput`
yyforyongyu Mar 19, 2024
4134b1c
sweep: make sure max fee rate can be reached
yyforyongyu Apr 3, 2024
07466c4
multi: query circuit map inside contractcourt
yyforyongyu Apr 4, 2024
c644deb
contractcourt: add locks in `SubscribeChannelEvents`
yyforyongyu Apr 8, 2024
acde08c
contractcourt: offer second-level outputs at CSV-1
yyforyongyu Apr 8, 2024
a2b8f4e
sweep: allow published input to be marked as `PublishFailed`
yyforyongyu Apr 8, 2024
94e0e32
multi: add itest `testSweepAnchorCPFPLocalForceClose`
yyforyongyu Apr 1, 2024
a1a480a
itest+lntest: add itest `testSweepHTLCs` to check HTLC sweepings
yyforyongyu Apr 3, 2024
6933c5a
itest: remove old CPFP tests
yyforyongyu Mar 21, 2024
ce58175
itest+lntest: fix channel force close itest
yyforyongyu Mar 26, 2024
f68c143
itest: fix multi-hop itest
yyforyongyu Apr 6, 2024
30c2b9f
itest: fix channel backup tests
yyforyongyu Apr 7, 2024
9c34eb7
itest: fix revocation itest
yyforyongyu Apr 8, 2024
d4de6dd
itest+lntest: fix onchain tests
yyforyongyu Apr 8, 2024
563a5ca
itest: fix watchtower test
yyforyongyu Apr 8, 2024
19a599a
sweep: catch third party spent in fee bumper for neutrino
yyforyongyu Apr 12, 2024
db3aad3
lnrpc+sweep: rename `Force` to `Immediate` for clarity
yyforyongyu Apr 2, 2024
b6a2984
sweep: allow specifying starting fee rate for fee func
yyforyongyu Apr 11, 2024
9be5b37
sweep+contractcourt: replace `ParamsUpdate` with `Params`
yyforyongyu Apr 9, 2024
94159e8
walletrpc: update `walletkit.proto` to use the new sweep params
yyforyongyu Apr 9, 2024
8804947
walletrpc+sweep: refactor `BumpFee` to properly handle sweep request
yyforyongyu Apr 11, 2024
11a276e
walletrpc+sweep: update `PendingSweeps` to return the new params
yyforyongyu Apr 11, 2024
6f55a7a
itest: add new test to check `BumpFee` and `PendingSweeps`
yyforyongyu Apr 11, 2024
7abefa7
cmd: update `bumpfee`, `bumpclosefee` and `bumpforceclosefee`
yyforyongyu Apr 2, 2024
49cfb91
contractcourt: make sure sweep happens immediately on startup
yyforyongyu Apr 2, 2024
871cab4
sweep: make sure inputs with different locktime values are not grouped
yyforyongyu Apr 11, 2024
f5a321d
contractcourt: remove unused param to please linter
yyforyongyu Apr 12, 2024
96883f3
itest+sweep: add itest `testSweepCommitOutputAndAnchor`
yyforyongyu Apr 12, 2024
a50cdd6
sweep: assign deadline values to inputs in `handleNewInput`
yyforyongyu Apr 15, 2024
e0f0f5c
sweep: skip wallet inputs in `isThirdPartySpent`
yyforyongyu Apr 17, 2024
d854c80
itest+lntest: fix itest re the new sweeping behavior
yyforyongyu Apr 17, 2024
4d96f9c
docs: add release notes
yyforyongyu Apr 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions chainntnfs/bitcoindnotify/bitcoind.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/btcsuite/btcwallet/chain"
"github.com/lightningnetwork/lnd/blockcache"
"github.com/lightningnetwork/lnd/chainntnfs"
"github.com/lightningnetwork/lnd/fn"
"github.com/lightningnetwork/lnd/queue"
)

Expand Down Expand Up @@ -1070,3 +1071,26 @@ func (b *BitcoindNotifier) CancelMempoolSpendEvent(

b.memNotifier.UnsubscribeEvent(sub)
}

// LookupInputMempoolSpend takes an outpoint and queries the mempool to find
// its spending tx. Returns the tx if found, otherwise fn.None.
//
// NOTE: part of the MempoolWatcher interface.
func (b *BitcoindNotifier) LookupInputMempoolSpend(
op wire.OutPoint) fn.Option[wire.MsgTx] {

// Find the spending txid.
txid, found := b.chainConn.LookupInputMempoolSpend(op)
if !found {
return fn.None[wire.MsgTx]()
}

// Query the spending tx using the id.
tx, err := b.chainConn.GetRawTransaction(&txid)
if err != nil {
// TODO(yy): enable logging errors in this package.
return fn.None[wire.MsgTx]()
}

return fn.Some(*tx.MsgTx().Copy())
}
48 changes: 41 additions & 7 deletions chainntnfs/btcdnotify/btcd.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ import (
"github.com/btcsuite/btcd/rpcclient"
"github.com/btcsuite/btcd/txscript"
"github.com/btcsuite/btcd/wire"
"github.com/btcsuite/btcwallet/chain"
"github.com/lightningnetwork/lnd/blockcache"
"github.com/lightningnetwork/lnd/chainntnfs"
"github.com/lightningnetwork/lnd/fn"
"github.com/lightningnetwork/lnd/queue"
)

Expand Down Expand Up @@ -58,7 +60,7 @@ type BtcdNotifier struct {
active int32 // To be used atomically.
stopped int32 // To be used atomically.

chainConn *rpcclient.Client
chainConn *chain.RPCClient
chainParams *chaincfg.Params

notificationCancels chan interface{}
Expand Down Expand Up @@ -127,21 +129,30 @@ func New(config *rpcclient.ConnConfig, chainParams *chaincfg.Params,
quit: make(chan struct{}),
}

// Disable connecting to btcd within the rpcclient.New method. We
// defer establishing the connection to our .Start() method.
config.DisableConnectOnNew = true
config.DisableAutoReconnect = false

ntfnCallbacks := &rpcclient.NotificationHandlers{
OnBlockConnected: notifier.onBlockConnected,
OnBlockDisconnected: notifier.onBlockDisconnected,
OnRedeemingTx: notifier.onRedeemingTx,
}

// Disable connecting to btcd within the rpcclient.New method. We
// defer establishing the connection to our .Start() method.
config.DisableConnectOnNew = true
config.DisableAutoReconnect = false
chainConn, err := rpcclient.New(config, ntfnCallbacks)
rpcCfg := &chain.RPCClientConfig{
ReconnectAttempts: 20,
Conn: config,
Chain: chainParams,
NotificationHandlers: ntfnCallbacks,
}

chainRPC, err := chain.NewRPCClientWithConfig(rpcCfg)
if err != nil {
return nil, err
}
notifier.chainConn = chainConn

notifier.chainConn = chainRPC

return notifier, nil
}
Expand Down Expand Up @@ -1127,3 +1138,26 @@ func (b *BtcdNotifier) CancelMempoolSpendEvent(

b.memNotifier.UnsubscribeEvent(sub)
}

// LookupInputMempoolSpend takes an outpoint and queries the mempool to find
// its spending tx. Returns the tx if found, otherwise fn.None.
//
// NOTE: part of the MempoolWatcher interface.
func (b *BtcdNotifier) LookupInputMempoolSpend(
op wire.OutPoint) fn.Option[wire.MsgTx] {

// Find the spending txid.
txid, found := b.chainConn.LookupInputMempoolSpend(op)
if !found {
return fn.None[wire.MsgTx]()
}

// Query the spending tx using the id.
tx, err := b.chainConn.GetRawTransaction(&txid)
if err != nil {
// TODO(yy): enable logging errors in this package.
return fn.None[wire.MsgTx]()
}

return fn.Some(*tx.MsgTx().Copy())
}
6 changes: 6 additions & 0 deletions chainntnfs/interface.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"github.com/btcsuite/btcd/btcutil"
"github.com/btcsuite/btcd/chaincfg/chainhash"
"github.com/btcsuite/btcd/wire"
"github.com/lightningnetwork/lnd/fn"
)

var (
Expand Down Expand Up @@ -849,4 +850,9 @@ type MempoolWatcher interface {
// CancelMempoolSpendEvent allows the caller to cancel a subscription to
// watch for a spend of an outpoint in the mempool.
CancelMempoolSpendEvent(sub *MempoolSpendEvent)

// LookupInputMempoolSpend looks up the mempool to find a spending tx
// which spends the given outpoint. A fn.None is returned if it's not
// found.
LookupInputMempoolSpend(op wire.OutPoint) fn.Option[wire.MsgTx]
}
123 changes: 123 additions & 0 deletions chainntnfs/mocks.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
package chainntnfs

import (
"github.com/btcsuite/btcd/chaincfg/chainhash"
"github.com/btcsuite/btcd/wire"
"github.com/lightningnetwork/lnd/fn"
"github.com/stretchr/testify/mock"
)

// MockMempoolWatcher is a mock implementation of the MempoolWatcher interface.
// This is used by other subsystems to mock the behavior of the mempool
// watcher.
type MockMempoolWatcher struct {
mock.Mock
}

// NewMockMempoolWatcher returns a new instance of a mock mempool watcher.
func NewMockMempoolWatcher() *MockMempoolWatcher {
return &MockMempoolWatcher{}
}

// Compile-time check to ensure MockMempoolWatcher implements MempoolWatcher.
var _ MempoolWatcher = (*MockMempoolWatcher)(nil)

// SubscribeMempoolSpent implements the MempoolWatcher interface.
func (m *MockMempoolWatcher) SubscribeMempoolSpent(
op wire.OutPoint) (*MempoolSpendEvent, error) {

args := m.Called(op)

if args.Get(0) == nil {
return nil, args.Error(1)
}

return args.Get(0).(*MempoolSpendEvent), args.Error(1)
}

// CancelMempoolSpendEvent implements the MempoolWatcher interface.
func (m *MockMempoolWatcher) CancelMempoolSpendEvent(
sub *MempoolSpendEvent) {

m.Called(sub)
}

// LookupInputMempoolSpend looks up the mempool to find a spending tx which
// spends the given outpoint.
func (m *MockMempoolWatcher) LookupInputMempoolSpend(
op wire.OutPoint) fn.Option[wire.MsgTx] {

args := m.Called(op)

return args.Get(0).(fn.Option[wire.MsgTx])
}

// MockNotifier is a mock implementation of the ChainNotifier interface.
type MockChainNotifier struct {
mock.Mock
}

// Compile-time check to ensure MockChainNotifier implements ChainNotifier.
var _ ChainNotifier = (*MockChainNotifier)(nil)

// RegisterConfirmationsNtfn registers an intent to be notified once txid
// reaches numConfs confirmations.
func (m *MockChainNotifier) RegisterConfirmationsNtfn(txid *chainhash.Hash,
pkScript []byte, numConfs, heightHint uint32,
opts ...NotifierOption) (*ConfirmationEvent, error) {

args := m.Called(txid, pkScript, numConfs, heightHint)
if args.Get(0) == nil {
return nil, args.Error(1)
}

return args.Get(0).(*ConfirmationEvent), args.Error(1)
}

// RegisterSpendNtfn registers an intent to be notified once the target
// outpoint is successfully spent within a transaction.
func (m *MockChainNotifier) RegisterSpendNtfn(outpoint *wire.OutPoint,
pkScript []byte, heightHint uint32) (*SpendEvent, error) {

args := m.Called(outpoint, pkScript, heightHint)
if args.Get(0) == nil {
return nil, args.Error(1)
}

return args.Get(0).(*SpendEvent), args.Error(1)
}

// RegisterBlockEpochNtfn registers an intent to be notified of each new block
// connected to the tip of the main chain.
func (m *MockChainNotifier) RegisterBlockEpochNtfn(epoch *BlockEpoch) (
*BlockEpochEvent, error) {

args := m.Called(epoch)
if args.Get(0) == nil {
return nil, args.Error(1)
}

return args.Get(0).(*BlockEpochEvent), args.Error(1)
}

// Start the ChainNotifier. Once started, the implementation should be ready,
// and able to receive notification registrations from clients.
func (m *MockChainNotifier) Start() error {
args := m.Called()

return args.Error(0)
}

// Started returns true if this instance has been started, and false otherwise.
func (m *MockChainNotifier) Started() bool {
args := m.Called()

return args.Bool(0)
}

// Stops the concrete ChainNotifier.
func (m *MockChainNotifier) Stop() error {
args := m.Called()

return args.Error(0)
}
Loading
Loading