Interested in working on the Kraken REST API

[ndopencode]

Hi I'm interested in working on the Kraken REST API but upon looking at the CCXT Based Data Loader Plugins the 5th bullet point regarding implementing the AbstractPaginationDetailSet functions is lacking clarity on how to start implementing it. How do I determine if an exchange supports these functions? Where should I look?

I was wondering, can someone explain what the function is doing and what I'm returning here? Is there already explanation somewhere in the docs that I should probably start looking at first? Sorry for the rather nascent question. Just trying to gather information.

[eprbell]

Thanks for volunteering to contribute code! @macanudo527 , this question is for you.

[macanudo527]

Thanks for volunteering! I designed the CCXT data loader, so I'll try to walk you through the process of building a plugin for it.

Basically, the AbstractPaginationDetailSet defines how an exchange handles pagination. There are 3 different ways exchanges do that. It looks like Kraken is date-based, which means you can use the Binance plugin as a template. Here is the relevant code:

    def _get_process_deposits_pagination_detail_set(self) -> Optional[AbstractPaginationDetailSet]:
        return DateBasedPaginationDetailSet(
            limit=_DEPOSIT_RECORD_LIMIT,
            exchange_start_time=self._start_time_ms,
            window=_NINETY_DAYS_IN_MS,
        )


    def _get_process_withdrawals_pagination_detail_set(self) -> Optional[AbstractPaginationDetailSet]:
        return DateBasedPaginationDetailSet(
            limit=_WITHDRAWAL_RECORD_LIMIT,
            exchange_start_time=self._start_time_ms,
            window=_NINETY_DAYS_IN_MS,
        )


    def _get_process_trades_pagination_detail_set(self) -> Optional[AbstractPaginationDetailSet]:
        return DateBasedPaginationDetailSet(
            limit=_TRADE_RECORD_LIMIT,
            exchange_start_time=self._start_time_ms,
            markets=self._get_markets(),
        )

The limit parameter is the number of records that can be retrieved in a single call. exchange_start_time is the date the exchange first came online. window is the largest window of time that can be pulled at one time in milliseconds. markets is the list of markets if the exchange pulls trades by market like Binance does. Most exchanges don't need that parameter though since you can usually pull all trades from all markets with one call.

You'll be one of the first people to build a CCXT based data plugin other than myself, so let me know what issues you have and I will try to be there for you.

Thanks again!

[ndopencode]

Thanks macanudo527! I feel like something isn't clicking for me. Here are few things I am trying to clear up in my head:

1. When I look at the Kraken API, it says that all calls will return a result of a maximum of 50. The window range I provide may have more than 50 items to return, so it'll take multiple calls within the window to get all records. I assume one way this is resolved is provide the start time for the subsequent call using the last record's timestamp.

2). What REST API is DateBasedPaginationDetailSet calling? I don't see a way in the code to pass in any one of the following to get the trade_history or the ledger for Kraken (i.e., private_post_tradeshistory(), private_post_ledgers())?

3). Another way Kraken allows pagination to work is to provide a ofs parameter. A count is provided in the results. Is there a pattern that CCXT follows where an exchange pagination is based on count and index?

4). What is the underlying data structure of AbstractPaginationDetailSet? Is it the full list of records from a specific API call after doing so or an intermediary object whose primary purpose is to provide state information to support pagination in API calls?

[macanudo527]

1. When I look at the Kraken API, it says that all calls will return a result of a maximum of 50. The window range I provide may have more than 50 items to return, so it'll take multiple calls within the window to get all records. I assume one way this is resolved is provide the start time for the subsequent call using the last record's timestamp.

Yeah, the AbstractCcxtInputPlugin will handle resizing the window if the limit is hit. Basically, it rolls through dates with the maximum window from the server start time. When the API returns 50 records, the limit, it will change the next window to be one millisecond after the last record. Here is the code.

2). What REST API is DateBasedPaginationDetailSet calling? I don't see a way in the code to pass in any one of the following to get the trade_history or the ledger for Kraken (i.e., private_post_tradeshistory(), private_post_ledgers())?

It might help if you familiarize yourself with CCXT. All of these exchange-specific calls are abstracted out to the unified API. If they aren't covered by the unified API, you can use the implicit api. Again, check the Binance plugin for an example with a full-featured exchange.

3). Another way Kraken allows pagination to work is to provide a ofs parameter. A count is provided in the results. Is there a pattern that CCXT follows where an exchange pagination is based on count and index?

This sounds like page-based pagination. That is not currently implemented. If you would like to make calls this way it would probably be faster. You would need to implement an iterator like what I made for the date-based pagination.

4). What is the underlying data structure of AbstractPaginationDetailSet? Is it the full list of records from a specific API call after doing so or an intermediary object whose primary purpose is to provide state information to support pagination in API calls?

It's the latter. AbstractPaginationDetailSet provides the information needed for each iteration when AbstractCcxtInputPlugin is making calls to the CCXT unified API.

[eprbell]

Also perhaps the pagination detail set documentation can be fleshed out a bit more to help onboard new coders more easily.

[ndopencode]

Hi guys, continuing development here. I had a question since I'm a little stumped. When I was looking at the margin faq on kraken https://support.kraken.com/hc/en-us/articles/360027712571-How-to-calculate-profits-losses-from-margin-trading-

It says that the profit/loss is already computed and provided in the ledger. From my understanding of rp2 is that gain/losses are calculated using matching with out transactions with its corresponding in transactions which implies asset in/outs are matched against each other. Margin is still a taxable event but it no longer applies to asset balance and rather its usually a quote balance change which if that is usually fiat. When I've tried out rp2, it doesn't seem to create a tab for fiat currencies like 'USD' in the output spreadsheets. First of all:

1). Why is fiat currency tabs not created in the output of rp2?
2). Since margin is no longer a spot transaction but a derivative, how to best represent this in dali/rp2? (Is there an OutTransaction analog that I am missing?)

[eprbell]

Good questions:

• margin: it's not supported yet. There was a discussion with a proposal, but it's not set in stone yet (Transaction Type - BORROW and REPAY LOAN #99). I am not a margin expert (I haven't used it personally), so this new functionality would have to be driven by the community and I'll be sure to review.
• missing fiat tab: fiat was never implemented fully, because I thought it didn't have impact on crypto taxes (obviously I hadn't thought about things like margin), so it never became a priority. Fiat is actually partially implemented, in the sense that some plugins capture some fiat transactions, but not all of them and so the DaLI input generator skips generation of the fiat tab (see https://github.com/eprbell/dali-rp2/blob/main/src/dali/ods_generator.py#L166), even though it has some (incomplete) fiat data.

I wonder, since margin is a fiat derivative if it wouldn't be best to leave it out of RP2. Here's the reasoning: the complexity of crypto taxes is that when you sell crypto, the exchange on which you sell doesn't necessarily know the cost basis of the sold coins (if they were bought on another exchange). So crypto exchanges can't typically provide a full 8849 form, like stock brokers can. So RP2 and DaLI help by gathering data from exchanges, normalizing it, resolving transactions, etc. In the case of margin on Kraken, isn't it true that Kraken has full details on what your margin-related gains are? If so, perhaps we can treat it as a separate issue: the user can just get the full documentation on their margin activity from Kraken and enter it into their tax forms. I guess, I'm not entirely sure margin is a crypto tax issue: it sounds more like a fiat tax issue. Anyway, not a strong opinion, just spitballing.

[ndopencode]

Yea from what I understand or rather what I am noticing among exchanges is that the gain/loss is denominated in a derivative that is in parity with a quote currency (usually 1 USD), so Crypto spot gain/losses are moot at this point. Additionally, the exchanges report the gain/loss for you so I don't think we need to do any cost basis analysis to calculate anything - its done for you. I would think that if you had a derivative long that resulted in +10 FIAT or a short that resulted in a + 10 or vicevesa (a loss), then you would just do simple addition/subtraction to get the final number and that is what rp2/dali should care about.

But I'm forgetting that once the gain/loss is computed, there is a change to your total crypto holdings if crypto is what funded your account - so there still needs to be a way to reflect that your total IN crypto balance has changed and then a cost basis for this needs to be understood when there is an OUT transaction. Sadly I'm confused, not sure what is the best way to handle this. Maybe a tax professional may help.

[macanudo527]

Yeah, first of all, a tax professional will of course be the final authority on this.
My guess though is that if you fund the trade with a crypto asset, you simply add one transaction. If it is a profitable trade, that is a capital gain. If it is a loss trade, that is a capital loss.

So, maybe all we need is a new InTransaction type for MARGIN_GAIN and an OutTransaction type for MARGIN_LOSS and implement a fiat tab. Then, add a tab to the US tax report in RP2 for margin trading. I've already implemented a fiat tab for JPY since I need to track payments for providing liquidity.

[ndopencode]

Very Neat. What about the one aspect where your total quantity of a crypto asset increases if you make capital gain and its settled in the crypto asset? Doesn't this count as an IN tx? Similarly, if you made a capital loss, its settled as a reduction of your total crypto asset - but since you claimed the tax loss as a MARGIN_LOSS a second OUT tx to account for the change in crypto balance is double dipping against you, so it should be a NO-OP IMO.

[eprbell]

CC: @topcoderasdf, who wrote up a proposal about margin and other things (see #99).

[ndopencode]

On the unique_id field topic and using the developer faq as reference.

1). What is the transaction hash?
1.a) Are we talking about the blockchain transaction hash? I would never have considered that but is that available to use?
1.b) If not that, then what is something that two exchanges might use that is global/unique to the transfer?

Thanks! The unique_id/transaction resolver has always been a mystery to me and somewhat I didn't think such a problem would be possible to solve but if this is indeed a problem that can be solved I'm on the edge to learn more. :)

[ndopencode]

Looking at the Kraken REST API I do not believe I will be able to get a global TX hash (assuming) and the wallet addresses that are provided may not be uniformly exposed information on another exchange/wallet. I think this is the inherent complexity of writing a tool like this.

One Idea I have is to come up with our own convention and compute our own unique_id something like:
hash(year-month-day_exchange-address_amount)

For example, let's say a trezor is to transmit asset A to an exchange B, an amount and exchange_address needs to be known. The day is also known. Even if there are multiple transfers with the same amount in the same day, it'll still resolve itself. Then at the receiving side, the same unique_id can be deduced because the receiving exchange knows its deposit address and the amount it received. It's not perfect but if its unique and deducible than I think it can act as a unique_id.

A few drawbacks:
The only issue I guess is if it happens on rollover in-between days or if a transaction takes more than 24 hours to be cleared on the other exchange and shows up in another day than the transmit day. Another is if network fees reduces the amount received so voiding the deducibility of the unique_id. Just brainstorming...

[eprbell]

Yes, it's the hash of the transaction that is publicly available on the blockchain: it's the only way to identify the two ends of a movement in a safe manner, because hashes are globally unique across exchanges. Other conventions for unique_id don't work because, as you pointed out they are not 100% safe and not globally unique. For more on unique_id and the transaction resolver read this FAQ.

Exchanges typically do expose transaction hashes in their REST responses: I took a quick look at the Kraken REST API and perhaps they call it refid, but you'll have to experiment to be sure. However if you're using CCXT, you won't use directly the Kraken API (except when you're using the implicit API to implement functionality that is not covered by CCXT, see @macanudo527's comment above about this). Additionally the CCXT superclass already factors a lot of the work for you, so you only have to implement what is missing for Kraken.

Hope this helps!

[ndopencode]

Yes Thank you. I've already took a look at the Kraken API, the FAQ you linked, and have been investigating the responses from Kraken API.
The refid is an internal identifier that is unique to Kraken. That would fulfill the condition and in regards to withdraw/deposit, I don't know which record type it is referring to.

Looking at the following APIs this is what I know:
Deposit Methods - this probably only provides data to supply to deposit addresses
Deposit Address - this supplies different deposit addresses that you use on another entity to supply to
Get Status of Recent Deposits - this is probably the closest one but unfortunately only provides last 90 days and doesn't seem to go further back in history

Withdraw is similar. None of the obvious ones (listed above) take in a refid as input field. Other API calls seem to focus on other types of info. So not sure what other one to try.

[eprbell]

@macanudo527, is there a standard way to retrieve the transaction hash via CCXT?

[macanudo527]

No, there isn't. I'm not sure why Kraken is so stingy about their API. Most exchanges provide this information. I think @ndopencode will have to manually link up deposits, unfortunately. Kind of disappointing of Kraken to be honest.

[eprbell]

Yes, major disappointment... I wonder how commercial software works around this.

[ndopencode]

Was thinking about this and realized that some commercial software out there requires some manual intervention. So probably not too much different from what we are dealing with.

[eprbell]

I looked at "Get Status of Recent Deposits": it's probably txid. I'm not sure about the 90-day limit though: does Kraken not provide full history of deposits? But again, if you're doing a subclass of CCXT (recommended), some of these problems are hopefully already resolved for you by the CCXT abstract superclass and by CCXT itself.

[ndopencode]

Yea I took a look. It doesn't seem available so I think I'm being forced to accept that the unique_id will be filled with the ledger ID which puts me on the manual post-processing path... :(

[ndopencode]

After taking a closer look of the AbstractCCXT super class and the CCXT implementation of the unified API, it calls the API "Get Status of Recent Deposits" ... which invariably is limited by the 90 day restriction so I think I'm SOL.

[macanudo527]

This seems to be a documented issue. Fetching of the ledger is supported by CCXT though for details follow the link, which also links to the CCXT manual for fetching the ledger.

You can override the superclass fetchDeposits() and have it pull from the ledger filtered for deposits. And do the same for withdrawals. You don't necessarily have to implement a pagination detail set.

[ndopencode]

I'm probably going to deviate from the Abstract class a bit here unfortunately, For example, when I get deposits from queryLedger(), it fills in the _type field with transaction instead of 'deposit'. I suspect I'll run into more subtleties like this.

[ndopencode]

@eprbell @macanudo527 I'm ready to open a pull request for the kraken plugin.

I did my best to incorporate the abstract super class but couldn't utilize everything. I did try to work on an Abstract Pagination object but had a few roadblocks so I kept with my original code. Maybe another week I can get on it but for now I just want to submit what I have which has the base functionality working. I'd like to submit and get feedback, I'm sure there are errors just like commercial software likely has errors so any feedback is appreciated. Look forward to making it better. Thanks!

Additionally, there are a few things I'm trying to understand better:

1. There is a concept with the double underscores "__" I don't understand. When I try to use self.__safe_api_call, I get the following error
   AttributeError: 'InputPlugin' object has no attribute '_InputPlugin__safe_api_call' when calling self.__safe_api_call
   I don't get why "__" makes the subclass unable to call the superclass's function. Unfortunately, I added a self._safe_api_call to the super class to get around this. I'm happy to fix this on my next commit, just need a little guidance.

2. I'm not sure what the added benefit is to the Threadpool. Wouldn't the Python GIL obsolete any type of threading? Are we using other cores with this code or is it the same python interpreter thread on the same core that is being utilized? I decided against incorporating into my code because I realized the way I implemented it was different from the Abstract class and I couldn't see a real usecase for my approach. If there is something I missed, please let me know and I'll try to work on it.

3. Finally, the last one doesn't have anything to do with my code but rather the documentation. Unfortunately, I feel little dumb here but could someone reiterate this bullet point from the developer's guide. I don't understand what is meant by "id-based hashing"

no id-based hashing: classes that are added to dictionaries and sets redefine __eq__(), __neq__() and __hash__();

Thank you guys, please let me know next steps.

[eprbell]

1. In Python prepending __ to a field makes it private, so you'll only be able to access it from within the class where it's defined. All class fields in RP2 and DaLI are private. To provide protected or public access, add read-only properties (more on this at https://github.com/eprbell/rp2/blob/main/README.dev.md#design-guidelines)
2. Multi-threading: data loaders are I/O bound so the GIL doesn't affect them so much. GIL is a concern only for CPU-bound threads.
3. hashing: Python objects come with an id (function id()), which is a unique number identifying the object. Using the object id to hash the object is inadvisable (it can lead to inefficient hashing), so all hashable classes in RP2 need to define those 3 functions in a way that leads to good hashing. Here's an example: https://github.com/eprbell/rp2/blob/main/src/rp2/computed_data.py#L92

As for having your code reviewed, open a PR and we'll take a look!

[ndopencode]

1. I thought in python "we are consenting adults", so language level enforcement of these qualifiers didn't exist and it was more of a hint to programmers rather than something the interpreter would formally pick up on. I guess maybe I missed something in the latest releases in Python.
   UPDATE: Ok nvm, I take back what I said. I looked up name mangling and I see how Python hides things with double underscores. It's not language enforced private but rather by convention. Clever.
2. Make sense
3. Ok I think it makes sense. I think what tripped me up was the part "classes that are added to dictionaries and sets", that seemed like a condition. Would it be more clear to say "Hashable classes must redefine __eq__(), __neq__(), __hash__(). For hash(), utilizing immutable (i.e., read-only) attributes is appropriate (e.g., example code)"? I think the part where if they are added to dictionaries/sets is assumed since that's where hash() will be used but doesn't need to be stated. Anyways, I'm still may not be understanding the intent of the statement.

I'll post a PR in a few hours.

[eprbell]

1. You're right: language-level enforcement of private doesn't exist. What the double underscore does is causing the field name to be mangled, so that if you really want to access it you can by following the mangling rules. However this is bad practice in production code and is used mostly in debugging and other special scenarios.
2. Your proposed doc line is an improvement over the existing text, so let's do it: feel free to fix it in a PR.

[ndopencode]

Thanks, I'll add it to the PR I'm trying to open today. I'm trying to push a branch to the repo but it says I don't have permission rights to push to this repo. How do I open a pull request?

1. Do I fork this repo, push my work onto my repo and then somehow am I able to point to your repo in a PR.
2. Require permission to push my work up and then create a pull request.
3. Or another set of steps I must have just glossed over :(

[macanudo527]

You will need to fork this project and make changes to a branch of that fork. Once you have done that you can logon to Github and there should be a notification at the top that asks if you want to propose a PR to the main repo that is protected.

[eprbell]

There are various tutorials online, here's one: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request