From 60eb45b7238799ca504936aac5f854bdf4cb4cf7 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Sun, 21 Oct 2018 11:08:37 +0200 Subject: [PATCH 01/23] Prevent FinTS dumper from failing if transaction type unknown (#32) --- lib/dumper/fints.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/dumper/fints.rb b/lib/dumper/fints.rb index 8066c4b..5b99864 100644 --- a/lib/dumper/fints.rb +++ b/lib/dumper/fints.rb @@ -82,9 +82,10 @@ def transaction_type(transaction) # change the hash returned by the `import_id` which # could will result in duplicated entries. - str = parse_transaction_at(0, transaction).encode('iso-8859-1') - .force_encoding('utf-8') - return nil unless str + str = parse_transaction_at(0, transaction) + return nil unless type + + str = str.encode('iso-8859-1').force_encoding('utf-8') str[1..-1] end From dccc0726c44bd8806668ad6a0062cb6e2f3d2ca9 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Sun, 21 Oct 2018 11:13:43 +0200 Subject: [PATCH 02/23] Fix variable (#33) --- lib/dumper/fints.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/dumper/fints.rb b/lib/dumper/fints.rb index 5b99864..43b5395 100644 --- a/lib/dumper/fints.rb +++ b/lib/dumper/fints.rb @@ -83,7 +83,7 @@ def transaction_type(transaction) # could will result in duplicated entries. str = parse_transaction_at(0, transaction) - return nil unless type + return nil unless str str = str.encode('iso-8859-1').force_encoding('utf-8') str[1..-1] From 8f300b0046d8bf9de57c08502382c46e9e65a893 Mon Sep 17 00:00:00 2001 From: Peter Jeschke Date: Mon, 22 Oct 2018 20:23:26 +0200 Subject: [PATCH 03/23] Fix digit-only usernames and passwords (fixes #34) (#35) If the username or password only contain digits, the ruby_fints library chokes because the YAML-parser returns the username/password as integer, while ruby_fints expects strings. --- lib/dumper/fints.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/dumper/fints.rb b/lib/dumper/fints.rb index 43b5395..f02912c 100644 --- a/lib/dumper/fints.rb +++ b/lib/dumper/fints.rb @@ -7,8 +7,8 @@ class Fints < Dumper def initialize(params = {}) @ynab_id = params.fetch('ynab_id') - @username = params.fetch('username') - @password = params.fetch('password') + @username = params.fetch('username').to_s + @password = params.fetch('password').to_s @iban = params.fetch('iban') @endpoint = params.fetch('fints_endpoint') @blz = params.fetch('fints_blz') From 004ff246ebfc173a99b3d40c3aa6340b87210c14 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Mon, 22 Oct 2018 20:26:31 +0200 Subject: [PATCH 04/23] Add a thank you Thanks to @peterjeschke for fixing a bug that happened when the FinTS username was an integer --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index dd707be..9ab49a0 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ I'm not that into hardware. It would be super awesome if someone could help maki # Thanks +* [@peterjeschke](https://github.com/peterjeschke) for fixing a bug that happened when the FinTS username was an integer [PR #35](https://github.com/schurig/ynab-bank-importer/pull/35) * [@derintendant](https://github.com/derintendant) for spotting and fixing edge cases [PR #27](https://github.com/schurig/ynab-bank-importer/pull/27) (improves error messages) and [PR #28](https://github.com/schurig/ynab-bank-importer/pull/28) (truncates the payee field if it's too long) * [@manuelgrabowski](https://github.com/manuelgrabowski) for implementing a fallback in the FinTS dumper [PR #26](https://github.com/schurig/ynab-bank-importer/pull/26) * [@markuspabst](https://github.com/markuspabst) for spotting an error in the readme [PR #11](https://github.com/schurig/ynab-bank-importer/pull/11) From 06100b1d206a868750409a1a2c8795f0e07c0f05 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Mon, 22 Oct 2018 20:36:24 +0200 Subject: [PATCH 05/23] Reference notes below dumpers in wiki This commit cleans up the README.md file and references the dumper notes to understand the limitations of each dumper --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 9ab49a0..5244b14 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ The script also includes some additional logic like detecting internal transacti # Known Problems * Currently no known problems +* Please read the notes in each Dumper _[(see Wiki)](https://github.com/schurig/ynab-bank-importer/wiki#supported-dumpers)_ to understand the limitations ____________________ @@ -39,8 +40,6 @@ ____________________ Support and contriubution of any kind is always welcome!!! -I'm not that into hardware. It would be super awesome if someone could help making this work on Raspbian. I already tried but building the docker container fails _(Dockerfile.rpi)_. The PR related to that you can find here: [18](https://github.com/schurig/ynab-bank-importer/pull/18) - # Thanks * [@peterjeschke](https://github.com/peterjeschke) for fixing a bug that happened when the FinTS username was an integer [PR #35](https://github.com/schurig/ynab-bank-importer/pull/35) From 5665024689928124118662b08f9f6270ea487404 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Wed, 2 Jan 2019 10:53:24 +0100 Subject: [PATCH 06/23] Introduce config flag for N26 to only import processed transactions (#38) Because the values that we get from the (not official) N26 API change from time to time, people often end up with duplicated transactions. This code change introduced a new config flag for the N26 Dumper called skip_pending_transactions. --- config.sample.yml | 1 + lib/dumper/n26.rb | 25 ++++++++------------ spec/dumper/n26_spec.rb | 52 ++++++++++++++++++++++++++++++++--------- 3 files changed, 52 insertions(+), 26 deletions(-) diff --git a/config.sample.yml b/config.sample.yml index 842f1ea..8407256 100644 --- a/config.sample.yml +++ b/config.sample.yml @@ -17,4 +17,5 @@ accounts: ynab_id: # last hash in the url when you click on the account in YNAB username: # n26 username password: # n26 password + skip_pending_transactions: false # default: false, only imports transactions when they're processed set_category: false # default: false, sets the N26 category name as category diff --git a/lib/dumper/n26.rb b/lib/dumper/n26.rb index 66e36e3..959ddc9 100644 --- a/lib/dumper/n26.rb +++ b/lib/dumper/n26.rb @@ -17,6 +17,8 @@ def initialize(params = {}) @password = params.fetch('password') @iban = params.fetch('iban') @set_category = params.fetch('set_category', false) + @skip_pending_transactions = params.fetch('skip_pending_transactions', + false) @categories = {} end @@ -28,10 +30,15 @@ def fetch_transactions end client.transactions(count: 100) - .reject { |t| t['pending'] } # Only transactions that aren't pending + .select { |t| accept?(t) } .map { |t| to_ynab_transaction(t) } end + def accept?(transaction) + return true unless @skip_pending_transactions + already_processed?(transaction) + end + private def check_authorization!(client) @@ -81,7 +88,7 @@ def withdrawal?(transaction) end def import_id(transaction) - data = [calculated_timestamp(transaction), + data = [transaction['visibleTS'], transaction['transactionNature'], transaction['amount'], transaction['accountId']].join @@ -89,22 +96,10 @@ def import_id(transaction) Digest::MD5.hexdigest(data) end - # N26 seems to have an internal timezone mismatch in their database. - # Transactions that are not processed yet have the `visibleTS` value - # in UTC but processed transactions have timezone Europe/Berlin. - # => This method checks if the transaction was processed or not. - # If it's already processed it will just take the value, if not it will - # add the current offset to make it Europe/Berlin timezone. - def calculated_timestamp(transaction) - return transaction['visibleTS'] if alread_processed?(transaction) - offset_to_utc = Time.now.in_time_zone('Europe/Berlin').utc_offset - transaction['visibleTS'] + offset_to_utc * 1000 - end - # All very recent transactions with the credit card have # the type value set to "AA". So we assume that this is an # indicator to check if a transaction has been processed or not. - def alread_processed?(transaction) + def already_processed?(transaction) transaction['type'] != 'AA' end end diff --git a/spec/dumper/n26_spec.rb b/spec/dumper/n26_spec.rb index 84c47e6..9362180 100644 --- a/spec/dumper/n26_spec.rb +++ b/spec/dumper/n26_spec.rb @@ -4,13 +4,15 @@ RSpec.describe Dumper::N26, vcr: vcr_options do subject(:object) { Dumper::N26.new(params) } + let(:skip_pending_transactions) { false } let(:params) do { 'ynab_id' => '123466', 'username' => 'username', 'password' => 'password', - 'iban' => 'DE89370400440532013000' + 'iban' => 'DE89370400440532013000', + 'skip_pending_transactions' => skip_pending_transactions } end @@ -159,19 +161,47 @@ it 'sets it correctly' do expect(method).to eq('46c9ccde424652bc013dca9b408dcdec') end + end + + describe '.accept?' do + subject(:accept?) { object.accept?(transaction) } - it 'is the same for a pending transaction and a processed transaction' do - expect( - object.send(:import_id, transaction_pending) - ).to eq(object.send(:import_id, transaction_processed)) + context 'when skip_pending_transactions feature is disabled' do + context 'when the transaction is pending' do + let(:transaction) { transaction_pending } + + it 'returns true' do + expect(accept?).to be_truthy + end + end + + context 'when the transaction is processed' do + let(:transaction) { transaction_processed } + + it 'returns true' do + expect(accept?).to be_truthy + end + end end - end - describe '#calculated_timestamp' do - it 'is the same for a pending transaction and a processed transaction' do - expect( - object.send(:calculated_timestamp, transaction_pending) - ).to eq(object.send(:calculated_timestamp, transaction_processed)) + context 'when skip_pending_transactions feature is disabled' do + let(:skip_pending_transactions) { true } + + context 'when the transaction is pending' do + let(:transaction) { transaction_pending } + + it 'returns false' do + expect(accept?).to be_falsy + end + end + + context 'when the transaction is processed' do + let(:transaction) { transaction_processed } + + it 'returns true' do + expect(accept?).to be_truthy + end + end end end end From dcfc54011fa62dd2add8b4c23aac3637f2a11705 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Wed, 2 Jan 2019 11:07:13 +0100 Subject: [PATCH 07/23] Add thanks for new N26 dumper config flag --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5244b14..25ecc42 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,7 @@ Support and contriubution of any kind is always welcome!!! # Thanks +* [@manuelgrabowski](https://github.com/manuelgrabowski), [@martinlabuschin](https://github.com/martinlabuschin) and [@peterjeschke](https://github.com/peterjeschke) for giving feedback on a new N26 dumper config flag that prevents transactions to be imported multiple times [PR #38](https://github.com/schurig/ynab-bank-importer/pull/38) * [@peterjeschke](https://github.com/peterjeschke) for fixing a bug that happened when the FinTS username was an integer [PR #35](https://github.com/schurig/ynab-bank-importer/pull/35) * [@derintendant](https://github.com/derintendant) for spotting and fixing edge cases [PR #27](https://github.com/schurig/ynab-bank-importer/pull/27) (improves error messages) and [PR #28](https://github.com/schurig/ynab-bank-importer/pull/28) (truncates the payee field if it's too long) * [@manuelgrabowski](https://github.com/manuelgrabowski) for implementing a fallback in the FinTS dumper [PR #26](https://github.com/schurig/ynab-bank-importer/pull/26) From 7badde269602db62c4684fa9a4b58d52e9e059c2 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Wed, 30 Jan 2019 23:22:51 +0100 Subject: [PATCH 08/23] Only enforce utf-8 encoding (#46) --- .travis.yml | 3 ++- Dockerfile | 2 +- lib/dumper/fints.rb | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.travis.yml b/.travis.yml index 055a278..f5a91a8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,7 +2,8 @@ language: ruby cache: bundler rvm: - 2.4.2 -- 2.5.0 +- 2.5.3 +- 2.6.1 env: - ENV=test before_script: diff --git a/Dockerfile b/Dockerfile index 29ab84e..68daa53 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM ruby:2.5.0 +FROM ruby:2.6.1 # Set the locale RUN apt-get clean && apt-get update && apt-get install -y locales diff --git a/lib/dumper/fints.rb b/lib/dumper/fints.rb index f02912c..cf0b7a0 100644 --- a/lib/dumper/fints.rb +++ b/lib/dumper/fints.rb @@ -85,7 +85,7 @@ def transaction_type(transaction) str = parse_transaction_at(0, transaction) return nil unless str - str = str.encode('iso-8859-1').force_encoding('utf-8') + str = str.force_encoding('utf-8') str[1..-1] end From 8929839debbb06e5f4f9935dfd40f5d87e2397f5 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Wed, 30 Jan 2019 23:24:47 +0100 Subject: [PATCH 09/23] Ignore .git directory in docker file --- .dockerignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.dockerignore b/.dockerignore index 2c23a39..744b479 100644 --- a/.dockerignore +++ b/.dockerignore @@ -5,6 +5,7 @@ docker-compose.yml config.yml config.sample.yml +.git .cache .export last_imported From 6489a424961111c054da2f0072544b0e4c2c37e8 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Wed, 30 Jan 2019 23:52:01 +0100 Subject: [PATCH 10/23] Fix ruby version in Dockerfile --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 68daa53..99c2ec4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM ruby:2.6.1 +FROM ruby:2.6 # Set the locale RUN apt-get clean && apt-get update && apt-get install -y locales From 3ff7ac0ae58b084065784d4fafb14c155cd71b10 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Thu, 31 Jan 2019 18:59:54 +0100 Subject: [PATCH 11/23] Update README.md --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 25ecc42..1d0d640 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,12 @@ This is a ruby script that **pulls your transactions from your banks** and impor --- +## Disclaimer + +**Using this tool is on your own risk.** I can **not** garantee you that this will work for you. Also I can **not** garantee that it won't up your YNAB data. I highly recommend to start with a new Budget or at least test it with a new one before you use your existing Budget. + +--- + ## Supported banks * Most German and Austrian banks _(all banks that implement the FinTS standard)_ From 5c5d6c93b005f1d06ae381e0454f07a90fda3bde Mon Sep 17 00:00:00 2001 From: mathijshoogland <7372934+mathijshoogland@users.noreply.github.com> Date: Fri, 8 Feb 2019 10:55:54 +0100 Subject: [PATCH 12/23] Updated packages, fixes #48. (#49) --- Gemfile.lock | 90 +++++++++++++++++++++++++++------------------------- 1 file changed, 47 insertions(+), 43 deletions(-) diff --git a/Gemfile.lock b/Gemfile.lock index d0d99ee..310bfd4 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,16 +1,17 @@ GEM remote: https://rubygems.org/ specs: - activesupport (5.1.5) + activesupport (5.2.2) concurrent-ruby (~> 1.0, >= 1.0.2) - i18n (~> 0.7) + i18n (>= 0.7, < 2) minitest (~> 5.1) tzinfo (~> 1.1) - addressable (2.5.2) + addressable (2.6.0) public_suffix (>= 2.0.2, < 4.0) ast (2.4.0) - bankscrap (2.0.6) + bankscrap (2.1.1) activesupport + json mechanize money thor @@ -18,69 +19,72 @@ GEM bankscrap-bbva (2.0.3) bankscrap (>= 2.0.0) base32 (0.3.2) - cmxl (0.2.1) + cmxl (0.2.2) rchardet19 colorize (0.8.1) - concurrent-ruby (1.0.5) + concurrent-ruby (1.1.4) + connection_pool (2.2.2) crack (0.4.3) safe_yaml (~> 1.0.0) deep_merge (1.2.1) diff-lcs (1.3) - domain_name (0.5.20170404) + domain_name (0.5.20180417) unf (>= 0.0.5, < 1.0.0) - ethon (0.11.0) + ethon (0.12.0) ffi (>= 1.3.0) - ffi (1.9.25) - hashdiff (0.3.7) + ffi (1.10.0) + hashdiff (0.3.8) http-cookie (1.0.3) domain_name (~> 0.5) - httparty (0.16.0) + httparty (0.16.3) + mime-types (~> 3.0) multi_xml (>= 0.5.2) - i18n (0.9.5) + i18n (1.5.3) concurrent-ruby (~> 1.0) json (2.1.0) - mechanize (2.7.5) + mechanize (2.7.6) domain_name (~> 0.5, >= 0.5.1) http-cookie (~> 1.0) mime-types (>= 1.17.2) net-http-digest_auth (~> 1.1, >= 1.1.1) - net-http-persistent (~> 2.5, >= 2.5.2) + net-http-persistent (>= 2.5.2) nokogiri (~> 1.6) ntlm-http (~> 0.1, >= 0.1.1) webrobots (>= 0.0.9, < 0.2) - mime-types (3.1) + mime-types (3.2.2) mime-types-data (~> 3.2015) - mime-types-data (3.2016.0521) - mini_portile2 (2.3.0) + mime-types-data (3.2018.0812) + mini_portile2 (2.4.0) minitest (5.11.3) - money (6.10.1) - i18n (>= 0.6.4, < 1.0) + money (6.13.2) + i18n (>= 0.6.4, <= 2) multi_xml (0.6.0) net-http-digest_auth (1.4.1) - net-http-persistent (2.9.4) - nokogiri (1.8.2) - mini_portile2 (~> 2.3.0) + net-http-persistent (3.0.0) + connection_pool (~> 2.2) + nokogiri (1.10.1) + mini_portile2 (~> 2.4.0) ntlm-http (0.1.1) - parallel (1.12.1) - parser (2.5.0.2) + parallel (1.13.0) + parser (2.6.0.0) ast (~> 2.4.0) - powerpack (0.1.1) - public_suffix (3.0.2) + powerpack (0.1.2) + public_suffix (3.0.3) rainbow (3.0.0) rchardet19 (1.3.7) - rspec (3.7.0) - rspec-core (~> 3.7.0) - rspec-expectations (~> 3.7.0) - rspec-mocks (~> 3.7.0) - rspec-core (3.7.1) - rspec-support (~> 3.7.0) - rspec-expectations (3.7.0) + rspec (3.8.0) + rspec-core (~> 3.8.0) + rspec-expectations (~> 3.8.0) + rspec-mocks (~> 3.8.0) + rspec-core (3.8.0) + rspec-support (~> 3.8.0) + rspec-expectations (3.8.2) diff-lcs (>= 1.2.0, < 2.0) - rspec-support (~> 3.7.0) - rspec-mocks (3.7.0) + rspec-support (~> 3.8.0) + rspec-mocks (3.8.0) diff-lcs (>= 1.2.0, < 2.0) - rspec-support (~> 3.7.0) - rspec-support (3.7.1) + rspec-support (~> 3.8.0) + rspec-support (3.8.0) rubocop (0.52.1) parallel (~> 1.10) parser (>= 2.4.0.2, < 3.0) @@ -90,26 +94,26 @@ GEM unicode-display_width (~> 1.0, >= 1.0.1) rubocop-rspec (1.23.0) rubocop (>= 0.52.1) - ruby-progressbar (1.9.0) + ruby-progressbar (1.10.0) ruby_fints (0.0.3) cmxl (~> 0.2) httparty (~> 0.10) safe_yaml (1.0.4) - thor (0.20.0) + thor (0.20.3) thread_safe (0.3.6) twentysix (0.1.1) deep_merge httparty - typhoeus (1.3.0) + typhoeus (1.3.1) ethon (>= 0.9.0) tzinfo (1.2.5) thread_safe (~> 0.1) unf (0.1.4) unf_ext unf_ext (0.0.7.5) - unicode-display_width (1.3.0) + unicode-display_width (1.4.1) vcr (4.0.0) - webmock (3.3.0) + webmock (3.5.1) addressable (>= 2.3.6) crack (>= 0.3.2) hashdiff @@ -138,4 +142,4 @@ DEPENDENCIES ynab (= 1.5.0) BUNDLED WITH - 1.16.2 + 1.17.3 From 595ab5b68fed3a55da2daaa4424cb1c7df8f3cec Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Fri, 8 Feb 2019 10:58:10 +0100 Subject: [PATCH 13/23] Add cheers --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 1d0d640..5e96322 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ Support and contriubution of any kind is always welcome!!! # Thanks +* [@mathijshoogland](https://github.com/mathijshoogland) for updating the dependencies [PR #49](https://github.com/schurig/ynab-bank-importer/pull/49) * [@manuelgrabowski](https://github.com/manuelgrabowski), [@martinlabuschin](https://github.com/martinlabuschin) and [@peterjeschke](https://github.com/peterjeschke) for giving feedback on a new N26 dumper config flag that prevents transactions to be imported multiple times [PR #38](https://github.com/schurig/ynab-bank-importer/pull/38) * [@peterjeschke](https://github.com/peterjeschke) for fixing a bug that happened when the FinTS username was an integer [PR #35](https://github.com/schurig/ynab-bank-importer/pull/35) * [@derintendant](https://github.com/derintendant) for spotting and fixing edge cases [PR #27](https://github.com/schurig/ynab-bank-importer/pull/27) (improves error messages) and [PR #28](https://github.com/schurig/ynab-bank-importer/pull/28) (truncates the payee field if it's too long) From 3739b4b56c25d255f44431ea9436cfba9922abb3 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Mon, 25 Feb 2019 18:37:54 +0100 Subject: [PATCH 14/23] Improve ATM detection for BBVA (#51) --- lib/dumper/bbva.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/dumper/bbva.rb b/lib/dumper/bbva.rb index 96c0d28..b9229c3 100644 --- a/lib/dumper/bbva.rb +++ b/lib/dumper/bbva.rb @@ -50,7 +50,9 @@ def amount(transaction) def withdrawal?(transaction) text = transaction.description.downcase - text.include?('cajero') || text.include?('withdrawal') + text.include?('cajero') || + text.include?('withdrawal') || + text.include?('efectivo') end def normalize_iban(iban) From 47bc36f4d472ba1bd15965312744a49737603bb9 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Fri, 8 Mar 2019 20:59:33 +0100 Subject: [PATCH 15/23] Improve fints dumper (#53) --- lib/dumper/fints.rb | 52 +++++++-------------------------------------- 1 file changed, 8 insertions(+), 44 deletions(-) diff --git a/lib/dumper/fints.rb b/lib/dumper/fints.rb index cf0b7a0..c7fe1a3 100644 --- a/lib/dumper/fints.rb +++ b/lib/dumper/fints.rb @@ -35,26 +35,22 @@ def date(transaction) end def payee_name(transaction) - parse_transaction_at(32, transaction).try(:strip) + transaction.name.try(:strip) end def payee_iban(transaction) - parse_transaction_at(31, transaction) + transaction.iban end def memo(transaction) - parse_transaction_at(20, transaction).try(:strip) + [ + transaction.description, + transaction.information + ].compact.join(' / ').try(:strip) end def amount(transaction) - amount = - if transaction.funds_code == 'D' - "-#{transaction.amount}" - else - transaction.amount - end - - (amount.to_f * 1000).to_i + (transaction.amount * transaction.sign * 1000).to_i end def withdrawal?(transaction) @@ -65,39 +61,7 @@ def withdrawal?(transaction) end def import_id(transaction) - data = [transaction_type(transaction), - transaction.date, - transaction.amount, - transaction.funds_code, - transaction.reference.try(:downcase), - payee_iban(transaction), - payee_name(transaction).try(:downcase), - @iban].join - - Digest::MD5.hexdigest(data) - end - - def transaction_type(transaction) - # Changing the result of this method will - # change the hash returned by the `import_id` which - # could will result in duplicated entries. - - str = parse_transaction_at(0, transaction) - return nil unless str - - str = str.force_encoding('utf-8') - str[1..-1] - end - - def parse_transaction_at(position, transaction) - # I don't know who invented this structure but I hope - # the responsible people know how inconvenient it is. - - seperator = transaction.details.seperator - array = transaction.details.source.split("#{seperator}#{position}") - return nil if array.size < 2 - - array.last.split(seperator).first + Digest::MD5.hexdigest(transaction.source) end end end From b7273b444eb2e1a15b552d6a839bd95410059566 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Sun, 24 Mar 2019 12:04:00 +0100 Subject: [PATCH 16/23] Fallback to Feb 28th if date is 29th in a non-leap-year (#56) --- lib/dumper/fints.rb | 38 +++++++++++++++++++++++++++++++++++++- spec/spec_helper.rb | 1 + 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/lib/dumper/fints.rb b/lib/dumper/fints.rb index c7fe1a3..910cb6a 100644 --- a/lib/dumper/fints.rb +++ b/lib/dumper/fints.rb @@ -31,7 +31,7 @@ def account_id end def date(transaction) - transaction.entry_date || transaction.date + entry_date(transaction) || to_date(transaction['date']) end def payee_name(transaction) @@ -63,5 +63,41 @@ def withdrawal?(transaction) def import_id(transaction) Digest::MD5.hexdigest(transaction.source) end + + # Patches + + # taken from https://github.com/railslove/cmxl/blob/master/lib/cmxl/field.rb + # and modified so that it changed Feb 29th to Feb 28th on non-leap-years. + # See issue: https://github.com/schurig/ynab-bank-importer/issues/52 + DATE = /(?\d{0,2})(?\d{2})(?\d{2})/ + def to_date(date, year = nil) + if match = date.to_s.match(DATE) + year ||= "20#{match['year'] || Date.today.strftime('%y')}" + month = match['month'] + day = match['day'] + + day = '28' if !Date.leap?(year.to_i) && match['month'] == '02' && match['day'] == '29' + + Date.new(year.to_i, month.to_i, day.to_i) + else + date + end + rescue ArgumentError # let's simply ignore invalid dates + date + end + + def entry_date(transaction) + data = transaction.data + date = to_date(data['date']) + + return unless transaction.data['entry_date'] && date + + entry_date_with_date_year = to_date(data['entry_date'], date.year) + if date.month == 1 && date.month < entry_date_with_date_year.month + to_date(data['entry_date'], date.year - 1) + else + to_date(data['entry_date'], date.year) + end + end end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index e371173..5e1ecfc 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -9,6 +9,7 @@ config.cassette_library_dir = 'spec/fixtures/vcr_cassettes' config.hook_into :webmock config.configure_rspec_metadata! + config.default_cassette_options = { record: :none } end # This file was generated by the `rspec --init` command. Conventionally, all From 55909d9cc0e87ed6a2b72d41c5d8a962cdc7f01f Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Sun, 24 Mar 2019 12:53:39 +0100 Subject: [PATCH 17/23] Fix not existing date errors (#57) Fixes #52 --- lib/dumper/fints.rb | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/lib/dumper/fints.rb b/lib/dumper/fints.rb index 910cb6a..ee17b37 100644 --- a/lib/dumper/fints.rb +++ b/lib/dumper/fints.rb @@ -67,7 +67,8 @@ def import_id(transaction) # Patches # taken from https://github.com/railslove/cmxl/blob/master/lib/cmxl/field.rb - # and modified so that it changed Feb 29th to Feb 28th on non-leap-years. + # and modified so that it takes the last day of the month if the provided day + # doesn't exist in that month. # See issue: https://github.com/schurig/ynab-bank-importer/issues/52 DATE = /(?\d{0,2})(?\d{2})(?\d{2})/ def to_date(date, year = nil) @@ -76,14 +77,15 @@ def to_date(date, year = nil) month = match['month'] day = match['day'] - day = '28' if !Date.leap?(year.to_i) && match['month'] == '02' && match['day'] == '29' - - Date.new(year.to_i, month.to_i, day.to_i) + begin + Date.new(year.to_i, month.to_i, day.to_i) + rescue ArgumentError + # Take the last day of that month + Date.civil(year.to_i, month.to_i, -1) + end else date end - rescue ArgumentError # let's simply ignore invalid dates - date end def entry_date(transaction) From 3b61facab2f6882e8c7ef00b0d6b5ffd6b9b5194 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Sun, 24 Mar 2019 12:58:28 +0100 Subject: [PATCH 18/23] Add credits --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 5e96322..957510e 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ Support and contriubution of any kind is always welcome!!! # Thanks +* [@moay](https://github.com/moay), [@yuvke](https://github.com/yuvke), [@BluetriX](https://github.com/BluetriX) for the help on debugging an error with not existing dates [Issue #52](https://github.com/schurig/ynab-bank-importer/issues/52) * [@mathijshoogland](https://github.com/mathijshoogland) for updating the dependencies [PR #49](https://github.com/schurig/ynab-bank-importer/pull/49) * [@manuelgrabowski](https://github.com/manuelgrabowski), [@martinlabuschin](https://github.com/martinlabuschin) and [@peterjeschke](https://github.com/peterjeschke) for giving feedback on a new N26 dumper config flag that prevents transactions to be imported multiple times [PR #38](https://github.com/schurig/ynab-bank-importer/pull/38) * [@peterjeschke](https://github.com/peterjeschke) for fixing a bug that happened when the FinTS username was an integer [PR #35](https://github.com/schurig/ynab-bank-importer/pull/35) From 04bb221055035c4add32ab694b8373a315d55476 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Mon, 1 Apr 2019 12:05:10 +0200 Subject: [PATCH 19/23] Use Feb 29/30 catch only as fallback but keep old behavior (#60) --- lib/dumper/fints.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/dumper/fints.rb b/lib/dumper/fints.rb index ee17b37..c5fce5d 100644 --- a/lib/dumper/fints.rb +++ b/lib/dumper/fints.rb @@ -31,6 +31,10 @@ def account_id end def date(transaction) + transaction.entry_date || transaction.date + rescue NoMethodError + # https://github.com/schurig/ynab-bank-importer/issues/52 + # Some banks think Feb 29 and 30 exist in non-leap years. entry_date(transaction) || to_date(transaction['date']) end From 8569abd0c8719618c1a52d647b34865dc18f4046 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Wed, 3 Apr 2019 11:34:55 +0200 Subject: [PATCH 20/23] Don't import transactions that have a nil date value (#61) --- lib/dumper.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/dumper.rb b/lib/dumper.rb index 210f363..9083022 100644 --- a/lib/dumper.rb +++ b/lib/dumper.rb @@ -16,7 +16,7 @@ def self.get_dumper(name) # rubocop:disable Metrics/MethodLength def to_ynab_transaction(transaction) - return nil if date(transaction) > Date.today + return nil if date(transaction).nil? || date(transaction) > Date.today ::TransactionCreator.call( account_id: account_id, date: date(transaction), From 2a0ca6cb58f63ab8b446e3e3782bac4a54fae3c5 Mon Sep 17 00:00:00 2001 From: dx Date: Tue, 11 Jun 2019 22:27:19 +0200 Subject: [PATCH 21/23] Move away from deprecated BulkTransactions API (#70) The old API still works and might be supported, but it's in the deprecated section of the docts and but POST to transactions should be more "future-proof" according to the 2018-10-04 release notes: https://www.youneedabudget.com/release-notes/#release_81452 --- run.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/run.rb b/run.rb index 254a3c2..0ee27a9 100644 --- a/run.rb +++ b/run.rb @@ -17,10 +17,10 @@ access_token = Settings.all['ynab'].fetch('access_token') ynab_api = YNAB::API.new(access_token) -bulk_transactions = YNAB::BulkTransactions.new(transactions: transactions) begin - ynab_api.transactions.bulk_create_transactions(budget_id, bulk_transactions) + ynab_api.transactions.create_transaction(budget_id, + transactions: transactions) rescue YNAB::ApiError => e ErrorMessage.new(e).print abort From f27fa069676d48ce1cdba0d34b1710fb6dedf08f Mon Sep 17 00:00:00 2001 From: dx Date: Tue, 11 Jun 2019 22:28:17 +0200 Subject: [PATCH 22/23] docker-compose.dev: bindmount the whole source tree (#69) So rebuilding the image isn't needed for development --- docker-compose.dev.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 792b6d5..f774c23 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -4,5 +4,5 @@ services: build: context: . volumes: - - ./config.yml:/usr/app/config.yml + - ./:/usr/app/ command: "ruby /usr/app/run.rb" From 8d0b47ec2b543cdb653a517aa3d1d00c4632d185 Mon Sep 17 00:00:00 2001 From: Martin Schurig Date: Tue, 11 Jun 2019 22:43:05 +0200 Subject: [PATCH 23/23] Add credits --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 957510e..0595e66 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ Support and contriubution of any kind is always welcome!!! # Thanks +* [@dequis](https://github.com/dequis) for preventing the script from failing in the future [PR #70](https://github.com/schurig/ynab-bank-importer/pull/70) and keeping an eye on the development experience [PR #69](https://github.com/schurig/ynab-bank-importer/pull/69) * [@moay](https://github.com/moay), [@yuvke](https://github.com/yuvke), [@BluetriX](https://github.com/BluetriX) for the help on debugging an error with not existing dates [Issue #52](https://github.com/schurig/ynab-bank-importer/issues/52) * [@mathijshoogland](https://github.com/mathijshoogland) for updating the dependencies [PR #49](https://github.com/schurig/ynab-bank-importer/pull/49) * [@manuelgrabowski](https://github.com/manuelgrabowski), [@martinlabuschin](https://github.com/martinlabuschin) and [@peterjeschke](https://github.com/peterjeschke) for giving feedback on a new N26 dumper config flag that prevents transactions to be imported multiple times [PR #38](https://github.com/schurig/ynab-bank-importer/pull/38)