today negative after import history

[decommentator]

Imported the history with removing all current data in HA. Direct after import everything looks ok.
After some time today is all negative as if HA thinks I'm delevering instead of using.
(Did the same yesterday with the same result)

Using domoticz 2023.2

[Johanbos]

It could be that the incorrect sensorid was used during import. In my HA there are different sensors for consumption and production of electricity.

Investigating on sql level would be the best option to find out which sensor is doing what job.

[decommentator]

the sensor is correct. Today everything is ok. History is ok. Except yesterday the time from import

same for gaz. Only wrong @importtime

[Johanbos]

Ah next hint: use the value divider/multiplier. You might need to go from Wh to KWh etc.

[decommentator]

it's strange as "today" is not in domoticz export
tried it with the different divider/multiplier options

[Erwin033]

At first, a big thank you for making this tool!
As decommentator, I have the same problem.

Would it be possible to delete al today's data?

[Johanbos]

It is very easy to skip/delete parts of the import sql script. Leave out the last insert into value line. It should be at the end of all the values, recognizable by the date.

Within HA, if you know Sql well enough, you can alter any weird value or delete record manually. Be careful ofcourse and make sure you have your backup.

[enduring78]

@Johanbos How would I exclude some period of data?

Like with many users, there is some overlap between my Home Assistant instance being created and my Domoticz instance shutting down.

Therefore, what I think is happening is when changing the date in the queries to update the data, the sum will not match so there will be a huge negative value the day after the period I imported the records of.

Unfortunately I know only very basic SQL. I tried experimenting with deleting the record of that bad day, but I can't get it to work

[Johanbos]

Ah, in that case.. Yes, I should add the option to set an end date. I understand now that the incorrect sum could cause these issues.

As soon as have added this, I'll update this issue.

[enduring78]

Thank you! I will wait importing my data then to avoid the head age of removing the huge incorrect value ;)

Another thing I was thinking about is adding some kind of helpful text around the sum setting. I have done a bit of googling to the sum value, and I still don't fully understand it.

[Johanbos]

Could you give https://github.com/Johanbos/home-assistant-import/releases/tag/v0.2.1-ci.140 a try?

It incudes the end date option and I also tried to improve the options page, including a better description about the 'sum' / relative existing data.

Any feedback to improve is appreciated!

[enduring78]

@Johanbos I tried the new version, but as soon as I try to load the Domoticz db, it crashes. This was working fine in the old version, so maybe a database parsing error?

I tried rebooting my machine and re-exporting the database, but it didn't work. I'm using Windows, it that matters.

[Johanbos]

Oops, I tested it with a pv export. I will test with domoticz soon.

[Johanbos]

Hmm, my Domoticz DB (from 2022, it's a bit old) is still working without any errors. Is there a error message you can share?

[enduring78]

I didn't get any error code, the program just quits.

I will try it again tonight on a clean VM with a new db export to rule out any other programs causing interference or something being wrong with the database.

[Johanbos]

I've created #174 for the application crash issues. Multiple people are experiencing them.

[Johanbos]

@enduring78 Please try https://github.com/Johanbos/home-assistant-import/releases/tag/v0.2.1-ci.151 with a specific end date.

[enduring78]

Seems to be working like expected!

Will do some more testing later tonight, but on a test copy of my HA install all the history shows up!
The history shows up on the graph between 04:00 and 05:00 PM instead of midnight (what I expected), but this isn't a problem since the day view isn't useful anyway since we can't import the data hour-by-hour.

One small thing that can be improved is that the 'end date' isn't really clear. By trial and error I found out that this is the day the script will not touch, but some users (like myself) might assume this is the last that that will be imported.

[Johanbos]

Updated end date parsing and description in https://github.com/Johanbos/home-assistant-import/releases/tag/v0.2.1-ci.159