Skip to content
This repository has been archived by the owner on Dec 23, 2024. It is now read-only.

export fetched data into a csv file #777

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

export fetched data into a csv file #777

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5525938
export fetched data into a csv file
Vebryn Aug 23, 2017
a867c06
clean old exportToCSV method
Vebryn Sep 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/AbstractSampleProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,19 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import de.greenrobot.dao.AbstractDao;
import de.greenrobot.dao.Property;
import de.greenrobot.dao.query.QueryBuilder;
import de.greenrobot.dao.query.WhereCondition;

import nodomain.freeyourgadget.gadgetbridge.database.DBHelper;
import nodomain.freeyourgadget.gadgetbridge.entities.AbstractActivitySample;
import nodomain.freeyourgadget.gadgetbridge.entities.DaoSession;
Expand All @@ -40,6 +48,7 @@
* @param <T> the sample type
*/
public abstract class AbstractSampleProvider<T extends AbstractActivitySample> implements SampleProvider<T> {
private static final Logger LOG = LoggerFactory.getLogger(AbstractSampleProvider.class);
private static final WhereCondition[] NO_CONDITIONS = new WhereCondition[0];
private final DaoSession mSession;
private final GBDevice mDevice;
Expand Down Expand Up @@ -90,6 +99,29 @@ public void addGBActivitySamples(T[] activitySamples) {
getSampleDao().insertOrReplaceInTx(activitySamples);
}

@Override
public void exportToCSV(T[] activitySamples, File outFile) {
String separator = ",";

LOG.info("Exporting samples into csv file: " + outFile.getName());
try {
BufferedWriter bw = new BufferedWriter(new FileWriter(outFile));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use
try (BufferedWriter bw = ....) {
and remove the bw.close()
That way the writer will always be closed (also when an exception occurs).

Copy link
Contributor Author

@Vebryn Vebryn Sep 17, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know this try block syntax. So, it's a try-with-resources statement. It closes bw at end of execution (cf. https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html). I understand so why you remove bw.close().


for (T sample : activitySamples){
String line = sample.getTimestamp() + separator + sample.getDeviceId() + separator + sample.getUserId() + separator + sample.getRawIntensity() + separator + sample.getSteps() + separator + sample.getRawKind() + separator + sample.getHeartRate();

//LOG.debug("Adding line into buffer: " + line);
bw.write(line);
bw.newLine();
}

bw.flush();
bw.close();
} catch (IOException e) {
LOG.error(e.getMessage());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use
LOG.error("Explain what failed", e);
otherwise, the stacktrace of the exception will not be logged, making debugging hard.

}
}

@Nullable
@Override
public T getLatestActivitySample() {
Expand Down
7 changes: 7 additions & 0 deletions app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/SampleProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import android.support.annotation.NonNull;
import android.support.annotation.Nullable;

import java.io.File;
import java.util.List;

import nodomain.freeyourgadget.gadgetbridge.entities.AbstractActivitySample;
Expand Down Expand Up @@ -88,6 +89,12 @@ public interface SampleProvider<T extends AbstractActivitySample> {
*/
void addGBActivitySamples(T[] activitySamples);

/**
* Exports samples into a CSV file
* @param activitySamples the samples to export
*/
void exportToCSV(T[] activitySamples, File outFile);

/**
* Factory method to creates an empty sample of the correct type for this sample provider
* @return the newly created "empty" sample
Expand Down
5 changes: 5 additions & 0 deletions app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/UnknownDeviceCoordinator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import android.support.annotation.NonNull;
import android.support.annotation.Nullable;

import java.io.File;
import java.util.List;

import nodomain.freeyourgadget.gadgetbridge.GBException;
Expand Down Expand Up @@ -69,6 +70,10 @@ public List getSleepSamples(int timestamp_from, int timestamp_to) {
return null;
}

@Override
public void exportToCSV(AbstractActivitySample[] activitySamples, File outFile) {
}

@Override
public void addGBActivitySample(AbstractActivitySample activitySample) {
}
Expand Down
11 changes: 11 additions & 0 deletions ...reeyourgadget/gadgetbridge/service/devices/miband2/operations/FetchActivityOperation.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.IOException;
import java.text.DateFormat;
import java.util.ArrayList;
Expand Down Expand Up @@ -57,6 +58,8 @@
import nodomain.freeyourgadget.gadgetbridge.service.devices.miband2.AbstractMiBand2Operation;
import nodomain.freeyourgadget.gadgetbridge.util.ArrayUtils;
import nodomain.freeyourgadget.gadgetbridge.util.DateTimeUtils;
import nodomain.freeyourgadget.gadgetbridge.util.FileUtils;
import nodomain.freeyourgadget.gadgetbridge.util.CSVExport;
import nodomain.freeyourgadget.gadgetbridge.util.GB;

/**
Expand Down Expand Up @@ -207,6 +210,14 @@ private GregorianCalendar saveSamples() {

timestamp.add(Calendar.MINUTE, 1);
}

// export csv on fetching
File myPath = null;
myPath = FileUtils.getExternalFilesDir();
File myFile = new File(myPath, startTimestamp.getTimeInMillis() / 1000 + ".csv");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a gadgetbridgedb_ prefix (or something like that, maybe shorter)?

But really, this does not belong here. It should be implemented in one place for all devices, not in the Mi Band 2 FetchActivityOperation. And I guess we should also have an option to turn this on and off.

Copy link
Contributor Author

@Vebryn Vebryn Sep 17, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, does data from other devices differs ? Where would you place on/off feature ? I would probably add into Mi Band menu since it's the only device that's supporting it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data from other devices may differ, yes. So either we use normalized values (= the same data ranges for all devices) or device specific raw values, or even both.

But it would be good to have this for all devices, not just one.

Since some devices provide the data in a different way, where we have no access to all samples at once, we might call a method like samplesAdded(device, start, end) which would then check if CSV export is enabled, then fetch the samples from the given date range from the db (probably cached in the session already) and perform the export.

Alternatively we could integrate with the new DB export functionality and perform the CSV export repeatedly. We would just need to remember the start and end dates so that we only export the incremental changes.

CSVExport.exportToCSV(samples.toArray(new MiBandActivitySample[0]), myFile);

// store samples on fetching
sampleProvider.addGBActivitySamples(samples.toArray(new MiBandActivitySample[0]));

saveLastSyncTimestamp(timestamp);
Expand Down
51 changes: 51 additions & 0 deletions app/src/main/java/nodomain/freeyourgadget/gadgetbridge/util/CSVExport.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package nodomain.freeyourgadget.gadgetbridge.util;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;

import nodomain.freeyourgadget.gadgetbridge.devices.AbstractSampleProvider;
import nodomain.freeyourgadget.gadgetbridge.entities.AbstractActivitySample;

/**
* Created by Vebryn on 29/05/17.
*/

public class CSVExport {
private static final Logger LOG = LoggerFactory.getLogger(AbstractSampleProvider.class);

public static void exportToCSV(AbstractActivitySample[] activitySamples, File outFile) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's sufficient to use ActivitySample here instead of AbstractActivitySample.

String separator = ",";
BufferedWriter bw = null;

LOG.info("Exporting samples into csv file: " + outFile.getName());
try {
bw = new BufferedWriter(new FileWriter(outFile));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above with try (BufferedWriter ...) and the bw.close()

bw.write("TIMESTAMP" + separator + "DEVICE_ID" + separator + "USER_ID" + separator + "RAW_INTENSITY" + separator + "STEPS" + separator + "RAW_KIND" + separator + "HEART_RATE");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also export the normalized values?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normalized values are deduced from a simple matching rule. I think correlation have to be done after exportation.

bw.newLine();

for (AbstractActivitySample sample : activitySamples){
String line = sample.getTimestamp() + separator + sample.getDeviceId() + separator + sample.getUserId() + separator + sample.getRawIntensity() + separator + sample.getSteps() + separator + sample.getRawKind() + separator + sample.getHeartRate();

//LOG.debug("Adding line into buffer: " + line);
bw.write(line);
bw.newLine();
}
} catch (IOException e) {
LOG.error(e.getMessage());
} finally {
if (bw != null){
try {
bw.flush();
bw.close();
} catch (IOException e) {
LOG.error(e.getMessage());
}
}
}
}
}