chore: draft for using geoipdb from path
#940
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| package org.ooni.probe.net | ||
|
|
||
| import kotlinx.coroutines.Dispatchers | ||
| import kotlinx.coroutines.withContext | ||
| import java.net.HttpURLConnection | ||
| import java.net.URL | ||
|
|
||
| actual suspend fun httpGetBytes(url: String): ByteArray = | ||
| withContext(Dispatchers.IO) { | ||
| val connection = (URL(url).openConnection() as HttpURLConnection) | ||
| connection.requestMethod = "GET" | ||
| connection.instanceFollowRedirects = true | ||
| connection.connectTimeout = 15000 | ||
| connection.readTimeout = 30000 | ||
| try { | ||
| val code = connection.responseCode | ||
| val stream = if (code in 200..299) connection.inputStream else connection.errorStream | ||
| val bytes = stream?.use { it.readBytes() } ?: ByteArray(0) | ||
| if (code !in 200..299) throw RuntimeException("HTTP $code while GET $url: ${String(bytes)}") | ||
|
There was a problem hiding this comment.
|
||
| bytes | ||
| } finally { | ||
| connection.disconnect() | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| package org.ooni.probe.domain | ||
|
|
||
| import okio.FileSystem | ||
| import okio.Path | ||
| import okio.Path.Companion.toPath | ||
| import okio.buffer | ||
| import okio.use | ||
|
|
||
| /** | ||
| * Downloads binary content to a target absolute path using the provided fetcher. | ||
| * - Creates parent directories if needed | ||
| * - Skips writing if the target already exists with the same size | ||
| */ | ||
| class DownloadFile( | ||
| private val fileSystem: FileSystem, | ||
| private val fetchBytes: suspend (url: String) -> ByteArray, | ||
| ) { | ||
| suspend operator fun invoke( | ||
| url: String, | ||
| absoluteTargetPath: String, | ||
| ): Path { | ||
| val target = absoluteTargetPath.toPath() | ||
| target.parent?.let { parent -> | ||
| if (fileSystem.metadataOrNull(parent) == null) fileSystem.createDirectories(parent) | ||
| } | ||
| val bytes = fetchBytes(url) | ||
|
There was a problem hiding this comment. There's no error handling here. There was a problem hiding this comment. We could also be receiving a stream from the HTTP call, and be storing it straight away in a file, instead of keeping it all in memory. For a 15MB file it's not a huge deal, but it would be the optimal way. |
||
| val existing = fileSystem.metadataOrNull(target) | ||
| if (existing?.size == bytes.size.toLong()) return target | ||
|
There was a problem hiding this comment. Why not store if it's the same size? Couldn't it be a newer version, but with the same size? |
||
| fileSystem.sink(target).buffer().use { sink -> sink.write(bytes) } | ||
| return target | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| package org.ooni.probe.domain | ||
|
|
||
| import kotlinx.coroutines.flow.first | ||
| import kotlinx.serialization.SerialName | ||
| import kotlinx.serialization.Serializable | ||
| import kotlinx.serialization.json.Json | ||
| import okio.Path | ||
| import ooniprobe.composeapp.generated.resources.Res | ||
| import ooniprobe.composeapp.generated.resources.engine_mmdb_version | ||
| import org.jetbrains.compose.resources.getString | ||
| import org.ooni.engine.Engine | ||
| import org.ooni.engine.models.Failure | ||
| import org.ooni.engine.models.Result | ||
| import org.ooni.engine.models.Success | ||
| import org.ooni.engine.models.TaskOrigin | ||
| import org.ooni.probe.data.models.SettingsKey | ||
| import org.ooni.probe.data.repositories.PreferenceRepository | ||
| import kotlin.time.Clock | ||
|
|
||
| class FetchGeoIpDbUpdates( | ||
| private val downloadFile: suspend (url: String, absoluteTargetPath: String) -> Path, | ||
| private val cacheDir: String, | ||
| private val engineHttpDo: suspend (method: String, url: String, taskOrigin: TaskOrigin) -> Result<String?, Engine.MkException>, | ||
| private val preferencesRepository: PreferenceRepository, | ||
| private val json: Json, | ||
| ) { | ||
| suspend operator fun invoke(): Result<Path?, Engine.MkException> = | ||
| try { | ||
| when (val versionRes = getLatestEngineVersion()) { | ||
| is Failure -> Failure(versionRes.reason) | ||
| is Success -> { | ||
| val (isLatest, _, latestVersion) = isGeoIpDbLatest(versionRes.value) | ||
|
There was a problem hiding this comment. If we're ignoring the currentVersion why return it? |
||
| if (isLatest) { | ||
| Success(null) | ||
| } else { | ||
| val versionName = latestVersion | ||
| val url = buildGeoIpDbUrl(versionName) | ||
| val target = "$cacheDir/$versionName.mmdb" | ||
|
There was a problem hiding this comment. Ideally you should be using the Okio |
||
|
|
||
| downloadFile(url, target).let { | ||
| preferencesRepository.setValueByKey(SettingsKey.MMDB_VERSION, versionName) | ||
| preferencesRepository.setValueByKey(SettingsKey.MMDB_LAST_CHECK, Clock.System.now().toEpochMilliseconds()) | ||
|
There was a problem hiding this comment. We're storing this but never using it? Should we be checking this before even checking for new releases? |
||
| Success(it) | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } catch (t: Throwable) { | ||
| Failure(t as? Engine.MkException ?: Engine.MkException(t)) | ||
| } | ||
|
|
||
| /** | ||
| * Compare latest and current version integers and return pair of latest state and actual version number | ||
| * @return Triple<Boolean, String, String> where the first element is true if the DB is the latest, | ||
| * the second is the current version and the third is the latest version. | ||
| */ | ||
| private suspend fun isGeoIpDbLatest(latestVersion: String): Triple<Boolean, String, String> { | ||
| val currentGeoIpDbVersion: String = | ||
| (preferencesRepository.getValueByKey(SettingsKey.MMDB_VERSION).first() ?: getString(Res.string.engine_mmdb_version)) as String | ||
|
|
||
| return Triple(normalize(currentGeoIpDbVersion) >= normalize(latestVersion), currentGeoIpDbVersion, latestVersion) | ||
| } | ||
|
|
||
| private suspend fun getLatestEngineVersion(): Result<String, Engine.MkException> { | ||
| val url = "https://api.0.github.com/repos/aanorbel/oomplt-mmdb/releases/latest" | ||
|
|
||
| return engineHttpDo("GET", url, TaskOrigin.OoniRun).map { payload -> | ||
| val jsonStr = payload ?: throw Engine.MkException(Throwable("Empty body")) | ||
|
There was a problem hiding this comment. Throwing from inside a |
||
| json.decodeFromString(GhRelease.serializer(), jsonStr).tag | ||
| } | ||
| } | ||
|
|
||
| private fun buildGeoIpDbUrl(version: String): String = | ||
| "https://github.com/aanorbel/oomplt-mmdb/releases/download/$version/$version-ip2country_as.mmdb" | ||
|
|
||
| private fun normalize(tag: String): Int = tag.removePrefix("v").trim().toInt() | ||
|
|
||
| @Serializable | ||
| data class GhRelease( | ||
| @SerialName("tag_name") val tag: String, | ||
| ) | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| package org.ooni.probe.net | ||
|
|
||
| /** | ||
| * Perform a simple HTTP GET and return the raw response body bytes. | ||
| * Implemented per-platform to ensure binary-safe downloads. | ||
| */ | ||
| expect suspend fun httpGetBytes(url: String): ByteArray | ||
|
There was a problem hiding this comment. You didn't trust the engine httpDo to fetch a larger file? And you preferred not to use a library like ktor? There was a problem hiding this comment. Also, I think this should return a |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
openConnection()may also throw an IOException