support datasource for github archive

[sankred9527]

run test:

cargo test -p fluxus-sources --features enable-gharchive test_local_source -- --nocapture
cargo test -p fluxus-sources --features enable-gharchive test_http_source -- --nocapture

[lispking]

You can consider adding an examples section to show how the entire workflow is used.

[lispking]

It's not recommended to use the unwrap method. If you use it, it'll be difficult to find the root cause when something goes wrong.

[lispking]

If this part supports both URLs and HTTP requests simultaneously, the functionality should be more streamlined.
For example, setting HTTP as the default option, and then providing a from_file method to handle local files.

[sankred9527]

I wanna support URI（Uniform Resource Identifier）, http res and local file are both URI .
so I use one new() method .

[lispking]

However, this will make the code complexity too high.

Alternatively, a dedicated resource processing module could be provided.

In that case, the new() method would seem easier to understand.

With the current implementation, it will be very difficult for oneself or others to understand the code after several months.

It is highly recommended to keep the functionality single - purposed. This makes it easier to further expand in the future. For example, is there a possibility that these files will appear in engines like S3 in the future? Or perhaps when retrieving these files from certain data sources, authentication is required.

[lispking]

It's not a good idea to hard - code the value as 10 seconds here.
It's recommended to make it a configuration so that users can set an appropriate value externally.
The parameter can default to 10 seconds.

[sankred9527]

no problem

[lispking]

The content of this function is too long. It is recommended to split it into two functions for processing. This way, it will be clearer to read.
For example,

if scheme == "http" || scheme == "https" {
   init_http();
} else if scheme == "file" {
   init_file();
}

[lispking]

This part is a bit wordy. It is recommended to change it to a form similar to the following.

self.reader
     .as_mut()
     .map_or_else(
          || Ok(None),
          |reader| async {
              let mut line = String::new();
              let read_result = reader.read_line(&mut line).await?;
              if read_result == 0 {
                  return Ok(None);
              }
              let event: Event = serde_json::from_str(&line).unwrap();
              Ok(Some(Record::new(event)))
          }.await
      )

[lispking]

The enable- prefix can be removed.

[lispking]

The issue of retrying needs to be considered, as HTTP requests may fail.

[lispking]

Certainly, for the current implementation of gharchive, it actually only supports the HTTP method.

You could even consider removing the file - related functionality first.

It's advisable not to over - design.

Instead, more thought should be given to how to solve the current timeout issue of gharchive, as gharchive has many limitations at the network level.

[lispking]

Certainly, for the current implementation of gharchive, it actually only supports the HTTP method.

You could even consider removing the file - related functionality first.

It's advisable not to over - design.

Instead, more thought should be given to how to solve the current timeout issue of gharchive, as gharchive has many limitations at the network level.

It would be better to complete the HTTP part in an independent issue #58 . Let's not deal with the retry problem here for now. After the HTTP issue is implemented later, we can come back to replace the relevant logic here.

So, please delete the file functionality first, or just handle it in a from_file way.

Currently, there is a user directly making an HTTP request, and there is no need for the file reading method for now.

[sankred9527]

I fixed the problem you required , add a new PR : #70