[feat] Decouple the project into modules/submodules/separate repos

[shivanshuraj1333]

Is your feature request related to a problem? Please describe.

Currently there are many issues with building the project, it is hard to understand which files are being used as dependencies for another project. For example k8s collector which is written in go https://github.com/open-telemetry/opentelemetry-network/tree/main/collector/k8s also contains c header files and some other code written in c.

It makes it harder to compile and debug.

Describe the solution you'd like

If it is possible, the project should be restructured in a way which makes debugging feasible for individual components, and make it easier to understand the dependencies.

Since the project is using Xtend, C, and go, and dependencies are coupled. It becomes hard for someone with less experience in C to build and debug issues.

Decoupling the project into modules/sub modules/directories/or different projects can help.

Describe alternatives you've considered

No response

Additional context

No response

[yonch]

Agreed.

The first obstacle to this vision appears to be the custom RPC implementation, that uses Xtend. This implementation is super low-overhead, which allows Otel-network to handle per-socket, per-second telemetry on 1k+ node clusters. However, it is a custom implementation, and requires the k8s-relay, and the C++ cloud-collector.

Avro seems to be a good potential fit for a replacement. However the C++ library does not support the RPC standard, only serialization and deserialization.

The Xtend implementation also supports keeping objects in sync between the two sides of the connection. When the connection disconnects, the objects are removed from the remote end. This handles cleanup in the reducer when collectors disconnect. We'll have to find a solution for that as well if we switch to a different library like Avro.

Once the implementation switches to a different RPC library, the k8s collector could be in pure Go (for example) and use that library to communicate to the non-Go parts. Its outputs could be tested by other components that have a library for that RPC (e.g., Go or Python).

@shivanshuraj1333 is that what you had in mind?