Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preallocate plus logrecord cow #26

Open
wants to merge 80 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
7fee0fc
inital commit
lalitb Jul 23, 2024
06aee01
more
lalitb Jul 23, 2024
ea7b2f8
more changes
lalitb Jul 23, 2024
bfb1297
changes
lalitb Jul 23, 2024
fcb0ed8
fix build
lalitb Jul 24, 2024
40b2b5f
changes
lalitb Jul 24, 2024
da8b8de
revert defaults
lalitb Jul 24, 2024
5019086
optionally create vec
lalitb Jul 24, 2024
17d1628
add intial capacity for vector
lalitb Jul 24, 2024
781c3ed
create vec of certain capacity
lalitb Jul 25, 2024
b9a90c2
changes
lalitb Jul 25, 2024
2bfde32
Merge branch 'main' into preallocate-attribute-mem
lalitb Jul 25, 2024
a95e681
cont..
lalitb Jul 25, 2024
9dc8a45
changes..
lalitb Jul 26, 2024
df66985
box enums
lalitb Jul 26, 2024
8f62fa3
make growable add inline
lalitb Jul 26, 2024
fb7a44c
update bench
lalitb Jul 26, 2024
8611bc0
Merge branch 'main' into preallocate-attribute-mem
lalitb Jul 27, 2024
2fabca1
remove experimenal metadata as default
lalitb Jul 27, 2024
e52d002
changes.
lalitb Jul 28, 2024
92d8689
Merge branch 'main' into preallocate-attribute-mem
lalitb Jul 28, 2024
877996e
revert back to option
lalitb Jul 29, 2024
8a86be3
Merge branch 'preallocate-attribute-mem' of github.com:lalitb/opentel…
lalitb Jul 29, 2024
283816b
revert the checkedin proto
lalitb Jul 29, 2024
77e1ae1
back
lalitb Jul 29, 2024
ad42e0b
trying again to revert unwanted otel-proto update
lalitb Jul 29, 2024
e99a957
another revert try
lalitb Jul 29, 2024
c97a948
ci errors
lalitb Jul 29, 2024
1adec8c
Merge branch 'main' into preallocate-attribute-mem
TommyCpp Jul 29, 2024
660a6d8
fix tests
lalitb Jul 29, 2024
9e09502
Merge branch 'preallocate-attribute-mem' of github.com:lalitb/opentel…
lalitb Jul 29, 2024
8e0bfaa
Update opentelemetry-sdk/src/logs/growable_array.rs
lalitb Jul 29, 2024
9d44e26
Update Cargo.toml
lalitb Jul 29, 2024
eed39a1
Update Cargo.toml
lalitb Jul 30, 2024
9275a11
review comments
lalitb Jul 31, 2024
8c28428
Merge branch 'main' into preallocate-attribute-mem
lalitb Jul 31, 2024
0f6b699
lint issues
lalitb Jul 31, 2024
1a2ec1f
comments
lalitb Jul 31, 2024
adc48ca
Update opentelemetry-sdk/src/logs/growable_array.rs
lalitb Jul 31, 2024
0a4715a
revert back box
lalitb Aug 1, 2024
b619409
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 1, 2024
6136678
change var names
lalitb Aug 1, 2024
1083e17
Merge remote-tracking branch 'refs/remotes/origin/preallocate-attribu…
lalitb Aug 1, 2024
7575f4b
Merge branch 'main' into preallocate-attribute-mem
TommyCpp Aug 1, 2024
cb0d05d
move growable array outside logs
lalitb Aug 2, 2024
196d2c7
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 2, 2024
c9816f0
review comments
lalitb Aug 2, 2024
2a149f9
use get_or_insert_with
lalitb Aug 2, 2024
3caf8b4
clioppy warning
lalitb Aug 2, 2024
0a876bf
remove growablearray from test
lalitb Aug 5, 2024
a0fc1f3
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 6, 2024
6c468e0
Update lib.rs
lalitb Aug 6, 2024
6f58225
Update lib.rs
lalitb Aug 6, 2024
b7ca124
Update lib.rs
lalitb Aug 6, 2024
846ca89
Update Cargo.toml
lalitb Aug 6, 2024
ee57967
merge conflict
lalitb Aug 6, 2024
f9dbdf0
Merge branch 'main' into preallocate-attribute-mem
cijothomas Aug 7, 2024
8911c39
Update record.rs
lalitb Aug 7, 2024
d500402
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 7, 2024
bbe5689
modify attributes_contain
lalitb Aug 7, 2024
4117af4
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 8, 2024
b256195
Update opentelemetry-sdk/src/growable_array.rs
lalitb Aug 8, 2024
40a97ac
fix
lalitb Aug 8, 2024
085f001
Update opentelemetry-sdk/src/growable_array.rs
lalitb Aug 8, 2024
cb200a5
fix attributes
lalitb Aug 8, 2024
da5a5ab
Merge branch 'preallocate-attribute-mem' of github.com:lalitb/opentel…
lalitb Aug 8, 2024
f3b63fb
fix
lalitb Aug 8, 2024
bdbaa87
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 8, 2024
8cccd9f
Merge branch 'main' into preallocate-attribute-mem
cijothomas Aug 9, 2024
d192971
Merge branch 'main' into preallocate-attribute-mem
cijothomas Aug 9, 2024
8d0fe0d
Merge branch 'main' into preallocate-attribute-mem
cijothomas Aug 9, 2024
2ada329
Merge branch 'main' into preallocate-attribute-mem
cijothomas Aug 9, 2024
6eb6d9d
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 9, 2024
aefc0b6
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 12, 2024
2f64386
Merge branch 'main' into preallocate-attribute-mem
lalitb Aug 12, 2024
b4e76bb
fix
lalitb Aug 15, 2024
a81ae72
more changes
lalitb Aug 15, 2024
f0793fe
fix
lalitb Aug 15, 2024
573f850
fix lint
lalitb Aug 15, 2024
ae33b5e
fix build error
lalitb Aug 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion opentelemetry-otlp/src/logs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ impl LogExporter {
impl opentelemetry_sdk::export::logs::LogExporter for LogExporter {
async fn export<'a>(
&mut self,
batch: Vec<std::borrow::Cow<'a, LogData>>,
batch: Vec<std::borrow::Cow<'a, LogData<'a>>>,
) -> opentelemetry::logs::LogResult<()> {
self.client.export(batch).await
}
Expand Down
34 changes: 34 additions & 0 deletions opentelemetry-proto/src/transform/common.rs
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,40 @@ pub mod tonic {
}
}

impl
From<(
Cow<'_, opentelemetry_sdk::InstrumentationLibrary>,
Option<Cow<'static, str>>,
)> for InstrumentationScope
{
fn from(
data: (
Cow<'_, opentelemetry_sdk::InstrumentationLibrary>,
Option<Cow<'static, str>>,
),
) -> Self {
let (library, target) = data;
if let Some(t) = target {
InstrumentationScope {
name: t.to_string(),
version: String::new(),
attributes: vec![],
..Default::default()
}
} else {
InstrumentationScope {
name: library.name.clone().into_owned(),
version: library
.version
.as_ref()
.map(ToString::to_string)
.unwrap_or_default(),
attributes: Attributes::from(library.attributes.clone()).0,
..Default::default()
}
}
}
}
/// Wrapper type for Vec<`KeyValue`>
#[derive(Default, Debug)]
pub struct Attributes(pub ::std::vec::Vec<crate::proto::tonic::common::v1::KeyValue>);
Expand Down
53 changes: 34 additions & 19 deletions opentelemetry-proto/src/transform/logs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ pub mod tonic {

impl
From<(
opentelemetry_sdk::export::logs::LogData,
opentelemetry_sdk::export::logs::LogData<'_>,
&ResourceAttributesWithSchema,
)> for ResourceLogs
{
Expand All @@ -164,30 +164,35 @@ pub mod tonic {
.clone()
.map(Into::into)
.unwrap_or_default(),
scope: Some((log_data.instrumentation, log_data.record.target.clone()).into()),
log_records: vec![log_data.record.into()],
scope: Some(
(
log_data.instrumentation.into_owned(),
log_data.record.target.clone(),
)
.into(),
),
log_records: vec![log_data.record.into_owned().into()],
}],
}
}
}

pub fn group_logs_by_resource_and_scope(
logs: Vec<opentelemetry_sdk::export::logs::LogData>,
logs: Vec<opentelemetry_sdk::export::logs::LogData<'_>>,
resource: &ResourceAttributesWithSchema,
) -> Vec<ResourceLogs> {
// Group logs by target or instrumentation name
let scope_map = logs.iter().fold(
HashMap::new(),
|mut scope_map: HashMap<
Cow<'static, str>,
Vec<&opentelemetry_sdk::export::logs::LogData>,
Vec<&opentelemetry_sdk::export::logs::LogData<'_>>,
>,
log| {
let key = log
.record
.target
.clone()
.unwrap_or_else(|| log.instrumentation.name.clone());
let key =
log.record.target.clone().unwrap_or_else(|| {
Cow::Owned(log.instrumentation.name.clone().into_owned())
});
scope_map.entry(key).or_default().push(log);
scope_map
},
Expand All @@ -197,13 +202,20 @@ pub mod tonic {
.into_iter()
.map(|(key, log_data)| ScopeLogs {
scope: Some(InstrumentationScope::from((
&log_data.first().unwrap().instrumentation,
Some(key),
Cow::Owned(
log_data
.first()
.unwrap()
.instrumentation
.clone()
.into_owned(),
),
Some(key.into_owned().into()),
))),
schema_url: resource.schema_url.clone().unwrap_or_default(),
log_records: log_data
.into_iter()
.map(|log_data| log_data.record.clone().into())
.map(|log_data| log_data.record.clone().into_owned().into())
.collect(),
})
.collect();
Expand All @@ -225,18 +237,21 @@ mod tests {
use opentelemetry::logs::LogRecord as _;
use opentelemetry_sdk::export::logs::LogData;
use opentelemetry_sdk::{logs::LogRecord, Resource};
use std::borrow::Cow;
use std::time::SystemTime;

fn create_test_log_data(instrumentation_name: &str, _message: &str) -> LogData {
fn create_test_log_data<'a>(instrumentation_name: &str, _message: &str) -> LogData<'a> {
let mut logrecord = LogRecord::default();
logrecord.set_timestamp(SystemTime::now());
logrecord.set_observed_timestamp(SystemTime::now());
LogData {
instrumentation: opentelemetry_sdk::InstrumentationLibrary::builder(
instrumentation_name.to_string(),
)
.build(),
record: logrecord,
instrumentation: Cow::Owned(
opentelemetry_sdk::InstrumentationLibrary::builder(
instrumentation_name.to_string(),
)
.build(),
),
record: Cow::Owned(logrecord),
}
}

Expand Down
3 changes: 1 addition & 2 deletions opentelemetry-sdk/benches/log.rs
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,7 @@ use opentelemetry::trace::Tracer;
use opentelemetry::trace::TracerProvider as _;
use opentelemetry::Key;
use opentelemetry_sdk::export::logs::LogData;
use opentelemetry_sdk::logs::LogProcessor;
use opentelemetry_sdk::logs::{Logger, LoggerProvider};
use opentelemetry_sdk::logs::{LogProcessor, Logger, LoggerProvider};
use opentelemetry_sdk::trace;
use opentelemetry_sdk::trace::{Sampler, TracerProvider};

Expand Down
10 changes: 5 additions & 5 deletions opentelemetry-sdk/src/export/logs/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ use std::fmt::Debug;
#[async_trait]
pub trait LogExporter: Send + Sync + Debug {
/// Exports a batch of [`LogData`].
async fn export<'a>(&mut self, batch: Vec<Cow<'a, LogData>>) -> LogResult<()>;
async fn export<'a>(&mut self, batch: Vec<Cow<'a, LogData<'a>>>) -> LogResult<()>;
/// Shuts down the exporter.
fn shutdown(&mut self) {}
#[cfg(feature = "logs_level_enabled")]
Expand All @@ -29,11 +29,11 @@ pub trait LogExporter: Send + Sync + Debug {

/// `LogData` represents a single log event without resource context.
#[derive(Clone, Debug)]
pub struct LogData {
/// Log record
pub record: LogRecord,
pub struct LogData<'a> {
/// Log record, which can be borrowed or owned.
pub record: Cow<'a, LogRecord>,
/// Instrumentation details for the emitter who produced this `LogEvent`.
pub instrumentation: InstrumentationLibrary,
pub instrumentation: Cow<'a, InstrumentationLibrary>,
}

/// Describes the result of an export.
Expand Down
Loading
Loading