27 Jun 14:17

russcam

00eaaa9

7.0.0

7.0.0-beta1...7.0.0

NEST & Elasticsearch.Net 7.0: Now GA!

After many months of work, two alphas and a beta, we are pleased to announce the GA release of the NEST and Elasticsearch.Net 7.0 clients.

The overall themes of this release have been based around faster serialization, performance improvements, codebase simplification, and ensuring parity with the many new features available in Elasticsearch 7.0.

Types removal

Specifying types within the .NET clients is now deprecated in 7.0, in line with the overall Elasticsearch type removal strategy.

In instances where your index contains type information and you need to preserve this information, one recommendation is to introduce a property to describe the document type (similar to a table per class with discriminator field in the ORM world) and then implement a custom serialization / deserialization implementation for that class.

This Elasticsearch page details some other approaches.

Faster Serialization

After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to announce that the next stage of serialization improvements have been completed in 7.0.

Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast serializer that works directly with UTF-8 binary. This has yielded a significant performance improvement, which we will be sharing in more detail in a later blog post.

With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have proven too onerous to carry forward at this stage.

JSON in the request is never indented, even if SerializationFormatting.Indented is specified. The serialization routines generated by Utf8Json never generate an IJsonFormatter<T> that will indent JSON, for performance reasons. We are considering options for exposing indented JSON for development and debugging purposes.
NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.
Serializer uses Reflection.Emit. Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees. Reflection.Emit is not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android.
Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List<object>. SimpleJson deserialized JSON arrays to object[], but Utf8Json deserializes them to List<object>. This change is preferred for allocation and performance reasons.
Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the C# POCO property name.

We believe that the trade-off of features vs. GA release has been worthwhile at this stage. We hold a view to address some of these missing features in a later release.

High to Low level client dispatch changes

In 6.x, the process of an API call within NEST looked roughly like this

client.Search()
  => Dispatch()
    => LowLevelDispatch.SearchDispatch()
      => lowlevelClient.Search()
        => lowlevelClient.DoRequest()

With 7.x, this process has been changed to remove dispatching to the low-level client methods. The new process looks like this

client.Search()
  => lowlevelClient.DoRequest()

This means that in the high-level client IRequest now builds its own URLs, with the upside that the call chain is shorter and allocates fewer closures. The downside is that there are now two URL building mechanisms, one in the low-level client and a new one in the high-level client. In practice, this area of the codebase is kept up to date via code generation, so it does not place any additional burden on development.

Given the simplified call chain and debugging experience, we believe this is an improvement worth making.

Namespaced API methods and Upgrade Assistant

As the API surface of Elasticsearch has grown to well over 200 endpoints, so has the number of client methods exposed, leading to an almost overwhelming number to navigate and explore through in an IDE. This is further exacerbated by the fact that the .NET client exposes both synchronous and asynchronous API methods for both the fluent API syntax as well as the object initializer syntax.

To address this, the APIs are now accessible through sub-properties on the client instance.

For example, in 6.x, to create a machine learning job

var putJobResponse = client.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
	.AnalysisConfig(a => a
        .BucketSpan("30m")
        .Latency("0s")
    	.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
	)
	.DataDescription(d => d.TimeField(r => r.Timestamp))
);

This has changed to

var putJobResponse = client.MachineLearning.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
	.AnalysisConfig(a => a
        .BucketSpan("30m")
        .Latency("0s")
    	.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
	)
	.DataDescription(d => d.TimeField(r => r.Timestamp))
);

Notice the client.MachineLearning.PutJob method call in 7.0, as opposed to client.PutJob in 6.x.

We believe this grouping of functionality leads to a better discoverability experience when writing your code, and improved readability when reviewing somebody else's.

The Upgrade Assistant

To assist developers in migrating from 6.x, we have published the Nest.7xUpgradeAssistant Nuget package. When included in your project and the using Nest.ElasticClientExtensions; directive is added, calls will be redirected from the old API method names to the new API method names in 7.0. The result is that your project will compile and you won't need to immediately update your code to use the namespaced methods; instead you'll see compiler warnings indicating the location of the new API methods in 7.0.

This package is to assist developers migrating from 6.x to 7.0 and is limited in scope to this purpose. It is recommended that you observe the compiler warnings and adjust your code as indicated.

Observability and DiagnosticSource

7.0 introduces emitting System.Diagnostics.DiagnosticSource information from the client, during a request. The goal is to enable rich information exchange with the Elastic APM .NET agent and other monitoring libraries.

We emit DiagnosticSource information for key parts of a client request via an Activity event, shipping support for Id, ParentId, RootId, as well as request Duration.

To facilitate wiring this up to DiagnosticListener.AllListeners, we ship both with static access to the publisher names and events through Elasticsearch.Net.Diagnostics.DiagnosticSources as well as strongly typed listeners, removing the need to cast the object passed to activity start/stop events.

An example listener implementation that writes events to the console is given below

private class ListenerObserver : IObserver<DiagnosticListener>
{
  public void OnCompleted() => Console.WriteLine("Completed");
  
  public void OnError(Exception error) => Console.Error.WriteLine(error.Message);
	
  public void OnNext(DiagnosticListener value)
  {
		void WriteToConsole<T>(string eventName, T data)
		{
			var a = Activity.Current;
			Console.WriteLine($"{eventName?.PadRight(30)} {a.Id?.PadRight(32)} {a.ParentId?.PadRight(32)} {data?.ToString().PadRight(10)}");
		}
		if (value.Name == DiagnosticSources.AuditTrailEvents.SourceName)
			value.Subscribe(new AuditDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));

		if (value.Name == DiagnosticSources.RequestPipeline.SourceName)
			value.Subscribe(new RequestPipelineDiagnosticObserver(
				v => WriteToConsole(v.Key, v.Value),
				v => WriteToConsole(v.Key, v.Value))
			);

		if (value.Name == DiagnosticSources.HttpConnection.SourceName)
			value.Subscribe(new HttpConnectionDiagnosticObserver(
				v => WriteToConsole(v.Key, v.Value),
				v => WriteToConsole(v.Key, v.Value)
			));

		if (value.Name == DiagnosticSources.Serializer.SourceName)
			value.Subscribe(new SerializerDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));
  }
}

Using the following example

var pool = new SniffingConnectionPool(new[] { node.NodesUris().First() });
var settings = new ConnectionSettings(pool).SniffOnStartup();
var client = new ElasticClient(settings);
var x = client.Search<object>(s=>s.AllIndices());
               	
Console.WriteLine(new string('-', Console.WindowWidth - 1));
                                  	
var y = client.Search<object>(s=>s.Index("does-not-exist"));

Emits the following console output:

SniffOnStartup.Start       	|59e275e-4f9c835d189eb14a.    	Event: SniffOnStartup
Sniff.Start                	|59e275e-4f9c835d189eb14a.1. 	|59e275e-4f9c835d189eb14a.   	GET _nodes/http,settings
Sniff.Start                	|59e...

Assets 2

12 Jun 13:11

codebrain

7.0.0-beta1

a9b2fb1

7.0.0-beta1 Pre-release

Pre-release

7.0.0-alpha2...7.0.0-beta1

Following the advice given in 7.0.0-alpha1 the namespacing change has been implemented for this release.

Namespaced API methods

In both Elasticsearch.Net and NEST clients, API methods are made available on the root of the client. As the API surface of Elasticsearch has grown to well over 200 endpoints, so has the number of API methods exposed, leading to an almost overwhelming number of methods to navigate through to choose the method aligning with the desired API call. This is further exacerbated by the fact that the client exposes synchronous and asynchronous API methods for both the fluent API syntax as well as the object initializer syntax.

To address this, the APIs are now accessible through sub-properties on the client instance.

For example, in 6.x, to create a Machine Learning job the code is:

var putJobResponse = client.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
    .AnalysisConfig(a => a
        .BucketSpan("30m")
        .Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
    )
    .DataDescription(d => d.TimeField(r => r.Timestamp));
);

This has changed to:

var putJobResponse = client.MachineLearning.PutJob<Metric>("id", c => c
    .Description("Lab 1 - Simple example")
    .ResultsIndexName("server-metrics")
    .AnalysisConfig(a => a
        .BucketSpan("30m")
        .Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
    )
    .DataDescription(d => d.TimeField(r => r.Timestamp));
);

Notice the client.MachineLearning.PutJob method call in 7.0.0-beta1, as opposed to client.PutJob in 6.x

Features & Enhancements

#3809 Add overloads of SerializeToString and SerializeToBytes
#3765 Clean up GET /_cluster/stats
#3807 Use indexers to set default mapping values
#3806 User SecureString for Basic and Proxy passwords

Bug Fixes

#3771 Breaking formatter backwards compatibility
#3805 Routing must be specified when doc used
#3706 Calling ConnectionSettings.DefaultMappingFor throws ArgumentException when the Type has been previously added
#3719 Wrong converting datetime fields
#3751 Unable to get raw document JSON without deserializing

View the full list of issues and PRs

Assets 2

29 May 06:31

codebrain

6.8.0

fd19c75

6.8.0

6.7.0...6.8.0

Features & Enhancements

#3757 Include stored_scripts response in cluster metadata response.
#3758 Add buckets to AutoDateHistogramAggregation.
#3759 Additional Rollup Stats
#3763 Attribute SuggestBucket for serialization
#3764 XPack info and usage improvements
#3766 Add pretty names (and other properties) to the cluster stats response
#3767 Add doc’s sequence number + primary term to get responses

Deprecations

#3770 Set obsolete warning on GeoShape properties.

Bug fixes

#3775 Convert rollover lifecycle action to accept max size as a string and not long?
#3776 Use a custom converter for GeoOrientation to tolerate alternative options in Elasticsearch

View the full list of issues and PRs

Assets 2

09 May 08:09

russcam

6.7.0

d17b00a

6.7.0

Major feature highlights include support for the Security Api Keys and Index Lifecycle Management endpoints.

6.6.0...6.7.0

Features & Enhancements

#3684 Security API Key functionality
#3671 Index Lifecycle functionality
#3627 Add Median Absolute Deviation Aggregation
#3632 Add Machine Learning filter APIs and detector custom rules
#3634 Add User Dictionary Rules to NoriTokenizer
#3635 Add prebuilt ICU Analyzer
#3686 & #3704 Add support for is_write_index for the create index API - thanks @KPStolk 👍
#3631 Remove delegate allocations in Fluent.Assign
#3653 Use ContractJsonConverterAttribute where possible
#3665 Remove superfluous pragma directives
#3727 Additional details on CCR auto follow stats
#3725 Add realm information for Authenticate API
#3723 Add password_hash parameter to PutUser
#3718 Allow geo_shape types to be coerced into valid shapes.

Bug Fixes

#3619 ignore_missing added to all ingest processors
#3629 Set the timeout on the bulk call to use the request timeout
#3696 More Like This Query Conditionless behaviour
#3699 Update ClusterHealthResponse.cs to align with ClusterHealthResponse.java - thanks @SpencerLN 👍
#3698 Add properties missing from ClusterHealthResponse.cs - thanks @SpencerLN 👍

View the full list of issues and PRs

Assets 2

08 May 08:57

russcam

7.0.0-alpha2

5eb459b

7.0.0-alpha2 Pre-release

Pre-release

This is another alpha release of the .NET client, as we move a step closer to GA. Thank you for all your feedback on the 7.0.0-alpha1 client, it has been incredibly helpful in squashing bugs in the new serializer implementation.

7.0.0-alpha1...7.0.0-alpha2

Features

#3680 Implement Intervals query
#3698 Add properties currently missing on ClusterHealthResponse. Thanks @SpencerLN 👍
#3714 Add SequenceNumber and PrimaryTerm to Hits, Get and MulitGetHit
#3704 Add is_write_index to aliases on create index API
#3668 Improve field resolver performance

Breaking Changes

#3670 Majority of response interfaces are gone

This reduces the interface type explosion within the client and simplifies the async API code paths. There is a discussion to keep interfaces for the most commonly mocked responses, like ISearchResponse<T>
#3685 Strongly typed property expressions

Benchmarking demonstrates a performance improvement in using Expression<Func<T, TValue>> instead of `Expression<Func<T, object>>, particularly on .NET Framework.

Bug Fixes

#3697 Fix MoreLikeThis query conditionless behaviour
#3715 Fix ExtendedStats and Percentiles aggregation deserialization
#3717 Fix empty geo_bounds aggregation
#3678 Fix DateHistogramBucket deserialization when there are no sub aggregations
#3700 Generate API paths without leading forward slashes
#3726 Remove _source_include and _source_exclude from generated types
#3679 Fix utf8json escape characters

Misc

#3730 Use the MemoryStream buffers, where possible, instead of copying byte array. Thanks @Henr1k80 👍
#3677 Refactor ServerError, Error and ErrorCause
#3702 Support deprecated Name property on ElasticsearchTypeAttribute
#3666 Rename SynchronizedCollection

View the full list of issues and PRs

Assets 2

15 Apr 02:46

russcam

7.0.0-alpha1

dec4e1e

7.0.0-alpha1 Pre-release

Pre-release

First prerelease of 7.x .NET clients

This is the first alpha prerelease of the client with support for Elasticsearch 7.x to Nuget.

The intention in releasing this package now is to elicit feedback from the community on some of the larger changes that have already gone into the 7.x branch, prior to putting out a GA release. Some of the large changes include:

#3493 Change internal serializer to Utf8Json

Following discussion and research with replacing Json.NET as the internal serializer, a fork of Utf8Json is source included into Elasticsearch.NET, replacing both the lowlevel serializer based on SimpleJson, and the high level serializer based on Json.NET. Early research indicated about a 2x performance improvement in serialization.

All unit tests and integration tests are passing with this new serializer, but there are some known existing features of the client that will not yet work:

#3654 JSON serialization is never indented, even if SerializationFormatting.Indented is specified

The serialization routines generated by Utf8Json never generate IJsonFormatter<T> that will indent JSON, for performance reasons. We expect to implement this feature in a later release.
#3655 NEST types cannot be extended by inheritance

With NEST 6.x, additional properties can be included for a type by deriving from that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work. We expect to implement this in a later release.
Serializer uses Reflection.Emit

Utf8Json uses Reflection.Emit to generate efficient formatters for serializing types that it sees. Reflection.Emit is not supported on all platforms e.g. UWP, Xamarin.iOS, Xamarin.Android.
Elasticsearch.Net.DynamicResponse deserializes JSON arrays to List<object>

SimpleJson deserialized JSON arrays to object[], but Utf8Json deserializes them to List<object>. We believe that that the change to returning List<object> would be preferred for allocation and performance reasons.

#3658 Removal of high level to low level dispatcher

In 6.x, the process of an API call within NEST looked roughly like

client.Search() 
=> Dispatch()
=> LowLevelDispatch.SearchDispatch()
=> lowlevelClient.Search() 
=> lowlevelClient.DoRequest()

With 7.x, this process has been changed to remove dispatching to the low level client methods and the middle layer. The new process looks like

client.Search() 
=> lowlevelClient.DoRequest()

This means that in the high level client IRequest now does it own url building, with the upside that the call chain is smaller and allocates a fewer lambda closures. The downside is that there are now 2 url building mechanisms, one in the low level client and a new one in the high level client, but we think this trade off is worth it.

#3213 Refactor `geo_shape` queries and geoshapes

Support for geo_shape queries has existed in NEST for a long time, but each geoshape had its own query type and API method to use to make a geo_shape query. With 7.x, these have all been refactored into one query, GeoShapeQuery, that accepts an IGeoShape, the shape to use within the query.

Feedback

We would love to get feedback from folks trying out the 7.0.0-alpha1 release of the client. If you come across an exception or bug, please open an issue to discuss.

Assets 2

21 Mar 05:48

russcam

6.6.0

d280796

6.6.0

6.5.1...6.6.0

Features & Enhancements

#3597 Add Update by query rethrottle API
#3595 Add Delete by query rethrottle API
#3571 Add ML Info API
#3569 Add ML Calendar APIs
#3609 Add Parent Aggregation
#3591 Add Delete forecast API
#3584 Add Security Privileges APIs
#3594 Add support for date histogram format.
#3558 Add keywords_pattern to Keyword Marker Token Filter
#3572 Add articles_case to elision token filter
#3593 Return statistics about forecasts as part of the job stats and usage API
#3603 Add node count to ML usage response
#3487 and #3598 BulkAll can now return the document _id's in each response
#3494 and #3599 Expose ability to set own derived ConnectionSettingsAwareContractResolver on JsonNetSerializer
#3512 and #3601 Expose _cluster on search response
#3540 and #3602 Support for Elasticsearch 6.3 "missing" in Composite Aggregation
#3589 and #3578 Add support for disabling id inference
#3592 Add fuzzy auto length support for NEST. Thanks @SukharevAndrey! 👍
#3604 and #3606 Add Score property to Nest.TermVectorTerm class

Bug Fixes

#3502 and #3600 Reindex attachment honours ingest-attachment field names
#3551 Allow keywords_pattern in keywords_marker token filter
#3554 GetManyAsync returns the changed values if they are similar to the time data
#3575 and #3605 Allow sending null values with CompositeKey and After() functionality
#3611 Use DateParseHandling.None when loading JObject for MultiGetHit<T>
#3617 JsonSerializationException: Self referencing loop detected with type 'Nest.Field'. Path '_source.includes'. Thanks @JulianRooze 👍

Misc

#3613 Reduce string allocations. Thanks @Henr1k80! 👍
#3557 Use the version number present in the global.json file when generating documentation

View the full list of issues and PRs

Assets 2

07 Mar 06:53

russcam

6.5.1

caf150d

6.5.1

6.5.0...6.5.1

Features

#3572 Add articles_case to elision token filter

Peformance

#3579 Avoid unnecessary string concatentation. Thanks @Henr1k80 👍
#3580 Avoid doing multiple dictionary lookups. Thanks @Henr1k80 👍

Bug Fixes

#3582 ElasticLowLevelClient with SniffingConnectionPool fails to make any request due to SniffResponse deserialization failure
#3586 fix #3582 low level client sniffing
#3588 fix #3585 VoidResponse should not be shared

View the full list of issues and PRs

Assets 2

07 Mar 11:06

russcam

5.6.6

4c2e1e7

5.6.6

5.6.5...5.6.6

Features

#3572 Add articles_case to elision token filter

Peformance

#3579 Avoid unnecessary string concatentation. Thanks @Henr1k80 👍
#3580 Avoid doing multiple dictionary lookups. Thanks @Henr1k80 👍

Bug Fixes

#3582 ElasticLowLevelClient with SniffingConnectionPool fails to make any request due to SniffResponse deserialization failure
#3586 fix #3582 low level client sniffing
#3588 fix #3585 VoidResponse should not be shared

View the full list of issues and PRs

Assets 2

29 Jan 06:49

codebrain

6.5.0

2234d5b

6.5.0

6.4.2...6.5.0

Features & Enhancements

#3519 Document how to use a custom authentication header e.g Kerberos
#3409 Can't create a role with Kibana privileges?
#3462 Add application privileges to Put Role API
#3490 Add a format option to docvalue_fields
#3533 Cross Cluster Replication support
#3534 Adds support for get_basic_license_status and start_basic_license API's
#3535 add support for the rollup index caps API
#3536 add support for the reload secure settings API
#3537 add support for the get ssl certificates API
#3545 fix #3544 JsonNetSerializer now also implements IPropertyMappingProvi…
#3491 Include Format property on Field for Doc Values fields
#3520 deprecate standard token filter as per elastic/elasticsearch#33468
#3521 Add support for the auto_date_histogram aggregations
#3522 support predicate_token_filter elastic/elasticsearch#33431
#3523 Add support for condition token filter
#3528 V65/processors
#3529 Add processor stats to node ingest stats
#3530 Add support for default pipeline on index settings
#3531 Add support for source only snapshot repository

Bug Fixes

#3378 StringResponse.TryGetServerError parse error on 401 response
#3402 Can't set refresh_interval to default value (null)
#3411 Cross Cluster support in IndexName prevents the use of ::
#3458 Fix IndexName Parse
#3460 Missing Enum Value in Nest.UnassignedInformationReason
#3461 Add missing enum values for UnassignedInformationReason
#3465 Allow null refresh_interval to be passed
#3466 try/catch when attempting to deserialize ServerError from response
#3548 Add backwards compatible overloads for Field format
#3546 fix #3505 daterange default DateTime should not serialize

View the full list of issues and PRs

Assets 2

Releases: elastic/elasticsearch-net

7.0.0

NEST & Elasticsearch.Net 7.0: Now GA!

Types removal

Faster Serialization

High to Low level client dispatch changes

Namespaced API methods and Upgrade Assistant

The Upgrade Assistant

Observability and DiagnosticSource

Uh oh!

7.0.0-beta1

Namespaced API methods

Features & Enhancements

Bug Fixes

Uh oh!

6.8.0

Features & Enhancements

Deprecations

Bug fixes

Uh oh!

6.7.0

Features & Enhancements

Bug Fixes

Uh oh!

7.0.0-alpha2

Features

Breaking Changes

Bug Fixes

Misc

Uh oh!

7.0.0-alpha1

First prerelease of 7.x .NET clients

#3493 Change internal serializer to Utf8Json

#3658 Removal of high level to low level dispatcher

#3213 Refactor geo_shape queries and geoshapes

Feedback

Uh oh!

6.6.0

Features & Enhancements

Bug Fixes

Misc

Uh oh!

6.5.1

Features

Peformance

Bug Fixes

Uh oh!

5.6.6

Features

Peformance

Bug Fixes

Uh oh!

6.5.0

Features & Enhancements

Bug Fixes

Uh oh!

#3213 Refactor `geo_shape` queries and geoshapes