Skip to content

Commit 77c5909

Browse files
umaannamalaiTimPansinoNayukenccedacero-nr
authored
Add support for aioredis. (#577)
* Add aioredis Instrumentation (#567) * Add aioredis instrumentation Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Nyenty Ayuk <[email protected]> Co-authored-by: ccedacero-nr <[email protected]> * [Mega-Linter] Apply linters fixes * Bump Tests * Fix double wrapping Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Nyenty Ayuk <[email protected]> Co-authored-by: ccedacero-nr <[email protected]> Co-authored-by: TimPansino <[email protected]> * Add aioredis test infrastructure. (#568) * Add aioredis test infra. * Fix flake8 errors. * Aredis concurrency bug reproduction. (#569) Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Uma Annamalai <[email protected]> * Add aioredis tests (#573) * Add get and set tests. * Add more testing for aioredis. * Add aioredis testing. Co-authored-by: Tim Pansino <[email protected]> Co-authored-by: Cristian Cedacero <[email protected]> Co-authored-by: Nyenty Ayuk-Enow <[email protected]> * Patch broken tests * Final aioredis testing cleanup Co-authored-by: Nyenty Ayuk <[email protected]> Co-authored-by: ccedacero-nr <[email protected]> Co-authored-by: Uma Annamalai <[email protected]> * Parametrize multiple db tests. * Add missing arg. * Fix typo. * Add missing comma. * Add background_task decorator. * Parametrize instance info tests. * Fix formatting Co-authored-by: Tim Pansino <[email protected]> Co-authored-by: Cristian Cedacero <[email protected]> Co-authored-by: Nyenty Ayuk-Enow <[email protected]> Co-authored-by: Tim Pansino <[email protected]> Co-authored-by: ccedacero-nr <[email protected]> Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Timothy Pansino <[email protected]> * Fix AIORedis Concurrency Bug (#574) * Add test for concurrency bug * Fix aioredis concurrency * Fix func signature * Fix ARedis Concurrency Bug (#570) * Patch aredis concurrency bug * Remove xfail marker * Format * Move fixture import Co-authored-by: Uma Annamalai <[email protected]> * Increase concurrency of redis tests (#575) * Instrument All Redis Client Methods (#576) * Initial test files * Fully instrument uninstrumented redis client methods Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: ccedacero-nr <[email protected]> Co-authored-by: Nyenty Ayuk <[email protected]> * Fix older redis client tests * Fix missing redis client method * Remove sentinel from commands list * Fix sentinels again Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: ccedacero-nr <[email protected]> Co-authored-by: Nyenty Ayuk <[email protected]> * Add aioredis v1 support (#579) * Add aioredis v1 tests * Fix aioredis v1 Co-authored-by: Uma Annamalai <[email protected]> * Adjust multiple dbs tests * Fix megalinter default base * Fix megalinter base take two * Fix aioredis version parser * Uncomment instance info tests * Fix import issues Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Timothy Pansino <[email protected]> Co-authored-by: Uma Annamalai <[email protected]> Co-authored-by: Nyenty Ayuk <[email protected]> Co-authored-by: ccedacero-nr <[email protected]> Co-authored-by: TimPansino <[email protected]> Co-authored-by: Tim Pansino <[email protected]> Co-authored-by: Cristian Cedacero <[email protected]> Co-authored-by: Tim Pansino <[email protected]>
1 parent 0dc8434 commit 77c5909

21 files changed

+2027
-190
lines changed

.github/workflows/mega-linter.yml

+1
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ jobs:
4040
# All available variables are described in documentation
4141
# https://megalinter.github.io/configuration/
4242
VALIDATE_ALL_CODEBASE: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} # Validates all source when push on main, else just the git diff with main. Set 'true' if you always want to lint all sources
43+
DEFAULT_BRANCH: ${{ github.event_name == 'pull_request' && github.base_ref || 'main' }}
4344
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
4445
# ADD YOUR CUSTOM ENV VARIABLES HERE TO OVERRIDE VALUES OF .mega-linter.yml AT THE ROOT OF YOUR REPOSITORY
4546

.github/workflows/tests.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -509,11 +509,11 @@ jobs:
509509

510510
redis:
511511
env:
512-
TOTAL_GROUPS: 1
512+
TOTAL_GROUPS: 2
513513

514514
strategy:
515515
matrix:
516-
group-number: [1]
516+
group-number: [1, 2]
517517

518518
runs-on: ubuntu-latest
519519
timeout-minutes: 30

newrelic/config.py

+32
Original file line numberDiff line numberDiff line change
@@ -2603,6 +2603,14 @@ def _process_module_builtin_defaults():
26032603
"instrument_aredis_connection",
26042604
)
26052605

2606+
_process_module_definition("aioredis.client", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_client")
2607+
2608+
_process_module_definition("aioredis.commands", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_client")
2609+
2610+
_process_module_definition(
2611+
"aioredis.connection", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_connection"
2612+
)
2613+
26062614
_process_module_definition(
26072615
"elasticsearch.client",
26082616
"newrelic.hooks.datastore_elasticsearch",
@@ -2691,6 +2699,30 @@ def _process_module_builtin_defaults():
26912699
"redis.commands.core", "newrelic.hooks.datastore_redis", "instrument_redis_commands_core"
26922700
)
26932701

2702+
_process_module_definition(
2703+
"redis.commands.sentinel", "newrelic.hooks.datastore_redis", "instrument_redis_commands_sentinel"
2704+
)
2705+
2706+
_process_module_definition(
2707+
"redis.commands.json.commands", "newrelic.hooks.datastore_redis", "instrument_redis_commands_json_commands"
2708+
)
2709+
2710+
_process_module_definition(
2711+
"redis.commands.search.commands", "newrelic.hooks.datastore_redis", "instrument_redis_commands_search_commands"
2712+
)
2713+
2714+
_process_module_definition(
2715+
"redis.commands.timeseries.commands", "newrelic.hooks.datastore_redis", "instrument_redis_commands_timeseries_commands"
2716+
)
2717+
2718+
_process_module_definition(
2719+
"redis.commands.bf.commands", "newrelic.hooks.datastore_redis", "instrument_redis_commands_bf_commands"
2720+
)
2721+
2722+
_process_module_definition(
2723+
"redis.commands.graph.commands", "newrelic.hooks.datastore_redis", "instrument_redis_commands_graph_commands"
2724+
)
2725+
26942726
_process_module_definition("motor", "newrelic.hooks.datastore_motor", "patch_motor")
26952727

26962728
_process_module_definition(

newrelic/hooks/datastore_aioredis.py

+127
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Copyright 2010 New Relic, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
from newrelic.api.datastore_trace import DatastoreTrace
16+
from newrelic.api.time_trace import current_trace
17+
from newrelic.api.transaction import current_transaction
18+
from newrelic.common.object_wrapper import wrap_function_wrapper
19+
from newrelic.hooks.datastore_redis import (
20+
_redis_client_methods,
21+
_redis_multipart_commands,
22+
_redis_operation_re,
23+
)
24+
25+
26+
def _conn_attrs_to_dict(connection):
27+
host = getattr(connection, "host", None)
28+
port = getattr(connection, "port", None)
29+
if not host and not port and hasattr(connection, "_address"):
30+
host, port = connection._address
31+
return {
32+
"host": host,
33+
"port": port,
34+
"path": getattr(connection, "path", None),
35+
"db": getattr(connection, "db", getattr(connection, "_db", None)),
36+
}
37+
38+
39+
def _instance_info(kwargs):
40+
host = kwargs.get("host") or "localhost"
41+
port_path_or_id = str(kwargs.get("port") or kwargs.get("path", 6379))
42+
db = str(kwargs.get("db") or 0)
43+
44+
return (host, port_path_or_id, db)
45+
46+
47+
def _wrap_AioRedis_method_wrapper(module, instance_class_name, operation):
48+
async def _nr_wrapper_AioRedis_method_(wrapped, instance, args, kwargs):
49+
transaction = current_transaction()
50+
if transaction is None:
51+
return await wrapped(*args, **kwargs)
52+
53+
with DatastoreTrace(product="Redis", target=None, operation=operation):
54+
return await wrapped(*args, **kwargs)
55+
56+
name = "%s.%s" % (instance_class_name, operation)
57+
wrap_function_wrapper(module, name, _nr_wrapper_AioRedis_method_)
58+
59+
60+
async def wrap_Connection_send_command(wrapped, instance, args, kwargs):
61+
transaction = current_transaction()
62+
if not transaction:
63+
return await wrapped(*args, **kwargs)
64+
65+
host, port_path_or_id, db = (None, None, None)
66+
67+
try:
68+
dt = transaction.settings.datastore_tracer
69+
if dt.instance_reporting.enabled or dt.database_name_reporting.enabled:
70+
conn_kwargs = _conn_attrs_to_dict(instance)
71+
host, port_path_or_id, db = _instance_info(conn_kwargs)
72+
except Exception:
73+
pass
74+
75+
# Older Redis clients would when sending multi part commands pass
76+
# them in as separate arguments to send_command(). Need to therefore
77+
# detect those and grab the next argument from the set of arguments.
78+
79+
operation = args[0].strip().lower()
80+
81+
# If it's not a multi part command, there's no need to trace it, so
82+
# we can return early.
83+
84+
if operation.split()[0] not in _redis_multipart_commands: # Set the datastore info on the DatastoreTrace containing this function call.
85+
trace = current_trace()
86+
87+
# Find DatastoreTrace no matter how many other traces are inbetween
88+
while trace is not None and not isinstance(trace, DatastoreTrace):
89+
trace = getattr(trace, "parent", None)
90+
91+
if trace is not None:
92+
trace.host = host
93+
trace.port_path_or_id = port_path_or_id
94+
trace.database_name = db
95+
96+
return await wrapped(*args, **kwargs)
97+
98+
# Convert multi args to single arg string
99+
100+
if operation in _redis_multipart_commands and len(args) > 1:
101+
operation = "%s %s" % (operation, args[1].strip().lower())
102+
103+
operation = _redis_operation_re.sub("_", operation)
104+
105+
with DatastoreTrace(
106+
product="Redis", target=None, operation=operation, host=host, port_path_or_id=port_path_or_id, database_name=db
107+
):
108+
return await wrapped(*args, **kwargs)
109+
110+
111+
def instrument_aioredis_client(module):
112+
# StrictRedis is just an alias of Redis, no need to wrap it as well.
113+
if hasattr(module, "Redis"):
114+
class_ = getattr(module, "Redis")
115+
for operation in _redis_client_methods:
116+
if hasattr(class_, operation):
117+
_wrap_AioRedis_method_wrapper(module, "Redis", operation)
118+
119+
120+
def instrument_aioredis_connection(module):
121+
if hasattr(module, "Connection"):
122+
if hasattr(module.Connection, "send_command"):
123+
wrap_function_wrapper(module, "Connection.send_command", wrap_Connection_send_command)
124+
125+
if hasattr(module, "RedisConnection"):
126+
if hasattr(module.RedisConnection, "execute"):
127+
wrap_function_wrapper(module, "RedisConnection.execute", wrap_Connection_send_command)

newrelic/hooks/datastore_aredis.py

+33-27
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,17 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
1615
from newrelic.api.datastore_trace import DatastoreTrace
16+
from newrelic.api.time_trace import current_trace
1717
from newrelic.api.transaction import current_transaction
1818
from newrelic.common.object_wrapper import wrap_function_wrapper
19-
from newrelic.hooks.datastore_redis import _conn_attrs_to_dict, _instance_info, _redis_client_methods, _redis_multipart_commands, _redis_operation_re
19+
from newrelic.hooks.datastore_redis import (
20+
_conn_attrs_to_dict,
21+
_instance_info,
22+
_redis_client_methods,
23+
_redis_multipart_commands,
24+
_redis_operation_re,
25+
)
2026

2127

2228
def _wrap_Aredis_method_wrapper_(module, instance_class_name, operation):
@@ -25,35 +31,18 @@ async def _nr_wrapper_Aredis_method_(wrapped, instance, args, kwargs):
2531
if transaction is None:
2632
return await wrapped(*args, **kwargs)
2733

28-
dt = DatastoreTrace(product="Redis", target=None, operation=operation)
29-
30-
transaction._nr_datastore_instance_info = (None, None, None)
31-
32-
with dt:
33-
result = await wrapped(*args, **kwargs)
34-
35-
host, port_path_or_id, db = transaction._nr_datastore_instance_info
36-
dt.host = host
37-
dt.port_path_or_id = port_path_or_id
38-
dt.database_name = db
39-
return result
34+
with DatastoreTrace(product="Redis", target=None, operation=operation):
35+
return await wrapped(*args, **kwargs)
4036

4137
name = "%s.%s" % (instance_class_name, operation)
4238
wrap_function_wrapper(module, name, _nr_wrapper_Aredis_method_)
4339

4440

45-
def instrument_aredis_client(module):
46-
if hasattr(module, "StrictRedis"):
47-
for name in _redis_client_methods:
48-
if hasattr(module.StrictRedis, name):
49-
_wrap_Aredis_method_wrapper_(module, "StrictRedis", name)
50-
51-
52-
async def _nr_Connection_send_command_wrapper_(wrapped, instance, args, kwargs):
41+
async def wrap_Connection_send_command(wrapped, instance, args, kwargs):
5342
transaction = current_transaction()
5443
if not transaction:
5544
return await wrapped(*args, **kwargs)
56-
45+
5746
host, port_path_or_id, db = (None, None, None)
5847

5948
try:
@@ -64,8 +53,6 @@ async def _nr_Connection_send_command_wrapper_(wrapped, instance, args, kwargs):
6453
except Exception:
6554
pass
6655

67-
transaction._nr_datastore_instance_info = (host, port_path_or_id, db)
68-
6956
# Older Redis clients would when sending multi part commands pass
7057
# them in as separate arguments to send_command(). Need to therefore
7158
# detect those and grab the next argument from the set of arguments.
@@ -76,6 +63,18 @@ async def _nr_Connection_send_command_wrapper_(wrapped, instance, args, kwargs):
7663
# we can return early.
7764

7865
if operation.split()[0] not in _redis_multipart_commands:
66+
# Set the datastore info on the DatastoreTrace containing this function call.
67+
trace = current_trace()
68+
69+
# Find DatastoreTrace no matter how many other traces are inbetween
70+
while trace is not None and not isinstance(trace, DatastoreTrace):
71+
trace = getattr(trace, "parent", None)
72+
73+
if trace is not None:
74+
trace.host = host
75+
trace.port_path_or_id = port_path_or_id
76+
trace.database_name = db
77+
7978
return await wrapped(*args, **kwargs)
8079

8180
# Convert multi args to single arg string
@@ -89,7 +88,14 @@ async def _nr_Connection_send_command_wrapper_(wrapped, instance, args, kwargs):
8988
product="Redis", target=None, operation=operation, host=host, port_path_or_id=port_path_or_id, database_name=db
9089
):
9190
return await wrapped(*args, **kwargs)
92-
91+
92+
93+
def instrument_aredis_client(module):
94+
if hasattr(module, "StrictRedis"):
95+
for name in _redis_client_methods:
96+
if hasattr(module.StrictRedis, name):
97+
_wrap_Aredis_method_wrapper_(module, "StrictRedis", name)
98+
9399

94100
def instrument_aredis_connection(module):
95-
wrap_function_wrapper(module.Connection, "send_command", _nr_Connection_send_command_wrapper_)
101+
wrap_function_wrapper(module.Connection, "send_command", wrap_Connection_send_command)

0 commit comments

Comments
 (0)