Skip to content
Open
Show file tree
Hide file tree
Changes from 70 commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
98ed28c
added initial watch impl
khchau7 Jul 13, 2021
7958042
debugging if we get to watch
khchau7 Jul 14, 2021
ef316a4
sanity check collection version is not reason for watch session ending
khchau7 Jul 14, 2021
7e91ef7
some logs won't be printed, why
khchau7 Jul 14, 2021
966f362
adding writing to file functionality from test app
khchau7 Jul 14, 2021
32598a0
added setup for kubeclient and log testing for watch
khchau7 Jul 14, 2021
08fdfa0
added kubclient support
khchau7 Jul 15, 2021
62c7c63
potentially fixed kubeclient issue
khchau7 Jul 16, 2021
1a0a7ab
debugging watch not being triggered
khchau7 Jul 16, 2021
006c868
porting working test code to agent
khchau7 Jul 19, 2021
ccc51fb
fixed yaml file
khchau7 Jul 19, 2021
d653e2b
e2e added - merging included
khchau7 Jul 20, 2021
6448767
fixed log statements - e2e should work!
khchau7 Jul 20, 2021
2af17b9
updated yaml
khchau7 Jul 20, 2021
0415b84
updated file write
khchau7 Jul 21, 2021
2973262
updated yaml test image 10
khchau7 Jul 21, 2021
18d3ca3
impl now uses getPodInventoryRecords to write to file
khchau7 Jul 21, 2021
3a3d671
fixed log statements, attempting to debug why merge_info write to fil…
khchau7 Jul 21, 2021
8a8f710
updated yaml for test-12
khchau7 Jul 21, 2021
03e9b60
added mmap2 gem and looking into merge logic by analyzing notice
khchau7 Jul 22, 2021
cb6601e
updated yaml test-13
khchau7 Jul 22, 2021
6e9b6ec
added notice record creation for merging logic impl
khchau7 Jul 22, 2021
9be1c64
added flag for mmap and mmap file impl
khchau7 Jul 22, 2021
9d1c907
updated yaml test-15
khchau7 Jul 22, 2021
3dfca9f
changes to mmap usage + removed some files
khchau7 Jul 23, 2021
3de0a39
added parse_and_emit_records for merge updates
khchau7 Jul 23, 2021
1eb6d29
fixed write to file bug when coming from merge_updates
khchau7 Jul 24, 2021
3635321
fixed a log statement bug
khchau7 Jul 24, 2021
10c538d
podInventoryHash is an instance variable + added logging for mmap san…
khchau7 Jul 25, 2021
8503ecf
checking if changing rw permissions on file.new fixes reading error
khchau7 Jul 25, 2021
0e8ae15
read write error potential fix with moving creation of file to start
khchau7 Jul 26, 2021
23af2bb
removing pretty gen and converting hash to string when writing to mmap
khchau7 Jul 26, 2021
78ac1c5
string might be frozen
khchau7 Jul 26, 2021
f45ecfb
podinvhash to json and removed dup syntax error
khchau7 Jul 26, 2021
f83f319
using yajl parser
khchau7 Jul 26, 2021
83c804a
added logging for size of file
khchau7 Jul 26, 2021
cc9dc1b
current version
khchau7 Jul 27, 2021
915c7eb
changed workspace and location of json file
khchau7 Jul 27, 2021
256e15d
writing to file now json rather than hash
khchau7 Jul 27, 2021
3e346d7
commented out merge logic to check if emitting works
khchau7 Jul 28, 2021
5515822
removed to_hash function
khchau7 Jul 28, 2021
a04d4d2
debugging parse and emit merge updates for failing each on string
khchau7 Jul 28, 2021
96c360d
adding back merging logic w mutex
khchau7 Jul 28, 2021
2e900a8
added fix for multiple json object error with mmap read
khchau7 Jul 28, 2021
0ae8a98
debugging why mmap error still occurs
khchau7 Jul 28, 2021
4cf149d
fixed mmap issue by changing file.new to file.open
khchau7 Jul 28, 2021
dd177c1
update record collection time with batch time
khchau7 Jul 28, 2021
c2be217
fixed logging -- checking if watch catches notifications
khchau7 Jul 29, 2021
27332e6
added logging to see why watch doesnt always emit, also removed insta…
khchau7 Aug 1, 2021
135cf21
added logging to check watch api loop
khchau7 Aug 3, 2021
e88b39e
add time logging to merge_updates
khchau7 Aug 3, 2021
64a4210
added append function for continuation token and only writes to file …
khchau7 Aug 3, 2021
9281815
changed sleep time, fixed shouldUpdateFile scope bug, collection vers…
khchau7 Aug 3, 2021
ea2e3bb
adding logging before and after sleep
khchau7 Aug 4, 2021
5b3863f
adding timeouts
khchau7 Aug 4, 2021
092fc64
appending function json fix
khchau7 Aug 4, 2021
ece8ac9
cleaned up code and changed read timeout to :open
khchau7 Aug 10, 2021
f6f2013
changing read timeout back to nil
khchau7 Aug 10, 2021
6d5c091
parse and emit merge error fix potentially
khchau7 Aug 10, 2021
fe7a1be
clean up and change open timeoiut to nil
khchau7 Aug 10, 2021
340a6ab
reverting to an old commit from last week to see if timeouts are stil…
khchau7 Aug 10, 2021
bb52ba4
testing w local kubeclient gem for tcp timeout
khchau7 Aug 13, 2021
fc326f2
updated dockerfile to support setup.sh changes for local kubeclient gem
khchau7 Aug 13, 2021
b5ebecb
read timeout is 4 mins
khchau7 Aug 17, 2021
430d859
watch restarts every 25 minutes
khchau7 Aug 18, 2021
ae90e21
changed sleep for 30 to 1 s
khchau7 Aug 18, 2021
08ee87a
eval in-mem approach
khchau7 Aug 18, 2021
7ed7cd4
changing setup.sh to use local kubeclient gem and remove podInvHash c…
khchau7 Aug 20, 2021
2c962f9
adding timeout seconds to watch call
khchau7 Aug 23, 2021
e6ef54a
cleaned up changes
khchau7 Aug 24, 2021
9b88d81
temp changes
khchau7 Aug 25, 2021
3f6869e
temp changes 2
khchau7 Aug 25, 2021
22e3281
code clean up using getPodInventoryRecords
khchau7 Aug 26, 2021
3b01db7
removing bookmarks and addressing noticetype bug
khchau7 Aug 26, 2021
f430ccf
moved telemetry in enumerate to merge updates
khchau7 Aug 26, 2021
6aa77ec
addressed comments, added telemetry, podInvHash reset, serviceRecords…
khchau7 Aug 27, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions build/linux/installer/datafiles/base_container.data
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,8 @@ chmod 666 /var/opt/microsoft/docker-cimprov/log/arc_k8s_cluster_identity.log
touch /var/opt/microsoft/docker-cimprov/log/fluentd.log
chmod 666 /var/opt/microsoft/docker-cimprov/log/fluentd.log

touch /var/opt/microsoft/docker-cimprov/log/testing-podinventory.json

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need these files?

chmod 666 /var/opt/microsoft/docker-cimprov/log/testing-podinventory.json

%Postuninstall_10
# If we're an upgrade, skip all of this cleanup
Expand Down
2 changes: 1 addition & 1 deletion kubernetes/linux/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ ENV KUBE_CLIENT_BACKOFF_BASE 1
ENV KUBE_CLIENT_BACKOFF_DURATION 0
ENV RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR 0.9
RUN /usr/bin/apt-get update && /usr/bin/apt-get install -y libc-bin wget openssl curl sudo python-ctypes init-system-helpers net-tools rsyslog cron vim dmidecode apt-transport-https gnupg && rm -rf /var/lib/apt/lists/*
COPY setup.sh main.sh defaultpromenvvariables defaultpromenvvariables-rs defaultpromenvvariables-sidecar mdsd.xml envmdsd $tmpdir/
COPY kubeclient-4.9.2.gem setup.sh main.sh defaultpromenvvariables defaultpromenvvariables-rs defaultpromenvvariables-sidecar mdsd.xml envmdsd $tmpdir/

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not urgent, but will have to update the build code to build the gem automatically instead of copying manually.

WORKDIR ${tmpdir}

# copy docker provider shell bundle to use the agent image
Expand Down
44 changes: 44 additions & 0 deletions kubernetes/linux/dockerbuild/existingClusterOnboarding.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
{

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not urgent, these files needs to be removed.

"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"aksResourceId": {
"type": "string",
"metadata": {
"description": "AKS Cluster Resource ID"
}
},
"aksResourceLocation": {
"type": "string",
"metadata": {
"description": "Location of the AKS resource e.g. \"East US\""
}
},
"workspaceResourceId": {
"type": "string",
"metadata": {
"description": "Azure Monitor Log Analytics Resource ID"
}
}
},
"resources": [
{
"name": "[split(parameters('aksResourceId'),'/')[8]]",
"type": "Microsoft.ContainerService/managedClusters",
"location": "[parameters('aksResourceLocation')]",
"apiVersion": "2018-03-31",
"properties": {
"mode": "Incremental",
"id": "[parameters('aksResourceId')]",
"addonProfiles": {
"omsagent": {
"enabled": false,
"config": {
"logAnalyticsWorkspaceResourceID": "[parameters('workspaceResourceId')]"
}
}
}
}
}
]
}
15 changes: 15 additions & 0 deletions kubernetes/linux/dockerbuild/existingClusterParam.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"aksResourceId": {
"value": "/subscriptions/b4b0cbab-64dc-42bc-9d95-ea7c8d0cf188/resourcegroups/khushi-watch-test/providers/Microsoft.ContainerService/managedClusters/khushi-watch-test"
},
"aksResourceLocation": {
"value": "West US 2"
},
"workspaceResourceId": {
"value": "/subscriptions/b4b0cbab-64dc-42bc-9d95-ea7c8d0cf188/resourcegroups/defaultresourcegroup-scus/providers/microsoft.operationalinsights/workspaces/defaultworkspace-b4b0cbab-64dc-42bc-9d95-ea7c8d0cf188-scus"
}
}
}
Binary file added kubernetes/linux/kubeclient-4.9.2.gem
Binary file not shown.
7 changes: 6 additions & 1 deletion kubernetes/linux/setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,12 @@ sudo apt-get install ruby2.6 ruby2.6-dev gcc make -y
gem install fluentd -v "1.12.2" --no-document
fluentd --setup ./fluent
gem install gyoku iso8601 --no-doc

# kubeclient gem
sudo apt-get install libmagickwand-dev -y
# sudo gem install kubeclient --no-document
sudo gem install kubeclient-4.9.2.gem
# mmap2 gem
sudo gem install mmap2

rm -f $TMPDIR/docker-cimprov*.sh
rm -f $TMPDIR/azure-mdsd*.deb
Expand Down
Loading