Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INTERNAL PR] Dotnet Counters Unification #2

Closed
wants to merge 134 commits into from

Conversation

kkeirstead
Copy link
Owner

@kkeirstead kkeirstead commented Sep 8, 2023

Internal PR to solidify things before bringing this code to the diagnostics repo. Note that I expect some things still need to be tweaked, but I wanted to put this up instead of just continuing to polish it. I also plan to do another round of manual testing on this when it's in a "done" state - I did manual testing to find some minor differences prior to the shared session logic, so just to be safe I'll likely do another sanity check. There's also an accompanying PR for dotnet-monitor with a few small changes here.

wiktork and others added 30 commits May 15, 2023 09:46
[main] Update dependencies from dotnet/aspnetcore
This pull request updates the following dependencies

[marker]: <> (Begin:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)
## From https://github.com/dotnet/runtime
- **Subscription**: e4bfb556-e13c-47f6-eb5a-08d8e4d5099b
- **Build**: 20230825.9
- **Date Produced**: August 26, 2023 2:22:36 AM UTC
- **Commit**: d379e3a1fe89d463ae55b86525fbd7713e983d9e
- **Branch**: refs/heads/release/8.0

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.NETCore.App.Runtime.win-x64**: [from 8.0.0-rc.1.23410.15
to 8.0.0-rc.2.23425.9][13]
- **VS.Redist.Common.NetCore.SharedFramework.x64.8.0**: [from
8.0.0-rc.1.23410.15 to 8.0.0-rc.2.23425.9][13]

[13]: dotnet/runtime@786b987...d379e3a

[DependencyUpdate]: <> (End)


[marker]: <> (End:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Mike McLaughlin <[email protected]>
Missed that during the eng/native sync, we made these replacements
upstream in
dotnet/runtime@9fc5fcb
to keep things distro-neutral.
[main] Update dependencies from dotnet/runtime
[main] Update dependencies from dotnet/arcade
[main] Update dependencies from dotnet/installer


 - Update Versions.props
dotnet#4081)

.net8 adds support for dotnet-gcdump on Mono, meaning dotnet-gcdump will
be used targeting mobile platforms. Currently dotnet-gcdump doesn't have
support needed since it can only connect using pid or process name and
has logic to make sure the process connected to has the same pid as the
-p argument passed to dotnet-gcdump.

On mobile platforms, runtime is running on device and communicates using
TCP/IP (over loopback interface, using Android adb port forwarding or
usbmux on iOS). All logic related to communicate with devices on
different platforms is handled by dotnet-dsrouter, that expose the same
IPC channels as a normal runtime would do, meaning most diagnostic tools
can connect to dotnet-dsrouter that will route all communication to/from
device. Tools like dotnet-trace can leverage
--diagnostic-port=<ipc>,connect instead of pid to connect to a named IPC
channel on dotnet-dsrouter (routed to a TCP/IP port on device). It is
also possible to connect directly towards dotnet-dsrouters pid since it
will masquerade like a regular runtime, but it will not alter the pid
passed in EventPipe messages, meaning that dotnet.gcdump's pid checks
currently breaks that scenario.

This PR extends diagnostic tooling support for mobile and
dotnet-dsrouter scenarios.

* Add support for --diagnostic-port argument in dotnet-gcdump, but only
support connect mode, since listen mode (reverse diagnostic server) is
mainly for startup tracing where GC dump is not full supported.
* Add support for new command, convert, in dotnet-gcdump that can take a
nettrace file as input and convert it into a gcdump file. Can be useful
if GC dumps have been captured by tools like dotnet-trace, meaning that
dotnet-gcdump will be able to convert it into a gcdump.
* Add ability to tell diagnostic tools currently supporting
--diagnostic-port command that process in -p is a dsrouter process. This
will make the tools construct a default IPC channel used by dsrouter
based on the -p parameter, simplify the connect scenarios against
dsrouter from tools a lot, since it can use the default IPC channel
setup by dsrouter.
* Always setup default IPC server channel in dsrouter if no specific
ipcs argument has been set.
* Fix dsrouter ctrl-c shutdown issue + additional error logging for
diagnostic.
[main] Update dependencies from dotnet/aspnetcore
[main] Update dependencies from dotnet/symstore
dotnet#4197)

This pull request updates the following dependencies

[marker]: <> (Begin:8fefa124-13dd-4c66-7dae-08d9c02d7834)
## From https://github.com/dotnet/source-build-reference-packages
- **Subscription**: 8fefa124-13dd-4c66-7dae-08d9c02d7834
- **Build**: 20230828.2
- **Date Produced**: August 28, 2023 1:15:33 PM UTC
- **Commit**: 26ce96327dd346534926c4551f8b8d62a6fc724f
- **Branch**: refs/heads/main

[DependencyUpdate]: <> (Begin)

- **Updates**:
-
**Microsoft.SourceBuild.Intermediate.source-build-reference-packages**:
[from 8.0.0-alpha.1.23424.1 to 8.0.0-alpha.1.23428.2][1]

[1]:
dotnet/source-build-reference-packages@93c2340...26ce963

[DependencyUpdate]: <> (End)


[marker]: <> (End:8fefa124-13dd-4c66-7dae-08d9c02d7834)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
dotnet-dsrouter can be a little hard to configure for mobile use cases
since it needs a number of arguments, both to setup its local IPC
client|server and the corresponding TCP client|server and what arguments
to use depends on what mobile platform and scenario the user runs.

There are currently a number of different scenarios described in
different sources of documentation:

Runtime docs:


https://github.com/dotnet/runtime/blob/main/docs/design/mono/diagnostics-tracing.md

iOS SDK docs:

https://github.com/xamarin/xamarin-macios/wiki/Profiling

Android SDK docs:


https://github.com/xamarin/xamarin-android/blob/main/Documentation/guides/tracing.md

They all fall into a number of predefined scenarios and a number of
"default" parameters that should be used.

This PR creates 4 new commands in dotnet-dsrouter to explicitly use the
defaults described in the documentation, ios-sim, ios, android-emu,
android. They all fallback to default IPC server listener for dsrouter
and in order to make that simpler in use with diagnostic tooling,
changes done in dotnet#4081 is also
needed to simplify the process.

So lets take an example form the docs, running an app on iOS simulator.
Before this PR the following dsrouter command needs to be run:

```
dotnet-dsrouter client-server -ipcc ~/my-sim-port -tcps 127.0.0.1:9000

launch app with DOTNET_DiagnosticPorts=127.0.0.1:9000

dotnet-trace collect --diagnostic-port ~/my-sim-port --format speedscope
```

With this PR (and dotnet#4081) the
above will look like:

```
dotnet-dsrouter ios-sim

launch app with DOTNET_DiagnosticPorts=127.0.0.1:9000

dotnet-trace collect -p:<dsrouter pid> --format speedscope
```

dontet-dsrouter will output both its pid as well as what
DOTNET_DiagnosticPorts that can be used to connect to it on startup.

Using a different mobile platform/scenario, like android emulator is
pretty much identical, just switch ios-sim to android-emu.
dotnet-dsrouter will output the exact DOTNET_DiagnosticPorts that should
be used with any of the configured scenarios.
[main] Update dependencies from dotnet/aspnetcore
[main] Update dependencies from dotnet/runtime
This pull request updates the following dependencies

[marker]: <> (Begin:cb58fe07-ae24-4e73-0e84-08d8e40a189f)
## From https://github.com/microsoft/clrmd
- **Subscription**: cb58fe07-ae24-4e73-0e84-08d8e40a189f
- **Build**: 20230829.1
- **Date Produced**: August 29, 2023 8:58:43 PM UTC
- **Commit**: 0c46ba9ce162de1cfc124379c0d9cde99ed4206f
- **Branch**: refs/heads/main

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.Diagnostics.Runtime**: [from 3.0.442202 to 3.0.442901][1]
- **Microsoft.Diagnostics.Runtime.Utilities**: [from 3.0.442202 to
3.0.442901][1]

[1]: microsoft/clrmd@c7ec730...0c46ba9

[DependencyUpdate]: <> (End)


[marker]: <> (End:cb58fe07-ae24-4e73-0e84-08d8e40a189f)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
[main] Update dependencies from dotnet/aspnetcore
[main] Update dependencies from dotnet/runtime
[main] Update dependencies from dotnet/runtime
[main] Update dependencies from dotnet/aspnetcore
[main] Update dependencies from dotnet/aspnetcore
dotnet#4208)

[main] Update dependencies from dotnet/source-build-reference-packages
dotnet-maestro bot and others added 29 commits October 3, 2023 13:21
[main] Update dependencies from dotnet/aspnetcore
This pull request updates the following dependencies

[marker]: <> (Begin:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)
## From https://github.com/dotnet/runtime
- **Subscription**: e4bfb556-e13c-47f6-eb5a-08d8e4d5099b
- **Build**: 20231002.11
- **Date Produced**: October 3, 2023 11:06:54 AM UTC
- **Commit**: ef6283ac0a14c78d9e9fef4841545099bd7ad12b
- **Branch**: refs/heads/release/8.0

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.NETCore.App.Runtime.win-x64**: [from 8.0.0-rtm.23478.17 to
8.0.0-rtm.23502.11][1]
- **VS.Redist.Common.NetCore.SharedFramework.x64.8.0**: [from
8.0.0-rtm.23478.17 to 8.0.0-rtm.23502.11][1]

[1]: dotnet/runtime@b20f704...ef6283a

[DependencyUpdate]: <> (End)


[marker]: <> (End:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
[main] Update dependencies from dotnet/aspnetcore
This pull request updates the following dependencies

[marker]: <> (Begin:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)
## From https://github.com/dotnet/runtime
- **Subscription**: e4bfb556-e13c-47f6-eb5a-08d8e4d5099b
- **Build**: 20231003.15
- **Date Produced**: October 4, 2023 5:51:18 AM UTC
- **Commit**: a84f8ffbf5d597b8a91e893a1f413466de017a68
- **Branch**: refs/heads/release/8.0

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.NETCore.App.Runtime.win-x64**: [from 8.0.0-rtm.23502.11 to
8.0.0-rtm.23503.15][1]
- **VS.Redist.Common.NetCore.SharedFramework.x64.8.0**: [from
8.0.0-rtm.23502.11 to 8.0.0-rtm.23503.15][1]

[1]: dotnet/runtime@ef6283a...a84f8ff

[DependencyUpdate]: <> (End)


[marker]: <> (End:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
…port (dotnet#4285)

Add extension load testing that includes duplicate command names.

Add internal CommandGroup class to CommandService

dotnet-dump analyze -c/--command will exit on any parsing errors or
exceptions in the command

command help not displayed on parsing errors/invalid options

Add INIT_API_CHECK_MANAGED native SOS command macro

Rename clrmodules command to assemblies and keep clrmodules as an alias

Support static HelpInvoke/FilterInvoke methods

Add command service testing

Better command service help interface. Returns help text instead of
printing it on the console directly. ICommandService.DisplayHelp() =>
GetHelpText(). Better help sorting.

Fix some SOS scripts

Return error code for command line option errors in native SOS

Load next to executing assembly first when hosted under desktop
Framework

Test using "clrthreads" instead of "Threads"

Replace testing "u" with "clru"

Add more general command filter mechanism. Add FilterType property to
Command attribute. Remove OS filter command flags.

NativeAddressHelper service work

Move Windows managed command stubs to separate file

Add SOS.Hosting services to SOS.Extensions for better native command
help. This requires the special ManagedOnlyCommandFilter service to
prevent recursion of the C++ commands. On Windows this recursion is from
"C++ command" -> "checking if managed version" -> "executing the command
in SOS.Hosting" -> "C++ command".

Help is now uniform across managed/native, alphabetized and filtered by
the current runtime.
This pull request updates the following dependencies

[marker]: <> (Begin:cb58fe07-ae24-4e73-0e84-08d8e40a189f)
## From https://github.com/microsoft/clrmd
- **Subscription**: cb58fe07-ae24-4e73-0e84-08d8e40a189f
- **Build**: 20231004.2
- **Date Produced**: October 4, 2023 10:57:18 PM UTC
- **Commit**: 10828baeeb7a144414259d9c00a9923c2fcb61e7
- **Branch**: refs/heads/main

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.Diagnostics.Runtime**: [from 3.0.450101 to 3.1.450402][3]
- **Microsoft.Diagnostics.Runtime.Utilities**: [from 3.0.450101 to
3.1.450402][3]

[3]: microsoft/clrmd@1c8f0f1...10828ba

[DependencyUpdate]: <> (End)


[marker]: <> (End:cb58fe07-ae24-4e73-0e84-08d8e40a189f)

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
This pull request updates the following dependencies

[marker]: <> (Begin:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)
## From https://github.com/dotnet/runtime
- **Subscription**: e4bfb556-e13c-47f6-eb5a-08d8e4d5099b
- **Build**: 20231004.8
- **Date Produced**: October 4, 2023 10:14:40 PM UTC
- **Commit**: 22b8a5665f5725a2c7bb09cfbe88f2cdc9847c1a
- **Branch**: refs/heads/release/8.0

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.NETCore.App.Runtime.win-x64**: [from 8.0.0-rtm.23503.15 to
8.0.0-rtm.23504.8][1]
- **VS.Redist.Common.NetCore.SharedFramework.x64.8.0**: [from
8.0.0-rtm.23503.15 to 8.0.0-rtm.23504.8][1]

[1]: dotnet/runtime@a84f8ff...22b8a56

[DependencyUpdate]: <> (End)


[marker]: <> (End:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
This pull request updates the following dependencies

[marker]: <> (Begin:319094f3-ed78-47c4-53e7-08d8e409d87d)
## From https://github.com/dotnet/aspnetcore
- **Subscription**: 319094f3-ed78-47c4-53e7-08d8e409d87d
- **Build**: 20231004.12
- **Date Produced**: October 4, 2023 8:08:32 PM UTC
- **Commit**: 91314b6bde6e9823f848fb37100aea5df4e44c99
- **Branch**: refs/heads/release/8.0

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.AspNetCore.App.Ref**: [from 8.0.0-rtm.23503.8 to
8.0.0-rtm.23504.12][1]
- **Microsoft.AspNetCore.App.Ref.Internal**: [from 8.0.0-rtm.23503.8 to
8.0.0-rtm.23504.12][1]

[1]:
dotnet/aspnetcore@2d2f880...91314b6

[DependencyUpdate]: <> (End)


[marker]: <> (End:319094f3-ed78-47c4-53e7-08d8e409d87d)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
[main] Update dependencies from dotnet/aspnetcore
This pull request updates the following dependencies

[marker]: <> (Begin:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)
## From https://github.com/dotnet/runtime
- **Subscription**: e4bfb556-e13c-47f6-eb5a-08d8e4d5099b
- **Build**: 20231006.12
- **Date Produced**: October 7, 2023 3:40:26 AM UTC
- **Commit**: b17a34c818bd5e01fdb9827fea64727a5fc51025
- **Branch**: refs/heads/release/8.0

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.NETCore.App.Runtime.win-x64**: [from 8.0.0-rtm.23504.8 to
8.0.0-rtm.23506.12][1]
- **VS.Redist.Common.NetCore.SharedFramework.x64.8.0**: [from
8.0.0-rtm.23504.8 to 8.0.0-rtm.23506.12][1]

[1]: dotnet/runtime@22b8a56...b17a34c

[DependencyUpdate]: <> (End)


[marker]: <> (End:e4bfb556-e13c-47f6-eb5a-08d8e4d5099b)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
This pull request updates the following dependencies

[marker]: <> (Begin:319094f3-ed78-47c4-53e7-08d8e409d87d)
## From https://github.com/dotnet/aspnetcore
- **Subscription**: 319094f3-ed78-47c4-53e7-08d8e409d87d
- **Build**: 20231006.5
- **Date Produced**: October 7, 2023 1:42:18 AM UTC
- **Commit**: 6bd37340c54e3b9690102886afca6198a461cb3e
- **Branch**: refs/heads/release/8.0

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.AspNetCore.App.Ref**: [from 8.0.0-rtm.23505.1 to
8.0.0-rtm.23506.5][1]
- **Microsoft.AspNetCore.App.Ref.Internal**: [from 8.0.0-rtm.23505.1 to
8.0.0-rtm.23506.5][1]

[1]:
dotnet/aspnetcore@51f80c7...6bd3734

[DependencyUpdate]: <> (End)


[marker]: <> (End:319094f3-ed78-47c4-53e7-08d8e409d87d)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
[main] Update dependencies from dotnet/arcade


 - Fix build from new analysis checks
This pull request updates the following dependencies

[marker]: <> (Begin:638f1194-0c1a-4d47-eb59-08d8e4d5099b)
## From https://github.com/dotnet/installer
- **Subscription**: 638f1194-0c1a-4d47-eb59-08d8e4d5099b
- **Build**: 20231006.1
- **Date Produced**: October 6, 2023 9:36:48 PM UTC
- **Commit**: 0ffc9fdc93e578268a09b0dccdc4c3527f4697f3
- **Branch**: refs/heads/release/8.0.1xx

[DependencyUpdate]: <> (Begin)

- **Updates**:
- **Microsoft.Dotnet.Sdk.Internal**: [from 8.0.100-rtm.23479.3 to
8.0.100-rtm.23506.1][1]

[1]: dotnet/installer@6dae849...0ffc9fd

[DependencyUpdate]: <> (End)


[marker]: <> (End:638f1194-0c1a-4d47-eb59-08d8e4d5099b)

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Mike McLaughlin <[email protected]>
SOS does get built dynamically linking to the C++ CRT in release so the
invalid parameter handler that gets installed affects all the code in
the process.

Fixes issue: dotnet#4070
[main] Update dependencies from microsoft/clrmd
[main] Update dependencies from dotnet/aspnetcore
[main] Update dependencies from dotnet/runtime
This pull request updates the following dependencies

[marker]: <> (Begin:678f7c5b-6647-4e77-0d75-08d8e40a4c7c)
## From https://github.com/dotnet/symstore
- **Subscription**: 678f7c5b-6647-4e77-0d75-08d8e40a4c7c
- **Build**: 20231009.1
- **Date Produced**: October 9, 2023 4:51:26 PM UTC
- **Commit**: a3b341f9e61c8d8e832c4acfeb5b3a2305e51bcc
- **Branch**: refs/heads/main

[DependencyUpdate]: <> (Begin)

- **Updates**:
  - **Microsoft.SymbolStore**: [from 1.0.446801 to 1.0.450901][1]

[1]: dotnet/symstore@8cc6f03...a3b341f

[DependencyUpdate]: <> (End)


[marker]: <> (End:678f7c5b-6647-4e77-0d75-08d8e40a4c7c)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
dotnet#4309)

This pull request updates the following dependencies

[marker]: <> (Begin:8fefa124-13dd-4c66-7dae-08d9c02d7834)
## From https://github.com/dotnet/source-build-reference-packages
- **Subscription**: 8fefa124-13dd-4c66-7dae-08d9c02d7834
- **Build**: 20231009.2
- **Date Produced**: October 9, 2023 1:15:04 PM UTC
- **Commit**: 8a2f652b1f23d493fcce31b73e829de56df38d5f
- **Branch**: refs/heads/main

[DependencyUpdate]: <> (Begin)

- **Updates**:
-
**Microsoft.SourceBuild.Intermediate.source-build-reference-packages**:
[from 9.0.0-alpha.1.23502.1 to 9.0.0-alpha.1.23509.2][1]

[1]:
dotnet/source-build-reference-packages@05ffbf9...8a2f652

[DependencyUpdate]: <> (End)


[marker]: <> (End:8fefa124-13dd-4c66-7dae-08d9c02d7834)

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
fixing some incorrect info for the explaination of the -allReady arg for the !fq cmd.
[main] Update dependencies from microsoft/clrmd
[main] Update dependencies from dotnet/aspnetcore
dotnet#4313)

[main] Update dependencies from dotnet/source-build-reference-packages
@kkeirstead kkeirstead closed this Oct 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.