From: Santiago Fernandez Madero Date: Wed, 4 Nov 2020 20:28:33 +0000 (-0800) Subject: Add debugging a ci dump documentation template and script to fill info and include... X-Git-Tag: submit/tizen/20210909.063632~4775 X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=070323db0d547cc0d97167c4d457b5fbca8cb527;p=platform%2Fupstream%2Fdotnet%2Fruntime.git Add debugging a ci dump documentation template and script to fill info and include it on azdo test tab (#43937) * Add debugging a ci dump documentation template and script to generate it on helix * PR Feedback and fix helix workitems * Fix python script and -buildid argument * Upload doc only if dumps were found * Pass down templatedir * PR feedback --- diff --git a/eng/testing/debug-dump-template.md b/eng/testing/debug-dump-template.md new file mode 100644 index 0000000..bc8a296 --- /dev/null +++ b/eng/testing/debug-dump-template.md @@ -0,0 +1,127 @@ +# Debugging a CI dump + +This document describes how to debug a CI/PR test dump by downloading assets from helix, using a dotnet tool called `runfo`. + +## What is runfo? + +Runfo is a dotnet global tool that helps get information about helix test runs and azure devops builds. For more information see [this](https://github.com/jaredpar/runfo/tree/master/runfo#runfo) + +### How do I install it? + +You just need to run: + +```script +dotnet tool install --global runfo +``` + +If you already have it installed, make sure you have at least version `0.6.1` installed, which contains support to download helix payloads. If you don't have the latest version just run: + +```script +dotnet tool update --global runfo +``` + +## Download helix payload containing symbols: + +You can just achieve this by running: + +```script +runfo get-helix-payload -j %JOBID% -w %WORKITEM% -o +``` + +> NOTE: if the helix job is an internal job, you need to pass down a [helix authentication token](https://helix.dot.net/Account/Tokens) using the `--helix-token` argument. + +This will download the workitem contents under `\workitems\` and the correlation payload under: `\correlation-payload\`. + +> The correlation payload is usually the testhost or core root, which contain the runtime and dotnet host that we use to run tests. + +Once you have those assets, you will need to extract the testhost or core root. Then extract the workitem assets into the same location where coreclr binary is. + +## Windows dump on Windows + +### Debug with WinDbg + +1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos). +2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios). +3. Load the dump with a recent WinDbg version for it to load sos automatically (if not you can run `.update sos`). It is important that bitness of WinDbg matches the bitness of the dump. +4. Then run the following commands: + +```script +!setclrpath +.sympath+ +``` +### Analyze with dotnet-dump + +1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump). +2. Run: `dotnet-dump analyze ` +3. Then run the following commands: + +```script +setclrpath (To verify an incorrect DAC hasn't been loaded). +setclrpath +setsymbolserver -directory +``` + +## Linux dumps on Windows + +In order to debug a Linux dump on Windows, you will have to first go to the PR/CI build +that sent the test run and download the cross DAC. + +Download the [`CoreCLRCrossDacArtifacts`](https://dev.azure.com/dnceng/public/_apis/build/builds/%BUILDID%/artifacts?artifactName=CoreCLRCrossDacArtifacts&api-version=6.0&%24format=zip), then extract it, and copy the matching flavor of the DAC with your dump and extract it in the same location where coreclr binary is. + +### Debug with WinDbg + +1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos). +2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios). +3. Load the dump with a recent WinDbg version for it to load sos automatically (if not you can run `.update sos`). It is important that bitness of WinDbg matches the bitness of the dump. +4. Then run the following commands: + +```script +!setclrpath +.sympath+ +``` +### Analyze with dotnet-dump + +1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump). +2. Run: `dotnet-dump analyze ` +3. Then run the following commands: + +```script +setclrpath (To verify an incorrect DAC hasn't been loaded). +setclrpath +setsymbolserver -directory +``` + +## Linux dumps on Linux + +### Debug with LLDB + +1. Install [dotnet-sos global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-sos). +2. Run `dotnet sos install` (This has an architecture flag to install diferent plugin versions for specific arch scenarios). +3. Load the dump by running `lldb -c ` +4. Run the following commands: + +```script +setclrpath +sethostruntime '' +setsymbolserver -directory +loadsymbols (if you want to resolve native symbols) +``` + +### Analyze with dotnet-dump + +1. Install [dotnet-dump global tool](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump). +2. Run: `dotnet-dump analyze ` +3. Then run the following commands: + +```script +setclrpath (To verify an incorrect DAC hasn't been loaded). +setclrpath +setsymbolserver -directory +``` + +## MacOS dumps + +Instructions for debugging dumps on MacOS the same as [Linux](#linux-dumps-on-linux); however there are a couple of caveats. + +1. It's only supported to debug them in `dotnet-dump` if it's a runtime generated dump. This includes hang dumps and dumps generated by `createdump`, `dotnet-dump` and the runtime itself. +2. If it's a system dump, then only `lldb` works. diff --git a/eng/testing/gen-debug-dump-docs.py b/eng/testing/gen-debug-dump-docs.py new file mode 100644 index 0000000..bd6798b --- /dev/null +++ b/eng/testing/gen-debug-dump-docs.py @@ -0,0 +1,104 @@ +import os +import sys +import platform + +build_id = '' +job_id = '' +workitem = '' +dump_dir = '' +template_dir = os.getcwd() +out_dir = template_dir +idx = 0 +args_len = len(sys.argv) +while idx < args_len: + arg = sys.argv[idx] + idx += 1 + if arg == '-buildid': + if idx >= args_len or sys.argv[idx].startswith('-'): + print("Must specify a value for -buildid") + exit(1) + + build_id = sys.argv[idx] + idx += 1 + + if arg == '-jobid': + if idx >= args_len or sys.argv[idx].startswith('-'): + print("Must specify a value for -jobid") + exit(1) + + job_id = sys.argv[idx] + idx += 1 + + if arg == '-workitem': + if idx >= args_len or sys.argv[idx].startswith('-'): + print("Must specify a value for -workitem") + exit(1) + + workitem = sys.argv[idx] + idx += 1 + + if arg == '-templatedir': + if idx >= args_len or sys.argv[idx].startswith('-'): + print("Must specify a value for -templatedir") + exit(1) + + template_dir = sys.argv[idx] + idx += 1 + + if arg == '-outdir': + if idx >= args_len or sys.argv[idx].startswith('-'): + print("Must specify a value for -outdir") + exit(1) + + out_dir = sys.argv[idx] + idx += 1 + + if arg == '-dumpdir': + if idx >= args_len or sys.argv[idx].startswith('-'): + print("Must specify a value for -dumpdir") + exit(1) + + dump_dir = sys.argv[idx] + idx += 1 + +found_dumps = False +if dump_dir != '': + for filename in os.listdir(dump_dir): + if filename.endswith('.dmp') or 'core.' in filename: + found_dumps = True + break + +if not found_dumps: + print("Did not find dumps, skipping dump docs generation.") + exit(0) + +if build_id == '': + print("ERROR: unespecified required argument -buildid") + exit(1) + +if workitem == '': + print("ERROR: unespecified required argument -workitem") + exit(1) + +if job_id == '': + print("ERROR: unespecified required argument -jobid") + exit(1) + +replace_string = '' +dir_separator = '/' if platform.system() != 'Windows' else '\\' +source_file = template_dir + dir_separator + 'debug-dump-template.md' +with open(source_file, 'r+') as f: + file_text = f.read() + + print('read file: ' + source_file) + + replace_string = file_text.replace('%JOBID%', job_id) + replace_string = replace_string.replace('%WORKITEM%', workitem) + replace_string = replace_string.replace('%BUILDID%', build_id) + +output_file = out_dir + dir_separator + 'how-to-debug-dump.md' +with open(output_file, 'w+') as output: + print('writing output file: ' + output_file) + write_file = output.write(replace_string) + +print('done writing debug dump information') diff --git a/src/libraries/sendtohelix.proj b/src/libraries/sendtohelix.proj index 29f1dcb..4d113c9 100644 --- a/src/libraries/sendtohelix.proj +++ b/src/libraries/sendtohelix.proj @@ -144,7 +144,22 @@ <_RuntimeInputs Condition=" '$(Scenarios)' != '' and '$(TargetsWindows)' != 'true' " Include="$(TestHostRootPath)**/*.sh" /> + + + + + + <_RuntimeInputs Include="@(DebugDocsFiles)" /> + + + + + + + $(HelixPostCommands); + %HELIX_PYTHONPATH% %HELIX_CORRELATION_PAYLOAD%\gen-debug-dump-docs.py -buildid $(BUILD_BUILDID) -workitem %HELIX_WORKITEM_ID% -jobid %HELIX_CORRELATION_ID% -outdir %HELIX_WORKITEM_UPLOAD_ROOT% -templatedir %HELIX_CORRELATION_PAYLOAD% -dumpdir %HELIX_DUMP_FOLDER% + + + $(HelixPostCommands); + $HELIX_PYTHONPATH $HELIX_CORRELATION_PAYLOAD/gen-debug-dump-docs.py -buildid $(BUILD_BUILDID) -workitem $HELIX_WORKITEM_ID -jobid $HELIX_CORRELATION_ID -outdir $HELIX_WORKITEM_UPLOAD_ROOT -templatedir $HELIX_CORRELATION_PAYLOAD -dumpdir $HELIX_DUMP_FOLDER + + +