Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Tests failing with 0xC0000142 14 Windows ARM64 checked coreclr test runs #54948

Closed
ericstj opened this issue Jun 30, 2021 · 8 comments
Closed

Tests failing with 0xC0000142 14 Windows ARM64 checked coreclr test runs #54948

ericstj opened this issue Jun 30, 2021 · 8 comments
Labels
area-Infrastructure-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly untriaged New issue has not been triaged by the area owner

Comments

@ericstj
Copy link
Member

ericstj commented Jun 30, 2021

The run log from helix is https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54890-merge-4746bd2a9072476897/profiler.gc/4a2812f6-4351-4f10-bb7f-d6339b705ee0.log?sv=2019-07-07&se=2021-07-20T03%3A47%3A09Z&sr=c&sp=rl&sig=DxAU9hk8WxZk31w0QVu09XUj6lqHsxrYIz8sgvrP%2BU4%3D

Interesting portion:

2021-06-30T05:05:13.574Z	INFO   	executor(771)	_execute_command	running %HELIX_CORRELATION_PAYLOAD%\scripts\950a34405e0340fea670d6e22f8326c7\execute.cmd in D:\h\w\A02A08CF\w\B08A097A\e max 1800 seconds
2021-06-30T05:05:14.320Z	INFO   	job(38)	wait	Work item's process exited normally.
...
2021-06-30T05:11:04.931Z	INFO   	executor(744)	_dump_file_upload	No dump files to upload
2021-06-30T05:11:04.937Z	INFO   	executor(815)	_execute_command	Finished _execute_command, exit code: 3221225794
2021-06-30T05:11:04.952Z	INFO   	executor(843)	_upload_single_file	Uploading the file console.845014af.log

Not sure what that last execute is since there was a previous log about the workitem completing successfully, but the return value 3221225794, is 0xC0000142 which seems to be the failure.

Log content is useless: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54890-merge-4746bd2a9072476897/profiler.gc/console.845014af.log?sv=2019-07-07&se=2021-07-20T03%3A47%3A09Z&sr=c&sp=rl&sig=DxAU9hk8WxZk31w0QVu09XUj6lqHsxrYIz8sgvrP%2BU4%3D
@dotnet-issue-labeler dotnet-issue-labeler bot added area-Diagnostics-coreclr untriaged New issue has not been triaged by the area owner labels Jun 30, 2021
@ghost
Copy link

ghost commented Jun 30, 2021

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

The run log from helix is https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54890-merge-4746bd2a9072476897/profiler.gc/4a2812f6-4351-4f10-bb7f-d6339b705ee0.log?sv=2019-07-07&se=2021-07-20T03%3A47%3A09Z&sr=c&sp=rl&sig=DxAU9hk8WxZk31w0QVu09XUj6lqHsxrYIz8sgvrP%2BU4%3D

Interesting portion:

2021-06-30T05:05:13.574Z	INFO   	executor(771)	_execute_command	running %HELIX_CORRELATION_PAYLOAD%\scripts\950a34405e0340fea670d6e22f8326c7\execute.cmd in D:\h\w\A02A08CF\w\B08A097A\e max 1800 seconds
2021-06-30T05:05:14.320Z	INFO   	job(38)	wait	Work item's process exited normally.
...
2021-06-30T05:11:04.931Z	INFO   	executor(744)	_dump_file_upload	No dump files to upload
2021-06-30T05:11:04.937Z	INFO   	executor(815)	_execute_command	Finished _execute_command, exit code: 3221225794
2021-06-30T05:11:04.952Z	INFO   	executor(843)	_upload_single_file	Uploading the file console.845014af.log

Not sure what that last execute is since there was a previous log about the workitem completing successfully, but the return value 3221225794, is 0xC0000142 which seems to be the failure.

Log content is useless: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54890-merge-4746bd2a9072476897/profiler.gc/console.845014af.log?sv=2019-07-07&se=2021-07-20T03%3A47%3A09Z&sr=c&sp=rl&sig=DxAU9hk8WxZk31w0QVu09XUj6lqHsxrYIz8sgvrP%2BU4%3D

Author: ericstj
Assignees: -
Labels:

area-Diagnostics-coreclr, untriaged
Milestone: -

@ericstj ericstj mentioned this issue Jun 30, 2021
Merged
@ericstj ericstj changed the title 'profiler.gc' test failure without log on Windows ARM64 Tests failing with 0xC0000142 14 Windows ARM64 checked coreclr test runs Jul 1, 2021
@ericstj
Copy link
Member Author

ericstj commented Jul 1, 2021

Seems to be similar issue effecting multiple tests.

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54950-merge-f076bf138036491bb1/profiler.transitions/e4c162e4-747e-49cb-abdb-f41c8fdf7ee9.log?sv=2019-07-07&se=2021-07-20T21%3A27%3A04Z&sr=c&sp=rl&sig=itwnIxMu85FQLVsObrGLZ%2B%2BcAYBsFJCuI6S5K61o9gA%3D

2021-06-30T21:50:04.929Z	INFO   	executor(815)	_execute_command	Finished _execute_command, exit code: 3221225794

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54950-merge-62be768a085a4868ab/PayloadGroup0/00d8698c-bbf1-425d-bc05-8bd9402b3034.log?sv=2019-07-07&se=2021-07-21T01%3A04%3A45Z&sr=c&sp=rl&sig=mRk%2ByTyPYELUmmFk2hJzrenPoFO3KuujTIUKYgt1I5o%3D

2021-07-01T01:21:09.749Z	INFO   	executor(815)	_execute_command	Finished _execute_command, exit code: 3221225794

0xC0000142 is either STATUS_DLL_INIT_FAILED or ERROR_DEVICE_NO_RESOURCES.

Given lack of a dump, I'm suspecting the latter. @MattGal @ViktorHofer do we have any resource issues on Windows ARM64 machines?

@ghost
Copy link

ghost commented Jul 1, 2021

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

The run log from helix is https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54890-merge-4746bd2a9072476897/profiler.gc/4a2812f6-4351-4f10-bb7f-d6339b705ee0.log?sv=2019-07-07&se=2021-07-20T03%3A47%3A09Z&sr=c&sp=rl&sig=DxAU9hk8WxZk31w0QVu09XUj6lqHsxrYIz8sgvrP%2BU4%3D

Interesting portion:

2021-06-30T05:05:13.574Z	INFO   	executor(771)	_execute_command	running %HELIX_CORRELATION_PAYLOAD%\scripts\950a34405e0340fea670d6e22f8326c7\execute.cmd in D:\h\w\A02A08CF\w\B08A097A\e max 1800 seconds
2021-06-30T05:05:14.320Z	INFO   	job(38)	wait	Work item's process exited normally.
...
2021-06-30T05:11:04.931Z	INFO   	executor(744)	_dump_file_upload	No dump files to upload
2021-06-30T05:11:04.937Z	INFO   	executor(815)	_execute_command	Finished _execute_command, exit code: 3221225794
2021-06-30T05:11:04.952Z	INFO   	executor(843)	_upload_single_file	Uploading the file console.845014af.log

Not sure what that last execute is since there was a previous log about the workitem completing successfully, but the return value 3221225794, is 0xC0000142 which seems to be the failure.

Log content is useless: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-54890-merge-4746bd2a9072476897/profiler.gc/console.845014af.log?sv=2019-07-07&se=2021-07-20T03%3A47%3A09Z&sr=c&sp=rl&sig=DxAU9hk8WxZk31w0QVu09XUj6lqHsxrYIz8sgvrP%2BU4%3D

Author: ericstj
Assignees: -
Labels:

area-Infrastructure-coreclr, untriaged
Milestone: -

@ericstj ericstj added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Jul 1, 2021
@ericstj ericstj mentioned this issue Jul 1, 2021
Closed
@hoyosjs hoyosjs added the tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly label Jul 1, 2021
@hoyosjs
Copy link
Member

hoyosjs commented Jul 1, 2021

Odd, 0xC0000142 is either STATUS_DLL_INIT_FAILED or ERROR_DEVICE_NO_RESOURCES. I'll ask dnceng to check on this device.

@ericstj
Copy link
Member Author

ericstj commented Jul 1, 2021

Yeah, all seem to be on DDARM64-119. 2 new instances here: https://dev.azure.com/dnceng/public/_build/results?buildId=1214460&view=ms.vss-test-web.build-test-results-tab&runId=36263728&resultId=102813&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

@ericstj ericstj mentioned this issue Jul 1, 2021
Merged
@runfoapp runfoapp bot mentioned this issue Jul 1, 2021
Closed
@MattGal
Copy link
Member

MattGal commented Jul 1, 2021

There were a bunch of files missing on that machine from %WINDIR%\system32, looks like @greenEkatherine already offlined it and we'll get it reimaged.

@hoyosjs
Copy link
Member

hoyosjs commented Jul 11, 2021

Closing as it hasn't happened again

@hoyosjs hoyosjs closed this as completed Jul 11, 2021
@ericstj
Copy link
Member Author

ericstj commented Jul 12, 2021

Thanks folks

@ghost ghost locked as resolved and limited conversation to collaborators Aug 11, 2021
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Assignees
No one assigned
Labels
area-Infrastructure-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly untriaged New issue has not been triaged by the area owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MattGal @ericstj @hoyosjs