Mark Russinovich Technical Fellow Windows Azure WCAB306 Outline Introduction Sluggish Performance Error Messages Application Crashes Blue Screens Case of the Unexplained This is the ID: 404979
Download Presentation The PPT/PDF document "The Case of the Unexplained…" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
The Case of the Unexplained…
Mark RussinovichTechnical FellowWindows Azure
WCA-B306Slide2
Outline
IntroductionSluggish PerformanceError Messages
Application
Crashes
Blue ScreensSlide3
Case of the Unexplained…
This is the 2013
version of the “case of the unexplained” talk series
Previous versions covered different cases
Can view webcast on Sysinternals->Mark’s webcasts
Based on real case studies
Some of these have been written up on my blogSlide4
Troubleshooting
Most applications do a poor job of reporting unexpected errors
Locked, missing or corrupt files
Missing or corrupt registry data
Permissions problems
Errors manifest in several different ways
Misleading error messages
Crashes or hangsSlide5
Purpose of Talk
Show you how to solve these classes of problems by peering beneath the surface
Interpreting process, file and registry activity
Interpreting call stacks
You’ll learn tools and techniques to help you solve seemingly unsolvable problemsSlide6
Tools We’ll Use
Sysinternals: www.microsoft.com/technet/sysinternals
(
\\
redmond
\files\SYSINTERNALS\LBI\Latest
)
Process Explorer – process/thread viewer
Process Monitor – file/registry/process/thread
tracing
Procdump
– process memory dumper
Autoruns
– displays all
autostart
locations
SigCheck
– shows file version information
PsExec
– execute processes remotely or in the system account
TcpView
– shows TCP/IP endpoints
Strings – dumps printable strings in any file
Zoomit
– presentation tool I’m using
Microsoft downloads:
Debugging Tools for Windows:
Windbg
application and kernel debugger:
www.microsoft.com/whdc/devtools/debugging
(
//dbg
) Slide7
The Sysinternals Administrator’s Reference
The official guide to the Sysinternals tools
Covers every tool, every feature, with tips
Written by
markruss
and aaronmar
Full chapters on the major tools:
Process Explorer
Process Monitor
Autoruns
Other chapters by tool group
Security, process, AD, desktop, …Slide8
Outline
Sluggish Performance
Error Messages
Application Crashes
Blue ScreensSlide9
The Case of the Sluggish Explorer
Explorer was very slow to display the contents of local filesCaptured a Process Monitor traceSlide10
Process Monitor
Process Monitor is a real-time file, registry, process and thread monitorWhen in doubt, run Process Monitor!
It will often show you the cause for error messages
It many times tells you what is causing sluggish performanceSlide11
Process Monitor Enhancements: Bookmarks
Bookmarking enables you to save markers in the trace:
Use F6 to find the next one, Shift+F6 to search upSlide12
The Case of the Sluggish Explorer (
Cont)Noticed numerous references to network path with NAME NOT FOUND:Slide13
The Case of the Sluggish Explorer (Cont)
Checked account mapping and saw that home drive was network share:
Confirmed in Process Explorer
current directory was the network
share:Slide14
The Case of the Sluggish Explorer: Solved
Opened command prompt, killed Explorer, and restarted from local directory: no delaysContacted IT and had them change home directory: problem solvedSlide15
Process Explorer
Process Explorer is a Task Manager replacement
You can literally replace Task Manager with Options->Replace Task Manager
Hide-when-minimized to always have it handy
Hover the mouse to see a tooltip showing the process consuming the most CPU
Open System Information graph to see CPU usage history
Graphs are time stamped with hover showing biggest consumer at point in time
Also includes other activity such as I/O, kernel memory limitsSlide16
Process Explorer v15:
GPU Monitoring and Windows 8
Captures GPU utilization and memory usage
System-wide
Per-ProcessSlide17
Process Explorer v15.2
Process timelines
Autostart
locationsSlide18
Process Explorer v15.3
Process view heat map columns
Bit-matching dump files
Disk counters fixed for Windows 8Slide19
The Case of the Spiking ICQ Process
System was sluggishOpened Process Explorer and saw Instant messaging application consuming CPU:Slide20
Processes and Threads
A process represents an instance of a running programAddress space
Resources (e.g., open handles)
Security profile (token)
A thread is an execution context within a process
Unit of scheduling (threads run, processes don’t run)
All threads in a process share the same per-process address space
The System process is the default home for kernel mode system threads
Functions in OS and some drivers that need to run as real threads
E.g., need to run concurrently with other system activity, wait on timers, perform background “housekeeping” work
Other host processes:
svchost
,
Iexplore
,
mmc
,
dllhostSlide21
Viewing Threads
Task Manager doesn’t show thread details within a processProcess Explorer does on “Threads” tab
Displays thread details such as ID, CPU usage, start time, state, priority
Start address is where the thread began running (not where it is now)
Click Module to get details on module containing thread start addressSlide22
Thread Start Functions and Symbol Information
Process Explorer can map the addresses within a module to the names of functions
This can help identify which component within a process is responsible for CPU usage
Configure Process Explorer’s symbol engine:
Download the latest Debugging Tools for Windows from Microsoft (free)
Use dbghelp.dll from the Debugging Tools
Point at the Microsoft public symbol server (or internal symbol server if you have access)Slide23
The Case of the Spiking ICQ Process (
Cont)Looked at threads tab
Start address of spiking threads were generic:Slide24
Viewing Call Stacks
Click Stack on the Threads tab to view a thread’s call stackNote that start address on Threads tab is different than first function shown in stack
This is because all threads created by Windows programs start in a library function in Kernel32.dll which calls the programmed start addressSlide25
The Case of the Spiking ICQ Process (
Cont)Looked at thread stack and saw video DLL:Slide26
The Case of the Spiking ICQ Process (
Cont)
Opened DLL view and saw that each time issue occurred, four video DLLs were loaded:
Noted that DLLs were not loaded on program startup
Opened Add/Remove programs, but saw no software from
ViscomSlide27
Autoruns
Shows every place in the system that can be configured to run something at boot & logon
Standard Run keys and Startup folders
Shell,
userinit
Services and drivers
Tasks
Winlogon
notifica`tions
Explorer and IE
addins
(toolbars, Browser Helper Objects, …)
More and ever growing…
Each startup category has its own tab and all items display on the Everything tab
Startup name, image description, company and pathSlide28
Identifying Malware Autostarts
Zoom-in on add-ons (including malware) by selecting these filter options:Verify Code Signatures
Hide Microsoft Entries
Select an item to see more in the lower window
Online search unknown images
Double-click on an item to look at where its configured in the Registry or file system
Has other features:
Can also show empty locations (informational only)
Includes compare functionality
Includes equivalent command-line version, Autorunsc.exeSlide29
The Case of the Spiking ICQ Process: Solved
Opened Autoruns and disabled all Viscom DLLs:
Problem solvedSlide30
The Case of the Company 15-Minute Logons
Logons were taking 5-15 minutes across a company for one OS imageThe other OS image had normal logon time
Execs started to complain
Admins tried everything:
Examined event viewer
Captured Wireshark traces
Checked for malware
Finally, another admin that had seen “Case of the Unexplained” decided to help:
Captured a Process Monitor boot traceSlide31
The Case of the Company 15-Minute Logons (
Cont)
Looked through process tree and noticed
Framepkg
taking four minutes:
Command-line showed it was McAfee A/V reinstallationSlide32
The Case of the Company 15-Minute Logons: Solved
Question was: why was script reinstalling A/V?Contacted server admins:
They discovered that Active Directory OU for OS image had GPO
GPO should have been removed from domain after Win7 rollout
Removed GPO: problem solvedSlide33
Outline
Sluggish Performance
Error Messages
Application
Crashes
Blue ScreensSlide34
The Case of the File in Use Error
Tried deleting PowerPoint deck I had been editing, but got an error:
No sign of PowerPoint running,
not even in Task Manager’s applications tabSlide35
The Case of the File in Use Error: Solved
Did a Handle Search in Process Explorer
Found hidden PowerPoint process:
Terminated process: problem solvedSlide36
The Case of the Mistaken Windows-to-Go
User tried to use the Windows 8 AppStore, but got an error message:
Had installed Windows on a Maxtor SATA 2 disk, but for some reason Windows thought it was a portable USB diskSlide37
The Case of the Mistaken Windows-to-Go: Solved
Captured a Process Monitor trace when trying to access the store and scanned the log
Found this reference:
Deleted the
PortableOperatingSystem
value: problem solvedSlide38
The Case of the Default Win8 RDP Lockscreen Background
Set Windows 8 lockscreen
to custom background
When
RDP’ing
into the system, default
lockscreen
showed
Needed to find out where default was coming from
Captured a Process monitor trace of locking system and
RDP’ing
into systemSlide39
The Case of the Default Win8 RDP Lockscreen Background (Cont)
Set a filter to “Path Begins With .jpg”:
RDP
LockSlide40
The Case of the Default Win8 RDP
Lockscreen Background: Solved
Had to replace system version of
lockscreen
background
Launched command prompt as system using
PsExec
:
Copied background over RDP
lockscreen
file:
Problem solvedSlide41
Outline
Sluggish Performance
Error Messages
Application Crashes
Blue ScreensSlide42
The Case of the Crashing IE Favorites
IE was crashing every time Favorites was clicked
Ran IE8 without Add-ons
Still Crashed
Upgraded to IE9
Still Crashed
Ran IE9 without Add-ons
Still
CrashedSlide43
The Case of the Crashing IE Favorites (
Cont)
Used Process Explorer to get the PID of iexplore.exe:Slide44
The Case of the Crashing IE Favorites (
Cont)Used ProcDump to capture a unhandled
exception (-e) mini dumpSlide45
The Case of the Crashing IE Favorites (
Cont)Changed to the Exception Context Record (.ecxr) and looked at the stack (k) – looked like it was sortingSlide46
The Case of the Crashing IE Favorites (
Cont)Suspected one favorite that had bullets:
Moved all of the Favorites to another folder
Still CrashedSlide47
The Case of the Crashing IE Favorites (
Cont)Captured a Process Monitor trace
Set a filter for IE, ‘Action’ is SUCCESS and ‘Path’ contains ‘Favorites’, and RegistrySlide48
The Case of the Crashing IE Favorites (cont.)
Saw Order REG_BINARY value read from HKCU\Software\Microsoft\Windows\CurrentVersion
\Explorer\
MenuOrder
\Favorites:Slide49
The Case of the Crashing IE Favorites: Solved
Used “Jump to” context menu to open Regedit
Renamed ‘Order’ value to ‘
xOrder
’
Restarted IE: problem solved
Moved the favorites back
Looked at
RegEdit
, new ‘Order’ value madeSlide50
Outline
Application Hangs
Sluggish Performance
Error Messages
Blue ScreensSlide51
Blue Screen Crashes
Windows has various components that run in Kernel Mode, the highest privilege mode of the OSOS components: Ntoskrnl.exe, Hal.dll
Drivers: Ntfs.sys, Tcpip.sys, device drivers
Kernel-mode components are privileged extensions to the OS have to adhere to various rules
Not accessing invalid memory
Accessing memory at the right “Interrupt Request Level”
Not causing resource deadlocks
When a kernel-mode component performs an illegal operation, Windows crashes (blue screens)
Crashing helps preserve the integrity of user data
A resource deadlock can hang the systemSlide52
Online Crash Analysis
When you reboot after a crash, Windows offers to upload it to Microsoft Online Crash Analysis (OCA)
Automated server generates a thumbprint of the crash and uses it as a key in a database
If the database has an entry, the user is told the cause and directed at a fixSlide53
Basic Crash Dump Analysis
Many times OCA doesn’t know the cause:
Basic crash dump analysis is easy and it might tell you the cause
Requires
Windbg
and symbol configuration
Dump files are in either:
\Windows\
Memory.dmp
: Vista+ and servers
\Windows\
Minidump
: Windows 2000 Pro, Windows XP, Vista+ Slide54
The Case of the Windows 8 IE BSODs
Users complained that Windows 8 would crash when running IE Administrator analyzed dump file:Slide55
The Case of the Windows 8 IE BSODs: Solved
Did a web search and found that Avast
had just posted an update:
Applied update: problem solvedSlide56
Summary and More Information
A few basic tools and techniques can solve seemingly impossible problems
I learn by always trying to determine the root cause
Resources:
Sysinternals Administrator’s Reference
Webcasts of two previous “Case of the Unexplained “ talked
Sysinternals
->Mark’s Webcasts
My blog
Windows Internals: understand the way the OS works
If you’ve solved one, send me a description, screenshots and log files!Slide57
Windows 8 Bluescreens