Hi all,
I will appreciate your help since not an expert with Backup Exec.
Backup Exec 2010 R3 v13 Rev 5204 fully updated
Backups were working fine and suddenly started to fail with:
Final error: 0x000fedd A failure occurred when trying to access the object list
Backup - System?State
V-79-57344-65245 - A failure occurred accessing the object list
Exceptions:
Backup - System?State
The item System?State\System Files\System Files in use - skipped
Troubleshooting steps:
As per what I've found online I have restarted BE services, make sure there's enough space on HDD (92GB) and also I enable logging in BE remote agent.
The logs around the time the error has been reported are:
[35356] 12/13/13 17:43:56 [fsys\brfs] - Status 0x0000007B returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\program files (x86)\kaseya\pcgr9908646926167327\..\knetcollector\kaseya.collector.exe in brUtil::GenerateFileList
[35356] 12/13/13 17:43:56 [fsys\brfs] - brUtil::SetObjectsList( ): Failed to apply the component exclusion list
[35356] 12/13/13 17:43:56 [fsys\shadow] - Status CODE (0x0000007B) generating Component System Files file list in SHADOW::OpenComponent
[35356] 12/13/13 17:43:56 [fsys\shadow] - Informational: calling IVssBackupComponents::SetBackupSucceeded with status FALSE for component System Files in SHADOW::OpenComponent
[35356] 12/13/13 17:43:56 [fsys\systemstate] - AD:Status FS_OBJECT_LIST_FAILURE (0xE000FEDD) calling FS_OpenObj in SystemState::OpenObj:169
[35356] 12/13/13 17:43:56 [fsys\shadow] - Writer SYSVOL include file list:
[35356] 12/13/13 17:43:56 [fsys\shadow] - Writer SYSVOL exclude file list:
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\ntfrs\jet\* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\Windows\debug\NtFrs\NtFrs*
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\sysvol\domain\DO_NOT_REMOVE_NtFrs_PreInstall_Directory\* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\sysvol\domain\NtFrs_PreExisting___See_EventLog\* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\sysvol\staging\domain\NTFRS_*
[35356] 12/13/13 17:43:56 [fsys\shadow] - Informational: Opening Writer 'System?State\SYSVOL'
[35356] 12/13/13 17:43:56 [fsys\shadow] - Informational: Closing Writer 'System?State\SYSVOL'
[35356] 12/13/13 17:43:56 [fsys\shadow] - Component cbe41c98-dec3-4cda-8f4a111bad43d345 file list:
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\sysvol\* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - Component cbe41c98-dec3-4cda-8f4a111bad43d345 Writer's exclude file list:
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\Windows\softwaredistribution\*.* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\*
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\StateData\*
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\Outbound\*
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\PublishedData\*
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\Temp\*
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\Windows\system32\config\Journal\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\Windows\system32\config\RegBack\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\Windows\system32\config\TxR\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: \*.config.cch /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: \*.config.cch.* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\ntfrs\jet\* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: C:\Windows\debug\NtFrs\NtFrs*
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\sysvol\domain\DO_NOT_REMOVE_NtFrs_PreInstall_Directory\* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\sysvol\domain\NtFrs_PreExisting___See_EventLog\* /s
[35356] 12/13/13 17:43:56 [fsys\shadow] - o: c:\windows\sysvol\staging\domain\NTFRS_*
[35356] 12/13/13 17:43:56 [fsys\shadow] - Informational: Opening Component 'cbe41c98-dec3-4cda-8f4a111bad43d345'
[35356] 12/13/13 17:43:56 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\Journal\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:43:56 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\RegBack\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:43:56 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\systemprofile\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:43:56 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\TxR\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:43:56 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\debug\NtFrs\NtFrs* for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:43:56 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\windows\sysvol\domain\NtFrs_PreExisting___See_EventLog\* for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:43:56 [fsys\brfs] - Informational status 0x00000002 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\windows\sysvol\staging\domain\NTFRS_* for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:24 [fsys\ntfs] - Entering SetupLocalShareInfo
[35356] 12/13/13 17:52:24 [fsys\ntfs] - Informational: Physical Memory:'16383' MB - Available Memory:'915' MB in SuggestBufferSizes:496
[35356] 12/13/13 17:52:24 [fsys\ntfs] - Informational: Async I/O Buffer Count:'32' in SuggestBufferSizes:568
[35356] 12/13/13 17:52:24 [fsys\ntfs] - Informational: Async I/O Buffer Size:'65536' bytes in SuggestBufferSizes:569
[35356] 12/13/13 17:52:24 [fsys\ntfs] - Informational: Enumerating DFS
[35356] 12/13/13 17:52:24 [fsys\ntfs] - Starting to initialize ntFsh->dfsJunctionsToAvoid
[35356] 12/13/13 17:52:26 [fsys\ntfs] - Successfully initialized ntFsh->dfsJunctionsToAvoid
[35356] 12/13/13 17:52:26 [fsys\shared] - Attach to E:
[35356] 12/13/13 17:52:44 [fsys\ntfs] - Informational: Object:'\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\windows\sysvol\staging areas\Myvista.local' MSFT Reparse Tag:'IO_REPARSE_TAG_MOUNT_POINT (0xA0000003)'
[35356] 12/13/13 17:52:44 [fsys\ntfs] - and target is [C:\Windows\sysvol\staging\domain]
[35356] 12/13/13 17:52:44 [engidrapi] - GetVirtualHardDisksNumbers() - Enter
[35356] 12/13/13 17:52:44 [engidrapi] - GetVirtualHardDisksNumbers() - Exit
[35356] 12/13/13 17:52:44 [fsys\shared] - CreateFile(\\.\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\windows\sysvol\staging areas\Myvista.local) failed. return code: 5
[35356] 12/13/13 17:52:44 [fsys\ntfs] - Informational: Object:'\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\windows\sysvol\sysvol\Myvista.local' MSFT Reparse Tag:'IO_REPARSE_TAG_MOUNT_POINT (0xA0000003)'
[35356] 12/13/13 17:52:44 [fsys\ntfs] - and target is [C:\Windows\sysvol\domain]
[35356] 12/13/13 17:52:44 [engidrapi] - GetVirtualHardDisksNumbers() - Enter
[35356] 12/13/13 17:52:44 [engidrapi] - GetVirtualHardDisksNumbers() - Exit
[35356] 12/13/13 17:52:44 [fsys\shared] - CreateFile(\\.\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\windows\sysvol\sysvol\Myvista.local) failed. return code: 5
[35356] 12/13/13 17:52:44 [fsys\shadow] - Informational: Closing Component 'cbe41c98-dec3-4cda-8f4a111bad43d345'
[35356] 12/13/13 17:52:44 [fsys\shadow] - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'cbe41c98-dec3-4cda-8f4a111bad43d345' in SHADOW::CloseComponent
[35356] 12/13/13 17:52:44 [fsys\shadow] - Writer Task Scheduler include file list:
[35356] 12/13/13 17:52:44 [fsys\shadow] - Writer Task Scheduler exclude file list:
[35356] 12/13/13 17:52:44 [fsys\shadow] - Informational: Opening Writer 'System?State\Task Scheduler'
[35356] 12/13/13 17:52:44 [fsys\shadow] - Informational: Closing Writer 'System?State\Task Scheduler'
[35356] 12/13/13 17:52:44 [fsys\shadow] - Component TasksStore file list:
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\Windows\System32\Tasks\* /s
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\Windows\Tasks\* /s
[35356] 12/13/13 17:52:44 [fsys\shadow] - Component TasksStore Writer's exclude file list:
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\Windows\softwaredistribution\*.* /s
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\*
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\StateData\*
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\Outbound\*
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\PublishedData\*
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\Temp\*
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\Windows\system32\config\Journal\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\Windows\system32\config\RegBack\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: C:\Windows\system32\config\TxR\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: \*.config.cch /s
[35356] 12/13/13 17:52:44 [fsys\shadow] - o: \*.config.cch.* /s
[35356] 12/13/13 17:52:44 [fsys\shadow] - Informational: Opening Component 'TasksStore'
[35356] 12/13/13 17:52:45 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\Journal\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:45 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\RegBack\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:45 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\systemprofile\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:45 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\TxR\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:46 [fsys\shadow] - Informational: Closing Component 'TasksStore'
[35356] 12/13/13 17:52:46 [fsys\shadow] - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'TasksStore' in SHADOW::CloseComponent
[35356] 12/13/13 17:52:46 [fsys\shadow] - Writer VSS Express Writer Store include file list:
[35356] 12/13/13 17:52:46 [fsys\shadow] - Writer VSS Express Writer Store exclude file list:
[35356] 12/13/13 17:52:46 [fsys\shadow] - Informational: Opening Writer 'System?State\VSS Express Writer Store'
[35356] 12/13/13 17:52:46 [fsys\shadow] - Informational: Closing Writer 'System?State\VSS Express Writer Store'
[35356] 12/13/13 17:52:46 [fsys\shadow] - Component WriterMetadataStore file list:
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\Windows\Vss\Writers\*.xml /s
[35356] 12/13/13 17:52:46 [fsys\shadow] - Component WriterMetadataStore Writer's exclude file list:
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\Windows\softwaredistribution\*.* /s
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\*
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\StateData\*
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\Outbound\*
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\PublishedData\*
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\ProgramData\Microsoft\RAC\Temp\*
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\Windows\system32\config\Journal\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\Windows\system32\config\RegBack\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: C:\Windows\system32\config\TxR\AppData\Local\Microsoft\Outlook\*.ost
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: \*.config.cch /s
[35356] 12/13/13 17:52:46 [fsys\shadow] - o: \*.config.cch.* /s
[35356] 12/13/13 17:52:46 [fsys\shadow] - Informational: Opening Component 'WriterMetadataStore'
[35356] 12/13/13 17:52:47 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\Journal\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:47 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\RegBack\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:47 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\systemprofile\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:47 [fsys\brfs] - Informational status 0x00000003 returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy471\Windows\system32\config\TxR\AppData\Local\Microsoft\Outlook\*.ost for exclusion in brUtil::GenerateFileList
[35356] 12/13/13 17:52:47 [fsys\shadow] - Informational: Closing Component 'WriterMetadataStore'
[35356] 12/13/13 17:52:47 [fsys\shadow] - Informational: calling IVssBackupComponents::SetBackupSucceeded with status 'SUCCESS (0x00000000)' for Component 'WriterMetadataStore' in SHADOW::CloseComponent
[35356] 12/13/13 17:54:32 [ndmp\tpfmt] - TF xfer time = 772 seconds.
[35356] 12/13/13 17:54:32 [ndmp\tpfmt] - WRITE: tpreceive_fail_count = 5601
[35356] 12/13/13 17:54:32 [ndmp\tpfmt] - WRITE: waiting_on_buffers_count = 5591
[35356] 12/13/13 17:54:32 [ndmp\tpfmt] - WRITE: buffers_written_count = 231425
[35356] 12/13/13 17:54:32 [ndmp\tpfmt] - TF_CloseSet()
[35356] 12/13/13 17:54:32 [ndmp\tpfmt] - FreeFormatEnv( cur_fmt=0 )
[35356] 12/13/13 17:54:32 [fsys\shared] - Detach from System?State
[35356] 12/13/13 17:54:32 [fsys\shared] - Detach from Shadow?Copy?Components
[35356] 12/13/13 17:54:33 [fsys\shared] - Detach from C:
[35356] 12/13/13 17:54:33 [fsys\shared] - Detach from \\?\VOLUME{03D003C7-7D08-11E1-A1E7-806E6F6E6963}
[35356] 12/13/13 17:54:33 [fsys\shared] - Detach from E:
[35356] 12/13/13 17:54:33 [ndmp\tpfmt] - TF_FreeDriveContext( 818A010 )
[35356] 12/13/13 17:54:33 [ndmp\tpfmt] - TF_FreeTapeBuffers: from 10 to 0 buffers
[35356] 12/13/13 17:54:33 [ndmp\ndmpsrvr] + ndmp_data.cpp (12055):
[35356] 12/13/13 17:54:33 [ndmp\ndmpsrvr] | Backup Job Stopped(0)
[35356] 12/13/13 17:54:33 [fsys\shared] - 獅健潲散獳㨺敒敬獡e:440 Release called with ref_cnt = 3
[35356] 12/13/13 17:54:33 [fsys\shared] - 獅健潲散獳㨺敒敬獡e:440 Release called with ref_cnt = 2
[35356] 12/13/13 17:54:33 [fsys\shared] - 獅健潲散獳㨺敒敬獡e:440 Release called with ref_cnt = 1
[35356] 12/13/13 17:54:33 [ndmp\tpfmt] - FreeFormatEnv( cur_fmt=0 )
[35356] 12/13/13 17:54:38 [besc] - VSS BackupComplete called with success.
[35356] 12/13/13 17:54:38 - VssSnapshotVolume::GetFailedWriterStatus() - waiting for IVssBackupComponents::GatherWriterMetadata to complete.
[35356] 12/13/13 17:54:38 - VssSnapshotVolume::GetFailedWriterStatus() - IVssBackupComponents::GatherWriterStatus complete.
[35356] 12/13/13 17:54:38 - VssSnapshotVolume::GetFailedWriterStatus() - Issuing Async QueryStatus check.
[35356] 12/13/13 17:54:38 [besc] - Attempting to delete VSS snapshot set: {4b49dfd5-93df-4d32-99c9-d78c09fa270c}
[35356] 12/13/13 17:54:39 [besc] - VSS snapshot set deleted. Snapshot SetID = {4b49dfd5-93df-4d32-99c9-d78c09fa270c}
[35356] 12/13/13 17:54:39 [ndmp\ndmpsrvr] - sslClose() : Closing SSL for: 00000000DF37612F
[35356] 12/13/13 17:54:39 [BESocket] - @@@@@@@MyCloseSocket called with sockfd = 648(0x288) retval = 0
[35356] 12/13/13 17:54:39 [ndmp\tpfmt] - FreeFormatEnv( cur_fmt=0 )
[35356] 12/13/13 17:54:39 [ndmp\tpfmt] - FreeFormatEnv( cur_fmt=0 )
[35356] 12/13/13 17:54:39 [ndmp\ndmpsrvr] - sslClose() : Closing SSL for: 00000000DF37612F
[35356] 12/13/13 17:54:39 [ndmp\ndmpsrvr] - sslClose() : certinfo = 0 ; sslConn = 0
[33448] 12/13/13 17:55:26 [ndmp\ndmpsrvr] - Freeing unsued COM libraries
[11804] 12/13/13 17:58:18 [fsys\shared] - 獅健潲散獳㨺楋汬牐捯獥sESE07 process not ready:297 Kill called with ref_cnt = 0
[11804] 12/13/13 17:58:18 [fsys\shared] - static_eseProcessMap user: myvista\administrator, EseProcess::KillProcess:302
[11804] 12/13/13 17:58:18 [fsys\shared] - Timer thread still running, tell it to close. EseProcess::KillProcess:319
[11804] 12/13/13 17:58:18 [fsys\shared] - Calling ProcessRequest( END_PROC, ESE_ABORT_TIMEOUT ) from EseProcess::KillProcess:336