I have 2 Server 2003 boxes that are running MSCS and serving as a SQL failover cluster.  Recently SQL crashed on one node, but I cannot figure out why.  I looked at the errorlog as well as the generated MDMP, but some of this is outside my comfort zone.  Here is the crash dump analysis:

**********

Microsoft (R) Windows Debugger Version 6.2.9200.16384 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\tsdrop\SQL dump\SQLDump0002.mdmp]
Comment: 'Stack Trace'
Comment: 'ex_terminator - Last chance exception handling'
User Mini Dump File: Only registers, stack and portions of memory are available

Symbol search path is: SRV*C:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Version 3790 (Service Pack 2) MP (7 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Fri Jan  4 11:49:54.000 2013 (UTC - 5:00)
System Uptime: not available
Process Uptime: 15 days 0:10:55.000
................................................................
................
Loading unloaded module list
.........
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1e88.1ee8): Unknown exception - code 000042ac (first/second chance not available)
eax=0000007c ebx=00000000 ecx=00000000 edx=001d0688 esi=0000114c edi=00000000
eip=7c82845c esp=02e9d880 ebp=02e9d8f0 iopl=0         nv up ei ng nz ac pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000297
ntdll!KiFastSystemCallRet:
7c82845c c3              ret

**********

This didn't give me anything super obvious, so I tried .excr:

**********

0:000> .ecxr
*** WARNING: Unable to verify timestamp for sqlservr.exe
eax=02e9de08 ebx=00000440 ecx=02a3f7fc edx=02a3f7fc esi=00000000 edi=027fab38
eip=77e4bef7 esp=02e9de04 ebp=02e9de58 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
kernel32!RaiseException+0x53:
77e4bef7 5e              pop     esi

**********

Then I tried !analyze:

**********

0:000> !analyze
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

*** WARNING: Unable to verify timestamp for msvcr80.dll
Probably caused by : sqlservr.exe ( sqlservr!CVariableInfo::CVarBlock::PvbJoin+5c )

Followup: MachineOwner
---------

********** 

And finally, !analyze -v:

**********

0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

FAULTING_IP:
sqlservr!CVariableInfo::CVarBlock::PvbJoin+5c
01004bc6 ??              ???

EXCEPTION_RECORD:  02e9efb0 -- (.exr 0x2e9efb0)
ExceptionAddress: 01004bc6 (sqlservr!CVariableInfo::CVarBlock::PvbJoin+0x0000005c)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 00000001
   Parameter[1]: b18316c8
Attempt to write to address b18316c8

DEFAULT_BUCKET_ID:  APPLICATION_FAULT

PROCESS_NAME:  sqlservr.exe

ERROR_CODE: (NTSTATUS) 0x42ac - <Unable to get error code text>

EXCEPTION_CODE: (Win32) 0x42ac (17068) - <Unable to get error code text>

NTGLOBALFLAG:  0

APP:  sqlservr.exe

CONTEXT:  02e9efcc -- (.cxr 0x2e9efcc)
eax=330e1e18 ebx=00000000 ecx=330e1c18 edx=b18316c0 esi=b1a6fc3e edi=330e0000
eip=01004bc6 esp=02e9f298 ebp=02e9f2ac iopl=0         nv up ei ng nz ac pe cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010297
sqlservr!CVariableInfo::CVarBlock::PvbJoin+0x5c:
01004bc6 ??              ???
Resetting default scope

WRITE_ADDRESS:  b18316c8

FOLLOWUP_IP:
sqlservr!CVariableInfo::CVarBlock::PvbJoin+5c
01004bc6 ??              ???

FAILED_INSTRUCTION_ADDRESS:
sqlservr!CVariableInfo::CVarBlock::PvbJoin+5c
01004bc6 ??              ???

IP_ON_HEAP:  330e1e18

FRAME_ONE_INVALID: 1

LAST_CONTROL_TRANSFER:  from 330e1e18 to 01004bc6

FAULTING_THREAD:  ffffffff

PRIMARY_PROBLEM_CLASS:  APPLICATION_FAULT

BUGCHECK_STR:  APPLICATION_FAULT_APPLICATION_FAULT

STACK_TEXT: 
02e9f298 01004bc6 sqlservr!CVariableInfo::CVarBlock::PvbJoin+0x5c
02e9f29c 330e1e18 unknown!unknown+0x0
02e9f2b4 0100450b sqlservr!CVarPageMgr::Release+0x16
02e9f2b8 330e1e20 unknown!unknown+0x0
02e9f2d0 01008988 sqlservr!CMemThread::Free+0x4f
02e9f2f0 0121daf4 sqlservr!CSqlHashBkt::PurgeDeadSql+0xf2
02e9f30c 0121da38 sqlservr!CSqlHashBkt::ReleaseSql+0x64
02e9f32c 01234205 sqlservr!CCompPlan::~CCompPlan+0xbd
02e9f39c 01234363 sqlservr!CCompPlan::`scalar deleting destructor'+0xd
02e9f3a8 010b048b sqlservr!CCacheObject::Release+0x36
02e9f3b8 01248827 sqlservr!CCacheObject::Destroy+0x30
02e9f3c8 01248995 sqlservr!SOS_CacheStore::RemoveDescriptor+0x1fd
02e9f420 012489de sqlservr!SOS_CacheStore::CacheEntryDescriptor::Destroy+0x40
02e9f440 024205e4 sqlservr!ClockHand::Move+0x38c
02e9f570 02420b6b sqlservr!ClockAlgorithm::MoveHand+0x43
02e9f594 02420a76 sqlservr!ClockAlgorithm::ProcessTick+0x149
02e9f5ec 0241f945 sqlservr!SOS_CacheStore::Notify+0x3f
02e9f604 0242536a sqlservr!ResourceMonitor::NotifyMemoryConsumers+0x2a4
02e9f6c8 02424f44 sqlservr!ResourceMonitor::ResourceMonitorTask+0x190
02e9f76c 021a9ef8 sqlservr!SetupResourceMonitorTaskContext+0x44e
02e9fd54 010067d3 sqlservr!SOS_Task::Param::Execute+0xe2
02e9fdc4 010068f9 sqlservr!SOS_Scheduler::RunTask+0xb9
02e9fdf8 01006609 sqlservr!SOS_Scheduler::ProcessTasks+0x141
02e9fe38 010daf6c sqlservr!SchedulerManager::WorkerEntryPoint+0x1ad
02e9fea0 010dae8c sqlservr!SystemThread::RunWorker+0x7f
02e9feb8 010dab54 sqlservr!SystemThreadDispatcher::ProcessWorker+0x246
02e9ff18 010dacf1 sqlservr!SchedulerManager::ThreadEntryPoint+0x143
02e9ff80 781329bb msvcr80!_callthreadstartex+0x1b
02e9ffb8 78132a47 msvcr80!_threadstartex+0x66
02e9ffc0 77e6482f kernel32!BaseThreadStart+0x34

STACK_COMMAND:  .cxr 02E9EFCC ; kb ; dps 2e9f298 ; kb

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  sqlservr!CVariableInfo::CVarBlock::PvbJoin+5c

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: sqlservr

IMAGE_NAME:  sqlservr.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  492b676e

FAILURE_BUCKET_ID:  APPLICATION_FAULT_42ac_sqlservr.exe!CVariableInfo::CVarBlock::PvbJoin

BUCKET_ID:  APPLICATION_FAULT_APPLICATION_FAULT_BAD_IP_sqlservr!CVariableInfo::CVarBlock::PvbJoin+5c

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/sqlservr_exe/2005_90_4035_0/492b676e/sqlservr_exe/2005_90_4035_0/492b676e/42ac/011bbce0.htm?Retriage=1

Followup: MachineOwner
---------

**********

I've analyzed Windows STOP errors (the DMP files that end up in the Minidump folder), but that's really all I have used Windows Debugger for.  I've never tried to analyze a SQL crash as I haven't had this problem before.  Any help would be much appreciated.

Sr System Engineer | Vision One IT Consulting | www.v1corp.com

Hi,

I just wanted to ask if there are any consequences to using a varbinary(120) column on a table and having a clustered index on it. How does this affect inserts etc.

This column is used in a checksum like manner, and the checksum values are unique.

Many thanks.

On running DBCC Opentran() I am getting below:

Transaction information for database 'ARSystem'.

Replicated Transaction Information:
Oldest distributed LSN : (34595:32751:24)
Oldest non-distributed LSN : (0:0:0)
DBCC execution completed. If DBCC printed error messages, contact your system administrator.

What is this?

Transaction Replication is working on this database.

Thanks

Hi,

It has now been few occurences where I see different query plans for the same stored procedure against different users. The case is somewhat like this:

A stored procedure is called by two database users - one regular user with 'exec' permission and another sysadmin. Both execute the same command like exec SP1 Parm1, Parm2 with same parameter values. However the instance executing under regular user has a different query plan than the other and takes longer.

I was wondering what could be the reason behind their having different query plans.

Thanks in advance.

Nirupam

Hi,

We are planning for upgrade from sql server 2005 to sql server 2012, used sql server upgrade advisor and it shows no issues to upgrade. On the applications side, development team is also telling that the applications(they developed) will not fail with the new version. unfortunately there are some old applications that are in heavy use but our development team doesn't have their source code, and their vendors are also out of scence now, and those are working very smoothly with sql server 2005.

In the testing there is no problem but when we will go for the production may be we face some problem (it is not sure)  and a rollback might be the option.

To apply this upgrade we must have a rollback plan (either it would be used or not).

Experts, please suggest some quick (instance have round 18 databases) and safe rollback plan.

Best Regards

khalil

Hi

I had taken a full backup of database, after 10 days transaction logs were truncated using

BACKUP LOG with TRUNCATE_ONLY; command without taking backup of transaction log.

then after few days one of the table got truncated, after truncation full database backup has been taken.

Kindly let me know is there any way to recover table data as our log chain has been broken.

Hi,

In our setup we are using SQL cluster active and passive and all of our cluster drives are on the SAN.

We are using SAN "E" drive for our application logs.

Last week, we were adding new drives to the database server. During this activity, accidently drive letter had changed of the “E” drive to “Y”. Later this issue was found & fixed and drive was re-named back to “E”.

Our application logs could not get stored in the database because of drive letter was changed and next day morning, we have found that our logs database is disconnect and this was happened that drive name was changed.

We have fixed this issue with offline and online the database. After that our application logging is started on the database.

Our application logs could not get stored in the database since drive letter had changed to until we have offline and online the database. (Around 15 hour’s application logs).

Is there any chance that SQL Server would have written these logs to a temporary file elsewhere (while the E drive was “missing”)?

I have attached the application event logs message:

FCB::Open failed: Could not open file E:\DB_LOGS\DB_LOGS.mdf for file number 1.  OS error: 3(The system cannot find the path specified.).

FileMgr::StartSecondaryDataFiles: Operating system error 2(The system cannot find the file specified.) occurred while creating or opening file 'E:\ DB_LOGS\DB_LOGS_xx.ndf'. Diagnose and correct the operating system error, and retry the operation.

Note: we are using MSMQ for transfer the logs from application server to DB server.

Thanks

How to quit a job if some condition is not satisfied in a job step?

I have multiple steps in a job and each step is individual and has one condition true or false. I wanted to add "xp_sendmail" end of each step, and if the condition is false with in the step execute "xp_sendmail" and quit the job.

My aim is to receive a mail with at which step the job is faild

for example like below.

Step 1

if my condition is true, then run some code and go to 2nd step.

if my condition is false execute the below code and quit the job but don't go to next step.

EXEC master.dbo.xp_sendmail
@recipients=N'sqldba@company.com',
@message=N'Job failed at step 1.'

now quit the job.

Step: 2

Here i ahve another condition if it is true run my statements and then "xp_sendmail"

EXEC master.dbo.xp_sendmail
@recipients=N'sqldba@company.com',
@message=N'Job completed successfully.'

If my condition is not true run the below statemets

EXEC master.dbo.xp_sendmail
@recipients=N'sqldba@company.com',
@message=N'Job failed at step 2.'

End of job.

Hi all, when -all- records from a table with a varbinary(max) column are deleted (not via truncate), the table properties still show a dataspace size from before the delete operation.

Inserting new blob records only leads in the growth of the allocated space withouth reusing the empty already allocated space.

My configuration is running SQL 2008 Std. x64 SP1 with DB is in simple recovery mode.

Runnning commands like dbcc updatusage/checkdb/cleantable/reindex or sp_spaceused @updateusage = N'TRUE' seem to have no effect.

Does anyone know when space allocated by a varbinary col. is released?

Only unfree many space if use dbcc shrinkfile empty_file.

I am Viewer in
FIX: The ghost row clean-up thread does not remove ghost rows on some data files of a database in SQL Server 2005
(http://support.microsoft.com/kb/932115/en-us)

In don´t test this fix with Sql Server 2005, this bug exists have in Sql Server 2008 ?

thanks,

Dan

ps. Sorry my English

We are maintaining an e-commerce site that occasionally has severe performance problems. Normally the site works fine for hours. But every once in a while the entire site completely locks for a few minutes and the works fine again. After the lock when I look at the log files from the web app it is filled with SQL Server timeout errors so the bottleneck is the database, also confirmed that the SQL Server process is consuming the CPU on the server (both the web app and the server are on the same server).

There was no specific action that caused this so it is not obviously bad SQL. Each individual SQL statement run is ok although not always optimal. I have previously been unable to to trace and see more details since it does not happen that often but today I was lucky. In the trace I found one statement that took about 4 minutes to run. It is a stored procedure which normally runs in less than a second. I tried this immediately after server was "unlocked" and it was back to normal. In the trace I read a CPU value of 362749 and the number of reads were 149987571 (0 writes) which is simply absurd. I ran the same statement now when the server is unlocked and the CPU value was 1310, reads were 483007 and writes were 898.

Our client is not happy and we struggle to find the causes of this. About three hours before this occasion I did a rebuild of all indexes so I do not expect fragmented indexes to be the problem. I have a suspicion though which is low memory. The server only has 2GB of memory and is constantly on 95% memory usage (SQL server has usually allocated about 1.2 GB). Could this low memory cause such problems or does anyone have other ideas of what is going on?

Hi All,

Hope this is a simple dumb question.. in sp_who2 i'm trying to get the correct explanation of what is seen say in the following instances:

1) 1 row for 1 SPID = this means one connection from a source application

2) Multiple rows for 1 SPID = this means multiple connections from the same source transaction, or does it mean multiple threads for the same transaction?

3) I always though that multiple rows for the same SPID meant it was executing the query in parallel (true?) - does this mean that i should only have as many rows as i do have logical CPUs? If i have 8 CPUs, then theoretically i should, at most, see 8 rows for the same SPID? I don't think i understand quite right b/c yesterday i saw one SPID with 15 rows, but there were only 8 CPUs on the server

Please clarify for me, much appreciated!

I have a table with blob data that has a truckload of unused space that I can't get rid of.  

SELECT * FROM sys.dm_db_index_physical_stats

    (DB_ID(N'Test_dbname'), OBJECT_ID(N'tableName'), NULL, NULL , 'DETAILED');
GO

Shows:

alloc_unit_type_descavg_page_space_used_in_percentrecord_countghost_record_count
IN_ROW_DATA  99.0901902643934125490720
IN_ROW_DATA  99.81305905609092272150
IN_ROW_DATA  50.17296763034354780
IN_ROW_DATA  0.39535458364220420
LOB_DATA63.1606004447739629731630

Thus I'm wasting huge amounts of space.  If I select all the data into another table it compacts at 100% give or take.  

SQL 2008, spk1.  I don't believe the table ever held text data.   Varbinary and Image are in the table.  

OS: Windows 2008 R2

SQL Server: 2008 R2 SP2

OS Memory: 16 GB

SQL Server Max Memory: 12 GB

Database is in SIMPLE recovery mode.

1. I saw some blocking on executing sp_who2 and ran Paul Randal's Where it hurts?

The results show, LCK_M_U and LCK_M_IX wait stats at 90%

2. I ran Glenn Berry's Memory DMV:

-- Good basic information about memory amounts and state
SELECT total_physical_memory_kb, available_physical_memory_kb,
       total_page_file_kb, available_page_file_kb,
       system_memory_state_desc
FROM sys.dm_os_sys_memory OPTION (RECOMPILE);

-- You want to see "Available physical memory is high"

The result was Available physical memory is high.

3. I ran Pinal Dave's DMV:

SELECT dm_ws.wait_duration_ms,
dm_ws.wait_type,
dm_es.status,
dm_t.TEXT,
--dm_qp.query_plan,
--dm_ws.session_ID,
--dm_es.cpu_time,
--dm_es.memory_usage,
--dm_es.logical_reads,
--dm_es.total_elapsed_time,
dm_es.program_name,
DB_NAME(dm_r.database_id) DatabaseName,
-- Optional columns
dm_ws.blocking_session_id--,
--dm_r.wait_resource,
--dm_es.login_name,
--dm_r.command,
--dm_r.last_wait_type
FROM sys.dm_os_waiting_tasks dm_ws
INNER JOIN sys.dm_exec_requests dm_r ON dm_ws.session_id = dm_r.session_id
INNER JOIN sys.dm_exec_sessions dm_es ON dm_es.session_id = dm_r.session_id
CROSS APPLY sys.dm_exec_sql_text (dm_r.sql_handle) dm_t
CROSS APPLY sys.dm_exec_query_plan (dm_r.plan_handle) dm_qp
WHERE dm_es.is_user_process = 1
order by wait_duration_ms desc
GO

The result showed an Update/Insert Trigger on a 34 million row table causing LCK_M_U and LCK_M_IX waits

4. I ran Glenn Berry's DMV for Signal Waits:

-- Signal Waits for instance
SELECT CAST(100.0 * SUM(signal_wait_time_ms) / SUM (wait_time_ms) AS NUMERIC(20,2))
AS [%signal (cpu) waits],
CAST(100.0 * SUM(wait_time_ms - signal_wait_time_ms) / SUM (wait_time_ms) AS NUMERIC(20,2))
AS [%resource waits]
FROM sys.dm_os_wait_stats

The result was 2% signal waits

5. dbcc loginfo returned 283 rows (VLFs).

On analyzing the logs, it was determined that 10% autogrowth was set for logs.
I changed the log growth from % to MB and brought the VLFs to close to 50.

6. I ran Glenn Berry's script for IO bottleneck:

-- Calculates average stalls per read, per write, and per total input/output for each database file.
SELECT DB_NAME(fs.database_id) AS [Database Name], mf.physical_name, io_stall_read_ms, num_of_reads,
CAST(io_stall_read_ms/(1.0 + num_of_reads) AS NUMERIC(10,1)) AS [avg_read_stall_ms],io_stall_write_ms,
num_of_writes,CAST(io_stall_write_ms/(1.0+num_of_writes) AS NUMERIC(10,1)) AS [avg_write_stall_ms],
io_stall_read_ms + io_stall_write_ms AS [io_stalls], num_of_reads + num_of_writes AS [total_io],
CAST((io_stall_read_ms + io_stall_write_ms)/(1.0 + num_of_reads + num_of_writes) AS NUMERIC(10,1))
AS [avg_io_stall_ms]
FROM sys.dm_io_virtual_file_stats(null,null) AS fs
INNER JOIN sys.master_files AS mf
ON fs.database_id = mf.database_id
AND fs.[file_id] = mf.[file_id]
ORDER BY avg_io_stall_ms DESC OPTION (RECOMPILE);

-- Helps you determine which database files on the entire instance have the most I/O bottlenecks
-- This can help you decide whether certain LUNs are overloaded and whether you might
-- want to move some files to a different location

The results are listed in the spreadsheet image below. Database1 is the database causing blocking. I don't know about what the numbers in the individual columns mean with regards to accepted scale of good to worse I/O but since Database1 is the first in the list, I am assuming that Database1 is causing the most I/O bottleneck.

Can I infer from the spreadsheet below that the LUN assigned to drive E:\ is overloaded and moving some files off the drive E:\ assigned LUN could potentially ease I/O bottlenecks?

This is how far my knowledge will take me.

I am starting to look into purging rows from the 34 million table. However, the purge will be a "future" fix as the vendor for the app needs to get in the game. Moving the drive with .mdf files to a faster storage (like Solid State Drives) is not an option but moving some files off the LUN assigned to E:\ could be an option.

Any guidance is appreciated.

Thanks in advance.

-Jeelani

Happy New Year all :)

I want to Index the content of email attachments like Word docs and Excel spreadsheets.

I understand that for this process I should store the content in a varbinary field, the file type in a varchar field and then set up a full text index (as perhttp://arcanecode.com/2009/05/28/full-text-searching-a-filestream-varbinary-max-column/_)

However I don't want the attachments stored in the database in this fashion - they will be stored in a MIME type field seperately - I just want their content indexed, so I can search on it.

Is there any way I can use an api in C# code to perform the same funtion in memory?

This will save me having to save unnecessary data in the database - I would just be saving the indexed data.

Recently we discovered a problem with the GUI when trying to restore a database to a point in time. The time happens to fall in between the start and end time of a differential backup. To make it clear, say the differential started at 5PM and take 10 minutes to complete. Our requested restore point is 5:01 PM. The GUI generated script suggest we do Full restore, Differential that started at 5PM then the first Log after the Diff backup. However, most of the time we encounter this error

Msg 4335, Level 16, State 1, Line 2
The specified STOPAT time is too early. All or part of the database is already rolled forward beyond that point.
Msg 3013, Level 16, State 1, Line 2
RESTORE LOG is terminating abnormally.

if we restore the log file without the STOPAT, then the restore can recovery the database.

We can work around it by forcing the restore to start from the previous differential backup and then all the Logs in between until the last one that contains the restore point.

This problem happens for both SQL 2005 and 2008R2 servers we tested, however, interestingly, we find the GUI generated script differ slightly in that a "STOPAT" is used in both the Differential and Log restore commands for SQL 2008 version.

I am wondering if this is a long term standing bug with SQL Server.

On SQL Server 2008, I just installed data collection.  When I look at Disk Usage, right click choose Reports, I only have the choice of Custom.  Why don't I get the two other choices of Historical and Disk Usage Summary?

Hi,

I want to know if there is options available to force SQL Server to use first some tempdb files before some others?

I have SSD drives, SAS and SATA drives.

the SSD is the better, but the smaller. for 90% of the needs, having the tempdb files here is enough.

but sometimes complex queries require more space from the disks, and the SSD drive will be filled.

So I have to use the SAS drives then the SATA drives...

if I create 1 tempdb file per disk, SQL Server use the 3 files at the same time and this reduces the speed of my most common queries.

so if I can force SQL Server to use the files in a sequential order rather then in parrallel it will be great.

is it possible to do this?

can I add a tempdb file when the SSD disk is near full and remove it later? (so add a file to the tempdb database when the disks is used at 90% or if the tempdb file is filled at 90%, and later in the day remove these files created on the SAS drive)

thanks.