Understanding Indexing in SQL Server: Types and Usage

What is an Index?   

An index in SQL Server is a data structure associated with a table or view that speeds up the retrieval of rows based on the values in one or more columns. It serves as a well-organized reference guide, allowing SQL Server to efficiently locate rows that match query criteria without scanning the entire table.

Types of Indexes:

1. Clustered Index: Determines the physical order of data in a table, affecting the order of data when modified.
2. Non-clustered Index: Creates a separate structure with sorted references to actual data rows, useful for enhancing SELECT query performance.
3. Unique Index: Ensures uniqueness of values in the indexed column(s) across the table, aiding in data integrity.
4. Covering Index: Includes all columns needed to fulfill a query, minimizing I/O operations and improving query performance.
5. Filtered Index: Includes only a subset of rows in the table based on a WHERE clause, useful for optimizing queries targeting specific subsets of data.
6. Spatial Index: Specialized for spatial data types, facilitating efficient spatial queries such as distance calculations and intersections.
7. Columnstore Indexes: Organizes data by columns, beneficial for analytical queries involving aggregations and scans across large datasets.

Usage of Indexes:

 Faster Data Retrieval: Provides a shortcut to desired rows, reducing the time to locate and retrieve data, particularly helpful for SELECT queries.  
Optimizing Joins: Indexes on join columns enhance performance by quickly identifying matching rows.  
Sorting and Grouping: Speed up ORDER BY and GROUP BY operations by efficiently retrieving and organizing data.  
Constraint Enforcement: Unique indexes ensure data integrity by preventing duplicate values in indexed columns.  
Covering Queries: Minimizes I/O operations and speeds up query execution by scanning the index alone.  
Reducing I/O Operations: Efficient use of indexes minimizes I/O operations required to satisfy a query.

Best Practices for Indexing:

1. Selective Indexing: Focus on columns frequently used in WHERE clauses, JOIN conditions, and ORDER BY clauses to avoid unnecessary overhead.
2. Regular Maintenance: Monitor and maintain indexes regularly, including rebuilding or reorganizing to minimize fragmentation.
3. Avoid Over-Indexing: Strike a balance between performance gains and maintenance overhead to avoid diminishing returns.
4. Consider Clustered Index Carefully: Choose based on typical table queries and access patterns.
5. Use Indexing Tools: Leverage tools such as the Database Engine Tuning Advisor to recommend appropriate indexes based on query performance analysis.
6. Understand Query Execution Plans: Analyse plans to identify areas where indexes can optimize query performance.

Conclusion:  

Indexes in SQL Server play a crucial role in enhancing query speed by enabling quicker data retrieval and minimizing the need for full-table scans. Selecting the right type of index and adhering to best practices, including regular maintenance and thorough understanding of database access patterns, are vital for extracting maximum benefits from indexing. 

How to identify duplicate indexes along with columns in SQL Server?

To get the key column list from indexes that are duplicates in SQL Server, you can use the following query:

use databasename
go

WITH DuplicateIndexes AS (
    SELECT 
        i.OBJECT_ID,
        i.index_id
    FROM 
        sys.index_columns ic
    JOIN 
        sys.indexes i ON i.OBJECT_ID = ic.OBJECT_ID 
                     AND i.index_id = ic.index_id
    WHERE 
        i.type_desc <> 'HEAP' 
		AND OBJECT_NAME(i.OBJECT_ID) NOT LIKE '%sys%' --excluding system tables
    GROUP BY 
        i.OBJECT_ID, i.index_id
    HAVING 
        COUNT(*) > 1 -- to check duplicates 
)

SELECT 
    SCHEMA_NAME(o.schema_id) AS SchemaName,
    OBJECT_NAME(ic.OBJECT_ID) AS TableName,
    i.name AS IndexName,
    STRING_AGG(c.name, ', ') WITHIN GROUP (ORDER BY ic.key_ordinal) AS IndexedColumns
FROM 
    sys.index_columns ic
JOIN 
    sys.indexes i ON i.OBJECT_ID = ic.OBJECT_ID 
                 AND i.index_id = ic.index_id
JOIN 
    sys.objects o ON o.OBJECT_ID = ic.OBJECT_ID
JOIN 
    sys.columns c ON ic.OBJECT_ID = c.OBJECT_ID 
                 AND ic.column_id = c.column_id
JOIN 
    DuplicateIndexes di ON di.OBJECT_ID = ic.OBJECT_ID 
                        AND di.index_id = ic.index_id
GROUP BY 
    o.schema_id, ic.OBJECT_ID, i.name;
  

This query first identifies the indexes that are duplicates, and then retrieves the table name, index name, and the key column list for each duplicate index.

Execute this query in your SQL Server management tool to get the key column list from indexes that are duplicates in your database.

How to Review transaction order and lock acquisition in SQL Server

In SQL Server, you can review the transaction order and lock acquisition by analysing the queries and transactions that are being executed against the database. Here are some approaches to review transaction order and lock acquisition:

1. Transaction isolation levels:

   • Review the transaction isolation levels used in your database transactions. Isolation levels such as Read Uncommitted, Read Committed, Repeatable Read, and Serializable can impact the order of lock acquisition and the behaviour of concurrent transactions.

2. Query execution plans:

   • Use SQL Server Management Studio (SSMS) or other database management tools to analyse the query execution plans for your transactions.
   • The execution plans can provide insights into the order in which data is accessed and the types of locks acquired during query execution.

3. Locking and blocking:

   • Monitor and analyse the locking and blocking behaviour of concurrent transactions using tools like SQL Server Profiler, Extended Events, or dynamic management views (DMVs) such as sys.dm_tran_locks and sys.dm_os_waiting_tasks.
   • Identify instances of blocking and analyse the lock types and resources involved to understand the order of lock acquisition.

4. Transaction log and history:

   • Review the transaction log and history to understand the sequence of transactions and their impact on lock acquisition.
   • SQL Server's transaction log and history can provide valuable information about the order in which transactions are executed and their associated locks.

By using these approaches, you can gain insights into the transaction order and lock acquisition behaviour in SQL Server, which can help in identifying potential issues related to deadlocks, blocking, and overall transaction concurrency.

What is deadlock priority and how to address in SQL Server

In SQL Server, deadlock priority is a mechanism that allows you to influence the selection of the transaction that will be chosen as the deadlock victim when a deadlock occurs. You can use deadlock priority to specify the importance of individual transactions in the event of a deadlock.

To address deadlock priority in SQL Server, you can consider the following:

1. Setting deadlock priority:
   • You can use the SET DEADLOCK_PRIORITY statement to specify the priority of a session or transaction.
   • The priority levels range from -10 to 10, where -10 is the lowest priority and 10 is the highest.
   • By setting the deadlock priority, you can influence the selection of the victim transaction when a deadlock occurs.

Here's an example of how to set the deadlock priority for a session:

SET DEADLOCK_PRIORITY LOW; -- Set the deadlock priority to low
  

2. Adjusting transaction logic:

   • Design your transaction logic to handle the potential impact of being chosen as the deadlock victim based on the assigned deadlock priority.
   • Consider implementing retry logic for transactions with lower deadlock priority after being chosen as the deadlock victim.

3. Analyzing and tuning deadlock priority:

   • Evaluate the impact of deadlock priority settings on your application's transactions and overall performance.
   • Tune the deadlock priority based on the specific requirements and characteristics of your application to effectively manage deadlocks.

It's important to carefully consider the implications of deadlock priority settings in SQL Server and design your transaction logic to handle deadlock situations appropriately. Understanding the behavior of deadlock priority in SQL Server is crucial for effectively addressing and managing deadlocks.

How to implement retry logic for DB Transactions

In SQL Server, you can implement retry logic for transactions using T-SQL and error handling. Here's an example of how you can create a stored procedure that includes retry logic for handling deadlock errors:

CREATE PROCEDURE usp_RetryTransaction
AS
BEGIN
    DECLARE @retryCount INT = 0
    DECLARE @maxRetries INT = 3

    WHILE @retryCount < @maxRetries
    BEGIN
        BEGIN TRY
            BEGIN TRANSACTION
            -- Your transactional logic goes here
            COMMIT TRANSACTION
            RETURN
        END TRY
        BEGIN CATCH
            IF ERROR_NUMBER() = 1205  -- Deadlock error number
            BEGIN
                ROLLBACK TRANSACTION
                SET @retryCount = @retryCount + 1
                WAITFOR DELAY '00:00:01'  -- Wait for 1 second before retrying
            END
            ELSE
            BEGIN
                -- Handle other types of errors
                THROW
            END
        END CATCH
    END
    -- If the maximum number of retries is reached, handle the situation as needed
    -- For example, raise an error or log the issue
END
  

In this example, the stored procedure attempts the transaction logic within a retry loop, and if a deadlock error (error number 1205) occurs, it rolls back the transaction, increments the retry count, and waits for a short duration before retrying the transaction. If the maximum number of retries is reached, you can handle the situation as needed based on your application's requirements.

You can then call this stored procedure whenever you need to perform a transaction with retry logic for deadlock handling.

Find tables or procedures that are associated in SQL Jobs via Query

Recently we need to look for a procedure where we are using in SQL Jobs. There is no easy way to find unless you script all jobs and find in the script.

But there is some easy way to find it using below query. You could also might have similar ask to find a procedure or table that you might have used in SQL Jobs in any of those steps. It could be any string like comment, procedure, function or table, this below query works.

USE msdb
GO

SELECT [sJOB].[job_id] AS [JobID]
	,[sJOB].[name] AS [JobName]
	,step.step_name AS JobStepName
	,step.command AS JobCommand
	,[sJOB].enabled AS ActiveStatus
FROM [msdb].[dbo].[sysjobs] AS [sJOB]
LEFT JOIN [msdb].dbo.sysjobsteps step ON sJOB.job_id = step.job_id
WHERE step.command LIKE '%uspPopulateAggregatorUsageData%' ----You can change here what you are searching for
ORDER BY [JobName]
  

Thank you

Use RAND() in User Defined Function

Here is the issue,  you cannot call a non-deterministic function from inside a user-defined function.

So there is a workaround, By creating a view to return RAND(), call RAND() function inside a view and use that view inside your function, something like below.

CREATE VIEW ReturnRANDValue
AS
SELECT RAND() AS Value
  

Here is the function where we call above view inside this function

CREATE FUNCTION [dbo].[ReturnMobileNotificationCount] (
	@MobilityOrderID INT,
	@MobilityOrderItemID INT,
	@LineStatusMasterID INT,
	@LineSubStatusMasterID INT
	)
RETURNS BIT
AS
BEGIN
	DECLARE @LineSubStatusMatch BIT = 0

	SELECT MobilityOrderID,
		MobilityOrderItemID,
		LineStatusMasterID,
		LineSubStatusMasterID,
		HistoryID,
		ROW_NUMBER() OVER (
			ORDER BY (
					SELECT Value
					FROM ReturnRANDValue
					) DESC
			) SerialNo
	FROM MobileNotificationCriteriaHistory MNC(NOLOCK)
	WHERE MobilityOrderID = @MobilityOrderID
		AND MobilityOrderItemID = @MobilityOrderItemID
	ORDER BY HistoryID DESC

	RETURN @LineSubStatusMatch
END
  

What is the Query to display the failed SQL Jobs

We can get this information by the standard reports available in SQL Server, right click on SQL server agent> Standard reports and select the desired report to see Job History

It can be frustrating to find recently failed jobs in the job history in SQL Server Management Studio. A quicker way to do it is to just run a query to see what jobs have failed recently.

Below Query will give you list of failed jobs, you can also filter by name.

select j.name
    ,js.step_name
    ,jh.sql_severity
    ,jh.message
    ,jh.run_date
    ,jh.run_time
FROM msdb.dbo.sysjobs AS j
INNER JOIN msdb.dbo.sysjobsteps AS js
   ON js.job_id = j.job_id
INNER JOIN msdb.dbo.sysjobhistory AS jh
   ON jh.job_id = j.job_id AND jh.step_id = js.step_id
WHERE jh.run_status = 0
--and name = 'jobName'
order by run_date desc
  

If you want to check results based on data range, you can use below query to find desired results.

-- Variable Declarations 
DECLARE @FinalDate INT;
SET @FinalDate = CONVERT(int
    , CONVERT(varchar(10), DATEADD(DAY, -2, GETDATE()), 112)
    ) -- last two days date as Integer in YYYYMMDD format

-- Final Logic 

SELECT  j.[name],  
        s.step_name,  
        h.step_id,  
        h.step_name,  
        h.run_date,  
        h.run_time,  
        h.sql_severity,  
        h.message,   
        h.server  
FROM    msdb.dbo.sysjobhistory h  
        INNER JOIN msdb.dbo.sysjobs j  
            ON h.job_id = j.job_id  
        INNER JOIN msdb.dbo.sysjobsteps s  
            ON j.job_id = s.job_id 
                AND h.step_id = s.step_id  
WHERE    h.run_status = 0 -- Failure  
         AND h.run_date > @FinalDate  
ORDER BY h.instance_id DESC;
  

Hope this helps!

How to Get the Last Day of the Month in T-SQL

We’ll use the function EOMONTH() to find the last day of the month.

DECLARE @fromdate DATETIME, @EOMMonthdate DATETIME

SET @fromdate = '11/01/2022'

SET @EOMMonthdate = EOMONTH(@fromdate,0) -- for current month

SELECT @EOMMonthdate as lastdayOfthemonth
  

If you want to return the last day of the second, third, etc. month from a given date, use EOMONTH()’s optional second argument: the number of months to add. Look at the examples for last day of next month or last day of the previous month.

DECLARE @fromdate DATETIME, @EOMMonthdate DATETIME

SET @fromdate = '11/01/2022'

SET @EOMMonthdate = EOMONTH(@fromdate,1) -- for next month last day

SELECT @EOMMonthdate as nextMonthLastDayOfthemonth

SET @EOMMonthdate = EOMONTH(@fromdate,-1) -- for previous month last day

SELECT @EOMMonthdate as previousMonthLastday
  

Hope this helps!!

How to get list of column names from Table Variable @table

A table variable is a variable data type which can be used to store temporary data. It's defined using the DECLARE keyword and the table's structure is defined in the declaration as shown below:

declare @ns_source table
(
  col1_id int, 
  col2_name varchar(50),
  col3_desc varchar(50)
)    

Unlike temporary tables, table variables are not affected by a rollback. As regular variables, they keep the data which was modified during the transaction even if the transaction is rolled back.

Coming to get columns from table varaiable, here is how we can get using below query.

declare @ns_source table
(
  col1_id int, 
  col2_name varchar(50),
  col3_desc varchar(50)
)

select DP.N.value('local-name(.)', 'sysname') as ColumnName
from 
  (
  select NS.*
  from (select 1) as D(N)
    outer apply (
                select top(0) *
                from @ns_source
                ) as NS
  for xml path(''), elements xsinil, type
  ) as LV(X)
cross apply LV.X.nodes('*') as DP(N)

Result looks like below

How to get only numbers from string sql

I have column where I need to get only numbers form the string.

Here is how we can do,

SUBSTRING(columnName, PATINDEX('%[0-9]%', columnName), LEN(columnName))

Here's the example with PATINDEX

use databasego
-- inventory Group, Employee , cost center
select [Cost Center], [Inventory Group], Employee, 
SUBSTRING(Employee, PATINDEX('%[0-9]%', Employee), LEN(Employee)) AS EmployeeID
from  tmp_BulkOrder_SP_4_16_21

Hope this helps 😀

Get Current TimeZone Name in SQL Server

Here is how we can get current time zone from sql server

DECLARE @TimeZone VARCHAR(50)
EXEC MASTER.dbo.xp_regread 'HKEY_LOCAL_MACHINE',
'SYSTEM\CurrentControlSet\Control\TimeZoneInformation',
'TimeZoneKeyName',@TimeZone OUT
SELECT @TimeZone

Quick note: How to find queries, SP's or jobs that use a linked server?

Here is the query below that can pull all above results.

Declare @VName varchar(256)
Declare Findlinked cursor
LOCAL STATIC FORWARD_ONLY READ_ONLY
     FOR
Select name
   From sys.servers
   Where is_linked = 1
      
Open Findlinked;
Fetch next from Findlinked into @VName;

while @@FETCH_STATUS = 0
Begin
   SELECT OBJECT_NAME(object_id) 
      FROM sys.sql_modules 
      WHERE Definition LIKE '%'+@VName +'%' 
      AND OBJECTPROPERTY(object_id, 'IsProcedure') = 1 ;
      
   Fetch next from Findlinked into @VName;
End

Close Findlinked
Deallocate Findlinked

SSIS Jobs are different, Here is one for SSIS jobs with Job Name and Details

Declare @VName varchar(256)
Declare Findlinked cursor
LOCAL STATIC FORWARD_ONLY READ_ONLY
     FOR
Select name AS name
   From sys.servers
   Where is_linked = 1
      
Open Findlinked;
Fetch next from Findlinked into @VName;

while @@FETCH_STATUS = 0
Begin
   SELECT OBJECT_NAME(object_id) as ProcedureName 
      FROM sys.sql_modules 
      WHERE Definition LIKE '%'+@VName +'%' 
      AND OBJECTPROPERTY(object_id, 'IsProcedure') = 1 ;
      
   Fetch next from Findlinked into @VName;
END
Close Findlinked

Open Findlinked;
Fetch next from Findlinked into @VName;

while @@FETCH_STATUS = 0
Begin
   SELECT j.name AS JobName,js.command 
      FROM msdb.dbo.sysjobsteps js
         INNER JOIN msdb.dbo.sysjobs j
            ON j.job_id = js.job_id
      WHERE js.command LIKE '%'+@VName +'%'
   Fetch next from Findlinked into @VName;
END

Close Findlinked
Deallocate Findlinked

Determining which version and edition of SQL Server Database Engine is running

Open SQL Server Management Studio (SSMS) and connect to SQL Server. Run below query to find version and edition of SQL server.

SELECT  
    SERVERPROPERTY('productversion') as 'Product Version', 
    SERVERPROPERTY('productlevel') as 'Product Level',  
    SERVERPROPERTY('edition') as 'Product Edition',
    SERVERPROPERTY('buildclrversion') as 'CLR Version',
    SERVERPROPERTY('collation') as 'Default Collation',
    SERVERPROPERTY('instancename') as 'Instance',
    SERVERPROPERTY('lcid') as 'LCID',
    SERVERPROPERTY('servername') as 'Server Name'

Configure the remote query timeout Server Configuration Option

Use below query to increase default time out or disable time out time.

USE ps;  
GO  
EXEC sp_configure 'remote query timeout', 0 ;  
GO  
RECONFIGURE ;  
GO 

The remote query timeout option specifies how long, in seconds, a remote operation can take before SQL Server times out. The default value for this option is 600, which allows a 10-minute wait. This value applies to an outgoing connection initiated by the Database Engine as a remote query. This value has no effect on queries received by the Database Engine. To disable the time-out, set the value to 0. A query will wait until it completes.

For more information, see Server Configuration Options (SQL Server).

How to get size of all tables in database

Here is how you can get table sizes via SQL query, this works from SQL 2005 and above

SELECT 
    t.NAME AS TableName,
    s.Name AS SchemaName,
    p.rows AS RowCounts,
    SUM(a.total_pages) * 8 AS TotalSpaceKB, 
    CAST(ROUND(((SUM(a.total_pages) * 8) / 1024.00), 2) AS NUMERIC(36, 2)) AS TotalSpaceMB,
    SUM(a.used_pages) * 8 AS UsedSpaceKB, 
    CAST(ROUND(((SUM(a.used_pages) * 8) / 1024.00), 2) AS NUMERIC(36, 2)) AS UsedSpaceMB, 
    (SUM(a.total_pages) - SUM(a.used_pages)) * 8 AS UnusedSpaceKB,
    CAST(ROUND(((SUM(a.total_pages) - SUM(a.used_pages)) * 8) / 1024.00, 2) AS NUMERIC(36, 2)) AS UnusedSpaceMB
FROM 
    sys.tables t
INNER JOIN      
    sys.indexes i ON t.OBJECT_ID = i.object_id
INNER JOIN 
    sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id
INNER JOIN 
    sys.allocation_units a ON p.partition_id = a.container_id
LEFT OUTER JOIN 
    sys.schemas s ON t.schema_id = s.schema_id
WHERE 
    t.NAME NOT LIKE 'dt%' 
    AND t.is_ms_shipped = 0
    AND i.OBJECT_ID > 255 
GROUP BY 
    t.Name, s.Name, p.Rows
ORDER BY 
    t.Name

Alternatively if you want to get size of one table you can do it by using sp_spaceused, this can get you information on the disk space used by a table, indexed view, or the whole database.

USE psdb  
GO

EXEC sp_spaceused contact
GO

You can use the sp_spaceused command to get all tables in database by using the below command.

USE psdb  
GO

sp_msforeachtable 'EXEC sp_spaceused [?]' 
GO

Hope this useful

EXCEPT and INTERSECT in T-SQL

The UNION, EXCEPT and INTERSECT operators of SQL enable you to combine more than one SELECT statement to form a single result set. The UNION operator returns all rows.

EXCEPT
Returns any distinct values from the query to the left of the EXCEPT operator that are not also returned from the right query.
INTERSECT
Returns any distinct values that are returned by both the query on the left and right sides of the INTERSECT operator.

The INTERSECT operator returns all rows that are in both result sets. The EXCEPT operator returns the rows that are only in the first result set but not in the second

EXCEPT operator is another most important feature in SQL Server which is used to returns distinct rows by comparing the results of two input queries. Both SQL queries within the EXCEPT query, the number and the order of the columns must be the same in the result sets within similar data types.  EXCEPT operator is a very quick and easy way to find differences, especially when needing to get all differences including null.

When using INTERSECT operator the number and the order of the columns must be the same in all queries as well data type must be compatible.

For more Information check MSDN site

How can I list all foreign keys referencing a given table in SQL Server?

Here is how you can get list of all foreign key references using below query for all tables in your database

SELECT  obj.name AS FK_NAME,
    sch.name AS [schema_name],
    tab1.name AS [table],
    col1.name AS [column],
    tab2.name AS [referenced_table],
    col2.name AS [referenced_column]
FROM sys.foreign_key_columns fkc
INNER JOIN sys.objects obj
    ON obj.object_id = fkc.constraint_object_id
INNER JOIN sys.tables tab1
    ON tab1.object_id = fkc.parent_object_id
INNER JOIN sys.schemas sch
    ON tab1.schema_id = sch.schema_id
INNER JOIN sys.columns col1
    ON col1.column_id = parent_column_id 
AND col1.object_id = tab1.object_id
INNER JOIN sys.tables tab2
    ON tab2.object_id = fkc.referenced_object_id
INNER JOIN sys.columns col2
    ON col2.column_id = referenced_column_id 
AND col2.object_id = tab2.object_id

Hope this helps!!!

How to search a column name within all tables of a database?

Here is how we can get table name and column which you are looking for,

SELECT table_name,column_name FROM information_schema.columns
WHERE column_name like '%engine%'
-- OR ---
SELECT c.name AS ColName, t.name AS TableName
FROM sys.columns c
    JOIN sys.tables t ON c.object_id = t.object_id
WHERE c.name LIKE '%engine%'

Sample OUTPUT for one of the query,

ColName    TableName
EngineType    DSS_Setup