Points to consider to increase performance in ADO.NET – Part II

In my previous article, ten points on the general considerations on increasing the performance of ADO.NET were discussed. Let us examine in more detail on other factors that affect the performance.

1. Use of Transactions
Transactions are important for ensuring data integrity. But they can also have an operational cost if they are not rightly used. You must select the right transaction management mechanism for your application to significantly improve scalability and performance.
The following points should be remembered during your decision of using transactions:

a) Use SQL transactions when:
· You need server controlled transactions
· You can complete the transaction operation on a single data store in just one simple call to the datastore
b) Use ADO.NET transactions when:
· You need client-controlled transactions
· When you want to make multiple calls to a single datastore
c) Keep transactions as short as possible with as little code as possible to avoid locks.
d) Use the appropriate isolation level.
SQL Server and other database systems support various levels of isolation for transactions. Isolation shields database operations from the effect of other concurrent transactions.
There are basically four levels of isolation-
· Read uncommitted,
· Read committed,
· Repeatable read
· Serializable

The highest isolation level, serializable, reduces concurrency and provides the highest level of data integrity.
The lowest isolation level, read uncommitted, gives the opposite result.

By selecting the correct level of isolation that pertains to your application, you can improve performance and scalability.

2. Avoid code that can lead to deadlock.
Sometimes your code may get trapped in a deadlock when it is highly data centric. To avoid deadlocks, always access tables in the same order across transactions in your application. The likelihood of a deadlock increases when you access tables in a different order each time you access them.

3. Consider SET NOCOUNT ON for SQL Server
When you use SET NOCOUNT ON, the message that indicates the number of rows that are affected by the T-SQL statement is not returned as part of the results. When you use SET NOCOUNT OFF, the count is returned. Using SET NOCOUNT ON can improve performance because network traffic can be reduced.

4. The Parameter Collection
Use the Parameters collection when you call a stored procedure and when you build SQL commands. Also ensure that you explicitly set the data type for each parameter. This will avoid the round trips to server by checking the data type on the client and the likelihood that the Parameter object could set an invalid type.

3. Cache stored procedure SqlParameter objects.
Often, applications must run commands multiple times. To avoid recreating the SqlParameter objects each time, cache them so that they can be reused later. You can use a Hashtable object to cache SqlParameter objects.
For code samples for the parameter caching approach, check out this link:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/daab-rm.asp.

4. Avoid moving Binary Large Objects Repeatedly
Avoid moving BLOB data more than one time from the server to client and vice versa.
For example, if you build a Web application that serves images, store the images on the file system and the file names in the database instead of storing the images as BLOBs in the database. Storing the images as BLOBs in the database means that you must read the BLOB from the database to the Web server and then send the image from the Web server to the browser. Reading the file name from the database and having the Web server send the image to the browser reduces the load on the database server.

5. Paging Records
Paging records is a common application scenario. The records that you need to page through can often be based on user input. For example, they can be based on a search keyword entered through a search screen. Or, the records can be common to all users, like a product catalogue.

There are two basic approaches to paging:
· You can return the whole result set from the database to the client. The client caches the result set and then displays the most relevant results to the user by using the correct filtering mechanism. But this will involve processing cost at the client and memory for caching.

· You can have the database assume the additional role of a filter by making the database return only the most relevant result set to the client. This will involve network cost for the amount of data sent across the network.

Depending upon the type of application and load on the server, we must try to decide the best between the above two approaches.

6. Exception Management
a) Avoid relying on an Exception error handler to detect connection state availability. use the ConnectionState.Open or ConnectionState.Close method to check the state before use.

b) Use try/finally more often than try/catch/finally. Using finally gives you the option to close the connection, even if an exception occurs.

c) Use specific handlers to catch specific exceptions. For example, if you know that your code may cause an SqlException, use catch(SqlException sqlEx) and then use a generic exception handler like catch(Exception ex).

I have covered most of the points I've learnt on increasing performance in ADO.NET.
One interesting topic which remains to be addressed is about the best use of DataSet and DataReader, which I think is addressed in numerous articles on web. Here is one:
http://geekswithblogs.net/ranganh/archive/2005/04/25/37618.aspx

Ten points to consider to increase performance in ADO.NET

Here is a consolidated list of how you can improve the performance of your .NET apps:

1. Design your data access layer based on how the data is used.

In larger applications, it’s always better to go for a separate Data Access Layer (DAL) to abstract the underlying data store complexity and to provide a logical separation. Having the data access logic in the same presentation layer may increase performance but at the cost of maintainability.

You can use the Microsoft Application Blocks for simplifying your tasks of Data Access and Exception handling. (I personally prefer this)

2. Cache your data to avoid unnecessary round trips and network overhead.

Try to cache data that is used across your application, in the layer that is close to the consumer of data. This will reduce the latency in network to fetch data. Note that if your data in cache needs to be updated too frequently, then better you don’t cache that data.

3. Acquire late, release early.

Open database connections right only when you need them. Close the database connections as soon as you are finished. Acquire locks late, and release them early.

4. Close disposable resources.

Make sure that you call either the Dispose or Close method on resources that are disposable, as soon as you are finished with using the resource.

5. Reduce round trips.

· If you have some 3 or 4 SQL statements, try to use batch sql statements in a stored procedure to decrease round trips.
· Use ExecuteScalar method for getting a single result.
· Use ExecuteNonQuery method when you want to execute any DDL statement.
· Use connection pooling to help avoid extra round trips. By reusing connections from a connection pool, you can avoid the round trips that are associated with connection establishment and authentication.

6.Return only the data you need.

Evaluate the data that your application actually requires and return only that data. This will minimize the bandwidth consumption in the network.

7. Use Windows authentication.

From a security perspective, use Windows authentication instead of SQL authentication. This ensures that credentials are not passed over the network, database connection strings do not contain credentials, and you can apply standard Windows security policies to accounts. Remember to use connection pooling with your connection.

8. Use stored procedures.

Avoid embedded SQL statements and use Store Procedures (SP) instead. This has the following advantages:
· A Logical separation of Data access code from your Business Logic code
· Queries can be optimized for performance from SQL server
· Deployment becomes easier as, for any change in SP you don’t need to redeploy you application.
· SPs allow the batch execution of SQL commands
· You can impose specific restrictions on selected stored procedures for security reasons. This is very difficult to be done from embedded SQL.
· You can avoid the most dangerous SQL Injection by using parameterized SPs.

9. Consider how to handle exceptions.

You can use try/finally blocks to ensure that connections and other resources are closed, regardless of whether exceptions are generated or not. The best way for abstracting all exceptions from user is to log them to a file or Windows Event log.

10. Use appropriate normalization.

You may want a normalized database to minimize data duplication but be aware that you don’t over-normalize. This can affect the performance and scalability of your application

The above list is purely based on my working experience and there are many more points to add to this, which I’ll be adding in the next part of this article.

What are Application Domains?

Many of us have come across the word "Application Domains" during our projects. But mostly I have seen many people having a wrong notion on what Application Domains really are.
Now lets examine what these App Domains are.

Application domains are basically logical boundaries which provide a secure, versatile unit of processing that the Common Language Runtime (CLR) uses to provide isolation between applications.

Thus, you can run several application domains in a single process with the same level of isolation that would exist in separate processes, but without incurring the additional overhead of making cross-process calls or switching between processes.
The ability to run multiple applications within a single process dramatically increases server scalability.

Thus, the Application domains provide an isolation that has the following advantages:

1. Application Security
2. Faults in one application cannot affect other applications.
3. Individual applications can be stopped without stopping the entire process.
4. Code running in one application cannot directly access code or resources from another application. The CLR enforces this isolation by preventing direct calls between objects in different application domains.
5. The Scope of the code is controlled by the Application domain. The Application domain provides configuration settings such as application version policies, the location of any remote assemblies it accesses, and information about where to locate assemblies that are loaded into the domain.
6. The Permissions granted to code can be controlled by the application domain in which the code is running.

HOW TO: Programmatically create SSIS packages using Whidbey

Hi,

If you had read my last two articles, you would now be familiar with SSIS and how to create simple packages Business Intelligence Development Studio.

In this article, let us now check how to programmatically create a package/edit an existing package using the APIs provided with SQL Server 2005.

Let us start with creating a simple Console Application in Whidbey.

First of all, we need to add the following references to your Whidbey project:

1. Microsoft.SqlServer.DTSPipelineWrap.dll
2. Microsoft.SQLServer.DTSRuntimeWrap.dll
3. Microsoft.SQLServer.ManagedDTS.dll

The reference dlls can be found in the following location:
C:\Program Files\Microsoft SQL Server\90\SDK\Assemblies

You must then add the namespace references in the Program.cs file as follows:

#region Using directives
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
#endregion

An SSIS package can be created using the Package class present in the Microsoft.SqlServer.Dts.Runtime namespace as follows:

Package pkg = new Package();
pkg.Name = "MyCreatedPackage";
pkg.ID = "MyPackageID";

Now lets see how to take an existing package, modify its properties and save the package to your file system or to Yukon server.

We begin with defining a Package object and an Application object.

Package pkg = new Package();
Application a = new Application();

The Application class is used to discover and access Package objects.
Next we declare the MainPipe interface which is used to programmatically configure the data flow task. The IDTSComponentMetaData90 Interface contains the definition of a data flow component; including the custom properties, inputs, outputs, and input and output columns defined on a data flow component.

IDTSComponentMetaData90 oledbSource;
IDTSComponentMetaData90 oledbDestination;
MainPipe dataFlow;

In case you are trying to modify the properties of an existing package, you must first load the external package that you already have as a template.

pkg = a.LoadPackage(@"C:\TestPackages\MyPackage.dtsx", null);

You must now add the connection using the ConnectionManager class and set connection properties. The ConnectionManager class contains all the information necessary to connect to a single type of data source.
//Add connections
ConnectionManager conMgr = pkg.Connections.Add("OLEDB");

/// Set stock properties
conMgr.Name = "OLEDBConnection";
conMgr.ConnectionString = "Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=Test;Data Source=mdpkb2e139;Auto Translate=False;";

The next step is to add/edit the data flow task. The following is the code to add a Data Flow Task and set its properties:

TaskHost th = pkg.Executables.Add("DTS.Pipeline") as TaskHost;
th.Name = "DataFlow";
dataFlow = th.InnerObject as MainPipe;

//set source component
oledbSource = dataFlow.ComponentMetaDataCollection.New();
oledbSource.ComponentClassID = "DTSAdapter.OLEDBSource";
oledbSource.Name = "OLEDBSource";
CManagedComponentWrapper instanceSource = oledbSource.Instantiate();

oledbSource.RuntimeConnectionCollection.New();
oledbSource.RuntimeConnectionCollection[0].ConnectionManagerID = pkg.Connections["OLEDBConnection"].ID;

oledbSource.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.ToConnectionManager90(pkg.Connections["OLEDBConnection"]);
instanceSource.ProvideComponentProperties();
instanceSource.SetComponentProperty("OpenRowset", "Emp");
instanceSource.SetComponentProperty("AccessMode", 0);

// Acquire Connections and reinitialize the component
instanceSource.AcquireConnections(null);
instanceSource.ReinitializeMetaData();
instanceSource.ReleaseConnections();

In case you already have a template with the data flow task, we can get the handle of it using the Executable collection. The following is the code for modifying a DataFlow task:

Executable exe = x.Executables["Data Flow Task"];
TaskHost th2 = exe as TaskHost; d
ataFlow = th2.InnerObject as MainPipe;
IDTSComponentMetaDataCollection90 metadataCollection = dataFlow.ComponentMetaDataCollection;
//set source component
oledbSource = dataFlow.ComponentMetaDataCollection[0];
if (oledbSource.RuntimeConnectionCollection.Count > 0)
{
oledbSource.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.ToConnectionManager90(x.Connections["OLEDBConnection"]);
oledbSource.RuntimeConnectionCollection[0].ConnectionManagerID = x.Connections["OLEDBConnection"].ID;
}
CManagedComponentWrapper instanceSource = oledbSource.Instantiate();

instanceSource.SetComponentProperty("OpenRowset", "Temp_1");
instanceSource.SetComponentProperty("AccessMode", 0);

// Acquire Connections and reinitialize the component
instanceSource.AcquireConnections(null);
instanceSource.ReinitializeMetaData();
instanceSource.ReleaseConnections();

The same steps are followed for setting the properties of the destination component. The following is the code for modifying the connection properties of OLEDB Destination component:

//set destination component
oledbDestination = dataFlow.ComponentMetaDataCollection[1];
if (oledbDestination.RuntimeConnectionCollection.Count > 0)
{
oledbDestination.RuntimeConnectionCollection[0].ConnectionManager = DtsConvert.ToConnectionManager90(x.Connections["OLEDBConnection"]);
oledbDestination.RuntimeConnectionCollection[0].ConnectionManagerID = x.Connections["OLEDBConnection"].ID;
}
CManagedComponentWrapper instanceDest = oledbDestination.Instantiate();
instanceDest.SetComponentProperty("OpenRowset", "Temp_2");
instanceDest.SetComponentProperty("AccessMode", 0);
// Acquire Connections and reinitialize the component
instanceDest.AcquireConnections(null);
instanceDest.ReinitializeMetaData();
instanceDest.ReleaseConnections();

The next step is to "Map" the columns of the source and destination components. The following is the code to map the input column collection (source collection) to the External Metadata column collection (Destination collection):

IDTSInput90 input = oledbDestination.InputCollection[0];
IDTSVirtualInput90 vInput = input.GetVirtualInput();
foreach (IDTSVirtualInputColumn90 vColumn in vInput.VirtualInputColumnCollection)
{
IDTSInputColumn90 vCol = instanceDest.SetUsageType(input.ID, vInput, vColumn.LineageID, DTSUsageType.UT_READWRITE);
instanceDest.MapInputColumn(input.ID, vCol.ID, input.ExternalMetadataColumnCollection[vColumn.Name].ID);
}

Finally we save the package to the file system.

string currentDirectory = System.IO.Directory.GetCurrentDirectory();
a.SaveToXml(currentDirectory + "\\DTSDataflow.dtsx", pkg, null);
Console.WriteLine("Successfully created an SSIS package");
Console.ReadLine();

In case you want to save the package directly to Yukon server, you can use the following code:

a.SaveToSqlServer(pkg, null, "mdpkb2e139", "", "");

Here it goes! You have now programmatically created a simple SSIS package.

SSIS in Yukon - Part II - Creating a Simple Package in SSIS

In the previous article we had discussed about the new IDE for SSIS.
Now let us see how to create a simple SSIS package. The following are the steps:

1. From the File menu, click New-> Project.
2. Under Projects, choose Business Intelligence Projects and under Templates, choose Integration Service Project, and then click OK.
3. By default a package named Package.dtsx is created under the folder SSIS Packages. Click View-> Solution Explorer to see this.
4. Right click on the Connections pane, and then click New OLE-DB Connection.
5. Create two new connections - one to your Yukon source server and another to your Yukon destination server. Name these connections as MySourceConnection and MyDestinationConnection.
6. From the View menu, click Toolbox.
7. Now drag and drop a Data Flow Task component on the Control Flow pane.
8. Double-click the Data Flow Task component to take you to the Data Flow pane.
9. From the toolbox, drag and drop a OLE DB Source component and OLE DB Destination component. Name them as MySource and MyDestination respectively.
10. Double-click MySource.
11. Set the connection of MySource to MySourceConnection.
12. Set the Data-access mode to Table or View.
13. Set the name of the Source table from the listed tables of the database.
14. By clicking on the Columns tab on the left, you will be able to see the list of columns for the table. You can uncheck the columns that are not needed and then click OK.
15. Set the properties of MyDestination OLEDB component in the same way as in 14.
16. Click on MySource component. Drag the Green arrow you notice and make it point to MyDestination component. You have now connected the Source and the Destination components.
17. From the Debug menu click Start to run your first SSIS project.

Once your package has executed successfully, you will see components (Dataflow task, MySource and MyDestination) coloured in Green. And if there is any failure during execution, the components are coloured in Red. The success or failure of the package can be monitored by clicking on the Progress tab in the project during execution.

Note: The Source and Destination components must contain the same number of columns and same column names (case sensitive) to avoid Validation errors.

You can also have Event Handlers at the Package level or for each component in the Event Handler tab.

So this is how we go about creating a simple package. My next article will focus on how you can programmatically create these SSIS packages using Whidbey.

SQL Server Integration Services (SSIS) in Yukon

Many of us are familiar with Data Transformation Services (DTS) in SQL Server 2000.

SQL Server 2005 (code named Yukon) provides a completely new platform called SQL Server Integration Services (SSIS) that serves as the primary enterprise platform Extract, Transform and Load (ETL).

There are a looot of enhancements that has happened to DTS in Sql 2000. Creating packages is now separated to a new Development environment called Business Intelligence Development Studio (bienv.exe). This gives the same look n feel as our Visual Studio Dev Environment.
We have everything similar to VS - the Solution, Project, toolbox, setting configurations, etc.

The toolbox contains various tasks and componets that can be used in the editor.

There are 4 tabs provided in the editor:

1. Control flow - For controlling the task flow
2. Data Flow - For Data pumping tasks
3. Event handlers - For handling events and errors
4. Package Explorer - Gives a treeview view of the various components in the package
5. Execution results - For viewing the status during execution

The connections form a separate section that is common to the above four tabs. Connection manages for File, SQL Server, OLE DB can be added here.


Another cool feature is the Logging feature. All we have to do is just Right click the editor pane and enable Logging with a specific connection. All events now get logged in the specified server.

Every package that is a part of a SSIS project can be executed from BI. The deployment of these packages can be done either to the file system or to SQL Server. The packages deployed in SQL Server will be saved in the DTS Server.

My next article will throw more light on working with SSIS and programmability in Whidbey for SSIS.

HOW TO : Call a C# function inside Yukon Stored Procedure

Recently I had an interesting requirement in my project to use a C# procedure inside SQL.
Prior to SQL Server 2005, we had to use only Extended store procedures..
But as Yukon is integrated with CLR, you can now call a C# Whidbey function inside SQL Server procedure.

The following is the way you can do it:

1. Start Visual Studio 2005.
2. On the File menu, click New Project.
3. Under Projects, choose Visual C# and under Templates choose SQL Server Project template. Click OK. By default a project named SqlServerProject1 is created.
4. You will now be prompted to specify a connection. Provide the connection of the Yukon server for which you need the C# function.

5. Right click SqlServerProject1 on Solution Explorer and click Add New Item. Choose any template that is provided.
6. Now replace the following code with the existing code provided with the template:

#region Using directives
using System;
using System.Collections.Generic;
using System.Text;
using System.Data;
using System.Data.Sql;
using System.Data.SqlServer;
#endregion
public class MyClass
{
public MyClass()
{
}
public static void GetDepartmentDetailsByNum(int DeptNo)
{
SqlPipe sp = SqlContext.GetPipe();
sp.Send("The Department number is " + DeptNo.ToString() + "\n");
SqlCommand cmd = SqlContext.GetCommand();
cmd.CommandText = "select * from dept where deptnum=" + DeptNo.ToString();
SqlDataReader rdr = cmd.ExecuteReader();
sp.Send(rdr);
}
public static void GetSum(int x, int y)
{
SqlPipe sp = SqlContext.GetPipe();
int z=x+y;
sp.Send(z.ToString());
}
}
7. Now save the project and Build the project.
8. Open SQL Server Management Studio and connect to the Yukon server.
9. Right click the Database where the C# function is needed and then click New Query.
10. In the query editor, paste the following code:

CREATE ASSEMBLY ManagedCsharpProcedure
FROM 'C:\Laksk\SqlServerProject1\bin\Debug\SqlServerProject1.dll'

CREATE PROCEDURE [dbo].[GetSum]
@x int ,@y int
AS
EXTERNAL NAME
SqlManaged.DeptDetails.GetSum
GO

11. Now test the GetSum procedure using the following statement :
exec GetSum 5,4

Thats it !!! You now have a C# method being called in SQL Server. The same method can also be used with recordsets as in the C# function GetDepartmentDetailsByNum that is included in the same class. So on passing the Dept Number, we get the Department details through the C# function.