Missing Excel Data Into SSIS (Jet Driver Config Issue)

By Mithun Bose at January 08, 2010 17:32
Filed Under: Integration Services, Troubleshooting

Couple of days ago while importing data from an Excel file to a SQL Server table using SSIS I ran into an interesting issue. I found out that some of the data in the Excel file were inserted as NULL in the SQL Server table. It was a bit confusing to start with, I thought I was doing something wrong but after a few checks I was convinced there is something behind the scenes which was causing this problem and there indeed was.

Issue:

SSIS inserts NULL value into a table even when there is good data in the corresponding cells in the excel source.

Reason:

SSIS uses the Jet driver under the hood to read data from an Excel file. During the import process the JET driver looks into the first 8 rows on each column to decide the corresponding data type. In my scenario the column was actually of type double; however, the first eight row was empty, so the JET driver treated that column as String data type or varchar. 

So, during the import process when the driver encountered a double data type, it decided that its a wrong type and it just silently ignored it and stored a NULL value in the table. If the column which you are inserting into does not allow nulls then the transformation will fail.

 

Fix:

There is a fix for this. The JET driver connection string has configuration named "Extended Property", this configuration has a flag name IMEX, if we set this to 1 then it instructs the driver to always read "intermixed" data as text. Intermixed mean columns that may contain numbers, strings etc data types. The connectionstrings.com website has the details on the different settings.

In SSIS you can access the ConnectionString property by clicking the Excel Connection Manager and then look for ConnectionString in the Properties window as shown in the image below

 

 

Set your connection string property similart to below:

Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\Work\SomeExcelFile.xls;Extended Properties="EXCEL 8.0;HDR=YES;IMEX=1";

 

Comments

1/28/2010 12:05:59 PM #

Nagendra

I have similar problem with SSIS. i am trying to transfer data from excel to sql table but data is getting truncated after 255 characters. Size of column in table is varchar(max) and in excel column is text. one of the row in excel(100 th row) is having 989 characters but when i am transfering the data from excel to table it is truncating the data from that row and only inserting 255 character. When i move that row to 1st or 2nd row in Excel sheet. Data is transferring correctly. But each time we can not do that. Could you please help me on this.  

Nagendra India |

1/28/2010 3:20:09 PM #

mithun

Hi Nagendra,

You have to use Data Conversion transform. The column in question - set it's output data type to DT_NTEXT in the data conversion transform and try. Hopefully that should fix the problem. Check this link msdn.microsoft.com/en-us/library/ms141706.aspx for more info on the data conversion transform.

Cheers,
Mithun
http://www.mithunbose.com
http://blog.mithunbose.com

mithun United Kingdom |

2/27/2010 9:48:33 PM #

money

I'm curious what CMS your website is built on? I really like how it looks all the visitor functions that are available. Sorry if this is the wrong place to ask this but I wasn't sure how to contact you - thanks.

money United States |

3/19/2010 1:56:21 PM #

Hugo Herrero

I like the blog, but could not find how to subscribe to receive the updates by email. Can you please let me know?

Hugo Herrero United States |

3/20/2010 7:44:18 AM #

Daniel Millions

Do have an email system where your blog posts emailed to me?

Daniel Millions United States |

3/20/2010 11:01:22 AM #

Bulgarian Property

I like the blog, but could not find how to subscribe to receive the updates by email.

Bulgarian Property United States |

3/25/2010 2:59:00 AM #

privat Krankenversicherung

by means of Search engines while searching for for initial help and your position looks really interesting for me. Good

privat Krankenversicherung Austria |

About the author

I am a professional software developer based in London, England. I specialize in developing enterprise application and integration framework in C#, SQL Server, ASP.NET. 

My area of interests include SOA using WCF, Web, Business Intelligence and Database programming. I am a Computer Science Graduate and Microsoft Certified Professional Developer in enterprise application.