Compress large files on windows server 2008

This started out as something else. We had a large database of about 100 GB that we wanted shift to our DR location. Due to bandwidth restrictions we started out to explore options on how we could reduce the overhead of transferring such a large file.

Now compressing a 100 GB file is not a joke. After some looking up I came across an interesting article by Chris over at solo-technology talking about Compression v/s Speed. It talked about a related issue but the same capability could be used for handling large files.

The best tool for doing this is 7Zip. It is an awesome freeware by Igor Pavlov handling a large number of compression formats. Check it out over here.



http://www.7-zip.org/


That aside, by using the command line version of this you could zip you file. When we started the zip file took 5 hours to finish just 30 % of compression. On check the task manager, I found that the whole process was just running at 100 % on a single thread of the 24 cores available on the server.

By using the following options,

I was able to compress the 100 GB SQL Server 2008 database backup file to just 16 GB ! in just 1 Hour !!!

This was way beyond my expectations :) Here is that technical syntax to use this


7z a -tbzip2 -mx=9 -mmt=on backup.zip backup.bak


"7z" is the program to initiate
"a" is to append the file
"-tbzip2" is to use the bzip2 format, this is important as only this format allows for multithreading for both the compression as well as the decompression
"-mx=9" is the number of passes, this is optional, I didn't use it, the system defaults this to 5
"-mmt=on" is to turn on the multithread capability, again this works only for the bzip2 format
"backup.zip" is the file that will get created
"backup.bak" is the file that needs to get backed up.

You could read through the technical literature available with 7zip to get a better hang of the application. This were just the specific options that are required to manage this requirement.

One other note, I did read the the bzip2 format has a restriction that it can include only 1 file at a time. I didn't try for multiple files.

Hope this helps in saving you precious bandwidth !!!

Microsoft Association Rules / Data mining in SQL Server 2008

Since the release of SQL Server 2005 Microsoft has dabbled quite a bit with Data mining models. While I have read that Data mining capability has been available with one of the later releases of SQL Server 2000, I haven't been able to come across too much material on the same.


First up I would like say the best place to start if you are serious about learning about Data mining is to go get yourself this really good book on the topic. Its extremely easy to read and the examples used are quite practical.

The book is

Data Mining with SQL Server® 2008

by
Jamie MacLennan
ZhaoHui Tang
Bogdan Crivat


Back to original reason of the post, being a techno functional individual, I was keen to see how we could effectively cross sell products to stores since we had their complete sales history. Ensuring that our consumers never left the store without the products they wanted goes a long way in ensuring consumer loyalty and company performance.

Having most of the data already available in SQL Server 2008 I was keen to see what we could use to help identify these gaps. In SQL Server 2008 Microsoft has the following Data Mining Models

Microsoft Naıve Bayes
Decision Trees Algorithm
Time Series Algorithm
Microsoft Clustering
Sequence Clustering
Association Rules
Neural Network and Logistic Regression

The best fit for the requirement I had was the Microsoft Association Rules. It allowed me to build the model utilizing the classification of outlets that we already had.

I built the model using three classifications we had for our outlets along with products that the outlet already sold. In order to do this I needed to create a nested table for the products as one outlet could sell multiple products. Also important to note that the data types for Data mining objects are slightly different from the data types that are used by standard SQL queries. The DMX I used is below.




Post creating the structure , I applied the Microsoft Association Rules Algorithm requested the prediction of the product utilizing the standard support and probability the model is generally used with.



The next step was to feed in data into the model to let it build the association rules. This was done using the following syntax. Am important point to note here is that the key data needs to be sorted. Sometimes in spite of sorting it still gives errors saying the data most probably is not sorted. The solution for this is to ensure that the key that is used is specifically a nvarchar source data type. I faced a problem when my key happened to have numeric data as well as string data. Needed to convert everything before I could successfully load the data into the mining structure.




Once the data is loaded into the model, the last step is to retrieve out the recommendations that the model would generate. I have done this for the store ABC by passing the store along with the products that the store purchased into the model and asking it to return 5 recommendations based on the model as well as the classification of the store and the products that the store sells.
This is done through the following DMX statement.




In case you want this data in the standard query windows you could always create a Linked server to OLAP where the Data Mining model exists.

Do let me know if you have any comments !

Acer eMachine E725 XP Drivers

Since this laptop is bundled with Linux, you need to have all the drivers in place before you can install XP on this E725 Laptop.

Acer has the drivers as part of the E720 release and you could get all of them on the below ftp site of acer itself

ftp://ftp.work.acer-euro.com/notebook/.em/E720/XP%20SP2/


Incase you want to do Vista on the Acer eMachine E725 instead , go to this location

ftp://ftp.work.acer-euro.com/notebook/.em/E720/Vista%20SP1/

Hope that helps, do leave a comment !!

Nokia 5230 Free Navigation / Maps

Recently Nokia made Navigation free atleast in India. However if you now tried to connect for free navigation it still asks for you go to the store. The reason for this is that Nokia has allowed free Navigation using Ovi Maps only on the new version of the software.

So if you have bought your phone couple of weeks ago there is a very good chance you are on the old version. What you need to do is go to the Nokia site and upgrade your version of Ovi maps to the latest version.

There are two ways of getting this done, one is to download the software on your computer and then Sync with your phone which would then automatically upgrade the version of Ovi Maps that is present on your phone.

The second option is to directly download the software to your phone and run it from there. However this would require a data connection from your service provider.

The link for downloading the software is here

If you want to get the map data stored directly on your computer you should go through this post.

http://piglings.blogspot.com/2010/01/nokia-5230-maps.html

Nokia 5230 Maps

My dad just got his Nokia 5230, nice phone that comes with Ovi Maps 3 installed however there were no map data available. Since he didn't have GPRS enabled on his phone I decided to do some digging around to make use of the GPS device bundled into the Nokia 5230.

Credit to symbian underground for helping me get this to work.


The first step is to have you your external card ready. Run Nokia Maps on your phone once before you add any maps or voices. This will create the required folder structure on your memory card. Quit the application after that.

On my Nokia the folder structure is cities with a subfolder called diskcache, filled with 16 subfolders called 0 to 9 and a to f, and each of these folders have another 16 subfolders 0 to 9 and a to f.


For India, download the following file from Nokia itself

India Maps

Extract the zip file into your local computer, once done copy drop the 16 folders called 0-9 and a-f into the diskcache folder of the Nokia 5230. You can do this by connecting your device as a external hard drive via the Micro USB port.

Thats it, maps should now work. You should now disconnect the connection to the computer and only then run maps again.

If you want voice as well download the following files

English or Hindi

Extract all the files into the cities folder. That is it.




For other countries check the maps links here
http://diabo.freehostia.com/symbian/indexmaps.htm

The voice links of other languages are here
http://diabo.freehostia.com/symbian/nokiamaps/voices001.html

If you want to enable free Navigation, do read this post
http://piglings.blogspot.com/2010/02/nokia-5230-free-navigation-maps.html

Hope it helps, do leave me a comments. No files are hosted on this site !

SQL Server 2008 Memory Leak / windows server 2008 high memory utilization

Having working with Cubes on SQL Server 2000, I had grown used to the fact that SQL will use and release memory depending on its need and I would be able to track the about of memory each program is utlizing through the task manager tab.

However its important to note the following. With windows server 2008 and SQL Server 2008 ( probably SQL Server 2005) the above mentioned assumptions are completely wrong. SQL Server 2008 will utilize as much memory that is available on the box, in my case 64 GB if any job works with high volumes of data but will NEVER EVER release this RAM back to the system once the job completes.

In short, the memory get reserved for SQL Server 2008 till the service / Server gets restarted. In case SQL needs to run some other job that requires data to be loaded, it internally will flush out the older data and load what it needs and yes even then nothing gets released back to the server.

Also to be noted is that in the task manager, none of this memory is ever shown against any of the processes that are running. Even if you use process explorer (sysinternals) this memory does not show up against SQL Server 2008 there as well.



Sadly this leads people ( including me) to come to the conclusion that there is a memory leak in SQL Server 2008. To convince yourself, just limit the amount of memory SQL Server can use and then check out the memory utilization in task manager. It will be that amount plus whatever is required to run your server.

There is article that describes this much more technical detail here ...

http://sqlblog.com/blogs/jonathan_kehayias/archive/2009/08/24/troubleshooting-the-sql-server-memory-leak-or-understanding-sql-server-memory-usage.aspx

Its a good idea to prevent SQL from utilizing the entire memory else the server can become unstable.

P.S. There is no memory leak :)

Excel 2007 OLAP Writeback SSAS 2008

We wanted to build an application based on the Writeback capability of Microsoft. The dream was to have a cube with multiple dimension allowing our users to analyze data by slicing / dicing this cube and based on their analysis make changes to the numbers at various levels.

Once they were happy with the data post changes we were to flow this data to a downstream system. It was to be a highly interactive system for a reasonably competent user to plan target setting by carrying out what if analysis on the existing data.

The front end for the user was Excel 2007 as the navigation / user interface was phenomenon and all our users in our company were fairly competent in Excel.

The path we started down was to have SQL Server Analysis Services 2005 as the backend sitting on a Hexacore 4 processor Windows Server 2008 box. We then enabled the cube for Writeback. The way this works according to me at a high level is that whatever numbers you choose to writeback, it does not make the change directly to your cube, what it does however is to create a writeback partitition that stores the net off data so that you could see the required number on your front end.

Let me explain that a little more with an example, lets say for one depot for one product the value initially was 100. As part of the writeback lets say you want it to be 70. What actually happens is that an entry takes place in the writeback partition for that depot and that product as -30. Based on a join of these two the number dispayed is 70.

While this works perfectly for small sets of data, it fails miserably for larger sets. The solution becomes impractical and very sluggish if the writeback is at an aggregate level and affecting large number of leafs / touples. Post engaging with Microsoft Consulting we confirmed that this was not going to work out for the application we had in mind.

OLAP Writeback sucks, period !!! and even Microsoft didn't have a response :) Now just because the writeback using the writeback partitition is hopeless it does not mean that its the only way to writeback user inputs into the cube.

The path we finally deployed our solution was a combination of the following.
VBA ( Macro coding) / Web services / SQL Server 2008 Integration services and SQL Server 2008 Analysis Services.

The excel that was shared with the users had a custom built macro code that essentially looped through the page / row / column fields to gather the user selections. Care had to taken to be able to loop through both the dimension member as well as the attributes. ( This is feature of SQL Server 2005 + , do read up if you are unaware) Also note it gets more complicated in Pivot 12 as there is extended capability in terms of Value and Label filters, both of which needs to be captured as well incase someone has applied them.

Once all the filters are looped and the details captured, the macro uses this data to frame a create sub cube query that gets passed to the server using a web service. This sub cube that gets created on the fly contains the entire data from the main cube that needs to undergo the writeback / change.

The next step is to strip this newly created sub cube to retrieve the underlying rowset of data that makes up the cube. Once we have that into a table, we have the option of applying pretty much any rule on this data, disaggregation / static values / percentage of existing value / percentage of another value.

Post applying the rule, the data gets updated into the base cube data and an incremental process is carried out and voila you have implemented your writeback.

Just to share some of the numbers, we were able to carry out a writeback affecting 3 Lakh records ( Thats three hundred thousand records :) in India) in a little over 2 minutes with the majority of the time in the incremental processing. This ofcourse will depend on the size and complexity of your own cube.

If your curious about the rest of the design, well since we had the changes in the table we could build an SSIS package to incrementally flow this information down stream into a DB2 environment using OLE/DB2 providers which ofcourse has its own problems in life :)