This started out as something else. We had a large database of about 100 GB that we wanted shift to our DR location. Due to bandwidth restrictions we started out to explore options on how we could reduce the overhead of transferring such a large file.
Now compressing a 100 GB file is not a joke. After some looking up I came across an interesting article by Chris over at solo-technology talking about Compression v/s Speed. It talked about a related issue but the same capability could be used for handling large files.
The best tool for doing this is 7Zip. It is an awesome freeware by Igor Pavlov handling a large number of compression formats. Check it out over here.
http://www.7-zip.org/
That aside, by using the command line version of this you could zip you file. When we started the zip file took 5 hours to finish just 30 % of compression. On check the task manager, I found that the whole process was just running at 100 % on a single thread of the 24 cores available on the server.
By using the following options,
I was able to compress the 100 GB SQL Server 2008 database backup file to just 16 GB ! in just 1 Hour !!!
This was way beyond my expectations :) Here is that technical syntax to use this
7z a -tbzip2 -mx=9 -mmt=on backup.zip backup.bak
"7z" is the program to initiate
"a" is to append the file
"-tbzip2" is to use the bzip2 format, this is important as only this format allows for multithreading for both the compression as well as the decompression
"-mx=9" is the number of passes, this is optional, I didn't use it, the system defaults this to 5
"-mmt=on" is to turn on the multithread capability, again this works only for the bzip2 format
"backup.zip" is the file that will get created
"backup.bak" is the file that needs to get backed up.
You could read through the technical literature available with 7zip to get a better hang of the application. This were just the specific options that are required to manage this requirement.
One other note, I did read the the bzip2 format has a restriction that it can include only 1 file at a time. I didn't try for multiple files.
Hope this helps in saving you precious bandwidth !!!
Wednesday, September 15, 2010
Friday, April 2, 2010
Microsoft Association Rules / Data mining in SQL Server 2008
Since the release of SQL Server 2005 Microsoft has dabbled quite a bit with Data mining models. While I have read that Data mining capability has been available with one of the later releases of SQL Server 2000, I haven't been able to come across too much material on the same.
First up I would like say the best place to start if you are serious about learning about Data mining is to go get yourself this really good book on the topic. Its extremely easy to read and the examples used are quite practical.
The book is
Data Mining with SQL Server® 2008
by
Jamie MacLennan
ZhaoHui Tang
Bogdan Crivat
Back to original reason of the post, being a techno functional individual, I was keen to see how we could effectively cross sell products to stores since we had their complete sales history. Ensuring that our consumers never left the store without the products they wanted goes a long way in ensuring consumer loyalty and company performance.
Having most of the data already available in SQL Server 2008 I was keen to see what we could use to help identify these gaps. In SQL Server 2008 Microsoft has the following Data Mining Models
Microsoft Naıve Bayes
Decision Trees Algorithm
Time Series Algorithm
Microsoft Clustering
Sequence Clustering
Association Rules
Neural Network and Logistic Regression
The best fit for the requirement I had was the Microsoft Association Rules. It allowed me to build the model utilizing the classification of outlets that we already had.
I built the model using three classifications we had for our outlets along with products that the outlet already sold. In order to do this I needed to create a nested table for the products as one outlet could sell multiple products. Also important to note that the data types for Data mining objects are slightly different from the data types that are used by standard SQL queries. The DMX I used is below.
Post creating the structure , I applied the Microsoft Association Rules Algorithm requested the prediction of the product utilizing the standard support and probability the model is generally used with.
The next step was to feed in data into the model to let it build the association rules. This was done using the following syntax. Am important point to note here is that the key data needs to be sorted. Sometimes in spite of sorting it still gives errors saying the data most probably is not sorted. The solution for this is to ensure that the key that is used is specifically a nvarchar source data type. I faced a problem when my key happened to have numeric data as well as string data. Needed to convert everything before I could successfully load the data into the mining structure.
Once the data is loaded into the model, the last step is to retrieve out the recommendations that the model would generate. I have done this for the store ABC by passing the store along with the products that the store purchased into the model and asking it to return 5 recommendations based on the model as well as the classification of the store and the products that the store sells.
This is done through the following DMX statement.
In case you want this data in the standard query windows you could always create a Linked server to OLAP where the Data Mining model exists.
Do let me know if you have any comments !
First up I would like say the best place to start if you are serious about learning about Data mining is to go get yourself this really good book on the topic. Its extremely easy to read and the examples used are quite practical.
The book is
Data Mining with SQL Server® 2008
by
Jamie MacLennan
ZhaoHui Tang
Bogdan Crivat
Back to original reason of the post, being a techno functional individual, I was keen to see how we could effectively cross sell products to stores since we had their complete sales history. Ensuring that our consumers never left the store without the products they wanted goes a long way in ensuring consumer loyalty and company performance.
Having most of the data already available in SQL Server 2008 I was keen to see what we could use to help identify these gaps. In SQL Server 2008 Microsoft has the following Data Mining Models
Microsoft Naıve Bayes
Decision Trees Algorithm
Time Series Algorithm
Microsoft Clustering
Sequence Clustering
Association Rules
Neural Network and Logistic Regression
The best fit for the requirement I had was the Microsoft Association Rules. It allowed me to build the model utilizing the classification of outlets that we already had.
I built the model using three classifications we had for our outlets along with products that the outlet already sold. In order to do this I needed to create a nested table for the products as one outlet could sell multiple products. Also important to note that the data types for Data mining objects are slightly different from the data types that are used by standard SQL queries. The DMX I used is below.
Post creating the structure , I applied the Microsoft Association Rules Algorithm requested the prediction of the product utilizing the standard support and probability the model is generally used with.
The next step was to feed in data into the model to let it build the association rules. This was done using the following syntax. Am important point to note here is that the key data needs to be sorted. Sometimes in spite of sorting it still gives errors saying the data most probably is not sorted. The solution for this is to ensure that the key that is used is specifically a nvarchar source data type. I faced a problem when my key happened to have numeric data as well as string data. Needed to convert everything before I could successfully load the data into the mining structure.
Once the data is loaded into the model, the last step is to retrieve out the recommendations that the model would generate. I have done this for the store ABC by passing the store along with the products that the store purchased into the model and asking it to return 5 recommendations based on the model as well as the classification of the store and the products that the store sells.
This is done through the following DMX statement.
In case you want this data in the standard query windows you could always create a Linked server to OLAP where the Data Mining model exists.
Do let me know if you have any comments !
Labels:
Data mining,
SQL Server 2005,
SQL Server 2008
Sunday, March 21, 2010
Acer eMachine E725 XP Drivers
Since this laptop is bundled with Linux, you need to have all the drivers in place before you can install XP on this E725 Laptop.
Acer has the drivers as part of the E720 release and you could get all of them on the below ftp site of acer itself
ftp://ftp.work.acer-euro.com/notebook/.em/E720/XP%20SP2/
Incase you want to do Vista on the Acer eMachine E725 instead , go to this location
ftp://ftp.work.acer-euro.com/notebook/.em/E720/Vista%20SP1/
Hope that helps, do leave a comment !!
Acer has the drivers as part of the E720 release and you could get all of them on the below ftp site of acer itself
ftp://ftp.work.acer-euro.com/notebook/.em/E720/XP%20SP2/
Incase you want to do Vista on the Acer eMachine E725 instead , go to this location
ftp://ftp.work.acer-euro.com/notebook/.em/E720/Vista%20SP1/
Hope that helps, do leave a comment !!
Labels:
Acer,
Acer eMachine E725,
E725,
Laptop,
Vista Drivers,
XP Drivers
Saturday, February 6, 2010
Nokia 5230 Free Navigation / Maps
Recently Nokia made Navigation free atleast in India. However if you now tried to connect for free navigation it still asks for you go to the store. The reason for this is that Nokia has allowed free Navigation using Ovi Maps only on the new version of the software.
So if you have bought your phone couple of weeks ago there is a very good chance you are on the old version. What you need to do is go to the Nokia site and upgrade your version of Ovi maps to the latest version.
There are two ways of getting this done, one is to download the software on your computer and then Sync with your phone which would then automatically upgrade the version of Ovi Maps that is present on your phone.
The second option is to directly download the software to your phone and run it from there. However this would require a data connection from your service provider.
The link for downloading the software is here
If you want to get the map data stored directly on your computer you should go through this post.
http://piglings.blogspot.com/2010/01/nokia-5230-maps.html
So if you have bought your phone couple of weeks ago there is a very good chance you are on the old version. What you need to do is go to the Nokia site and upgrade your version of Ovi maps to the latest version.
There are two ways of getting this done, one is to download the software on your computer and then Sync with your phone which would then automatically upgrade the version of Ovi Maps that is present on your phone.
The second option is to directly download the software to your phone and run it from there. However this would require a data connection from your service provider.
The link for downloading the software is here
If you want to get the map data stored directly on your computer you should go through this post.
http://piglings.blogspot.com/2010/01/nokia-5230-maps.html
Labels:
5230,
Navigation,
Nokia,
Nokia maps,
Ovi Maps
Saturday, January 9, 2010
Nokia 5230 Maps
My dad just got his Nokia 5230, nice phone that comes with Ovi Maps 3 installed however there were no map data available. Since he didn't have GPRS enabled on his phone I decided to do some digging around to make use of the GPS device bundled into the Nokia 5230.
Credit to symbian underground for helping me get this to work.
The first step is to have you your external card ready. Run Nokia Maps on your phone once before you add any maps or voices. This will create the required folder structure on your memory card. Quit the application after that.
On my Nokia the folder structure is cities with a subfolder called diskcache, filled with 16 subfolders called 0 to 9 and a to f, and each of these folders have another 16 subfolders 0 to 9 and a to f.
For India, download the following file from Nokia itself
India Maps
Extract the zip file into your local computer, once done copy drop the 16 folders called 0-9 and a-f into the diskcache folder of the Nokia 5230. You can do this by connecting your device as a external hard drive via the Micro USB port.
Thats it, maps should now work. You should now disconnect the connection to the computer and only then run maps again.
If you want voice as well download the following files
English or Hindi
Extract all the files into the cities folder. That is it.
For other countries check the maps links here
http://diabo.freehostia.com/symbian/indexmaps.htm
The voice links of other languages are here
http://diabo.freehostia.com/symbian/nokiamaps/voices001.html
If you want to enable free Navigation, do read this post
http://piglings.blogspot.com/2010/02/nokia-5230-free-navigation-maps.html
Hope it helps, do leave me a comments. No files are hosted on this site !
Credit to symbian underground for helping me get this to work.
The first step is to have you your external card ready. Run Nokia Maps on your phone once before you add any maps or voices. This will create the required folder structure on your memory card. Quit the application after that.
On my Nokia the folder structure is cities with a subfolder called diskcache, filled with 16 subfolders called 0 to 9 and a to f, and each of these folders have another 16 subfolders 0 to 9 and a to f.
For India, download the following file from Nokia itself
India Maps
Extract the zip file into your local computer, once done copy drop the 16 folders called 0-9 and a-f into the diskcache folder of the Nokia 5230. You can do this by connecting your device as a external hard drive via the Micro USB port.
Thats it, maps should now work. You should now disconnect the connection to the computer and only then run maps again.
If you want voice as well download the following files
English or Hindi
Extract all the files into the cities folder. That is it.
For other countries check the maps links here
http://diabo.freehostia.com/symbian/indexmaps.htm
The voice links of other languages are here
http://diabo.freehostia.com/symbian/nokiamaps/voices001.html
If you want to enable free Navigation, do read this post
http://piglings.blogspot.com/2010/02/nokia-5230-free-navigation-maps.html
Hope it helps, do leave me a comments. No files are hosted on this site !
Subscribe to:
Posts (Atom)