Why I love Open Source:
At work, there is a layer of burocracy when it comes to forced software to use to do your job. At my previous job, I was extremely limited to what I could to and more often then not, was forced to come up with my own solutions to problems – and this is where open source came in. I was able to design / execute and implement a true 1 to 1 marketing vehicle based solely in open source applications (php, apache and mysql) and I’m not talking about a small task – the datasets I was working with where reaching hundreds of millions of records and worked out to be over 100 gigs in size. I was also able to analyze the results in a fraction of time. The whole solution took me a month to develop, a month to tweak, a month of tuning my queries and it never crashed. My only problem was running the model off a truly underpowered machine so runtime was severely impacted – but at the end of the day it didn’t cost the company any money (except my professional time). Right before I left they were in the process of building me a server with SAMP (solaris / apache /mysql / php) to help me crunch the data.
I think the main reason why projects fail is there is constant back and forth with the business and IT. This game turns into a “you didn’t write this in the spec” aspect and the business always comes back with “well we didn’t think of that when we had to write the spec a year earlier” which leads to failure.
The best part of this whole experience was I being able to implement a system / manage a catalogue of inventory / increase response rate without harassing our customers with no complaints from our customers. I never had one complaint from anyone (internal or external) throughout the whole process except for some collegues not understanding how the process worked and were confused (which was a good learning in itself).
Currently, my situation is even worse. There are people who don’t question the data (so they assume it’s correct) and we are extremely hampered in terms of the tools / resources we need to do our job efficiently. For example the business definition is different than what is in the data warehouse (Teradata) so you can imagine writing multiple syntax in a slow system to begin with to just pull out the right information. So I’ve decided to pull the data I need from the warehouse (at the lowest level), and store this in my own warehouse (guess what Mysql) using open source tools. I’m using Kettle (again open source) to move data / roll data from the warehouse into my own warehouse. Then managing my own tables of business logic to automate the process. The worst part of this whole process is it works.
The other people around here never think to look outside of their universe to come up with solutions to problems.
I think the main reason why projects fail is there is constant back and forth with the business and IT. This game turns into a “you didn’t write this in the spec” aspect and the business always comes back with “well we didn’t think of that when we had to write the spec a year earlier” which leads to failure.
The best part of this whole experience was I being able to implement a system / manage a catalogue of inventory / increase response rate without harassing our customers with no complaints from our customers. I never had one complaint from anyone (internal or external) throughout the whole process except for some collegues not understanding how the process worked and were confused (which was a good learning in itself).
Currently, my situation is even worse. There are people who don’t question the data (so they assume it’s correct) and we are extremely hampered in terms of the tools / resources we need to do our job efficiently. For example the business definition is different than what is in the data warehouse (Teradata) so you can imagine writing multiple syntax in a slow system to begin with to just pull out the right information. So I’ve decided to pull the data I need from the warehouse (at the lowest level), and store this in my own warehouse (guess what Mysql) using open source tools. I’m using Kettle (again open source) to move data / roll data from the warehouse into my own warehouse. Then managing my own tables of business logic to automate the process. The worst part of this whole process is it works.
The other people around here never think to look outside of their universe to come up with solutions to problems.
posted by CutlaS at 7:37 AM
0 Comments:
Post a Comment
<< Home