Steven S. Ross - Home

Dynamic SQL jumping a hoop (with a loop)

2009-04-15T15:57:00Z

Ok ignore the title, SQL is all business so lets get back to the “serious” SQL train.

So maybe you have a table that is very flat, someone did something like data1, data2, data3 as column names and you want to loop over it and insert it into a new table that can hold an infinite amount of meta-data with a relationship. Hey after all we are programming with relational databases these days.

So here are the basics, the loop (from 1 to 10):


DECLARE @intFlag INT
SET @intFlag = 1
WHILE (@intFlag <11)
BEGIN
-- DO LOOP STUFF HERE
SET @intFlag = @intFlag + 1
CONTINUE;
END
GO

So that is useful and all but, how do we create some dynamic sql with lovely concatenations?

Like so:


Declare @SQL VarChar(1000)
Select @SQL = 'INSERT new_table_name(description, status, date_due, date_complete, date_start, ref_id)'
Select @SQL = @SQL + ' SELECT descrip' + CAST(@intFlag AS varchar) + ', status' + CAST(@intFlag AS varchar) + ', due' + CAST(@intFlag AS varchar) + ', complete' + CAST(@intFlag AS varchar) + ', start' + CAST(@intFlag AS varchar) + ', old_id'
Select @SQL = @SQL + ' FROM old_table_name'
Select @SQL = @SQL + ' WHERE required_column' + CAST(@intFlag AS varchar) + ' IS NOT NULL'
Exec ( @SQL)

So now we just combine it all:


DECLARE @intFlag INT
SET @intFlag = 1
WHILE (@intFlag <11)
BEGIN
Declare @SQL VarChar(1000)
Select @SQL = 'INSERT new_table_name(description, status, date_due, date_complete, date_start, ref_id)'
Select @SQL = @SQL + ' SELECT descrip' + CAST(@intFlag AS varchar) + ', status' + CAST(@intFlag AS varchar) + ', due' + CAST(@intFlag AS varchar) + ', complete' + CAST(@intFlag AS varchar) + ', start' + CAST(@intFlag AS varchar) + ', old_id'
Select @SQL = @SQL + ' FROM old_table_name'
Select @SQL = @SQL + ' WHERE required_column' + CAST(@intFlag AS varchar) + ' IS NOT NULL'
Exec ( @SQL)
SET @intFlag = @intFlag + 1
CONTINUE;
END
GO

Ask away if there are any questions.

OOP - it's not perfect

2009-03-13T13:41:00Z

Ok, not trying to start a flame war but, I’m republishing this, it was from an interview with Rob Pike (works for google, really smart, had his hands in the early days of Unix). This is a great response to a question posed to him and for the record the original article is here Rob Pike Responds but, I am just publishing one part of it. I’d encourage you to read it as we should always try to solve a problem the best way possible and that is what this is about.

Question Hello!

Maybe this is an overly-asked question, but I still often ponder it. Does object-oriented design negate or diminish the future prospects of Unix’s continuing popularity?

I’ve developed in C (which I still love), but lately, I’ve been doing a lot of purely object-oriented development in Java. Using things like delegation and reusable classes have made life so much easier in many respects. Since the *nixes are so dependent upon C, I was wondering what future you see in C combined with Unix. Like I said, I love C and still enjoy developing in Unix, but there has to be a point where you build on your progress and the object-oriented languages, in my opinion, seem to be doing that.

Thank you for all your contributions!!!

Pike:

The future does indeed seem to have an OO hue. It may have bearing on Unix, but I doubt it; Unix in all its variants has become so important as the operating system of the internet that whatever the Java applications and desktop dances may lead to, Unix will still be pushing the packets around for a quite a while.

On a related topic, let me say that I’m not much of a fan of object-oriented design. I’ve seen some beautiful stuff done with OO, and I’ve even done some OO stuff myself, but it’s just one way to approach a problem. For some problems, it’s an ideal way; for others, it’s not such a good fit.

Here’s an analogy. If you want to make some physical artifact, you might decide to build it purely in wood because you like the way the grain of the wood adds to the beauty of the object. In fact many of the most beautiful things in the world are made of wood. But wood is not ideal for everything. No amount of beauty of the grain can make wood conduct electricity, or support a skyscraper, or absorb huge amounts of energy without breaking. Sometimes you need metal or plastic or synthetic materials; more often you need a wide range of materials to build something of lasting value. Don’t let the fact that you love wood blind you to the problems wood has as a material, or to the possibilities offered by other materials.

The promoters of object-oriented design sometimes sound like master woodworkers waiting for the beauty of the physical block of wood to reveal itself before they begin to work. “Oh, look; if I turn the wood this way, the grain flows along the angle of the seat at just the right angle, see?” Great, nice chair. But will you notice the grain when you’re sitting on it? And what about next time? Sometimes the thing that needs to be made is not hiding in any block of wood.

OO is great for problems where an interface applies naturally to a wide range of types, not so good for managing polymorphism (the machinations to get collections into OO languages are astounding to watch and can be hellish to work with), and remarkably ill-suited for network computing. That’s why I reserve the right to match the language to the problem, and even – often – to coordinate software written in several languages towards solving a single problem.

It’s that last point – different languages for different subproblems – that sometimes seems lost to the OO crowd. In a typical working day I probably use a half dozen languages – C, C++, Java, Python, Awk, Shell – and many more little languages you don’t usually even think of as languages – regular expressions, Makefiles, shell wildcards, arithmetic, logic, statistics, calculus – the list goes on.

Does object-oriented design have much to say to Unix? Sure, but no more than functions or concurrency or databases or pattern matching or little languages or….

Regardless of what I think, though, OO design is the way people are taught to think about computing these days. I guess that’s OK – the work does seem to get done, after all – but I wish the view was a little broader.

Fast csv import mysql in rails or any language

2009-03-12T13:57:00Z

Been working a lot with rails lately and I needed to import a large dataset (almost 3200 records). If you do something like this in rails you have to use the fastercsv gem, as, well it is faster. The thing is though, ActiveRecord isn’t. So although I can read the csv file in about 2-3 seconds creating the records ends up taking about 10-12. I didn’t really like this performance wise so I poked around on the web and found this method using “LOAD DATA INFILE” – a feature in mysql that lets you specify a csv to load and map to columns and boy, it is fast. The whole process takes about as long as it takes fastercsv to read the file (2-3 seconds now).

So here is the method:



def fast_import
    csv_string = FasterCSV.generate do |csv| 
      FasterCSV.foreach( absolute_path, {:headers => true, :return_headers => false, :header_converters => :symbol} ) do |row|
        csv << [row[:firstname],row[:lastname],row[:doctor_name],row[:homephone],self.id,row[:insurance],row[:mi],row[:dob],row[:address1],row[:address2],row[:city],row[:state],row[:zip],row[:id],row[:salutation],row[:survey_result_code] ]
      end
    end
    tmp_file = absolute_path + '.tmp'
    File.open(tmp_file, 'w') {|f| f.write(csv_string) }
    ActiveRecord::Base.connection.execute(load_data_infile(tmp_file))
    system("rm", tmp_file) # remove the temp file
  end

private
def load_data_infile(temp_path)
           <<-EOF
                 LOAD DATA INFILE "#{temp_path}" 
                          INTO TABLE attendees
                          FIELDS TERMINATED BY ','
                             (first_name, last_name, doctor_name, home_phone, event_id, insurance, middle_initial, date_of_birth, address, address_continued, city, state, zip_code, barcode_id, salutation, survey_result_code)
                            SET created_at = '#{Time.current.to_s(:db)}',
                            updated_at = '#{Time.current.to_s(:db)}';
            EOF
        end

Ok all we are doing is using fastercsv to parse the uploaded file, read it and then write it back to the disk so that mysql can import it. The first method is rails specific but the second could be used in any language as it is really just some mysql code. It is pretty self explanatory. If you need some help getting it working, just leavea comment!

Savannah - dying of the fountains

2009-03-09T23:54:00Z

Well St. Pats is officially on its way and just in case you haven't heard Savannah is crazy for St. Patrick's Day. Check out the video of the fountain being dyed green.

<embed src="http://services.brightcove.com/services/viewer/federated_f8/1184614610" height="412" width="486"></embed>

Learning git? This might help.

2009-02-24T14:25:00Z

I stumbled upon this after browsing some podcasts on rails the other day. The learning rails podcast has an episode on git (a distributed version control system). If you have heard the buzz (git is basically the new SVN, at least in hype). Go check it out

Quick tip for mysql speed

2009-02-23T15:55:00Z

I was doing some query tuning the other day (live search on a page and I needed it to go as fast as possible) I was searching about 3000 records and seeing how much performance I could squeeze out of mysql and a simple LIKE search. Put it simply if you want the most fastest (yes I just said that) you need to switch your tables from InnoDB to MyIsam because MyIsam doesn’t use transactions. Could be a no go for you but, if you need to do a fast search against the db it is your best bet. I saw my searches improve in speed about 2-3 times (however your mileage may vary).

Kitesurfing: where do you get the time?

2009-02-12T15:10:00Z

Ok, this is directed at a fan, well 3 comments to my dribble, I consider that a fan, hah! Anyway, Diane posed the question in response to my kitesurfing experience: “Where do you get the time?” So I thought heavily about the question (really I didn’t, I immediately knew the answer). The answer is simple, I consider my free time more valuable than anything else and one day while I was working on-site in Cleveland, OH I was telling someone about kiteboarding. Then it just struck me, why haven’t I taken lessons? It was something that I wanted to try the first time I saw it and I just let it fester in the back of my mind. So I did it, I made a call and scheduled a lesson, end of story.

So the long and short of it is that I just figured that at the end of my life I don’t want to say “I wish I had tried that” so I don’t, I just try it and man am I ever glad that I did. Well there is one drawback, kiteboarding is like crack apparently and I am hooked!

So here is an update from my kiteboarding experience: I can now jump, do back rolls (sort of an off axis flip/360) and this cool thing called a railey. Ok, now when I say jump I really mean fly, because a jump is probably a few feet off the water and with the kite attached to you on a windy day (like yesterday) my “jumps” were 20-30 feet off the water.

I’m also on twitter now so give me a follow if you want. Adios!

Now running on passenger (and ruby EE)

2009-02-06T20:22:00Z

If you use rails, or are considering it, it just got a whole lot easier to deploy using apache and mod_rails a.k.a Phusion Passenger

I think the best part of the experience was the installer, if you have ever compiled anything on linux you know it can be a pain when you don’t know if you have the required libraries. So the Phusion Passenger folks wrote their installer in ruby and it tells you what is missing (if there is anything). Sweet stuff.

Its much easier than running on mongrel, especially dealing with restarts and reboots (if you ever have to reboot, typically apache is already configured to start on startup). So how to restart an app? no shutdown you just navigate to the rails directory and issue a “touch tmp/restart.txt” next request the app will restart. Slick, wish everything was that simple.

So all in all (with out ruby enterprise) it took me 15 mins to install it and switch all my apps over from Mongrel to Phusion Passenger.

New site launched: http://zerium.com

2009-02-03T19:47:00Z

Just thought I’d put this out there that my company, zerium Interactive, has a new site: http://zerium.com and is geared toward helping small business eliminate some of their website issues. Each web package contains most of the tools you need to help your business succeed on the web.

We also do custom application design, development and hosting, so if you don’t just need the basics we can get it done.

Linux / OS X - shell scripting with sudo

2009-01-29T15:20:00Z

Ever need to create a shell script and use the sudo command? here is how you do it:

read $passwd

echo $passwd | sudo -S touch test

WDDXConfig - replaces SimpleConfig.cfc now with more simple!

2009-01-20T19:31:00Z

Well I know I just released the SimpleConfig.cfc but, I didn’t think it was as flexible as some might need. The new version uses one config file and can load any datatype that ColdFusion will understand in WDDX format, so if you already have configuration variables in your App Scope you can just write them out as a WDDX file and copy the values into “config.wddx” under the “Base” node and then just add your instances under the INSTANCES node in “config.wddx”

Click to download source files

I think this is pretty self explanatory when you take a peek at the code, I even simplified the required application.cfc code to be as easy as possible:

Application.cfc:


<cffunction name="OnApplicationStart">
        <cflock type="exclusive" scope="Application" timeout="60">
            <cfset WDDXConfig = createObject("component","org.ross.WDDXConfig") />
            <!--- optional parameters to WDDXConfig are as follows:
                 WDDXConfig.init("/config/", "config.wddx", "config")
                "/config/" is the mapping where your config files are
                "config.wddx" is your configuration file name
                "config" is the name of your Application variable where your configuration options are going
                 --->
            <cfset WDDXConfig.init() />

        </cflock>
    </cffunction>

config.wddx

This file is pretty simple, if you need some help understanding what it will translate to when you run the app, just run it and see the dump of “Application”


<?xml version="1.0"?>
<wddxPacket version="1.0">
    <header/>
    <data>
        <struct>
            <var name="INSTANCES">
                <!-- don't for get to iterate the length below as you add instances -->
                <array length="1">
                    <struct>
                        <var name="instanceType"><string>QA</string></var>
                        <!-- because you could be running multiple instances on the same machine
                         and thus the physical name could be the same (and we can't rely on CGI variables to be clean)
                         we use an identifying directory name here and the identifier will be placed
                         into the app scope as "ACTUALPATH" incase you are unsure of the physical path to your Application.cfc
                         C:\mysites\steve\appname\Application.cfc or mac: /mysites/steve/appname/Application.cfc -->
                        <var name="identifier"><string>/Volumes/Case Sensitive/www/simpleconfig/html/</string></var>
                        <var name="OVER_RIDE">
                            <struct>
                                <!-- override your base settings here -->
                                <var name="DSNs">
                                    <struct>
                                        <var name="marketing"><string>marketingDSNLocal</string></var>
                                    </struct>
                                </var>
                            </struct>
                        </var>
                    </struct>
                </array>
            </var>
            <!-- all of your base configuration data will go here -->
            <var name="BASE">
                <struct>
                    <var name="instanceType"><string>Development</string></var>
                    <var name="DSNs">
                        <struct>
                            <var name="marketing"><string>marketingPROD</string></var>
                        </struct>
                    </var>
                    <var name="additionalConfigFiles">
                        <array length="1">
                            <!-- as an example I'm just reloading the config.wddx file into another struct,
                             you can do as many additional wddx based config files as you want
                              (structName is the substruct name and fileName is the file name)-->
                            <struct>
                                <var name="structName"><string>sample</string></var>
                                <var name="fileName"><string>config.wddx</string></var>
                            </struct>
                        </array>
                    </var>
                </struct>
            </var>
        </struct>
    </data>
</wddxPacket>

SimpleConfig.cfc - Managing multiple instances of your application in Coldfusion

2009-01-16T22:08:00Z

When it comes time to deploy your application it is often difficult to manage application variables that change per instance. SimpleConfig aim’s to reduce the pain in running different environments. It does this by reading two xml files and taking those variables and placing them into your application scope for you to reference by using the prefix: Application.Configuration anywhere in your application.

source files: click to download

Setup and Configuration

Ok so lets dig into the file structure. When you download the zip file you will find a few directories listed as such:

/simpleconfig/config_files

 /simpleconfig/html

The “html” directory is where you want to point Apache/IIS as the web root of your application. All you need to do is add a mapping of “/config” and point it to wherever you want to put the “config_files” directory and you are ready to start tweaking your config files.

Ok so on to configuring this puppy. Point Apache/IIS to the extracted directory /simpleconfig/html as your web root (again you can skip that step for demonstration purposes and just drop the contents of the directory “simpleconfig” onto your existing web root).

Defining your environments

Now go take a look at the file: /simpleconfig/config_files/config-instances.xml:   


<configuration>
     <option> 
        <description>Local Instance</description>
         <name><![CDATA[simpleconfig.local]]></name>
         <value><![CDATA[config-dev.xml]]></value> 
    </option> 
    <option>
         <description>Configuration Files that pertain to all instances</description>
         <name><![CDATA[BASE_CONFIGURATION_FILES]]></name> 
        <value> 
            <key>
                 <name><![CDATA[Base Config File]]></name>
                 <value><![CDATA[config-base.xml]]></value> 
            </key> 
        </value>
     </option>
 </configuration>

As you can see there is just some xml in here that defines each environment that you will be running. When you want to add an environment you simply add another node with the appropriate details. So to edit this for your own local machine you probably want to change the node to the host name of your machine, because CGI can be mucked with too easily on the client side I opted to use the java way at getting the machine name, most likely you will just have to run the app and see the dump to get your machine’s name so you can put the appropriate name in the config-instances.xml file. Next after that is the config file that pertains to your local instance. In this case we have “config-dev.xml”.

The idea here is to use this configuration file to define any differences from your base-config for your application. So say you have an email address that you want to use in your application, you probably don’t want to use your production account info while you are developing locally. However you probably do want the same DSN name and that would be put in your config-base.xml and not overridden in your config-dev.xml.

You should also note that the first word in the description will be the name of that environment type. So if you put into the description “QA” the “CurrentMode” variable in the configuration would become “QA” instead of “Local”. This is useful when you have to add more environments that you didn’t account for initially.

Also, this gives you the ability to quickly switch and test configurations by simply changing the xml file that you are loading for that instance. So if you had a “config-qa.xml” you could easily test that configuration locally by specifying that xml file instead of the “config-dev.xml”

h2. Configuring each environment The config-base.xml file:


<configuration>
    <option>
        <description>Application Status Struct</description>
        <name><![CDATA[applicationStatus]]></name>
        <value>
            <key>
                <name><![CDATA[isAvailable]]></name>
                <value><![CDATA[true]]></value>
            </key>
            <key>
                <name><![CDATA[unAvailableMessageHeader]]></name>
                <value><![CDATA[Website Unavailable]]></value>
            </key>
            <key>
                <name><![CDATA[unAvailableMessageBody]]></name>

                <value><![CDATA[Application is temporarily unavailable.]]></value>
            </key>
        </value>
    </option>
    <option>
        <description>An application variable</description>
        <name><![CDATA[somevariable]]></name>
        <value><![CDATA[something you need in the app scope]]></value>
    </option>
    <option>
        <description>Some struct that you want in the app scope</description>
        <name><![CDATA[theStructName]]></name>
        <value>
            <key>
                <name><![CDATA[structkeyOne]]></name>
                <value><![CDATA[~now()]]></value>
            </key>
            <key>
                <name><![CDATA[structKeyTwo]]></name>
                <value><![CDATA[some string]]></value>
            </key>
            <key>
                <name><![CDATA[structKeyThree]]></name>
                <value><![CDATA[some other string]]></value>
            </key>
        </value>
    </option>
</configuration>

This file is a bit bigger and it contains some interesting stuff. Basically you can put pretty much anything into your application scope. You can even put calls to coldfusion functions in your xml that will be called when the xml is read and placed into your application scope. You can see a few lines up where there is the coldfusion function “now()” being called on “structKeyThree”. You tell SimpleConfig to evaluate the function by placing a tilde (~) before the function. I think the other huge advantage here is being able to define a struct that will go into your Application.Configuration object.

You can see a struct being defined in the applicationstatus node. You see that the first node contains a node then what will be created is a struct. The first node will be the struct name ie: Application.Configuration.MyStruct and then the additional nodes under the nodes will be the key in the struct and the will be the value for that key. This is much easier to see in action. So make a change to one of the struct keys or their values and you can see it work.

About the applicationstatus node

I built this node into the guts of the application. If for some reason you need to take your application offline (some back-end system is down perhaps). You can edit the applicationstatus node and set the isAvailable node to false and the Application.cfc will throw an exception (for every request) which you can then catch and display to your users on your own custom error page. Granted you don’t have to go this route but, it is there for you to use.

Reloading your configuration files

To reload the application you simply tack on a ?reload to the query string. Note that this only works when the application is not in production mode and also when trusted cache is off.

SimpleConfig.cfc

If you take a look at org/ross/SimpleConfig.cfc there are two places for you to add in your own methods that can be called when the application is starting up. The first place is the setup() method and the second is the start() method. The setup method is designed to be used when you need to call some other code before variables are placed into the Application scope and the start() method is designed to be used after those variables have been placed into the Application scope. Both of these methods are called each time the Application is reloaded (which you can see in the Application.cfc).

I hope this is enough to get you started. It really is a simple and elegant way to handle configuration variables. I will be adding additional text as I get time that will explain how you can add more configuration data into the application scope just by adding your own xml files and editing the config-instances.xml. Here is a hint: The SimpleConfig.cfc will keep loading any of the defined xml files in the BASE_CONFIGURATION_FILES node, just make sure you structure them exactly like the other config files.

This software is licensed under the CC-GNU GPL.

Available for Hire: Coldfusion, Ruby on Rails, AJAX / JS, CSS, Flex, even Design work.

2009-01-02T14:50:00Z

With things a bit slow right now I’m putting myself out there for any contract work. You can view my resume here:

Here are a few things that I can help you with: Debugging / Tuning, Architecture, Linux Administration, UI, layout, and site implementation (templatizing)

http://blog.stevensross.com/resume

If you just need a few hours help on a problem to a full blown site please don’t hesitate to contact me, my rates are reasonable.

Open an ssh Tunnel to Administer MySQL (or anything else)

2008-11-03T16:17:00Z

Pretty simple stuff, say you are using mysql administrator and you want to point it at a server that you have ssh access to but the firewall prevents you from making a direct connection over the internet (which it should).

Here is what you do (if mysql is running on its default port):


ssh -N -L3306:localhost:3306 username@server.com

You can also pass in a port variable if you are running ssh on a non-starndard port (which you should also do) all you do is add ” -p 1234” (1234 would be the port number where ssh is wanting you to connect)

Now just point mysql administrator at 127.0.0.1 and supply your credentials. Bang. You’re in.

Sync Google Contacts with your Mac/iPhone

2008-10-23T14:06:00Z

Just passing this along. Found this article at google and yay! Sync’ing is cool! read