Kopongo - Home

Accessing Trac with Ruby's HTTP Library

2008-08-08T09:06:00Z

I needed to access a Trac repository which was password-protected with basic authentication. Using ruby's standard standard http libraries, my first try did not work.

This is what does not work:

def trac_scraper_that_not_work
    http = Net::HTTP.new('some_trac_hoster.com', 443)
    http.use_ssl = true
    http.start do |http|

      login_location = '/trac/<something>/login'
      req = Net::HTTP::Get.new(login_location)
      req.basic_auth 'username', 'secret'
      response = http.request(req)    
      return nil if response.class == Net::HTTPUnauthorized

      actual_page_to_fetch_url = 'reports/or/whatever...'   
      req = Net::HTTP::Get.new(actual_page_to_fetch_url)
      response = http.request(req)
      #puts "response is #{response.body}"
      response.body
    end
end

Turns out that Trac uses cookies, so you need to provide the cookie you are given. First you to provide the authentication using the basic_auth method, then you copy the cookie you are given when provide it in the next request. Like this:

def trac_scraper_that_works
    http = Net::HTTP.new('some_trac_hoster.com', 443)
    http.use_ssl = true
    http.start do |http|

      login_location = '/trac/<something>/login'
      req = Net::HTTP::Get.new(login_location)
      req.basic_auth 'username', 'secret'
      response = http.request(req)

      return nil if response.class == Net::HTTPUnauthorized

      cookie = response.response['set-cookie']      
      headers = {
        'Cookie' => cookie
      }
      actual_page_to_fetch_url = 'reports/or/whatever...'   
      req = Net::HTTP::Get.new(actual_page_to_fetch_url, headers)
      response = http.request(req)
      #puts "response is #{response.body}"
      response.body
    end
end

postgres RI_ConstraintTrigger Error

2008-07-25T13:12:00Z

A problem that I recently started running into is a strange Postgres FK constraints error, which only shows up when running tests. The error looks something like this:

PGError: ERROR:  permission denied: "RI_ConstraintTrigger_XXXXX" is a system trigger

This is something that came in with FoxyFixtures, and there's a discussion about this problem and a patch in the rails trac. So what's this about?

When loading fixtures, rails tries to disable all foreign keys so that there are no problems inserting the data in whatever order. The calls work like this:

ALTER TABLE table_name DISABLE TRIGGER ALL
insert a bunch of fixtures 
ALTER TABLE table_name ENABLE TRIGGER ALL

However, postgres keeps some kind of metadata regarding the FKs to be enforced called RI_Constraint_something. Disabling the triggers on a table causes these to be deleted as well. The problems is that these other triggers belong to the superuser and not you. So, you got this permissions problem.

You may not have superuser permissions for your postgres instance, but it's probably best that the database you work with does not have superuser permissions, since that's the way it's going to be in production.

One workaround for to disable the foreign key checking in a different way. Instead of running the DISABLE TRIGGER, you can make the constraints DEFERRED postgres docs. This way, you run like this:

SET CONSTRAINTS ALL DEFERRED
insert a bunch of fixtures 
SET CONSTRAINTS ALL IMMEDIATE

This allows you to load your fixtures, but keep all contraint-checking.

The easiest way to patch this is to override the behavior of the disable_referential_integrity method. I created a new file, active_record_fk_hack.rb, stuck this in there:

module ActiveRecord
  module ConnectionAdapters
    class PostgreSQLAdapter < AbstractAdapter
      def disable_referential_integrity(&block)
         transaction {
           begin
             execute "SET CONSTRAINTS ALL DEFERRED"
             yield
           ensure
             execute "SET CONSTRAINTS ALL IMMEDIATE"
           end
         }
      end
    end
  end
end

then, in your environment.rb, add this at the end:

require 'active_record_fk_hack'

Query for Postgres database sizes

2008-04-16T11:11:00Z

If you need to figure out how big your postgres databases are, this query can come in pretty useful.

SELECT 
   pg_database.datname,
   pg_size_pretty(pg_database_size(pg_database.datname)) AS size
FROM 
  pg_database                                                                                                                                       
JOIN 
  pg_shadow ON pg_database.datdba = pg_shadow.usesysid                                                                                                     
ORDER BY 
  pg_database_size(pg_database.datname) desc;

You may need to log in as postgres to get the privilege to look at the pg_shadow tables.

Google Charts API rails plugin

2007-12-09T21:00:33Z

I bundled the code from the previous post into a plugin, called Chartr. This makes it easy to create graphs within rails.

Here's how to use it.

First, install the plugin. (It's probably a good idea to install it with '-x' since it's likely to be updated. Also, I should mention that this is my first plugin.)

    ruby script/plugin install -x svn://syvera.com/plugins/chartr

Right, so now we need to graph something. You can put a graph into any page, but let's create a page that definitely deserves a graph:

    ruby script/generate controller graphs index

This gives you an index.html.erb page where you can put your graph. In that page, we're going to use Chartr to give us a random graph. This is the code:

    <%= Chartr.make_simple_line_chart Array.new(5) {|i| rand(10)}, 
                                      ['one', 'two', 'three', 'four', 'five'],
                                      'stuff to graph' %>

So that's going to give you a graph like this:

Sure, that's cool. But what the heck were those arguments?

The first one is for your values:

    Array.new(5) {|i| rand(10)}  # returns an array of 5 random values between 0 and 9

The second, also an array, is for the legend. No explanation needed for that. The last is for the html 'alt' tag.

You could provide a fourth one for the size of the graph. If not, the default is 200x100.

Google Charts API

2007-12-07T22:48:00Z

I just noticed that google has published another cool API, this time for charts. It makes very fine looking charts, and has a simple API. Getting the values into the chart is very strange at first, but the constraints that you have on the values actually help in that your chart ends up looking better.

Here's how to use it. First we need some values to chart. A couple of weeks ago I wrote about how to extract some data out of subversion and put in into postgres. I still have those values around, so I for values I can use the number of commits per month in the rails project since it began. If you want to know more about that, read svn tricks. The strange part, as I mentioned before, is that you don't pass numbers to the chart. You pass some kind of encoding which has either 62, 1000, or 4096 values. For many uses, the first one is good enough.

Ok, so first we need to generate the encoding for the values. The Google maps API webpage shows a sample javascript function to generate that. I translated that to ruby. I think it's more or less correct, and goes like this:

    def simpleEncode(values, maxValue)
      simpleEncoding = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';

      chartData = ['s:'];
      values.each do |v|
        val = Float(v)
        if (!val.nan? && v >= 0)   
          chartData.push(simpleEncoding[( (simpleEncoding.length-1) * val / maxValue).round, 1]);
        else
          chartData.push('_');
        end
      end
      return chartData.join('');
    end

Right, so now how do we get our values to this function. As mentioned before, our values would come out of the database like this:

    % psql -d work_activity -t -c "select date_trunc('month', date), count(*) from svn_activity group by 1 order by 1;"
     2004-11-01 00:00:00 |    30
     2004-12-01 00:00:00 |   259
     2005-01-01 00:00:00 |   218
     etc
     ...

In order to generate the graph, we need to stuff those values into the funny URL google wants. So we hack a little chunk of code to parse the stuff.

    a = STDIN.read
    values = []
    dates = []
    a.each do |line|
      next if line.chomp.length < 1
      date, value = line.split('|')
      value.chomp!; date.chomp! #get rid of all the chars
      values << value.to_i 
      dates << date.split[0].split('-')[1] # just take month, legend too busy otherwise
    end

That gives us two arrays, one with the values, and one with the dates. Notice that I chopped off the year and the day off the dates. Before I did that, the labels on the X axis were so busy with text that you could not even make sense out of any of them.

Now that we have our values, we make the img tag with all the funny values in it. We also decide the size of the chart and the 'alt' value here. There are plenty of other things that you can modify, but this does most of what you want:

    image_str =<<-START
    <img src="http://chart.apis.google.com/chart?
    chs=500x225
    &amp;chd=#{simpleEncode(values, values.max)}
    &amp;cht=lc
    &amp;chxt=x,y
    &amp;chxl=0:|#{dates.join('|')}|1:||#{values.max}"
    alt="Rails Subversion Commits" />
    START
    puts image_str

Now we put all those bits of code above into a single file, and call it makegoogleurl.rb. We run the query as before, and pipe it to our nifty new program like this:

    psql -d work_activity -t -c "select date_trunc('month', date), count(*) from svn_activity group by 1 order by 1"| ruby make_google_url.rb

This is our result:

    <img src="http://chart.apis.google.com/chart?
    chs=500x225
    &amp;chd=s:FummojRe1Lu2vQOd9cLURcmXYRbQSLiUHIxpS
    &amp;cht=lc
    &amp;chxt=x,y
    &amp;chxl=0:|11|12|01|02|03|04|05|06|07|08|09|10|11|12|01|02|03|04|05|06|07|08|09|10|11|12|01|02|03|04|05|06|07|08|09|10|11|1:||347"
    alt="Rails Subversion Commits" />

And if you put that in your page, it looks like:

deploying with google maps API keys

2007-12-02T13:00:00Z

If you are using google maps in your apps, you get special key from them that works for the URL where you want to use the maps (ie www.yoursite.com). Since you probably test your code on localhost before you deploy it, you have a problem. The key they give you only works at that URL. What you need to do is get another key that works with your testing on your localhost, for example for localhost:3000.

What this means that when you deploy, you have to switch them. Of course, you do not want to do this by hand.

Here's a simple way to do this with capistrano:

    task :after_update_code, :roles => [:app, :db, :web] do

       layout_file = File.read('app/views/layouts/standard.rhtml')
       google_map_key = File.read("config/google_map_key.txt")
       layout_file.sub!(/file=api&amp;v=2&amp;key=(\w+)/, "file=api&amp;v=2&amp;key=#{google_map_key}")
       put(layout_file,
           "#{release_path}/app/views/layouts/standard.rhtml",
           :mode => 0444)
    end

So, what's going on?

First, save the production key on some file, like config/googlemapkey.txt. Don't think that is is like saving database passwords in SVN, since this key appears on all your webpages anyway.

I put my key in the site layout I use, standard.rhtml. By default, you want to keep development version of the key in there. When we deploy, we read the key from googlemapkey.txt, use a regexp to switch the development key with the production key. Then we send that file over with the put command.

You could be more clever by changing the file that's already on the server, instead of changing the local file and sending it over, but this works well enough.

svn tricks and rails on sundays

2007-11-25T16:00:27Z

I've got a few projects that I work on when I get the time. Since I usually work on all of them at the same time, it seems none of them moves forward very fast. I got curious to see how much work I am actually doing over time, and came up with a few little SVN hacks.

First, get the svn logs, pipe into a file:

% cd <head_of_the_svn_tree>
% svn log -q | egrep '^r' > activity.csv

Right, that gives us a file with all of the project checkins. The 'egrep' part strips out all of the annoying dashes that come with the svn log. The data looks like of like this:

r2 | danielw | 2006-12-20 00:38:13 +0200 (Wed, 20 Dec 2006)
r1 | danielw | 2006-12-20 00:33:41 +0200 (Wed, 20 Dec 2006)

Now, with some command-line tricks I can break down the activity a little more:

% svn log -q | egrep '^r' | cut -d '|' -f 2 | sort | uniq -c | sort -n

This breaks down the log and counts the number of checkins per person. You can point it to a URL as well. Results on one of my SVN trees gives something like this:

6  carl 
123  danielw

What I am really interested in is how this activity progresses over time. I don't know how to do this on the command line, but SQL could do this in no time. We need to create a database and a table to hold the data. In postgres, like this:

 % createdb work_activity
 % psql -d work_activity
 work_activity => create table svn_activity (revision varchar, who varchar, date timestamp);

Now we need to populate this with data. Since the end of that SVN line has got some funny timestamps, we'll get AWK to strip that out for us. Also, since the standard postgres column delimiter is the tab (\t), we'll delimit our records like that. Also, let's use the rails project to get more interesting stats.

% svn log -q http://svn.rubyonrails.org/rails/trunk > activity_rails.txt
% cat activity_rails.txt | egrep '^r' | awk '{print $1"\t"$3"\t"$5}' > activity_rails.data

This puts all of the data into a file, which we can now load into the DB in a single easy command:

% psql -d work_activity -c 'COPY svn_activity FROM STDIN' < activity_rails.data

Now it's all in the database, and we can do loads of fancy queries on it:

% psql -d work_activity -c "select date_trunc('month', date), count(*) from svn_activity group by 1 order by 1;"

     date_trunc      | count 
---------------------+-------
 2004-11-01 00:00:00 |    30
 2004-12-01 00:00:00 |   259
 2005-01-01 00:00:00 |   218
 2005-02-01 00:00:00 |   219
 2005-03-01 00:00:00 |   227
 2005-04-01 00:00:00 |   199
 2005-05-01 00:00:00 |    99
 2005-06-01 00:00:00 |   172
 2005-07-01 00:00:00 |   304
 2005-08-01 00:00:00 |    63
 2005-09-01 00:00:00 |   263
 2005-10-01 00:00:00 |   306
 2005-11-01 00:00:00 |   265
 2005-12-01 00:00:00 |    93
 2006-01-01 00:00:00 |    79
 2006-02-01 00:00:00 |   163
 2006-03-01 00:00:00 |   347
 2006-04-01 00:00:00 |   162
 2006-05-01 00:00:00 |    60
 2006-06-01 00:00:00 |   116
 2006-07-01 00:00:00 |    96
 2006-08-01 00:00:00 |   162
 2006-09-01 00:00:00 |   216
 2006-10-01 00:00:00 |   130
 2006-11-01 00:00:00 |   139
 2006-12-01 00:00:00 |    97
 2007-01-01 00:00:00 |   155
 2007-02-01 00:00:00 |    92
 2007-03-01 00:00:00 |   101
 2007-04-01 00:00:00 |    65
 2007-05-01 00:00:00 |   192
 2007-06-01 00:00:00 |   115
 2007-07-01 00:00:00 |    39
 2007-08-01 00:00:00 |    43
 2007-09-01 00:00:00 |   278
 2007-10-01 00:00:00 |   236
 2007-11-01 00:00:00 |   105

Looks like a very healthy project. Ok, let's find out on what day of the week rails developers have been most prolific:

psql -d work_activity -c "select extract(dow from date) as day, count(*) from svn_activity group by 1 order by 1;"  

 day | count 
-----+-------
   0 |  1040
   1 |   969
   2 |   874
   3 |   755
   4 |   790
   5 |   688
   6 |   789
(7 rows)

Day 0 is sunday! Thanks for the hard work, guys.

Useful Capistrano tricks

2007-11-13T13:23:00Z

I just ran into a neat little capistrano trick. I have been making some changes to an app, and knew it had been a while since I had made a release to one of the apps I am working on. But which version do I have installed?

I tried the cap diff_from_last_deploy, but it was a little too much information. I ran into capistrano's stream command, and that proved to be just right:

    desc "Find out svn version on server"
    task :what_version, :roles => [:app] do
        stream <<-CMD
            svn info #{current_path}/app
        CMD
    end

It's also useful for those one-off commands you might want to run on your server, like seeing how many users registered at your site:

    desc "Find out how many users are registered"
    task :how_many_users, :roles => [:app] do
        stream <<-CMD
            psql -U user -d yourapp_production -c 'select count(login) from users'
        CMD
    end

Rails Inflector problems

2007-11-04T16:23:21Z

I knew that the rails pluralization did some cool fancy stuff, but never really had to mess with it until today. I was trying to duplicate some bug I ran into, so I created a brand new rails app to recreate the bug in there. I had to create some model, so created "Dive", and did the whole scaffold thing that goes along with it.

I ran the tests, and they then started to complain about "Dife" not being found.

Curious, I went to the console:

    % "dives".singularize   # => dife

Huh? What's a 'dife'? Even dictionary.com does not even know what a 'dife' is! No matter, I knew that you could fix this up in the environment.rb file. I went in there and uncommented that default block in there, and added my little contribution to rails' knowledge of the english language.

     Inflector.inflections do |inflect|
        inflect.singular /dives/i, 'dive'
     end

I then ran the console again, just to make sure. First thing I see are all these errors on the command line:

    ./script/../config/../config/environment.rb:55:NameError: uninitialized constant Inflector
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.5/lib/action_controller/assertions/selector_assertions.rb:525:NoMethodError:      undefined method `camelize' for "top":String
    ..../app/controllers/application.rb:4:NameError: uninitialized constant ActionController::Base

On the prompt, I try the 'dives'.singularize again, but only get an error.

    >> "dives".singularize
    NoMethod Error: undefined method `singularize' for "dives":String
      from (irb):2

I look in the awdwr book and some other places, and find that what I added really does seem to be correct. A little googling on the error finally takes me to someone else that ran into the same problem.

It's really simple. The Inflector block needs to be outside the whole 'Rails::Initializer.run' block, not inside. Once you move it to the right place, you get what you expect:

    >> "dive".pluralize
    => "dives"
    >> "dives".singularize
    => "dive"

Rails 2.0.0_PR and Globalize breakage

2007-10-30T15:25:00Z

My functional tests broke when I went to Rails 2.0. It took a while to grok the stack trace:

ArgumentError: wrong number of arguments (2 for 1)
..../vendor/rails/activerecord/lib/active_record/base.rb:1965:in `attributes_with_quotes'
    ..../vendor/rails/activerecord/lib/active_record/base.rb:1965:in `update_without_lock'

Turns out the problem was that in Rails 2.0, ActiveRecord's 'attributeswithquotes' method now takes 2 arguments instead of a single one. The problem is that globalize overrides this method in the db_translate.rb, so things break.

It's easy to fix, though. Adding the extra argument to globalize's db_translate.rb seems to do fix what hurts.

basic plugins

2007-10-18T14:45:00Z

Some plugins are plain good. I tend to forget where they are, and then have to go around hunting for them again. Here's so that I don't have to hunt around for these next time.

    ruby script/plugin install exception_notification
    ruby script/plugin install tztime
    ruby script/plugin install tzinfo_timezone
    ruby script/plugin install http://svn.techno-weenie.net/projects/plugins/restful_authentication/
    ruby script/plugin install http://svn.pragprog.com/Public/plugins/annotate_models
    ruby script/plugin install svn://errtheblog.com/svn/plugins/will_paginate
    ruby script/plugin install svn://caboo.se/plugins/court3nay/spider_test
    ruby script/plugin install http://terralien.com/svn/projects/plugins/query_trace/

Also

    svn propset svn:ignore "*.log" log/
    svn propset svn:ignore "ruby_sess*" tmp/sessions/   
    svn propset svn:ignore "*" tmp/pids/
    svn propset svn:ignore "*" tmp/cache/
    svn propset svn:ignore "*" tmp/sockets/

Learning Javascript

2007-07-17T15:07:00Z

I have avoided javascript for a while, but finally decided to find out a little more about it. A few videos out there made the process painless.

First, Douglas Crockford's excellent presentations.

Basic Javascript Part (split into 4 parts)
Theory of the DOM (split into 3)
Advanced Javascripts (split into 3)

After this, I felt I had inkling of what the language was about. Still not sure how I would write a large chunk of code with it, but it's good to know what's going on. He's got some more videos which I will try to come back to, but at this point I want to move on to apply some of this stuff.

What I really wanted was to be able to use some of these new javascript libraries like Prototype, but I did not want to start blindly using then without knowing a little about what was going on in the background.

A good introduction to those libraries is Peepcode's Prototype video. It costs money, but it's certainly worth it.

Moving from rails 1.2.3 to rails edge

2007-07-11T16:22:00Z

Moving to edge broke a bunch of stuff, especially a bunch of that code that I had managed to completely forget about.

It's like this. You've been rolling along fine on 1.2.3, and, for some reason, you go to the edge release.

       % rake rails:freeze:edge

Some stuff will break. So, here's what's going out, and how to replace it:

Un-initialized active_resource

This one might happen early. The symptom:

% ruby script/server
   Exiting
   /usr/local/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/commands/servers/mongrel.rb:15: warning: already initialized constant OPTIONS
   /usr/local/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/commands/servers/mongrel.rb:18: undefined method `options' for []:Array (NoMethodError)

Weird. Something with mongrel? Let's try webrick:

% ruby script/server webrick
  => Booting WEBrick...
   /usr/local/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/commands/servers/webrick.rb:11: warning: already initialized constant OPTIONS
   /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require': no such file to load -- active_resource (MissingSourceFile)

This seems to be because the original rake rails:freeze:edge done forgot to bring the activeresource library into the mix. The fix is quick:

  % svn export http://dev.rubyonrails.org/svn/rails/trunk/activeresource vendor/rails/activeresource

Cookies needed some new password.

What is this? Turns out all you need to do for this is to add a little config item to the environment.rb Like this:
```
config.action_controller.session = {:session_key => "_myapp_session", :secret => 'this_can_be_long'}
```
pagination

Remember all that code the scaffolding used to spit out. Some of it, including the pagination, is still sitting around in a lot of places. It seems that pagination got a bad rap in the meantime, and it has now been taken out of edge. So, how do you replace them? The rails deprecations page is not exactly very helpful.

Well, quite a few choices are around. Quite a few people seem to have stepped in to provide this lost functionality, and it's a matter of choosing the plugin. One is [will_paginate][willpaginate], another is paginatingfind, and yet another is the paginator gem.

I did not have the time to test them all out (trying to move on, you know?), so I looked through a couple articles and discussions, and picked the will_paginate. Hardly any changes needed, and seems to work fine. Nice work.
Webservices

Yep, webservices are still around, but edge is not all that interested in loading them. The initialization failed. Looking in the code, the webservices were not even in the rake:freeze list .

Ok, so do it manually:
```
svn export http://dev.rubyonrails.org/svn/rails/trunk/actionwebservice/ vendor/rails/actionwebservice -r7144
```
Try again: nothing. Fine, google the thing. Looks like someone ran into this problem. He's got the fix all documented there.
push_with_attributes changes

The push_with_attributes call is out. If you have a join table with any extra non-joining columns in it (and therefore not a 'pure' join), you cannot use the push_with_attributes call to save those values anymore.

saving objects:

Something that used to work in 1.2.3 was this: let's say you have you have some golf website, where you are keeping track of how far a player hits a ball with a given club.

Your database would be setup something like this: table player (id, name), table clubs (id, name), and finally table player_club_performance (id, playerid, clubid, distance) to tie them together.
```
player = Player.new(:name => 'Tiger')
player.club_performance.create(:club_id => 7, :distance => 190)
```
This seems to save the record (the player_club_performance row) immediately. Since player has not yet been saved, it does not have an id, assigned to it. Therefore, if you have any db constraints on that table, something like club_id not null (you should, it's safer that way), this will crash.

If you use method build instead, it just creates the object. When you call p.save (the player), it will save the other associated objects as well.
```
player = Player.new(:name => 'Me')
player.club_performace.build(:club_id => 7, :distance => 300)
player.save # saves player and club_performance
```
Sessions

Don't know what happened to this one. I have always used the default file session, but this broke. First, I never noticed any files getting generated in the tmp/sessions directory. Second, on some actions where a bunch of stuff is saved in the session, I would get this:
```
[2007-07-06 16:50:13] FATAL  DISPATCHER FAILSAFE RESPONSE (has cgi) Fri Jul 06 16:50:13 +0000 2007
Status: 500 Internal Server Error
     CGI::Session::CookieStore::CookieOverflow             
       .../vendor/rails/actionpack/lib/action_controller/session/cookie_store.rb:91:in 'close'
```
This one might be a bug in the edge. I tried setting it to CGI::Session::PStore (as per AWDWR 2, pg 440), but it seemed not to know what that was. It seems defined in the vendor/rails/actionpack/lib/actioncontroller/cgiext/session.rb, but did not pick it up. I looked around at the loading/initialization of this for a bit, the just switched to the database session, which worked fine the first time.
Other random stuff
- assert_invalid_column_on_record Let's say that your Person model validates age, and says it's got to be a number or less than 1000 or something. It used to be okay to do something like this in a test
```
  post :create, :person => { :name => "elvis", age: => "10000"}
  assert_invalid_column_on_record "person", "age"
```
  That assert is out. The new way to do it is:
```
 post :create, :person => { :name => "elvis", age: => "asdfg"}
 assert(assigns(:person).errors.invalid?('age'))
```

Cleaning up postgres

2007-03-16T20:35:00Z

When you use postgres, every now and then, you need to clean out your database. Of course, as the project moves on, the tables that you need to clean change as well: some get added, some dropped. And then there's the indexes.

I usually do not get around to updating the cleaning scripts unless I notice something is wrong with the database, and then I have to deal with figuring out all the new tables and such.

This can be a lot simpler if you use the postgres metadata tables. The 'information_schema.tables' table has the list of all tables for a specific database. You can therefore get a list of tables with a simple:

    select
        table_name 
    from 
        INFORMATION_SCHEMA.TABLES 
    where 
         table_schema = 'public'

This gives you the list of tables, like this:

      table_name           
      -------------------------------
      first_table
      second_table 
      (2 rows)

So how do you get the VACUUM?

Change the select to this:

    select 
         'VACUUM ANALYZE VERBOSE ' || table_name || ';' 
    from 
         INFORMATION_SCHEMA.TABLES 
    where 
        table_schema = 'public'

The '||' does string concatenation in SQL. Not the quite the most obvious thing, but what are you gonna do?

So this that last command gives you this:

    ?column?                        
    -------------------------------------------------------
    VACUUM ANALYZE VERBOSE first_table;
    VACUUM ANALYZE VERBOSE second_table;

So now you can try this one by hand, or put it in a cron job somewhere...

    % psql -d mydb_production -f sql_script.sql -o vacuuum_tables.sql
    % psql -d mydb_production -f vacuum_tables.sql

I also like to recompute my indexes after I vacuum. I add this to the first cleaning script, a simple thing like so:

     select 
         'REINDEX INDEX ' || indexname || ';' 
     from 
         pg_indexes 
     where 
         schemaname ='public';

Now you get freshly computed indexes every time you run this. Nice!

ImageMagick RMagick Bus Error

2006-12-30T12:36:00Z

Installing software on the mac again. This one was not fun, but it did work out fine.

One of my projects uses the very cool RMagick gem, which requires ImageMagick or GraphMagick to be installed. This has always been a step I dread, ever since I unsuccessfully tried to install that whose thing on cygwin a year ago despite much effort. I have since installed all this on various linux boxes, and it’s usually quite simple.

Not on the Mac.

I first followed the very clear instructions I found at Hivelogic http://hivelogic.com/articles/2006/06/10/rmagick_os_x.

Unfortunately, that did not work. The RMagick gem did not install properly, and I got the now infamous bus error. Something about misc.rb, line 317.

I found some hints here http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/216875, and tried to install RMagick from source. No good. Same issue.

Then I tried looking around the web, and found many people asking the same questions. Going down further on the google list of issues for this, I started finding articles telling me to use GraphicsMagick. http://poocs.net/2005/7/6/making-rmagick-suck-less-with-tiger

Often the solution was something like ‘use graphmagick instead’. Fine, I am sure that would work, but my Linux production servers are setup with ImageMagick. Do I want to use different software in development than I use in production? No way. This has to compile.

I tried to figure out exactly what was going on in this line 317 which was such a problem. Who was not cooperating?

I wondered why hivelogic’s instruction did not quite match up with the compilation instructions at the RMagick site. Finally, I found this: http://blog.labratz.net/articles/2006/10/10/really-truly-getting-imagemagick-rmagick-working-on-osx-from-source-without-using-macports-darwinports-or-fink I t’s basically the hivelogic instruction plus the wmf and lcms libraries, and ImageMagick compiled with the instructions at the RMagick site. I added the ghostscript and the fonts as well.

And it works great!