..::Planet PostgreSQL::..

Explaining MVCC in Postgres: system defined columns

2010-09-01T15:00:57+00:00

I’m playing around with some diagrams for explaining MVCC that I’ll be posting here over the next few days. Not sure if I’ll end up giving up on slides and just use a whiteboard for the talk. I made an illustrated shared buffers deck to go along with Greg Smith’s excellent talk on shared buffers a while back. This is the beginning of a talk that I hope will emulate that.

Here are my first few slides, showing the system-defined columns. The next few slides will describe optimizations PostgreSQL has for managing the side effects of our pessimistic rollback strategy, and reducing IO during vacuuming and index updates.

Using logger with pg_standby

2010-08-31T23:49:14+00:00

Piping logs to syslog is pretty useful for automating log rotation and forwarding lots of different logs to a central log server.

To that end, the command-line utility ‘logger’ is nice for piping output from utilities like pg_standby without having to add syslogging code to the utility itself. Another thing is that logger comes by default with modern packages of syslog.

Here’s an easy way to implement this:

restore_command = 'pg_standby -d -s 2 -t /pgdata/trigger /shared/wal_archive/ %f %p %r 2>&1 | logger -p local3.info -t pgstandby'

PostgreSQL @ FrOSCamp 2010 in Zurich

2010-08-31T06:54:00+00:00

Andreas 'ads' Scherbaum

There's a new FOSS conference taking place at the ETH in Zurich, called FrOSCamp. It's a two day conference but unlike many others it's on friday and saturday, september 17th/18th. FUDCon, the Fedora conference, is running parallel to FrOSCamp. Same time, same location.

The PostgreSQL guys from Switzerland and myself submitted a project booth and several talks and workshops. Four talks and workshops as well as the booth got accepted. You can attend:

PostGIS/PostgreSQL for interactive maps (talk, speaker: Stefan Keller)
Replication with PostgreSQL 9.0 (workshop, speaker: Andreas Scherbaum)
PostgreSQL administration and optimization (talk, speaker: Stephan Wagner)
What's new in PostgreSQL 9.0? (talk, speaker: Andreas Scherbaum)

Here is the full schedule.

Come and visit us in Zurich!

Variable substitution with psql

2010-08-31T01:53:43+00:00

Updated: Thanks @johto for s/:bar/:foo/.

A coworker asked about variable substitution with psql using \set, and so I looked into it a bit further.

You definitely can do things like this:
16:55 sdeckelmann@[local]:5432|postgres=> \set test 'select * from :foo limit 10;' 16:56 sdeckelmann@[local]:5432|postgres=> \set foo 'test' 16:56 sdeckelmann@[local]:5432|postgres=> :test myint ------- 1 2 3 4 5 6 7 8 9 10 (10 rows)

But, what about something like this:

=> \set test 'select * from :var limit 10;' => :test mytable

Unfortunately, this isn’t supported.

The best you could do is something pathological like:

=> \set s 'select * from ' => \set pr ' limit 10;' => :s mytable :pr => :s test :pr myint ------- 1 2 3 4 5 6 7 8 9 10 (10 rows)

Happy Numbers

2010-08-30T08:00:00+00:00

After discovering the excellent Gwene service, which allows you to subscribe to newsgroups to read RSS content (blogs, planets, commits, etc), I came to read this nice article about Happy Numbers. That's a little problem that fits well an interview style question, so I first solved it yesterday evening in Emacs Lisp as that's the language I use the most those days.

A happy number is defined by the following process. Starting with any positive integer, replace the number by the sum of the squares of its digits, and repeat the process until the number equals 1 (where it will stay), or it loops endlessly in a cycle which does not include 1. Those numbers for which this process ends in 1 are happy numbers, while those that do not end in 1 are unhappy numbers (or sad numbers).

Now, what about implementing the same in pure SQL, for more fun? Now that's interesting! After all, we didn't get WITH RECURSIVE for tree traversal only, did we?

Unfortunately, we need a little helper function first, if only to ease the reading of the recursive query. I didn't try to inline it, but here it goes:

create or replace function digits(x bigint)
  returns setof int
  language sql
as $$
  select substring($1::text from i for 1)::int
    from generate_series(1, length($1::text)) as t(i)
$$;

That was easy: it will output one row per digit of the input number — and rather than resorting to powers of ten and divisions and remainders, we do use plain old text representation and substring. Now, to the real problem. If you're read what is an happy number and already did read the fine manual about Recursive Query Evaluation, it should be quite easy to read the following:

with recursive happy(n, seen) as (
    select 7::bigint, '{}'::bigint[]
  union all
    select sum(d*d), h.seen || sum(d*d)
      from (select n, digits(n) as d, seen
              from happy
           ) as h
  group by h.n, h.seen
    having not seen @> array[sum(d*d)]
)
  select * from happy;
  n  |       seen
-----+------------------
   7 | {}
  49 | {49}
  97 | {49,97}
 130 | {49,97,130}
  10 | {49,97,130,10}
   1 | {49,97,130,10,1}
(6 rows)

Time: 1.238 ms

That shows how it works for some happy number, and it's easy to test for a non-happy one, like for example 17. The query won't cycle thanks to the seen array and the having filter, so the only difference between an happy and a sad number will be that in the former case the last line output by the recursive query will have n = 1. Let's expand this knowledge into a proper function (because we want to be able to have the number we test for happiness as an argument):

create or replace function happy(x bigint)
  returns boolean
  language sql
as $$
with recursive happy(n, seen) as (
    select $1, '{}'::bigint[]
  union all
    select sum(d*d), h.seen || sum(d*d)
      from (select n, digits(n) as d, seen
              from happy
           ) as h
  group by h.n, h.seen
    having not seen @> array[sum(d*d)]
)
  select n = 1 as happy
    from happy
order by array_length(seen, 1) desc nulls last
   limit 1
$$;

We need the desc nulls last trick in the order by because the array_length() of any dimension of an empty array is NULL, and we certainly don't want to return all and any number as unhappy on the grounds that the query result contains a line input, {}. Let's now play the same tricks as in the puzzle article:

=# select array_agg(x) as happy from generate_series(1, 50) as t(x) where happy(x);
              happy
----------------------------------
 {1,7,10,13,19,23,28,31,32,44,49}
(1 row)

Time: 24.527 ms

=# explain analyze select x from generate_series(1, 10000) as t(x) where happy(x);
                      QUERY PLAN
----------------------------------------------------------------------------------------
 Function Scan on generate_series t  (cost=0.00..265.00 rows=333 width=4)
                          (actual time=2.938..3651.019 rows=1442 loops=1)
   Filter: happy((x)::bigint)
 Total runtime: 3651.534 ms
(3 rows)

Time: 3652.178 ms

(Yes, I tricked the EXPLAIN ANALYZE output so that it fits on the page width here). For what it's worth, finding the first 10000 happy numbers in Emacs Lisp on the same laptop takes 2830 ms, also running a recursive version of the code.

performance issue of SQL functions

2010-08-30T00:38:31+00:00

I though so sql functions is always faster than plpgsql function. But it's not true. Only when sql function is inlined, then function is faster, else plpgsql can be faster. Now I know, so sql functions where parameter is twice and more used are not inlined - because it means double parameter evaluation.you can see:


CREATE OR REPLACE FUNCTION is_empty(text) 
RETURNS bool AS $$ 
SELECT $1 IS NULL OR $1 = '' 
$$ LANGUAGE sql;

This function returns true when parameter is empty string or is null. The first parameter is used twice and then the body of this function isn't inlined.


postgres=# EXPLAIN VERBOSE SELECT count(is_empty(CASE WHEN random()  0.5 THEN NULL ELSE 'x' END)) FROM generate_series(1,100000);
                                                 QUERY PLAN                                                 
------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=12.50..12.77 rows=1 width=0)
   Output: count(is_empty(CASE WHEN (random()  0.5::double precision) THEN NULL::text ELSE 'x'::text END))
   ->  Function Scan on pg_catalog.generate_series  (cost=0.00..10.00 rows=1000 width=0)
         Output: generate_series
         Function Call: generate_series(1, 100000)
(5 rows)

The execution time is 458ms. When I modify the body for one time paremeter using:


CREATE OR REPLACE FUNCTION is_empty(text) 
RETURNS bool AS $$
SELECT COALESCE($1,'') = '' 
$$ LANGUAGE sql;

Then function is inlined:


postgres=# EXPLAIN VERBOSE SELECT count(is_empty(CASE WHEN random()  0.5 THEN NULL ELSE 'x' END)) FROM generate_series(1,100000);
                                                            QUERY PLAN                                                             
-----------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=12.50..12.52 rows=1 width=0)
   Output: count((COALESCE(CASE WHEN (random()  0.5::double precision) THEN NULL::text ELSE 'x'::text END, ''::text) = ''::text))
   ->  Function Scan on pg_catalog.generate_series  (cost=0.00..10.00 rows=1000 width=0)
         Output: generate_series
         Function Call: generate_series(1, 100000)
(5 rows)

and execution time is only 68ms (5 x faster than not inlined function).Attention: this note is valid for some more complex parameters - like volatile functions. The life isn't simple - for basic parameters (like columns or simple expression), the most simple SQL function (in my article function with OR operator and twice used parameter) can be better - wirking better with planner.

PostgreSQL Weekly News August 29th 2010

2010-08-29T08:00:00+00:00

PostgreSQL 9.0 RC1 out soon. The more you test, the sooner 9.0.0 comes out! http://www.postgresql.org/developer/beta/

Online aggregation paper from 1997 and PSU’s database reading group

2010-08-28T23:03:44+00:00

A couple weeks ago, Mark Wong and I took a field trip over to the Database Reading Group at Portland State University. It’s a group of students and professors that meet weekly throughout the school year to go over research papers. The papers are picked by the participants, and vary in topic from obscure to very practical.

This week’s paper reading was led by Professor Len Shapiro, and titled “Online Aggregation“. The paper is considered a foundational paper about SQL aggregates (like COUNT() or AVERAGE), and was published in 1997 by researchers from UC Berkeley and IBM. It’s also precursor to research into query parallelization and streaming databases. It was also awarded the SIGMOD “Test of Time” award in 2007, and is cited by over 170 other papers in the ACM archive.

The basic idea behind the paper centered around how to improve user experience in reporting results of aggregate queries – asking questions about how to solve three key problems when solving aggregates: blocking, fairness and control (from a user’s perspective). Roughly: Blocking is what happens when some part of the system waits and doesn’t return results to the user as a result of the waiting. Fairness concerns whether certain types of operations prevent certain groups of data from being processed (the example given had to do with GROUP BY and groups being processed one at a time). Control concerns whether or not a user can exert control over the speed of computation applied to a group (example given being a lever that “speeds up” processing of a set).

One insight from the paper is how online aggregates should be treated differently than traditional query processing – which might favor expensive plans involving sorts so that the output is ordered. When you’re dealing with online aggregates, you prefer unordered, or ideally random order, because your intermediate results will be more representative of the ultimate result. I guess that’s probably obvious once you think about it, but the paper provided some concrete examples.

Another interesting thought experiment involving the planner is how you pick plans that favor non-blocking, fairness and user control. Each of those properties is not narrowly defined, and changes based on individual user expectation. Professor Kristen Tufte mentioned that she’d be interested in how the ideas presented in this paper would be applied today, and Professor David Meier brought up that we might most be interested in applications involving managing Hadoop.

Prof Meier also brought up an interesting paper involving alternating nested loop joins during a discussion about optimizing JOIN algorithms for online aggregates. Another cool thing about the paper is that it involved modifications to Postgres! Granted, it was Postgres95, which doesn’t resemble the modern PostgreSQL 9.0 very much. But it was nice to revisit research that used Postgres that’s still relevant today.

Playing with bit strings

2010-08-26T14:45:00+00:00

The idea of the day ain't directly from me, I'm just helping with a very thin subpart of the problem. The problem, I can't say much about, let's just assume you want to reduce the storage of MD5 in your database, so you want to abuse bit strings. A solution to use them works fine, but the datatype is still missing some facilities, for example going from and to hexadecimal representation in text.

create or replace function hex_to_varbit(h text)
 returns varbit
 language sql
as $$
  select ('X' || $1)::varbit;
$$;

create or replace function varbit_to_hex(b varbit)
 returns text
 language sql
as $$
  select array_to_string(array_agg(to_hex((b << (32*o))::bit(32)::bigint)), '')
    from (select b, generate_series(0, n-1) as o
            from (select $1, octet_length($1)/4) as t(b, n)) as x
$$;

To understand the magic in the second function, let's walk through the tests one could do when wanting to grasp how things work in the bitstring world (using also some reading of the fine documentation, too).

=# select ('101011001011100110010110'::varbit << 0)::bit(8);
   bit
----------
 10101100
(1 row)

=# select ('101011001011100110010110'::varbit << 8)::bit(8);
   bit
----------
 10111001
(1 row)

=# select ('101011001011100110010110'::varbit << 16)::bit(8);
   bit
----------
 10010110
(1 row)

=# select * from *TEMP VERSION OF THE FUNCTION FOR TESTING*
 o |                b                 |    x
---+----------------------------------+----------
 0 | 10101100101111010001100011011011 | acbd18db
 1 | 01001100110000101111100001011100 | 4cc2f85c
 2 | 11101101111011110110010101001111 | edef654f
 3 | 11001100110001001010010011011000 | ccc4a4d8
(4 rows)

What do we get from that, will you ask? Let's see a little example:

=# select hex_to_varbit(md5('foo'));
                                                          hex_to_varbit
----------------------------------------------------------------------------------------------------------------------------------
 10101100101111010001100011011011010011001100001011111000010111001110110111101111011001010100111111001100110001001010010011011000
(1 row)

=# select md5('foo'), varbit_to_hex(hex_to_varbit(md5('foo')));
               md5                |          varbit_to_hex
----------------------------------+----------------------------------
 acbd18db4cc2f85cedef654fccc4a4d8 | acbd18db4cc2f85cedef654fccc4a4d8
(1 row)

Storing varbits rather than the text form of the MD5 allows us to go from 6510 MB down to 4976 MB on a sample table containing 100 millions rows. We're targeting more that that, so that's a great win down here!

In case you wonder, querying the main index on varbit rather than the one on text for a single result row, the cost of doing the conversion with varbit_to_hex seems to be around 28 µs. We can afford it.

Hope this helps!

Explain Plans PostgreSQL 9.0 - Part 2: JSON and JQuery Plan Viewer

2010-08-26T08:13:00+00:00

In part 1 of this series on PostgreSQL 9.0 planner outputs, we demonstrated how to render explain plans in YAML, JSON, and XML using the new explain features in PostgreSQL 9.0. In this second part, we'll demonstrate how to build a user interface that allows you input a JSON formatted explain plan and have it render into a printable, navigateable display using JQuery, javascript and a little bit of HTML coding. In part 3 we'll do something similar using XML and XSLT programming.

For those who aren't familiar with JQuery, it is an MIT licensed javascript library that is fairly light weight and allows you to inspect and change html elements with fairly intuitive syntax, has some nice ajax methods and tools for converting xml/json to native objects that can be manipulated. You can check it out at JQUERY.

We are not experts in JQuery, but from what we have used of it, we really like it and the fact the base package is MIT licensed, fairly light weight and lots of plugins available for it are real pluses.

The most difficult thing I think most people find about reading explain plans is that they are upside down; it starts with a conclusion and backtracks how to arrive at it. Humans by nature think about planning steps from start to finish. In order to make an explain plan understandable to mere mortals, we generally display them upside down or having the child-nodes shown left most. We shall follow that approach.

Continue reading "Explain Plans PostgreSQL 9.0 - Part 2: JSON and JQuery Plan Viewer"

PgWest: 2010 Call for Papers (2nd call)

2010-08-25T19:14:10+00:00

Yes, it is the second call. That means some of you haven't submitted after the first call. Of course, I haven't submitted mine either; so it is time for everyone to get on it. West is just around the corner and from all observations this West stands to be the largest PostgreSQL Conference, ever. (O.k. we might not over take Brazil).

Here is the announcement for everyone to review, enjoy and click on the CFP link:

Following on the smashing success of PostgreSQL Conference East, PostgreSQL Conference West, The PostgreSQL Conference for Decision Makers, End Users and Developers, is being held at the Sir Francis Drake Hotel in San Francisco from November 2nd through 4th 2010. Please join us in making this the largest PostgreSQL Conference to date!

Main conference site

Call for Papers

Thank you to our sponsors: Founding: Command Prompt Diamond: EnterpriseDB

Time line:

July 14th: Talk submission opens Sept 5th: Talk submission closes Sept 10th: Speaker notification

This year we will be continuing our trend of covering the entire PostgreSQL ecosystem. We would like to see talks and tutorials on the following topics:

      * General PostgreSQL: 
              * Administration 
              * Performance 
              * High Availability 
              * Migration 
              * GIS 
              * Integration 
              * Solutions and White Papers 
      * The Stack: 
              * Python/Django/Pylons/TurboGears/Custom 
              * Perl5/Catalyst/Bricolage 
              * Ruby/Rails 
              * Java (PLJava would be great)/Groovy/Grails 
              * Operating System optimization
                (Linux/FBSD/Solaris/Windows) 
              * Solutions and White Papers

Waiting for 9.1 – concat, concat_ws, right, left, reverse

2010-08-24T11:59:20+00:00

On 24th of August, Takahiro Itagaki committed patch: Log Message: ----------- Add string functions: concat(), concat_ws(), left(), right(), and reverse(). Pavel Stehule, reviewed by me. What are these functions? concat() is just functional counterpart to || operator, with the ability to handle many strings at the same time: $ SELECT concat( 'post', 'gres', 'ql' [...]

Adding enums

2010-08-23T21:06:42+00:00

While on a 15 hour plane trip from LA to Sydney the other day I broke the back of making enums extensible, thus proving that there is some good to be had from both long plane flights and tiny netbooks.

I posted a work in progress patch on the -hackers list yesterday. In it current form, it will allow you to do these things:

ALTER TYPE myenum ADD 'newlabel';

ALTER TYPE myenum ADD 'newlabel' BEFORE 'existinglabel';

ALTER TYPE myenum ADD 'newlabel' AFTER 'existinglabel';

The last one is not strictly necessary, and there is some opinion both for and against having it.

If labels are added at the end of the list (the first form does this) then, unless there is Oid wraparound, the enum Oids are in sorted order, which makes sorting and comparison operations very fast, as now. Labels added earlier in the list will sort and compare correctly, but not quite as fast.

Note that I have not provided for dropping existing labels, nor for changing the sort order of existing labels. The reason is that this would involve rewriting tables and indexes, where the current proposal will not require any such thing.

Npgsql receives donation of an MSDN subscription!!

2010-08-23T20:10:38+00:00

Hi all!

It all started when Josh Cooley told me about this post: http://devlicio.us/blogs/tuna_toksoz/archive/2010/07/27/codebetter-devlicio-us-msdn-ultimate-giveaways.aspx

I sent a mail talking about Npgsql and how a VS.Net would help us to add design time support and today I received a very nice mail saying that Npgsql was choosen to receive an MSDN subscription!

I'd like to thank the people who contributed to make this possible: Codebetter Crew: Ben Hall, Ward Bell, James Kovacs. Devlicious Crew: Hadi Hariri, Christopher Bennage, Tim Barcz, Rob Reynolds. Lostechies Crew: Eric Hexter, Jimmy Bogard, Keith Dahlby and Josh Cooley for heads up!

Thank you very much! Your support to OSS projects is awesome!

And stay tuned for better support of Npgsql inside VS.Net :)

How to manage changes to your database?

2010-08-22T14:39:49+00:00

Every now and then somebody asks how to make diff of database schemata. Usual background is like: we have production database, and development database, and we want to see what is different on development to be able to change production in the same way. Personally I think that such approach is inherently flawed. Why? First [...]

PostgreSQL Weekly News August 22nd 2010

2010-08-22T08:00:00+00:00

Call for Papers for PGDay.EU 2010 held on December 6-8 in Stuttgart, Germany is open. http://2010.pgday.eu/callforpapers

Using LTree to Represent and Query Hierarchy and Tree Structures

2010-08-22T05:15:00+00:00

PostgreSQL offers several options for displaying and querying tree like structures. In Using Recursive Common Table Expressions (CTE) to represent tree structures we demonstrated how to use common table expressions to display a tree like structure. Common Table Expressions required PostgreSQL 8.4 and above but was fairly ANSI standards compliant. In addition to that approach you have the option of using recursive functions. There is yet another common approach for this which is specific to PostgreSQL. This is using the ltree contrib datatype that has been supported for sometime in PostgreSQL. For one of our recent projects, we chose ltree over the other approaches because the performance is much better when you need to do ad-hoc queries over the tree since it can take advantage of btree and gist indexes and also has built-in tree query expressions that make ad-hoc queries simpler to do; similar in concept to the tsearch query syntax for querying text.

In this article we'll demonstrate how to use ltree and along the way also show the PostgreSQL 9.0 new features conditional triggers and ordered aggregates.

Continue reading "Using LTree to Represent and Query Hierarchy and Tree Structures"

OmniPITR – hot backup on slave

2010-08-18T12:36:52+00:00

Well, the biggest information is that hot-backups on slave work. And they work fine. Really fine. Some more information (with nice graph!): Background: hot backup is backup of database server, done with backing up data files, and not issuing pg_dump. There are certain benefits of doing it – for example the fact that if you’d [...]

New Android Blog

2010-08-17T20:40:56+00:00

Hi all!

I've been playing with Android for quite some time now and so I decided to create a blog where I'll talk about my experience with the platform and the projects I'm developing for it.

The blog page is: http://franciscodroid.blogspot.com

I invite you all who are interested in the platform to have a look at it.

This blog will continue to be about Npgsql, Mono and other relevant things. I decided to create another blog so I could concentrate posts about Android there.

Feedback, as always, is very welcome.

divide a table into a partitions

2010-08-17T05:59:37+00:00

HelloI had to verify a speed of two methods of dividing data to partion. First method is based on dynamic SQL, second is based on cursor scan and prepared statements. I was surprised - both variants has a same speed.


CREATE OR REPLACE FUNCTION public.foo()
 RETURNS void
 LANGUAGE plpgsql
AS $function$
BEGIN
  FOR i IN 0..9
  LOOP
    RAISE NOTICE '***** % *****', i;
    EXECUTE 'TRUNCATE TABLE ' || ('x' || i)::regclass;
    EXECUTE 'INSERT INTO ' || ('x' || i)::regclass ||
                ' SELECT * FROM x WHERE a >= $1 AND a  $1 + 100000'
       USING i * 100000;
  END LOOP;
  RETURN;
END;
$function$

This variant is shorter, but needs a active index on source table.


CREATE OR REPLACE FUNCTION public.foo1()
 RETURNS void
 LANGUAGE plpgsql
AS $function$
DECLARE
  c CURSOR FOR SELECT * FROM x;
  r RECORD;
BEGIN
  TRUNCATE TABLE x0;
  TRUNCATE TABLE x1;
  TRUNCATE TABLE x2;
  TRUNCATE TABLE x3;
  TRUNCATE TABLE x4;
  TRUNCATE TABLE x5;
  TRUNCATE TABLE x6;
  TRUNCATE TABLE x7;
  TRUNCATE TABLE x8;
  TRUNCATE TABLE x9;

  FOR r IN c
  LOOP
    CASE
        WHEN r.a BETWEEN 0 AND 99999 THEN
            INSERT INTO x0 VALUES(r.a);
        WHEN r.a BETWEEN 100000 AND 199999 THEN
            INSERT INTO x1 VALUES(r.a);
        WHEN r.a BETWEEN 200000 AND 299999 THEN
            INSERT INTO x2 VALUES(r.a);
        WHEN r.a BETWEEN 300000 AND 399999 THEN
            INSERT INTO x3 VALUES(r.a);
        WHEN r.a BETWEEN 400000 AND 499999 THEN
            INSERT INTO x4 VALUES(r.a);
        WHEN r.a BETWEEN 500000 AND 599999 THEN
            INSERT INTO x5 VALUES(r.a);
        WHEN r.a BETWEEN 600000 AND 699999 THEN
            INSERT INTO x6 VALUES(r.a);
        WHEN r.a BETWEEN 700000 AND 799999 THEN
            INSERT INTO x7 VALUES(r.a);
        WHEN r.a BETWEEN 800000 AND 899999 THEN
            INSERT INTO x8 VALUES(r.a);
        WHEN r.a BETWEEN 900000 AND 1000000 THEN
            INSERT INTO x9 VALUES(r.a);
    END CASE;
  END LOOP;
END;
$function$

This variant is longer, but doesn't need a active index on source table.

pgAdmin GSoC: Database Designer

2010-08-16T21:47:00+00:00

Luis Ochoa just finished his GSoC. I really think it is a great success. Wanna get a look at it? just watch this nice video.

Unfortunately, I can't commit it right away. There is still work to do. First, I want to read all the code. I know most of it, but I need to get a deeper understanding. Then, the UI needs a lot of work, as Thom Brown already said on the pgAdmin hackers list.

The first thing we need to decide is how to work together. I want Luis to continue his work if he wants too. As pgAdmin is now using Git as its main source code repository, we could probably create a gsoc_dd branch where I could put Luis's current work and where Luis and I can work together to enhance it. Then, when it's ready, we'll merge this branch with the master. Need to get Dave's opinion on this. And I'm still wondering what we should do with the branch once we merged it.

Anyway, even if there's still work to do, it's really exciting. It could be the major new feature for pgAdmin 1.14. And not the only one. There's also the other GSoC project for pgAdmin. Unfortunately I don't have a video link for this one, I didn't even see it live. Just hope for the best. There will be other major features, in the server status window and in the query tool (EXPLAIN XML, I need to find time to work on this one). Yeah, this is really exciting.

Sponsoring PGXN

2010-08-16T15:58:08+00:00

JD Wrote:

We are pleased to announce that the PgUS Board has decided to sponsor PGXN. For those who don't know what PGXN is, from the website:

PGXN, the PostgreSQL Extension network, is a central distribution system for open-source PostgreSQL extension libraries. In its first iteration it will consist of four basic parts:

An upload and distribution infrastructure for extension developers.
A centralized index and API of distribution metadata.
A website for searching extensions and perusing their documentation.

read more

Now What? (wrt OpenSolaris and your database)

2010-08-16T13:45:11+00:00

Last week's "announcement" of the death of OpenSolaris has steered a lot of questions my way about where people should go, and/or where OmniTI will go, now that OpenSolaris future looks non-existent. As one of the more open users of Solaris related technology, and running some beefy loads on top of it, it makes sense that people would be curious as to what we might be doing next. I would start with saying that as a company, we don't have an official policy on this yet, and probably won't. We evaluate each situation on a customer by customer basis, so what follows here is more my personal feelings on what people should do at this current point in time.

The one thing I have noticed from the people I have already spoken with is that there seem to be two major camps, an over simplification to be sure, but I break this down into the free software camp (those motivated by a desire to remain on open source, and/or support, free software as a primary driver of technology decisions), and those more interested in the technology than the ideals behind it. Depending on where you fall into that spectrum, you have different options available to you, and will likely reach very different conclusions.

Too Soon?

The first thing I have said to everyone is that it is honestly too soon to make any moves. Oracle is notorious for being poor communicators, and at this point I don't think we've seen enough official communication to really know what's going to happen. This doesn't mean you can't start planning though! We've been looking at some of the available options since before the Oracle/Sun merger was closed, so it doesn't hurt to start evaluating the options out there. However there's no need to rush in to things; it is possible that the announcement of OpenSolaris's death might be premature. I personally don't believe Solaris can't survive based on the model we've just seen laid out; there are too many people learning the gnu tool chain who won't be willing to invest big money into a tool that is hard for them to use. They need a low cost / free option for people to familiarize themselves on (and all the better if it installs gnu tools by default). There's an outside chance Oracle might come to this conclusion, which would give new life to OpenSolaris.

A more likely alternative to that theory is that some other group might pick up OpenSolaris maintenance and start pushing it forward. Certainly not an easy task, but there are already several different distribution of OpenSolaris available, so the userland level management has the resources, we mostly would need to figure out how to handle the more core technologies that have been maintained by Sun. I think this might also be possible, as there are numerous companies already heavily invested in OpenSolaris technology, and there are Solaris internals hackers looking to move out of Oracle, it's not an impossible leap to think we might see something worked out. And if Oracle continues to make technology available via the CDDL (which most of the current signs seem to indicate), this could work out. I would say that this might not resemble the OpenSolaris as it is now, but could definitely be an option for current users who'd like to remain on the OpenSolaris platform.

Other Options?

Of course, you might not want to put all your eggs in that basket. So what other options do we have? Well, that mostly depends on what you're getting out of OpenSolaris now, and what you want out of your OS going forward. For many people, I suspect that Solaris 11 Express might be a suitable replacement, especially for those running mixed OpenSolaris / Solaris environments. Migrating up to full Solaris 11 will also cover most of your technology needs, so depending on pricing I suspect people may find that a cheaper alternative to migrating to a new platform. Of course, if you want to stick with a free software solution, this won't really be an option.

FreeBSD seems to be the most obvious alternative platform. If you're currently taking advantage of dtrace, zfs, and zones, FreeBSD gives you options to cover all three. It won't be the same; the dtrace and zfs implementations are pretty close aiui, but for zones you'll probably have to use either Jails or OpenVS, neither of which am I a fan of. I think you'd also find a larger overlap in system utilities (tar, find, grep, etc..) between FreeBSD and Solaris, so for people (and scripts) making the transition, this might be an easier move. The big question here is probably hardware support; if you can't get FreeBSD running on your hardware, that's likely to be a show stopper, unless you can work out a new hardware purchase in the transition

So, if you don't want to go closed Solaris, and FreeBSD isn't an option, that probably leaves you on Linux. People sometimes think I don't like Linux; I'm actually very comfortable on it. My first "unix" was Linux, and we run some extremely demanding systems on Linux and it has performed well in those cases. However if you're trying to do deep introspection, systemtap is a poor man's dtrace. And if you are relying on zfs, you'll have a hard time finding a suitable replacement amongst the current Linux options. Personally I am most comfortable on ext3, but I tend to give up on file system snapshots, which is a painful submission if you have to make it. XFS is probably the next most common option, and generally I've no bones about using it if you want to avoid ext3. Of the three "advanced" replacements; ext4, btrfs, and zfs on linux; I think ext4 is probably your best bet, but only because zfs is too new for any serious database systems, and if you are moving off OpenSolaris to get away from Oracle, "butter" seems like an odd choice.

And so...

I think it's wise to keep things in perspective. There are some cases where you want to be a technology leader (we've been running Postgres 9 for months on some systems), but generally speaking when it comes to picking the operating system and filesystem for your database, it's best to tread lightly. Now is a fine time to start evaluating your options; at least figure out what features are critical to your enterprise that you'll need to replace (and don't just think about database, you might be relying on crossbow for something, or who knows what else). We'll certainly be watching the current options available, and I suspect diversifying a little, over the next 6 months, as we wait for the picture to clear up where we can. We're not in a hurry (after all, we do have the source code of what we're running now), and I don't see much reason for others to be either.

PostgreSQL Weekly News August 15th 2010

2010-08-15T08:00:00+00:00

The Call for Papers for West is open until September 5, 2010. Details at: http://www.postgresqlconference.org/

Security Meeting in Maryland

2010-08-15T03:45:01+00:00

I have been invited to attend a Postgres security meeting at OmniTI in Maryland on September 8. The meeting will include other local Postgres community members, government security experts, and contractors.

The goal of the meeting is for everyone to meet in person and to plan for necessary Postgres security additions and secure installation guidelines. While this will be US-government specific, I expect all work products to be publicly released and available to benefit the larger Postgres global community.

Npgsql 2.0.10 and NpgsqlParameter.Value

2010-08-15T01:13:59+00:00

Hi all!

As you may already know, Npgsql 2.0.10 is out. Besides the usual batch of bug fixes and enhancements, this release has a new feature which needs more attention as it may give you backward compatibility problems.

What happened?

With this release, we decided to implement a property in NpgsqlParameter which returns provider specific objects, just like SQLClient does.

NpgsqlParameter now features a property called NpgsqlValue. As you may guess, when you use it, you will receive a value whose type is Npgsql specific.

Why this change?

In the past, we had a lot of problems dealing how we would handle values which are specific to Npgsql, like NpgsqlTimestamp from others which are represented in CLR, like DateTime. This situation led to cast exceptions bugs. This was the main motivation for this.

Also, we think that by implementing this we would be giving our users a much consistent way of getting values in CLR type and in Npgsql-specific type. Another benefit is that now Npgsql can behave better inside DataSet and DataRow objects as they use Value property and expect CLR types being returned.

What is the problem and what can I do about it?

The problem lies in the fact that now, NpgsqlParameter.Value returns only CLR types. So, if your code was expecting to receive Npgsql specific types you may need to change your code to either use CLR type or change the call to NpgsqlValue property.

I think the first option is a good one as it will make your code more portable between other providers. If you are using only Npgsql, second option is a better choice as it will allow you to use directly Npgsql specific types.

We would like to hear from you how this modification is working for you.

Please, post your questions and suggestions in our user forums.

Thanks in advance.

Is Google a big enough Billy Goat Gruff?

2010-08-14T13:40:08+00:00

So, now it looks like Oracle's new business model for actually making money out of their investment in Sun is to become a patent troll. They seem to be going out of their way to make themselves pariahs in the Open Source world, as hated as SCO. Personally, I was always sceptical of their ability or intention to act well. They are simply not an organization that is geared to operating in this milieu. And there are plenty of people who should be at least a bit scared by this development, as Steven J Vaughan-Nichols points out.

I spent a year or two writing Java, and I get a bit impatient with people who like to rag on it. Their information is often out of date and ill-informed, in my experience. But anyone who now bases a major application, especially one which they sell, on Java would need to be very careful about how they proceed. And that involves a world of trouble and disputation. Surely it would be much simpler for any new project simply to say "OK, we'll use other technology, where there is less risk of running foul of patent lawsuits." That would be my reaction.

Incidentally, this is similar to the reasoning that led me to get involved with Postgres years ago. The company I was working for looked at shipping MySQL as a reference database with their product, and we were not interested in paying for a commercial license. Some people said we could, some said we couldn't. Our take was that we didn't want the hassle. Unless it was beyond dispute, we'd look elsewhere, and the next place we looked, naturally, was Postgres. By the time we decided against using Postgres, because there was then no Windows port, it was too late - I was hooked

Starting PostgreSQL in windows without install

2010-08-12T23:25:00+00:00

This is a question that comes up quite often by windows users, so thought we would share how we normally do it. The question is can you run a PostgreSQL server on your windows desktop/server box without having to install anything? The answer is yes and quite easily. Why would you need to do this. There are a couple of cases -- one you are developing a single user app that you want users to be able to run from anywhere without having to install it first. The other common reason is, you aren't allowed to install anything on a user's pc and you also want to package along a database you already have created.

For our purposes, many of our developers develop on portable WAMP like things, and for some of our applications, they need to work in both MySQL and PostgreSQL, so we need an easy way during development to swap one out for the other.

Continue reading "Starting PostgreSQL in windows without install"

Creativity with fuzzy string search

2010-08-12T14:36:04+00:00

PostgreSQL provides a useful set of contrib modules for "fuzzy" string searching; that is, searching for something that sounds like or looks like the original search key, but that might not exactly match. One place this type of searching shows up frequently is when looking for peoples' names. For instance, a receptionist at the dentist's office doesn't want to have to ask for the exact spelling of your name every time you call asking for an appointment, so the scheduling application allows "fuzzy" searches, and the receptionist doesn't have to get it exactly right to find out who you really are. The PostgreSQL documentation provides an excellent introduction to the topic in terms of the available modules; This blog post also demonstrates some of the things they can do.

The TriSano application was originally written to use soundex search alone to find patient names, but that proved insufficient, particularly because common-sounding last names with unusual spellings would be ranked very poorly in the search results. Our solution, which has worked quite well in practice, involved creative use of PostgreSQL's full-text search combined with the pg_trgm contrib module.

A trigram is a set of three characters. In the case of pg_trgm, it's three adjacent characters taken from a given input text. The pg_trgm module provides easy ways to extract all possible trigrams from an input, and compare them with similar sets taken from other inputs. Two strings that generate similar trigram lists are, in theory, similar strings. There's no particular reason you couldn't use two, four, or some other number of characters instead of trigrams, but you'd trade sensitivity and variability. And as the name implies, pg_trgm only supports trigrams.

Straight trigram search didn't buy us much on top of soundex, so we got a bit more creative. A trigram is just a set of three characters, which looks pretty much just like a word, so we thought we'd try using PostgreSQL's full text search on trigram data. Typically full text search has a list of "stop words": un-indexed words judged too common and too short to contribute meaningfully to an index. Our words would all be three characters long, so we had to create a new text search configuration using a dictionary with an empty stop word list. With that text search configuration, we could index trigrams effectively.

This search helped, but wasn't quite good enough. We finally borrowed a simplified version of a data mining technique called "boosting", which involves using multiple "weak" classifiers or searchers to create one relatively good result set. We combined straightforward trigram, soundex, and metaphone searches with a normal full text search of the unmodified name data and a full text search over the trigrams generated from the names. The data sizes in question aren't particularly large, so this amount of searching hasn't proven unsustainably taxing on processor power, and it provides excellent results. The code is on github; feel free to try it out.

Update: One of the comments suggested a demonstration of the results, which of course makes perfect sense. So I resurrected some of the scripts I used when developing the technique. In addition to the scripts used to install the fuzzystrmatch and pg_trgm modules and the name_search.sql script linked above, I had a script that populated the people table with a bunch of fake names. Then, it's easy to test the search mechanism like this:

select * from search_for_name('John Doe')
as a(id integer, last_name text, first_name text, sources text[], rank double precision);

 id  |  last_name  | first_name |                     sources                     |        rank        
-----+-------------+------------+-------------------------------------------------+--------------------
 167 | Krohn       | Javier     | {trigram_fts,name_trgm,trigram_fts,trigram_fts} |  0.281305521726608
 228 | Jordahl     | Javier     | {trigram_fts,name_trgm,trigram_fts}             |  0.237995445728302
  59 | Pesce       | Dona       | {trigram_fts}                                   |  0.174265757203102
 185 | Finchum     | Dona       | {trigram_fts}                                   |  0.174265757203102
 104 | Rumore      | Dona       | {trigram_fts}                                   |  0.174265757203102
 250 | Dumond      | Julio      | {name_trgm,trigram_fts,trigram_fts}             |   0.16849160194397
 200 | Dedmon      | Javier     | {name_trgm,trigram_fts,trigram_fts}             |  0.163729697465897
 230 | Dossey      | Malinda    | {name_trgm,trigram_fts}                         |  0.158055320382118
  50 | Dress       | Darren     | {name_trgm,trigram_fts}                         |  0.153293430805206
 136 | Doshier     | Neil       | {name_trgm,trigram_fts}                         |  0.148531511425972
 165 | Donatelli   | Lance      | {name_trgm,trigram_fts}                         |  0.132845237851143
 280 | Dollinger   | Clinton    | {name_trgm,trigram_fts}                         |  0.132845237851143
 273 | Dimeo       | Milagros   | {name_trgm,trigram_fts}                         | 0.0866267532110214
  49 | Dawdy       | Christian  | {name_trgm,trigram_fts}                         | 0.0866267532110214
 298 | Elswick     | Jami       | {trigram_fts}                                   | 0.0845221653580666

This isn't all the results it returned, but it gives an idea what the results look like. The rank value ranks results based on the rankings given by each of the underlying search methods, and the sources column shows which of the search methods found this particular entry. Some search methods may show up twice, because that search method found multiple matches between the input text and the result record. These results don't look particularly good, because there isn't really a good match for "John Doe" in the data set. But if I horribly misspell "Jamie Elswick", the search does a good job:

select * from search_for_name('Jomy Elswik') as a(id integer, last_name text,                                                 
first_name text, sources text[], rank double precision)

 id  |  last_name  | first_name |                     sources                     |        rank        
-----+-------------+------------+-------------------------------------------------+--------------------
 298 | Elswick     | Jami       | {trigram_fts,name_trgm,trigram_fts,trigram_fts} |  0.480943143367767
 312 | Elswick     | Kurt       | {name_trgm,trigram_fts}                         |  0.381967514753342
 228 | Jordahl     | Javier     | {trigram_fts,name_trgm,trigram_fts}             |  0.197063013911247
 403 | Walberg     | Erik       | {trigram_fts}                                   |  0.145491883158684
 309 | Hammaker    | Erik       | {trigram_fts}                                   |  0.145491883158684

PostgreSQL at FroSCon 2010

2010-08-10T16:02:20+00:00

Like last year, the German PostgreSQL User Group will have a Devroom at FroSCon (Sankt Augustin, Germany) this year. There will be talks around PostgreSQL, ASP.NET, the upcoming feature set in 9.0 and a workshop how to setup PostgreSQL 9.0 replication. The talks in detail can be found here (look for room C117).

For people interested in technical discussions or brainstorming, there's a booth at the conference, too, where people can meet PostgreSQL community members to discuss their questions. See you there!

BuildFarm Client release 4.1 Announcement

2010-08-09T22:39:57+00:00

There is a new release of the buildfarm client. There are no new features, but some cleanup and bug fixes.

The code can be downloaded from pgFoundry

Here are the release notes / change logs:

Use git clean to remove build products left by vpath builds, following a suggestion from Robert Haas.
provide for "default" entry in force_every setting
Accomodate oddities in at least some versions of File::Copy
If the web transaction fails, put back the old timestamps so the next run works on the same basis
provide enough log to avoid server failure if we get an SCM error (bug found following report from Greg Sabino Mullane)
work with older versions of git (bug found following report from Greg Sabino Mullane)

Financial Support for United States PostgreSQL User Groups

2010-08-09T18:08:30+00:00

JD Wrote:

As the United States PostgreSQL Association continues to grow we are extending the services we provide to our members and the PostgreSQL.Org community in the United States. I am pleased to announce that we have budgeted for 4000.00 USD to be provided to United States based PostgreSQL User Groups from September 2010 through December 2010.

Why Is standard_conforming_strings Enabled in Postgres 9.1?

2010-08-09T16:00:00+00:00

'depesz' already wrote a clear blog entry explaining the effect of enabling standard_conforming_strings in Postgres 9.1. My blog entry explains why it will be changed in Postgres 9.1.

To understand standard_conforming_strings, you have to know some Postgres history. Postgres is written in the C language, and some C syntax was added to Postgres for convenience. For example, Postgres supports the C syntax for not-equal, != , as well as the SQL-standard non-equal syntax, <>. This is a good addition of C syntax because it does not affect SQL-standard-compliant behavior.

Managing Key/Value Pairs in PostgreSQL

2010-08-09T13:00:00+00:00

Let's say that you've been following the latest research in key/value data storage and are interested in managing such data in a PostgreSQL database. You want to have functions to store and retrieve pairs, but there is no natural way to represent pairs in SQL. Many languages have hashes or or data dictionaries to fulfill this role, and you can pass them to functional interfaces. SQL's got nothin’. In PostgreSQL, have two options: use nested arrays (simple, fast) or use a custom composite data type (sugary, legible).

Editing constants in constraints

2010-08-09T11:45:00+00:00

We're using constants in some constraints here, for example in cases where several servers are replicating to the same federating one: each origin server has his own schema, and all is replicated nicely on the central host, thanks to Londiste, as you might have guessed already.

For bare-metal recovery scripts, I'm working on how to change those constants in the constraints, so that pg_dump -s plus some schema tweaking would kick-start a server. Here's a PLpgSQL snippet to do just that:

  FOR rec IN EXECUTE
$s$
SELECT schemaname, tablename, conname, attnames, def
  FROM (
   SELECT n.nspname, c.relname, r.conname,
          (select array_accum(attname)
             from pg_attribute
            where attrelid = c.oid and r.conkey @> array[attnum]) as attnames,
          pg_catalog.pg_get_constraintdef(r.oid, true)
   FROM pg_catalog.pg_constraint r
        JOIN pg_class c on c.oid = r.conrelid
        JOIN pg_namespace n ON n.oid = c.relnamespace
   WHERE r.contype = 'c'
ORDER BY 1, 2, 3
       ) as cons(schemaname, tablename, conname, attnames, def)
WHERE attnames @> array['server']::name[]
$s$
  LOOP
    rec.def := replace(rec.def, 'server = ' || old_id,
                                'server = ' || new_id);

    sql := 'ALTER TABLE ' || rec.schemaname || '.' || rec.tablename
        || ' DROP CONSTRAINT ' || rec.conname;
    RAISE NOTICE '%', sql;
    RETURN NEXT;
    EXECUTE sql;

    sql := 'ALTER TABLE ' || rec.schemaname || '.' || rec.tablename
        || ' ADD ' || rec.def;
    RAISE NOTICE '%', sql;
    RETURN NEXT;
    EXECUTE sql;

  END LOOP;

This relies on the fact that our constraints are on the column server. Why would this be any better than a sed one-liner, would you ask me? I'm fed up with having pseudo-parsing scripts and taking the risk that the simple command will change data I didn't want to edit. I want context aware tools, pretty please, to feel safe.

Otherwise I'd might have gone with pg_dump -s| sed -e 's:$server =$ 17:\1 18:' but this one-liner already contains too much useless magic for my taste (the space before 17 ain't in the group match to allow for having \1 18 in the right hand side. And this isn't yet parametrized, and there I'll need to talk to the database, as that's were I store the servers name and their id (a bigserial — yes, the constraints are all generated from scripts). I don't want to write an SQL parser and I don't want to play loose, so the PLpgSQL approach is what I'm thinking as the best tool here. Opinionated answers get to my mailbox!

Waiting for 9.1 – Recognize functional dependency on primary keys.

2010-08-08T20:43:33+00:00

Yesterday (August, 7th), Tom Lane committed: Log Message: ----------- Recognize functional dependency on primary keys. This allows a table's other columns to be referenced without listing them in GROUP BY, so long as the primary key column(s) are listed in GROUP BY. Eventually we should also allow functional dependency on a UNIQUE constraint when [...]

Planning for when the git repo goes live

2010-08-08T18:39:35+00:00

The cutover is due to happen soon,

Currently I have the following setup:

On Fedora 6: a CVS mirror and a buildfarm animal, dungbeetle that uses it. This machine will retire some time in the next few months, and will probably be replaced by a machine running CentOS 6 when it is released.

On Fedora 11, a mirror of my git repo that reflects the CVS state on github, and a buildfarm member quoll, that uses it. This is also where the github repo gets built.

On Windows NT, another mirror of my github repo, and three buildfarm members, dawn_bat, red_bat and brown_bat, that use it

When CVS goes quiet, I will need to do the following:

On all these machines, create mirrors of the new community repo, and cron jobs to keep them updated. The good news is that git is a whole lot more efficient about this than CVS is, so runs of "git fetch" won't place much stress on either then client or the server.

Blow away all the per-branch repositories for all the buildfarm members, and change the config settings for 'scm', 'scmrepo' and 'scm_url'. The repo will point to the machine-local mirror. On dungbeetle, that means also removing the legacy 'cvsmethod' and 'cvsrepo' config settings.

I'll probably be doing this while I'm a long way from home, but with luck all will run smoothly.

When everything is settled down I will turn off my CVS mirrors.

Speaking in New York City

2010-08-08T16:00:00+00:00

I am speaking at the New York PostgreSQL Meetup Group this Tuesday about the new features in Postgres 9.0. (It will be similar to a webcast I did a few months ago.)

I have a few interesting conferences coming up: SURGE (Baltimore), OpenSQL Camp (Massachusetts), PgWest 2010 (San Francisco), and perhaps PGDay Europe 2010 (Germany).

Waiting for 9.1 – Reduced lock levels for ALTER TABLE

2010-08-08T13:20:33+00:00

On 28th of July, Simon Riggs committed patch which: Log Message: ----------- Reduce lock levels of CREATE TRIGGER and some ALTER TABLE, CREATE RULE actions. Avoid hard-coding lockmode used for many altering DDL commands, allowing easier future changes of lock levels. Implementation of initial analysis on DDL sub-commands, so that many lock levels are now [...]

PostgreSQL Weekly News August 8th 2010

2010-08-08T08:00:00+00:00

Bruce Momjian's will walk through PostgreSQL 9.0 at the next NYC PostgreSQL Meetup, August 10th at 6:00 pm. RSVP below: http://www.meetup.com/postgresql-3/calendar/14310923/

Tail_n_mail and the log_line_prefix curse

2010-08-06T12:15:23+00:00

span{font-family:sans-serif;}span.c{color:red;}span.t{font-family:sans-serif;color:green;}span.o{font-family:sans-serif;color:blue;}

One of the problems I had when writing tail_n_mail (a program that parses log files and mails interesting lines to you) was getting the program to understand the format of the Postgres log files. There are quite a few options inside of postgresql.conf that control where the logging goes, and what it looks like. The basic three options are to send it to a rotating logfile with a custom prefix at the start of each line, to use syslog, or to write it in CSV format. I'll save a discussion of all the logging parameters for another time, but the important one for this story is log_line_prefix. This is what gets prepended to each log line when using 'stderr' mode (e.g. regular log files and not syslog or csvlog). By default, log_line_prefix is an empty string. This is a very useless default.

What you can put in the log_line_prefix parameter is a string of sprintf style escapes, which Postgres will expand for you as it writes the log. There are a large number of escapes, but only a few are commonly used or useful. Here's a log_line_prefix I commonly use:


log_line_prefix = '%t [%p] %u@%d '

This tells Postgres to print out the timestamp, the PID aka process id (inside of square brackets), the current username and database name, and finally a single space to help separate the prefix visually from the rest of the line. The above will generate lines that look like this:


2010-08-06 09:24:57.714 EDT [7229] joy@joymail LOG: execute dbdpg_p7228_5: SELECT count(id) FROM joymail WHERE folder = $1
2010-08-06 09:24:57.714 EDT [7229] joy@joymail DETAIL:  parameters: $1 = '4'

As you might imagine, the customizability of log_line_prefix makes parsing the log files all but impossible without some prior knowledge. I didn't want to go the pgfouine route and make people change their log_line_prefix to a specific setting. I think it's kind of rude to force your database to change its logging to accommodate your tools :). The original quick solution I came up with was to have a set of predefined regular expressions and the user would pick one that most closely matched their logs. For tail_n_mail to work properly, it needs to pick up at least the PID so it can tell when one statement ends a new one begins. For example, if you chose "regex #1", the log parsing regex would look like this:


(\d\d\d\d\-\d\d\-\d\d \d\d:\d\d:\d\d).+?(\d+)

This works fine on the example above, and gets us the timestamp and the PID from each line. The stock regexes worked for many different log_line_prefixes I came across that our clients were using, but I was never very happy with this solution. Not only was it susceptible to failing completely when a client was using a log_line_prefix not fitting into the current list of regexes, but there was no way to know exactly where the prefix ended and the statement began, which is important for the formatting of the output and the canonicaliztion of similar queries.

Enter the current solution: building a regex on the fly. Since we don't have a connection to the database at all, merely to the the log files, this requires that the user enter in their current log_line_prefix. This is a simple entry into the tailnmailrc file that looks just like the entry in postgresql.conf, e.g.:


log_line_prefix = '%t [%p] %u@%d '

The tail_n_mail script uses that variable to build a custom regex specifically tailored to that log_line_prefix and thus to the Postgres logs being used. Not only can we grab whatever bits we want (currently we only care about the timestamp (%t and %m) and the PID (%p)), but we can now cleanly break apart each line in the log into the prefix and the actual statement. This means the canonicalization/flattening of the queries is more effective, and allows us to only output the prefix information once. The output of tail_n_mail looks something like this:


Date: Fri Aug  6 11:01:03 2010 UTC                                                        
Host: whale.example.com
Unique items: 7
Total matches: 85
Matches from [A] /var/log/pg_log/postgresql-2010-08-05.log: 61
Matches from [B] /var/log/pg_log/postgresql-2010-08-06.log: 24

[1] From files A to B (between lines 14,205 of A and 527 of B, occurs 64 times)
First: [A] 2010-08-05 16:52:11 UTC [1602]  postgres@mydb
Last:  [B] 2010-08-06 01:18:14 UTC [20981] postgres@mydb
ERROR: syntax error at or near ")" 
STATEMENT: INSERT INTO mytable (id, foo, bar) VALUES (?,?,?))
-
ERROR: syntax error at or near ")"
STATEMENT: INSERT INTO mytable (id, foo, bar) VALUES (123,'chocolate','donut'));

[2] From file A (line 12,172)                                                                                                
2010-08-05 12:27:48 UTC [2906] bob@otherdb
ERROR: invalid input syntax for type date: "May" 
STATEMENT: UPDATE personnel SET birthdate='May' WHERE id = 1234;

(plus five other entries)

For the entry in the above example, we are able to show the complete prefix of the log lines where the error first occurred and where it most recently occurred. The next two lines show the "flattened" version of the query that tail_n_mail uses to group together similar errors. We then show a non-flattened example of an actual query from that group. In this case, someone added an extra closing paren in their application somewhere, which gives the same error each time, although the exact output changes depending on the values used. In the second example, because there is only one match, we don't bother to show the flattened version at all.

So in theory tail_n_mail should be now be able to handle any Postgres log you care to throw at it (yes, it can read syslog and csvlog format as well). As my coworker pointed out, parsing log files in this way is something that should probably be abstracted into a common module so other tools like pgsi can take advantage of it as well.

debian packaging PostgreSQL extensions

2010-08-06T10:00:00+00:00

In trying to help an extension debian packaging effort, I've once again proposed to handle it. That's because I now begin to know how to do it, as you can see in my package overview page at debian QA facility. There's a reason why I proposed myself here, it's that yet another tool of mine is now to be found in debian, and should greatly help extension packaging there. You can already check for the postgresql-server-dev-all package page if you're that impatient!

Back? Ok, so I used to have two main gripes against debian support for PostgreSQL. The first one, which is now feeling alone, is that both project release support policy aren't compatible enough for debian stable to include all currently supported stable PostgreSQL major version. That's very bad that debian stable will only propose one major version, knowing that the support for several of them is in there.

The problem is two fold: first, debian stable has to maintain any distributed package. There's no deprecation policy allowing for droping the ball. So the other side of this coin is that debian developers must take on themselves maintaining included software for as long as stable is not renamed oldstable. And it so happens that there's no debian developer that feels like maintaining end of lined PostgreSQL releases without help from PostgreSQL Core Team. Or, say, without official statement that they would help.

Now, why I don't like this situation is because I'm pretty sure there's very few software development group offering as long and reliable maintenance policy as PostgreSQL is doing, but debian will still happily distribute unknown-maintenance-policy pieces of code in its stable repositories. So the uncertainty excuse is rather poor. And highly frustrating.

Note: you have to admit that the debian stable management model copes very well with all the debian included software. You can't release stable with a new PostgreSQL major version unless each and every package depending on PostgreSQL will actually work with the newer version, and the debian scripts will care for upgrading the cluster. Where it's not working good is when you're using debian for a PostgreSQL server for a proprietary application, which happens quite frequently too.

The consequence of this fact leads to my second main gripe against debian support for PostgreSQL: the extensions. It so happens that the PostgreSQL extensions are developped for supporting several major versions from the same source code. So typically, all you need to do is recompile the extension against the new major version, and there you go.

Now, say debian new stable is coming with 8.4 rather than 8.3 as it used to. You should be able to just build the extensions (like prefix), without changing the source package, nor droping postgresql-8.3-prefix from the distribution on the grounds that 8.3 ain't in debian stable anymore.

I've been ranting a lot about this state of facts, and I finally provided a patch to the postgresql-common debian packaging, which made it into version 110: welcome pg_buildext. An exemple of how to use it can be found in the git branch for prefix, it shows up in debian/pgversions and debian/rules files.

As you can see, the pg_buildext tool allows you to list the PostgreSQL major versions the extension you're packaging supports, and only those that are both in your list and in the current debian supported major version list will get built. pg_buildext will do a VPATH build of your extension, so it's capable of building the same extension for multiple major versions of PostgreSQL. Here's how it looks:

        # build all supported version
        pg_buildext build $(SRCDIR) $(TARGET) "$(CFLAGS)"

        # then install each of them
        for v in `pg_buildext supported-versions $(SRCDIR)`; do \
                dh_install -ppostgresql-$$v-prefix ;\
        done

And the files are to be found in those places:

dim ~/dev/prefix cat debian/postgresql-8.3-prefix.install
debian/prefix-8.3/prefix.so usr/lib/postgresql/8.3/lib
debian/prefix-8.3/prefix.sql usr/share/postgresql/8.3/contrib

dim ~/dev/prefix cat debian/postgresql-8.4-prefix.install
debian/prefix-8.4/prefix.so usr/lib/postgresql/8.4/lib
debian/prefix-8.4/prefix.sql usr/share/postgresql/8.4/contrib

So you still need to maintain debian/pgversions and the postgresql-X.Y-extension.* files, but then a change in debian support for PostgreSQL major versions will be handled automatically (there's a facility to trigger automatic rebuild when necessary).

All this ranting to explain that pretty soon, the extenion's packages that I maintain will no longer have to be patched when dropping a previously supported major version of PostgreSQL. I'm breathing a little better, so thanks a lot Martin!

PDXPUG August Meeting in two weeks

2010-08-05T03:56:19+00:00

When: 7pm Aug 19th
Where: FreeGeek, 1731 SE 10th Avenue
What: PL/Parrot and PL/Perl6 - Parrots and Butterflies in your Database
Who: Jonathan "Duke" Leto

PL/Parrot is the Parrot Virtual Machine, embedded into the PostgreSQL relational database. This means that any Parrot language has the opportunity to become a PostgreSQL Procedural Language (PL). PL/Parrot supports writing stored procedures in the following languages:

* PL/PIR(U) - Parrot Intermediate Representation (PIR)
* PL/Perl6(U) - Rakudo Perl 6

This talk will give a quick introduction to Parrot VM and Rakudo Perl 6, and why you would want them in your database, including a status update of the current feature set of PL/Parrot and PL/Perl6. Example stored procedures in each language will be shown and, time permitting, the internals of how PL/Parrot works will be described. More information can be found at http://pl.parrot.org .

Jonathan is an open source hacker who currently focuses on the Parrot Virtual Machine and Rakudo Perl 6, as well as being the maintainer of many CPAN modules. He has been involved with Google Summer of Code since 2008 and is currently organization administrator for The Perl Foundation and Parrot Foundation, as well as a mentor for RTEMS on Parrot. Jonathan received a masters in mathematics from University of Central Florida and has published several papers in the field of differential equations. He enjoys discovering wheels within wheels.

PGXN Blog and Twitterstream

2010-08-04T16:51:00+00:00

I crated the PGXN Blog yesterday. Tune in there for news and announcements. I’ll also be posting status reports once development gets underway, so that all you fans out there can follow my progress. Once the site is done (or at 1.0 anyway), the blog will be used for announcements, discussion of support issues, etc. So tune in!

Finding Xmax

2010-08-04T16:00:00+00:00

Postgres implements Multiversion Concurrency Control (MVCC) using several normally-invisible fields, notably xmin and xmax. The xmin column records the transaction id that created the row, and xmax records the transaction id that expired the row, either through an UPDATE or DELETE.

I often demonstrate MVCC by showing the xmin and xmax columns:

        SELECT xmin, xmax FROM mytable;
         xmin | xmax
        ------+------
          664 |    0
        (1 row)

Reliability Humor

2010-08-04T16:00:00+00:00

As a followup to my previous blog post, this humorous Slashdot comment under the title "Reliability" warrants coverage:

Continue Reading »

Database Reliability

2010-08-04T16:00:00+00:00

The database is usually a critical piece of infrastructure in an organization; when the database is down, many things stop working, so database reliability is often a serious concern. While the reliability of database software is important, for Postgres it is often the infrastructure that Postgres depends on that causes outages, not Postgres itself. We see this regularly on the Postgres email lists.

To get started, a fundamental assumption has to be discarded — that computers are abstract machines and always do what they are told. While we often treat hardware as abstract devices, in reality they are physical, and are susceptible to failure just like any physical entity.

Customizing the RPMs from pgrpms.org

2010-08-04T00:33:57+00:00

To pick up where Devrim left off in customizing RPMs, here are some more tips for getting your very own RPMs built:

Create a VM with your favorite operating system (I’m using versions of CentOS). I need both 32-bit OS and 64-bit OS. This is much easier to manage with separate, local VMs.
Install spectool (available here), and SVN
The other dependancies were: gcc glibc-devel bison flex python-devel tcl-devel readline-devel zlib-devel openssl-devel krb5-devel e2fsprocs-devel libxml2-devel libxslt-devel pam-devel
Edit the postgresql-$VERSION.spec file to your liking: If you’re adding patches, you need to add them in TWO places – first in the Patch#: group, and then again below where the %patch# series starts. Finally, if you’re adding an entirely new package (say in 8.2, pg_standby in contrib), you’ll need to also add the binary (or library, or whatever) to the appropriate %files clause later in the spec file. It’s also a good idea to modify ‘Release’. Here’s a sample diff of my spec file:

--- postgresql-8.2.spec (revision 188) +++ postgresql-8.2.spec (working copy) @@ -74,7 +74,7 @@ Summary: PostgreSQL client programs and libraries Name: postgresql Version: 8.2.17 -Release: 1PGDG%{?dist} +Release: 1test%{?dist} License: BSD Group: Applications/Databases Url: http://www.postgresql.org/ @@ -95,7 +95,9 @@ Patch4: postgresql-test.patch Patch6: postgresql-perl-rpath.patch Patch8: postgresql-prefer-ncurses.patch +Patch7: postgresql-pgstat-dir.patch Patch9: postgresql-use-zoneinfo.patch +Patch10: pg_standby.patch

Buildrequires: perl glibc-devel bison flex
Requires: /sbin/ldconfig initscripts
@@ -282,7 +284,9 @@
%patch4 -p1
%patch6 -p1
%patch8 -p1
+%patch7 -p1
%patch9 -p1
+%patch10 -p1

pushd doc
tar -zcf postgres.tar.gz *.html stylesheet.css
@@ -604,6 +608,7 @@
%{_bindir}/pg_controldata
%{_bindir}/pg_ctl
%{_bindir}/pg_resetxlog
+%{_bindir}/pg_standby
%{_bindir}/postgres
%{_bindir}/postmaster
%{_mandir}/man1/initdb.*

How have you customized RPMs using this repo? Share your .spec files!

Development Process Article

2010-08-03T16:00:00+00:00

I wrote an article about the Postgres development process for the Japan PostgreSQL Users Group (JPUG) and they have now published it in English and Japanese. It highlights many of the processes that make Postgres open source development different, frustrating, and effective, with section titles like "No Single Authority", "No Bug Tracking System", and "No Development Roadmap".

Form Follows Failure

2010-08-03T16:00:00+00:00

While Steve Jobs tried to blame everyone but Apple for poor iPhone 4 reception, this article calls this long-term good news for the iPhone and smartphones in general because smartphone designers will increase their focus on call quality. This mirrors the competing interests that are often involved in technological progress:

"Almost everything about technology, to me, is a response to a negative. Something isn't working right. Something doesn't look right. Something doesn't perform right," Petroski told me. "And you try to remove that negative quality and that change is presumably an improvement. But then you release the new and improved product and people find faults with it. Because nothing is perfect. You always have competing constraints."

ORMs and Their Discontents

2010-08-03T16:00:00+00:00

Object-Relational Mappers are often a hot topic of discussion among database geeks. At OSCON's PGDay this year, Christophe Pettus presented the clearest explanation I have ever heard of the complexities of ORMs and the basically unattainable expectations users have for them. The basic problem is that it is easy to map simple ORM operations into SQL, but complex cases and mass changes require access to SQL for performance. Christophe explained why ORMs increase programmer productivity, but also why ORMs will never be able to do everything SQL can, so organizations should be prepared to add SQL to their ORM systems where appropriate.

A video recording was made of this talk and hopefully it will be posted soon.

View Comments

Not impressed

2010-08-02T23:20:37+00:00

Last Friday the CEO of Continuent sent out an apology for spamming the PostgreSQL community. Leaving aside the fact that the apology was issued on the postgresql-www mailing list, which is hardly the right forum, his apology left quite a number of questions unanswered. Many of those questions were subsequently asked explicitly by Robert Haas. So far there has been no response.

Also, it doesn't explain why I was spammed today, three days after the apology was issued.

Right now I'm thinking of putting Continuent on my automatic delete list.

I'm certainly not impressed by their technical expertise if they can't even get this right.

Cleaning up the Oracle mess

2010-07-30T18:11:55+00:00

I've been caught off-guard by the number of news stories that have been posted about Oracle turning off their Solaris boxes that participated in the PostgreSQL Buildfarm and the fact that I've been working to replace them. I think most of us in the community assumed it was going to happen sooner or later - Sun were contributors to PostgreSQL, but once they bought MySQL I think it was the beginning of the end of their contributions to the project. Being acquired by Oracle was the solid-gold nail in the coffin.

Anyway, just to clarify a few points in some of the articles I've seen:

My employer and corporate contributor to PostgreSQL, EnterpriseDB is providing the hardware for the new buildfarm members. We try to give back to the community wherever we can, and this is one thing that we can do quite easily.
So far I've enabled two Solaris SPARC buildfarm members, one running the Sun Studio compiler, and one using GCC. There are equivalent Solaris Intel members being setup at the moment, as well as a couple of Windows 7 ones using VC++ 2008 32 and 64 bit.
Contrary to reports and what might be inferred from the short bio on my blog, I actually work for EnterpriseDB on a full time basis. The "in my spare time I work for..." bit was my poor British attempt at humour. Sorry 'bout that - it won't happen again!

Well, that's about it I guess. We now return you to your scheduled programming.

The Pain of Software Development

2010-07-30T16:00:00+00:00

Adding features to Postgres is often a frustrating experience for all involved. Some features are clear-cut, but many are subject to interpretation and involve much email discussion. The email discussion is often draining, but this comment from a Slashdot thread about Linux desktop innovation is illuminating. The comment is about human interface engineereers asking for tiny application changes to improve usability, and while the application developer was originally "distracted trying to decide a fitting way to end the e-mail authors life", he ultimately agreed that the changes improved the user experience. I think Postgres developers often feel the same way.

View Comments

Explain Plans PostgreSQL 9.0 Text, JSON, XML, YAML - Part 1: You Choose

2010-07-29T21:58:03+00:00

One of the new features of PostgreSQL 9.0 is the ability to specify the format of an explain plan. In prior versions your only choice was text (and graphic explain with tools like PgAdmin III and other GUIS), but in 9.0 on, you have the additional options of Javascript Object Notation (JSON) which some people have a thing or two to say about them, YAML Ain't Markup Language (YAML) or eXtended Markup Language (XML). The new explain options are itemized in PostgreSQL 9.0 EXPLAIN.

The main benefit of the JSON, XML, YAML formats is that they are easier to machine parse than the default text version. This will allow for creative renderings of planner trees with minimal coding.

In Part 1 of this series, we'll demonstrate how to output the plans in these various formats and what they look like. In later parts of this series -- we'll demonstrate how to use Javascript, XSL and other scripting/markup languages to transform these into works of art you can hang on your wall.

-- START POSTGIS IN ACTION ASIDE --

We just submitted the third major revision of Chapter 3 Data Modeling of our upcoming PostGIS in Action book. The second major revision we never submitted and threw it out because it wasn't worldly enough and was too involved. We may use it later on for an example. Chapter 3 should be up on Manning Early Access Program (MEAP) soon. If you haven't bought the book yet Buy now. You don't want to miss out on a major masterpiece in the making. Okay we exaggerate a bit.

-- END POSTGIS IN ACTION ASIDE --
Continue reading "Explain Plans PostgreSQL 9.0 Text, JSON, XML, YAML - Part 1: You Choose"

We make the big time!

2010-07-29T17:16:02+00:00

Australian IT News has picked up on my recent post about Oracle shutting down their buildfarm servers. It's good publicity for PostgreSQL but I doubt it will shame them into continuing to support us.

Meanwhile, EnterpriseDB through Dave Page have added some more Solaris members. Thanks, guys.

Distributed Transactions and Two-Phase Commit

2010-07-29T17:46:15+00:00

The typical example of a transaction involves Alice and Bob, and their bank. Alice pays Bob $100, and the bank needs to debit Alice and credit Bob. Easy enough, provided the server doesn't crash. But what happens if the bank debits Alice, and then before crediting Bob, the server goes down? Or what if they credit Bob first, and then try to debit Alice only to find she doesn't have enough funds? A transaction allows the debit and credit operations to happen as a package ("atomically" is the word commonly used), so either both operations happen or neither happens, even if the server crashes halfway through the transaction. That way the bank never credits Bob without debiting Alice, or vice versa.

That's simple enough, but the situation can become more complex. What if, for instance, for buzzword-compliance purposes, the bank has "sharded" its accounts database by splitting it in pieces and putting each piece on a different server (whether this is would be smart or not is outside the scope of this post). The typical transaction handles statements issued only for one database, so we can't wrap the debit and credit operations within a single BEGIN/COMMIT if Alice's account information lives on one server and Bob's lives on another.

Enter "distributed transactions". A distributed transaction allows applications to group multiple transaction-aware systems into a single transaction. These systems might be different databases, or they might include other systems such as message queues, in which case the transaction concept means a message would get delivered if and only if the rest of the transaction completed. So with a distributed transaction, the bank could debit Alice's account in one database and credit Bob's in another, atomically.

All this comes at some cost. Distributed transactions require a "transaction manager", an application which handles the special semantics required to commit a distributed transaction. Second, the systems involved must support "two-phase commit" (which was added to PostgreSQL in version 8.1). Distributed transactions are committed using PREPARE TRANSACTION 'foo' (phase 1), and COMMIT PREPARED 'foo' or ROLLBACK PREPARED 'foo' (phase 2), rather than the usual COMMIT or ROLLBACK.

The beginning of a distributed transaction looks just like any other transaction: the application issues a BEGIN statement (optional in PostgreSQL), followed by normal SQL statements. When the transaction manager is instructed to commit, it runs the first commit phase by saying "PREPARE TRANSACTION 'foo'" (where "foo" is some arbitrary identifier for this transaction) on each system involved in the distributed transaction. Each system does whatever it needs to do to determine whether or not this transaction can be committed and to make sure it can be committed even if the server crashes, and reports success or failure. If all systems succeed, the transaction manager follows up with "COMMIT PREPARED 'foo'", and if a system reports failure, the transaction manager can roll back all the other systems using either ROLLBACK (for those transactions it hasn't yet prepared), or "ROLLBACK PREPARED 'foo'". Using two-phase commit is obviously slower than committing transactions on only one database, but sometimes the data integrity it provides justifies the extra cost.

In PostgreSQL, two-phase commit is supported provided max_prepared_transactions is nonzero. A PREPARE TRANSACTION statement persists the current transaction to disk, and dissociates it from the current session. That way it can survive even if the database goes down. The current session no longer has an active transaction. However, the prepared transaction acts like any other open transaction in that all locks held by the prepared transaction remain held, and VACUUM cannot reclaim storage from that transaction. So it's not a good idea to leave prepared transactions open for a long time.

Distributed transactions are most common, it seems, in Java applications. Full J2EE application servers typically come with a transaction manager component. For my examples I'll use an open source, standalone transaction manager, called Bitronix. I'm not particularly fond of using Java for simple scripts, though, so I've used JRuby for this demonstration code.

This script uses two databases, which I've called "athos" and "porthos". Each has same schema, which provides a simple framework for the sharded bank example described above. This schema provides a table for account names, another for ledger information, and a simple trigger to raise an exception when a transaction would bring a person's balance below $0. I'll first populate athos with Alice's account information. She gets $200 to start. Bob will go in the porthos database, with no initial balance.

5432 josh@athos# insert into accounts values ('Alice');
INSERT 0 1
5432 josh@athos*# insert into ledger values ('Alice', 200);
INSERT 0 1
5432 josh@athos*# commit;
COMMIT5432 josh@athos# \c porthos
You are now connected to database "porthos".
5432 josh@porthos# insert into accounts values ('Bob');
INSERT 0 1
5432 josh@porthos*# commit;
COMMIT

Use of Bitronix is pretty straightforward. After setting up a few constants for easier typing, I create a Bitronix data source for each PostgreSQL database. Here I have to use the PostgreSQL JDBC driver's org.postgresql.xa.PGXADataSource class; "XA" is Java's protocol for two-phase commit, and requires JDBC driver support. Here's the code for setting up one data source; the other is just the same.

ds1 = PDS.new
ds1.set_class_name 'org.postgresql.xa.PGXADataSource'
ds1.set_unique_name 'pgsql1'
ds1.set_max_pool_size 3
ds1.get_driver_properties.set_property 'databaseName', 'athos'
ds1.get_driver_properties.set_property 'user', 'josh'
ds1.init

Then I simply get a connection from each data source, instantiate a Bitronix TransactionManager object, and begin a transaction.

c1 = ds1.get_connection
c2 = ds2.get_connection
btm = TxnSvc.get_transaction_manager
btm.begin

Within my transaction, I just use normal JDBC commands to debit Alice and credit Bob, after which I commit the transaction through the TransactionManager object. If this transaction fails, it raises an exception, which I can capture using Ruby's begin/rescue exception handling, and roll back the transaction.

begin
  s2 = c2.prepare_statement "INSERT INTO ledger VALUES ('Bob', 100)"
  s2.execute_update
  s2.close

  s1 = c1.prepare_statement "INSERT INTO ledger VALUES ('Alice', -100)"
  s1.execute_update
  s1.close

  btm.commit
  puts "Successfully committed"
rescue
  puts "Something bad happened: " + $!
  btm.rollback
end

When I run this, Bitronix gives me a bunch of output, which I haven't bothered to suppress, but among it all is the "Successfully committed" string I told it to print on success. Since Alice is debited $100 each time we run this, and she started with $200, we can run it twice before hitting errors. On the third time, we get this:

Something bad happened: org.postgresql.util.PSQLException: ERROR: Rejecting operation; account owner Alice's balance would drop below 0

This is our trigger firing, to tell us that we can't debit Alice any more. If I look in the two databases, I can see that everything worked as planned:

5432 josh@athos*# select get_balance('Alice');
 get_balance 
-------------
           0
(1 row)

5432 josh@athos*# \c porthos 
You are now connected to database "porthos".
5432 josh@porthos# select get_balance('Bob');
 get_balance 
-------------
         200
(1 row)

Remember I've run my script three times, but Bob has only been credited $200, because that's all Alice had to start with.

PostgreSQL and Flymake

2010-07-28T23:49:13+00:00

Flymake is an on-the-fly syntax checker for Emacs. Here is how you can use it for editing the PostgreSQL source code. Add this to your src/Makefile.custom:

check-syntax:
        $(COMPILE.c) -fsyntax-only $(CHK_SOURCES)

Then you can activate the flymake minor mode either manually using M-x flymake-mode, or by adding it to your Emacs customization for PostgreSQL. I use:

(defun pgsql-c-mode ()
  "C mode adjusted for PostgreSQL project"
  (interactive)
  (c-mode)
  (flymake-mode)

  ; .. and the rest ...
)

Here is a screenshot:

Notice the marked line with the syntax error and the tooltip/popup with the error message, which appears when the mouse hovers over the marked line.

Note, however, that since activating flymake will essentially cause a compiler to be running continuously in the background, this is not what you want to use when hacking PostgreSQL on the road. ;-)

PostgreSQL: per-version .psqlrc

2010-07-28T21:31:29+00:00

File this under "you learn something new every day." I came across this little tidbit while browsing the source code for psql: you can have a per-version .psqlrc file which will be executed only by the psql associated with that major version. Just name the file .psqlrc-$version, substituting the major version for the $version token. So for PostgreSQL 8.4.4, it would look for a file named .psqlrc-8.4.4 in your $HOME directory.

It's worth noting that the version-specific .psqlrc file requires the full minor version, so you cannot currently define (say) an 8.4-only version which applies to all 8.4 psqls. I don't know if this feature gets enough mileage to make said modification worth it, but it would be easy enough to just use a symlink from the .psqlrc-$majorversion to the specific .psqlrc file with minor version.

This seems of most interest to developers, who may simultaneously run many versions of psql which may have incompatible settings, but also could come in handy to regular users as well.

FOSSExperts, day 2

2010-07-28T16:35:05+00:00

I expected feedback from the community on FOSSExperts. I did not expect feedback with such immediacy. All the feedback I have received so far is positive. Which is a great feeling. Here are the key points that are coming back.

How do deal with disagreement about the deliverable:

This is an interesting one. I wanted to keep FOSSExperts simple. That is why the deliverable on the ALTER TABLE project is simple, committed to PostgreSQL Core.

That may not work in all circumstances. So I am considering one of two options. The first option is courtesy of Josh Berkus. The idea would be to have a board of people that determine whether or not the deliverable has been met. This has merit because you have a panel of experts that make the determination. It can make the review process painless but is also takes the power out of both the funders and developers.

The second one is to take a vote. It would work something like this.

Developer states project is complete and demonstrates completeness based on the deliverable.
Every person who funded the project votes on whether or not the project is complete.
- If 66% vote the project complete, developer gets their money.
- If less than 66% vote project incomplete, developer doesn't get their money.
There would have to be some caveats. First the funders need to be able to communicate with the developer because they may not have understood part of the spec. Further as we are working with Open Source, the end deliverable may have been changed based on the will of the community versus the developer (see the Hot Standby work with PostgreSQL).
The voting would also need to be limited to a period of time. I was thinking 14 days. The 66% would be tallied against those that voted in that 14 days.
I like this idea because it removes the third party and it stops a single funder from calling foul as they are part of a collective vote. What do you think?

Other than that the feedback has been extremely positive. I even posted to the LedgerSMB list and multiple people are excited to see this opportunity.

If you have ideas, please share them. We have setup a flame page just for this.

Remember a lot of your questions can be answered in the FAQ as well.