..::Planet PostgreSQL::..

Peter Eisentraut: Where have all the translations gone?

peter_e@gmx.net (Peter Eisentraut) — Thu, 02 Jul 2009 20:00:06 +0000

If you have downloaded PostgreSQL 8.4.0 and are wondering where so many of the translations have gone: The translation team has decided not to ship translations anymore that are not translated at least about 80%. (See the commit message for the list of victims.) This is so that incidental users of stale translations are not presented with a confusing and distracting mix of translated and untranslated messages all the time. So right now we are only shipping a full or almost full set of translations into German, Spanish, French, Japanese, Portuguese, and Turkish.

To get the translations into other languages back into the release, go to http://babel.postgresql.org/ and start submitting updates. Updates may be included as early as release 8.4.1 in a few months.

I hope in particular that we might get the Chinese, Italian, and Russian translations back into shape.

By the way, if you want to start (or continue) translating, I suggest that you approximately follow this priority order: libpq, psql, pgscripts, pg_dump, initdb, postgres. This or a similar order will make the translations useful to the most users with the least amount of work.

Robert Gravsjö: Finally a real world webservice that enterprises can use to save money

Thu, 02 Jul 2009 11:17:42 +0000

Meet the Meeting Ticker.

Robert Gravsjö: "No to SQL"

Thu, 02 Jul 2009 08:54:26 +0000

Every time I see something or hear something like this I sigh a little bit. Not only when it's related to SQL but in the world of computer professionals in general. "The right tool for the job" seems to be a hard concept to understand sometimes. I wonder ...

David Wheeler: Committed: pgTAP Result Set Assertion Functions

Wed, 01 Jul 2009 21:32:00 +0000

Regular readers will know that I've been thinking a lot about testing SQL result sets and how to how to name result testing functions, and various implementation issues. I am very happy to say that I've now committed the first three such test functions to the Git repository. They've been tested on 8.4 and 8.3. Here's what I came up with.

David Fetter: WITH (so much drama in the CTE)

nospam@example.com (David Fetter) — Wed, 01 Jul 2009 17:18:57 +0000

By now, you've probably seen that PostgreSQL 8.4 can produce Mandelbrot sets
like the one below, but what are Common Table Expressions really about?

Continue reading "WITH (so much drama in the CTE)"

Andreas Scherbaum: PostgreSQL 8.4: Column Permissions

nospam@example.com (Andreas 'ads' Scherbaum) — Wed, 01 Jul 2009 17:00:00 +0000

Andreas 'ads' Scherbaum

Up to PostgreSQL 8.3 it was only possible to grant (and revoke) permissions on the entire table. If column level permissions were needed, a workaround like a view solved (more or less) the problem: create the view with the required (allowed) columns, revoke all permissions from the underlaying table, grant permissions to the view.

This - of course - is uneloquent, error prone and does not scale well. For different users requiring access to different columns, a big number of views is needed.

PostgreSQL 8.4 solves the problem with a shiny new feature: column level permissions.

Continue reading "PostgreSQL 8.4: Column Permissions"

US PostgreSQL Association: PostgreSQL 8.4 Released!

Wed, 01 Jul 2009 16:35:50 +0000

JD wrote:

For those sleeping in PostgreSQL.org just released PostgreSQL 8.4. This is an exciting release with many new features including:

Parallel Database Restore, speeding up recovery from backup up to 8 times
Per-Column Permissions, allowing more granular control of sensitive data
Per-database Collation Support, making PostgreSQL more useful in multi-lingual environments
In-place Upgrades through pg_migrator (beta), enabling upgrades from 8.3 to 8.4 without extensive downtime

read more

Robert Gravsjö: PostgreSQL 8.4 released

Wed, 01 Jul 2009 16:30:46 +0000

Spread the word, PostgreSQL 8.4 is out!

Josh Berkus: pg_stat_statements in 8.4

Wed, 01 Jul 2009 12:52:42 +0000

Now that PostgreSQL 8.4 is out, I thought I'd write a little about my favorite 8.4 feature. As Mister Performance Whack-a-Mole, what makes me happy about 8.4 is the ability to whack moles faster ... which is why I'm very fond of pg_stat_statements.

Robert Lor: How to add DTrace probes to your application

Tue, 30 Jun 2009 15:22:16 +0000

MC Brown and I co-presented a session at CommunityOne West on how to add probes to applications, using MySQL and PostgreSQL as case study. In the presentation, I used a very simple example to demonstrate how easy it is to add probes. If you want to try out yourself, here is the code. Extract the files, run gmake (or gnumake on OS X) to build, and run the executable in one terminal and the DTrace script in another terminal to see the output from the probes. To see more complicated examples, checkout the MySQL or PostgreSQL source code.

Leo Hsu and Regina Obe: Managing disk space using table spaces

nospam@example.com (Leo Hsu and Regina Obe) — Tue, 30 Jun 2009 04:00:00 +0000

We have covered this briefly before, but its an important enough concept to cover again in more detail.

Problem: You are running out of disk space on the drive you keep PostgreSQL data on

Solution:

Create a new tablespace on a separate drive and move existing tables to it, or create a new tablespace and use for future tables.

What is a tablespace and how to create a tablespace

A tablespace in PostgreSQL is similar to a tablespace in Oracle and a filegroup in SQL Server. It segments a piece of physical disk space for use by the PostgreSQL process for holding data. Below are steps to creating a new tablespace. Tablespaces have existed since PostgreSQL 8.0.

More about tablespaces in PostgreSQL is outlined in the manual PostgreSQL 8.3 tablespaces

While it is possible to create a table index on a different tablespace from the table, we won't be covering that.

Continue reading "Managing disk space using table spaces"

Robert Lor: More on the demos from PgCon 2009

Mon, 29 Jun 2009 18:17:44 +0000

At PgCon 2009 in Ottawa, I did a lightning talk on DTrace probes in PostreSQL 8.4. I wanted to show several demos but ran out of time. If you want to try them out, use the scripts below.

Here is the script, query_time.d, used in slide 14. This script is used to identify slow queries by printing out the query execution time.


#!/usr/sbin/dtrace -s
#pragma D option quiet

dtrace:::BEGIN
{
  printf("Tracing... Hit Ctrl-C to end.\n");
}

postgresql*:::query-start
{
    self->query = copyinstr(arg0);
    self->ts = timestamp;
}

postgresql*:::query-done
/self->ts/
{
    @query_time[self->query] = avg(timestamp - self->ts);
    self->query = 0;
    self->ts = 0;
}

dtrace:::END
{
     printf("%10s %s\n", "TIME (ns)", "QUERY");
     printf("==============================================================\n");
     printa("%@10d %s\n", @query_time);
}

Here is the second script, sort.d, used in slide 16. This script tells the type of sort, whether the sort was done in memory or on disk, and the time to perform the sort.


#!/usr/sbin/dtrace -qs

dtrace:::BEGIN
{
        sorttype[0] = "TUPLE";
        sorttype[1] = "INDEX";
        sorttype[2] = "DATUM";

        sortmethod[0] = "INTERNAL";
        sortmethod[1] = "EXTERNAL";
}

postgresql*:::sort-start
{
        self->ts = timestamp;
        printf("\nBegin %s sort, workmem = %d KB \n", sorttype[arg0], arg3);
}

postgresql*:::sort-done
/self->ts && arg0 == 0/
{
        /* Internal Sort */
        printf("%s sort ended, space used = %d KB \n", sortmethod[arg0], arg1);
        printf("Sort time = %d ms\n\n", (timestamp - self->ts) / 1000000);
}

postgresql*:::sort-done
/self->ts && arg0 == 1 /
{        /* External Sort */
        printf("%s sort ended, space used = %d disk blocks\n", sortmethod[arg0], 
arg1);
        printf("Sort time = %d ms\n\n", (timestamp - self->ts) / 1000000);
}

Below is the last script, query_trace.d, used in slide 23. This script provides useful data that will allow you to dig down deeper. In this example, the buffer reads to table 16397 (this is the OID) is huge. This signals a red flag that an index may be needed for this table. To find out the table name from OID, run "SELECT relname FROM pg_class WHERE relfilenode=16397" in psql.


#!/usr/sbin/dtrace -qs

postgresql*:::query-start
{
        self->ts = timestamp;
        self->pid = pid;
}

postgresql*:::buffer-read-start
/self->pid/
{
        self->readts = timestamp;
}

postgresql*:::buffer-read-done
/self->pid && arg7/
{
        /* Buffer cache hit */
        @read_count[arg2, arg3, arg4] = count();
        @read_hit_total["Total buffer cache hits      : "] = count();
        @read_hit_time["Average read time from cache : "] = avg (timestamp - self->readts);
        self->readts = 0;
}

postgresql*:::buffer-read-done
/self->pid && !arg7/
{
        /* Buffer cache miss */
        @read_count[arg2, arg3, arg4] = count();
        @read_miss_total["Total buffer cache misses    : "] = count();
        @read_miss_time["Average read time from disk  : "] = avg (timestamp - self->readts);
        self->readts = 0;
}

postgresql*:::buffer-flush-start
/self->pid/
{
        self->writets = timestamp;
}

postgresql*:::buffer-flush-done
/self->pid/
{
        @write_count[arg2, arg3, arg4] = count();
        @write_time["Average write time to disk   : "] = avg (timestamp - self->writets);
        self->writets = 0;
}

postgresql*:::query-done
/self->ts && self->pid == pid/
{
        printf("\n============ Buffer Read Counts ============\n");
        printf("%10s %10s %10s %10s\n","Tablespace", "Database", "Table", "Count");
        printa("%10d %10d %10d %@10d\n",@read_count);

        printf("\n======= Buffer Write Request Counts ========\n");
        printf("%10s %10s %10s %10s\n","Tablespace", "Database", "Table", "Count");
        printa("%10d %10d %10d %@10d\n",@write_count);

        printf("\n========== Additional Statistics ===========\n");

        printf ("Backend PID    : %d\n", pid);
        printf ("SQL Statement  : %s\n", copyinstr(arg0));
        printf ("Execution time : %d.%03d sec (%d ns)\n", (timestamp - self->ts) / 1000000000, ((timestamp - self->ts) / 1000000) % 1000, timestamp - self->ts);
        printa("\n%19s %@8d\n",@read_hit_total);
        printa("%19s %@8d\n",@read_miss_total);
        printa("%19s %@8d (ns)\n",@read_hit_time);
        printa("%19s %@8d (ns)\n",@read_miss_time);
        printa("%19s %@8d (ns)\n",@write_time);
        printf("\n\n");

        trunc(@read_count);
        trunc(@write_count);
        trunc(@read_hit_total);
        trunc(@read_miss_total);
        trunc(@read_hit_time);
        trunc(@read_miss_time);
        trunc(@write_time);

        self->ts = 0;
        self->pid = 0;
}

To see more sample scripts as well as a GUI tool, check out the PostgreSQL DTrace Toolkit.

Robert Lor: PostgreSQL DTrace Toolkit

Mon, 29 Jun 2009 04:19:03 +0000

As many of you know, PostgreSQL 8.4 has quite a few more DTrace probes. See my previous blog post for more details . To use the probes, you need to write DTrace scripts, which is quite easy to do, but to make it easier to use the probes (especially for those who are new to DTrace), I have written some scripts that you can just run from the command line. In addition, I've integrated some of those script with Chime to make it even easier to visualize the data. Check out the toolkit on PgFoundry.

Andrew Dunstan: Parallel pg_restore for PostgreSQL 8.4

nospam@example.com (Andrew Dunstan) — Sun, 28 Jun 2009 12:24:52 +0000

I try to complete at least one significant feature item per PostgreSQL release. This time the feature is making pg_restore run in parallel. This is quite important for many users, particularly some large enterprise users.

It's important that people understand what this will do and what it won't do. pg_restores runs a number of steps. In conventional mode it simply runs them all in a single connection to the database, one after the other. In parallel mode it first runs all the quick and easy steps, essentially those that don't involve any data access, such as table and function creation, in a single connection, just like conventional mode. Then it runs the remaining steps each in its own connection. The steps are the same, and there is no parallelism within a given step. For example, a single COPY to a table is not parallelised. Rather, we run it in parallel with other data intensive steps.

The maximum amount of parellelism is controlled by the user. This will involve some experimentation to get to the sweet spot for your setup. A good place to start is the number of physical processors you have available. The idea here is to improve the situation where the CPU is the limiting factor, and allow you to drive the restoration rate up to where IO is in fact the limiting factor. With very high end hardware we believe that you can drive the parallelism quite high.

Like many performance features, this one might well require several releases to tweak it for optimal performance gain. The program works by keeping a pool of slots to be used for the steps that are run in parallel. One possible area for improvement is in the algorithm that selects the item to be used for a slot as it it becomes available. Currently we keep a queue of items that have no remaining unrestored dependencies. An item gets put on the queue as soon as all the items it depends on have been restored. This is likely to be a fairly good approximation of an optimal algorithm, but there might well be a way of tweaking it. Another possible area of optimsation would be to take some notice of the tablespace that each item affects, and try to balance these, so we use as many IO channels as possible.

What is important is that we have now got the basic framework of parallel restore, so that some researchers can easily experiment with various tweaks to improve the performance.

pg_restore is going to be with us for quiter a long time. Even if we manage to get pg_upgrade working pretty well, that will take quite a bit of time, and there is currently no guarantee that it will for for every release. So I expect pg_restore to be the most common method of upgrading for quite some time, making it run as fast as possible is thus still a significant requirement.

I'm proud to have been able to contribute this feature to Postgres, and look forward to other people improving it further as time goes by.

PostgreSQL Weekly News: PostgreSQL Weekly News June 28th 2009

Sun, 28 Jun 2009 08:00:00 +0000

8.4.0 will be out soon. Meanwhile, keep testing! http://www.postgresql.org/developer/beta

Josh Berkus: Anyone Have a Timer for Lightning Talks?

Sat, 27 Jun 2009 17:31:36 +0000

I run a lot of lightning talks, and one tool I haven't been able to find a satisfactory solution for is the timer for the talks ...

Andreas Scherbaum: Call for Papers for the PostgreSQL-Devroom at FrOSCon 2009

nospam@example.com (Andreas 'ads' Scherbaum) — Fri, 26 Jun 2009 17:27:59 +0000

Andreas 'ads' Scherbaum

The PostgreSQL Project will have a dev-room at FrOSCon on sunday, august 23, 2009. Talks wanted!

The theme should be PostgreSQL-related, please submit the talk(s) by using the FrOSCon Pentabarf:

https://pentabarf.froscon.org/submission/froscon2009/

Procedure:

Create account (if not yet done) and follow the confirmation link in the email

Create a new event

Choose track "PostgreSQL"

All submitters will receive a confirmation timely, if the talk is accepted. Who wants to submit a talk about databases in general - or a talk about another database - may choose the "OpenSQL Camp" track.

Kenny Gorman: pg_reorg 1.0.4

Fri, 26 Jun 2009 00:33:11 +0000

At Hi5, we currently use pg_reorg1.0.3 in order to organize data in a clustered fashion. I posted previously about the strategy. Our version is slightly modified, the modifications I made to the C code essentially allow pg_reorg to spin/wait for locks on the objects to be released before proceeding.

The good news is the folks at NTT have incorporated a similar change in pg_reorg 1.0.4. This is a fantastic improvement, and frankly implemented in a cleaner way than my changes.

The crux of the issue is the situation where a database is being auto-vacuumed, you can’t be guaranteed that pg_reorg and the vacuum will not collide. In theory you should not need to vacuum a table which you are pg_reorg’ing because that is the point of a pg_reorg, it’s essentially a vacuum full w/ extra features because the table is being rebuilt from scratch. However in an environment where auto-vacuum is being utilized to keep tables vacuumed, both will need to co-exist.

The change is simple, use the NOWAIT option of lock table to fail if the lock can not be obtained. This is wrapped in a loop until the lock is granted. The effect is pg_reorg patiently sits and waits while your vacuums complete and then it can finish it’s work. The downside is if any of these operations run for too long, then the journal table may grow very large. So there should be some monitoring wrapped around the code if it’s intended to run in the background. For the future we need a backoff algorithm as well as perhaps a limit to the number of spin/sleep cycles, but hey this is excellent progress.

This tool is essential in my humble opinion for everyone running PostgreSQL in a high transaction/high availability environment. By the way, pg_reorg works seamlessly with Slony-I.

The code addition does the following:

for (;;)                        
        {
                command("BEGIN ISOLATION LEVEL READ COMMITTED", 0, NULL);
                res = execute_nothrow(table->lock_table, 0, NULL); 
                if (PQresultStatus(res) == PGRES_COMMAND_OK)
                {
                        PQclear(res);
                        break;
                }
                else if (sqlstate_equals(res, SQLSTATE_LOCK_NOT_AVAILABLE))
                {
                        /* retry if lock conflicted */ 
                        PQclear(res);
                        command("ROLLBACK", 0, NULL);
                        sleep(1);
                        continue;
                }
                else
                {
                        /* exit otherwise */
                        printf("%s", PQerrorMessage(connection));
                        PQclear(res);
                        exit(1);        
                }
        }

The text below is a snip of the strace on pg_reorg while it’s waiting for the lock:

rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_DFL}, 8) = 0
sendto(3, "P\0\0\0008\0SELECT reorg.reorg_apply($"..., 529, 0, NULL, 0) = 529
rt_sigaction(SIGPIPE, {SIG_DFL}, {SIG_IGN}, 8) = 0
poll([{fd=3, events=POLLIN|POLLERR, revents=POLLIN}], 1, -1) = 1
recvfrom(3, "1\0\0\0\0042\0\0\0\4T\0\0\0$\0\1reorg_apply\0\0\0\0"..., 16384, 0, NULL, NULL) = 77
rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_DFL}, 8) = 0
sendto(3, "P\0\0\0\177\0SELECT 1 FROM pg_locks WHE"..., 178, 0, NULL, 0) = 178
rt_sigaction(SIGPIPE, {SIG_DFL}, {SIG_IGN}, 8) = 0
poll([{fd=3, events=POLLIN|POLLERR, revents=POLLIN}], 1, -1) = 1
recvfrom(3, "1\0\0\0\0042\0\0\0\4T\0\0\0!\0\1?column?\0\0\0\0\0\0\0"..., 16384, 0, NULL, NULL) = 74
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({1, 0}, {1, 0})

Your Postgresql log file will show the following:

Jun 25 17:09:33 <dbname> postgres[7825]: [37-2] 2009-06-25 17:09:33 PDTSTATEMENT:  LOCK TABLE <tablename> IN ACCESS EXCLUSIVE MODE NOWAIT
Jun 25 17:09:34 <dbname> postgres[7825]: [38-1] 2009-06-25 17:09:34 PDTERROR:  could NOT obtain LOCK ON relation "<tablename>"

Josh Berkus: Normalization, Performance, and Virtual Private Databases

Thu, 25 Jun 2009 19:01:22 +0000

Whenever I encounter a pervasive query performance problem, one of the first things I ask is, "is this database really normalized?" Usually the answer is no ...

Hubert Lubaczewski: explain.depesz.com – update

Thu, 25 Jun 2009 13:04:44 +0000

I just updated explain.depesz.com with bugfix, which changes the way Bitmap Index Scan and Bitmap Heap Scan are displayed. Apparently index and table names were not shown previously. Thanks go to Viktor Rosenfeld for spotting and reporting the bug.

Joshua D. Drake: PostgreSQL Conference West 2009 Call for Papers

Wed, 24 Jun 2009 19:00:00 +0000

PostgreSQL Conference West 2009 Call for Papers June 24th, 2009, the PostgreSQL Conference U.S. team is pleased to announce the West 2009 venue and call for papers. This year the premiere West Coast PostgreSQL Conference will be leaving its roots at Portland State University and moving north to sunny Seattle, Washington. The event this year is being held at Seattle Central Community College from October 16th through 18th. The move to Seattle opens up a larger metropolitan area for continuing to expose databases users, developers, and administrators to the World's Most Advanced Open Source Database. Following previously successful West Coast conferences, we will be hosting a series of 3-4 hour tutorials, 90 minute mini-tutorials, and 45 minute talks. This year we will be continuing our trend of covering the entire PostgreSQL ecosystem. We would like to see talks and tutorials on the following topics: General PostgreSQL:

Administration
Performance
High Availability
Migration
GIS
Integration
Solutions and White Papers

The Stack:

Python/Django/Pylons/TurboGears/Custom
Perl5/Catalyst/Bricolage
Potato
Ruby/Rails
Java (PLJava would be great)/Groovy/Grails
Operating System optimization (Linux/FBSD/Solaris/Windows)
Solutions and White Papers

If you are using PostgreSQL as your platform, you need to be presenting at this conference! Submit your talk (You must be have an account on the site)
*** The PostgreSQL Conference U.S. series is an autonomous Educational Project used to educate all comers on the use of The World's Most Advanced Open Source Database. Proceeds from the event are donated directly to United States PostgreSQL; the 501c3 non-profit for PostgreSQL education and advocacy in the United States.

Josh Berkus: OSCON for Postgres Geeks

Wed, 24 Jun 2009 18:13:32 +0000

In addition to the pgDay San Jose, there are several talks at OSCON which will be of interest to PostgreSQL community ...

Andrew Dunstan: Workation

nospam@example.com (Andrew Dunstan) — Wed, 24 Jun 2009 12:28:23 +0000

Tomorrow we pack the family and pets up and head off to Hilton Head Island for eight days. A little swimming, a little jetskiing, probably too much good dining, lots of relaxation.

I have been prepping a laptop to take with me. Unfortunately, several clients want me to be able to do things if an emergency arises, so I need the laptop to be able to talk to their VPNs. That means I need to support whatever they use. Everybody seems to use something different. So far I am supporting Windows and Checkpoint VPN on WIndows (since that's the only place the client runs) and OpenVPN, CiscoVPN and Juniper on Linux. Many of these have the very annoying habit of locking the client machine out from any access to anything other than the VPN. That can be more than a small inconvenience. I complained about it to one client the other day and the response was "Yes, we hate it too." Not enough, apparently, to change it. Overall, I have found OpenVPN to be the nicest VPN so far to use on Linux. On Windows, the native VPN works pretty simply too, and is a good fit if you're only going to be talking to Windows boxes via Remote Desktop.

I decided yesterday to upgrade the Linux install on the laptop from Fedora 10 to Fedora 11. You would think that after all this time they would be able to handle upgrading their own packages nicely, but it didn't work any better than the last time I tried to upgrade a RedHat machine, back around the time of RedHat 9. So I backed up what little stuff I had that wasn't actually a Fedora artifact, and reinstalled. The other thing about upgrading that made it so much fun was when anaconda decided to barf when trying to mount my swap file, which it couldn't see. I had to comment it out of the fstab. Why anaconda needs to mount the swap file to upgrade, when it doesn't need any to install, AFAIK, is beyond me.

One nice thing I noticed is that Fedora 11 has a much nicer naming scheme for volume groups and logical volumes. Good work there.

Anyway, now I have F11, with all the VPNs working and tested. I'm hoping that customers won't be too demanding so that I can grab a few minutes to continue with making the buildfarm client support git.

US PostgreSQL Association: PostgreSQL at SELF

Tue, 23 Jun 2009 14:22:55 +0000

Michael Brewer wrote:

On Saturday, June 13th, I wound up manning the PostgreSQL booth at SouthEast LinuxFest, in Clemson, South Carolina. This free conference drew a larger crowd than I'd expected; organizers told me there had been some 450 registrants by the day before, and they were expecting a final total of over 500 (with walk-ups).

Hubert Lubaczewski: Tips N’ Tricks – Generating readable reports with plain SQL

Tue, 23 Jun 2009 11:51:43 +0000

Let’s say you imported some data, but it contains duplicates. You will have to handle them in some way, but to make sensible choice on how to handle it, you need more information. So, let’s start. We have table: # \d users [...]

Dimitri Fontaine: prefix extension reaches 1.0 (rc1)

Tue, 23 Jun 2009 07:53:00 +0000

At long last, after millions and millions of queries just here at work and some more in other places, the prefix project is reaching 1.0 milestone. The release candidate is getting uploaded into debian at the moment of this writing, and available at the following place: prefix-1.0~rc1.tar.gz.

If you have any use for it (as some VoIP companies have already), please consider testing it, in order for me to release a shiny 1.0 next week! :)

Recent changes include getting rid of those square brackets output when it's not neccesary, fixing btree operators, adding support for more operators in the GiST support code (now supported: @>, <@, =, &&). Enjoy!

David Wheeler: pgTAP Set-Testing Update

Mon, 22 Jun 2009 17:34:00 +0000

I've been thinking more about testing SQL result sets and how to name functions that do such testing, and I've started to come to some conclusions. Some of the constraints I'm looking at:

PostgreSQL Weekly News: PostgreSQL Weekly News June 21st 2009

Sun, 21 Jun 2009 08:00:00 +0000

The Call for Papers for pgday.eu is out. Submit! http://2009.pgday.eu/

Robert Hodges: When SANs Go Bad

noreply@blogger.com (Robert Hodges) — Sat, 20 Jun 2009 11:27:34 +0000

They sometimes go bad in completely unpredictable ways. Here's a problem I have now seen twice in production situations. A host boots up nicely and mounts file systems from the SAN. At some point a SAN switch (e.g., through a Fibrechannel controller) fails in such a way that the SAN goes away but the file system still appears visible to applications.

This kind of problem is an example of a Byzantine fault where a system does not fail cleanly but instead starts to behave in a completely arbitrary manner. It seems that you can get into a state where the in-memory representation of the file system inodes is intact but the underlying storage is non-responsive. The non-responsive file system in turn can make operating system processes go a little crazy. They continue to operate but show bizarre failures or hang. The result is problems that may not be diagnosed or even detected for hours.

What to do about this type of failure? Here are some ideas.

Be careful what you put on the SAN. Log files and other local data should not go onto the SAN. Use local files with syslog instead. Think about it: your application is sick and trying to write a log message to tell you about it on a non-responsive file system. In fact, if you have a robust scale-out architecture, don't use a SAN at all. Use database replication and/or DRBD instead to protect your data.
Test the SAN configuration carefully, especially failover scenarios. What happens when the host fails from access one path to another? What happens when another host picks up the LUN from a "failed" host? Do you have fencing properly enabled?
Actively look for SAN failures. Write test files to each mounted file system and read them back as part of your regular monitoring. That way you know that the file system is fully "live."

The last idea gets at a core issue with SAN failures--they are rare, so it's not the first thing people think of when there is a problem. The first time this happened on one of my systems it was around 4am in the morning. It took a really long time to figure out what was going on. We didn't exactly feel like geniuses when we finally checked the file system.

SANs are great technology, but there is an increasingly large "literature" of SAN failures on the net, such as this overview from Arjen Lentz and this example of a typical failure. You need to design mission-critical systems with SAN failures in mind. Otherwise you may want to consider avoiding SAN use entirely.

PDXPUG: OSBridge Pg BOF!

Fri, 19 Jun 2009 04:31:07 +0000

Reporting LIVE from OSBridge...we had the PostgreSQL BOF at the same time as our monthly PDXPUG meeting. 6 regulars, a new person, and a couple of out-of-towners were in attendance. Josh B gave us a quick overview of 8.4. 293 (or 297!) new features. In addition to the big stuff (like windowing functions) there are a ton of little administrative tweaks. I am particularly excited about the new functionality of \df - displays only user-defined fuctions.

Announcements:
PgDay is Saturday Sept 19 in Athens, GA! Make your plans now.

Michael Brewer highly recommends the use of a tuba as a booth prop. Ask him to tell you the story.

Selena & I are working on PostgreSQL equivalents for the queries in Stephan Faroult's _Refactoring SQL Applications_.

I was disappointed to learn that TrustTheVote is running on MySQL. Granted, it's an improvement over current voting software, but maybe we want to talk to them about running on a different, say, more elephantine database?

Future plans:
Josh told us about the patch review process. You don't have to be able to read C! If you can read the spec, apply the patch, verify that it works, proof the docs - you can be a reviewer! All you need to do is write a report. Webb suggested PDXPUG have a patch review party - sounds like a great idea to me!

Leo Hsu and Regina Obe: Restore of functional indexes gotcha

nospam@example.com (Leo Hsu and Regina Obe) — Fri, 19 Jun 2009 03:04:00 +0000

This has been bugging me for a long time and I finally complained about it and Tom Lane kindly gave a reason for the problem and that its by design and not a bug.

So I thought I would post the situation here without getting into too many embarassing specifics in case others have suffered from a similar fate and can learn from this.

The situation:

You create a function lets call it myniftyfunc() in the public schema.
Then you create another function that depends on myniftyfunc(), lets call it mysuperniftyfunc() also in public schema.
Then because your function is such a super nifty function, you decide to create a functional index with that super function on your table that sits in mysuperdata schema - mysuperdata.mysupertable

Your super nifty function is doing its thing; your table is happy; the planner is spitting out your queries lightning fast using the super nifty index on your super table; The world is good.

One day you decide to restore your nifty database backup and to your chagrin, your nifty index is not there. The planner is no longer happily spitting out your queries lighting fast and everything has come to a painful crawl. Your super nifty index is gone. What happened to super nifty functional index?

I have to admit that I'm the type of person that assumes the public schema is always there and always in search_path and that my assumption is a flawed one. After all the public schema is there by default on new databases for convenience, but one can change it not to be in the search_path and in fact pg_dump does just that. So if everything you have is kept in public schema -- you don't run into this particular misfortune. If however you have your functions in public and your tables in different schemas, during restore -- the search path is changed to the schema being restored and your super functional indexes based on super functions that depend on other super functions fail because public is no longer in the search_path.

Continue reading "Restore of functional indexes gotcha"

Andrew Dunstan: Misuse of SQL

nospam@example.com (Andrew Dunstan) — Thu, 18 Jun 2009 22:12:41 +0000

Today I was reminded of something I saw about 15 years ago, when working as a systems adminstrator. One of the application developers had a large amount of data that needed to be sorted. Knowing SQL, he wrote a program in C with embedded SQL which created a table, read in the data, and then selected it with an ORDER BY clause. Problem solved.

Talk about a sledgehammer to crack a nut. Switching metaphors, I guess it's a case of everything looking like a nail if all you have is a hammer.

Note that 10 lines of C code would have sorted the data in a fraction of the time. Or a simple call to the unix sort utility. Even if the data were too big to fit in memory, something much lighter like using a Berkeley DB btree or hash would have hardly been much more trouble.

Today I saw something similar. Someone created an in-memory SQLite table, and set up an index on it, just for fast random access to it. And this was in PHP which has associative arrays which presumably offer fast keyed access (by hashing the keys I guess - I know nothing of the innards of PHP.)

After rolling on the floor laughing for a while, I ripped the guts out of this thing and just emulated it using an array of arrays. Guess what? Performance was fine and the program no longer relies on SQLite, the non-PDO version of which isn't available on the version of PHP installed on the machine in question.

SQL is not a general purpose tool. It's a tool for manipulating data with quite specialised needs, and using it for something else is just silly.

Gabriele Bartolini: Free PostgreSQL support for the students of the University of L'Aquila

Thu, 18 Jun 2009 08:12:17 +0000

We all know about the terrible events following the earthquake that shook the city of L'Aquila and the entire Abruzzo region last April.
In assisting the university to reinstate its teaching program, the company I work for (2ndQuadrant Italia) will offer students a free and preferential consulting service on PostgreSQL. Back in February and March, we had had several e-mail exchanges and a few Skype sessions with Professor Paolino Di Felice from the "Università degli Studi dell'Aquila".
I was supposed to visit their University in May, together with my colleague Gianni Ciolli. The aim of the visit was to personally get to know Prof. Di Felice and the students of his database systems class, and to introduce them to PostgreSQL.
Unfortunately, we all know what happened afterwards.
Our initiative aims to help students gain practical skills and knowledge in installation procedures, configuration and the usage of PostgreSQL.
This embraces also Italian PostgreSQL Users Group's objective for the introduction of PostgreSQL in the Italian schools and universities.
The joint agreement will take effect from 1 October 2009 and has a duration of 15 months. More information are in the Italian press announcement.
Finally, I am also happy to restart my PlanetPostgreSQL.org activity with such great news for the whole PostgreSQL community!

Robert Hodges: Lots of New Tungsten Builds--Get 'Em While They're Hot

noreply@blogger.com (Robert Hodges) — Thu, 18 Jun 2009 07:01:38 +0000

There is a raft of new Tungsten open source builds available for your replication and clustering pleasure. Over the last couple of days we uploaded new binary builds for Tungsten Replicator, Tungsten Connector, Tungsten Monitor, and Tungsten SQL Router. These contain the features described in my previous blog article, including even more bug fixes (36 on Tungsten Replicator alone) than I had expected as we had a debugging fest over the last few days that knocked off a bunch of issues. You can pick up the builds on the Tungsten download page. Docs are posted on the Tungsten wiki.

If you have questions, see problems with the builds, or just want to tell us how great they are, please post on the community forums or on the tungsten-discuss mailing list.

Our next open source release will be the Tungsten Manager, which is long overdue to join the family of regular builds. We are doing some polishing work on the state machine processing and group communications, after which the Manager will go out along with documentation on how to use it.

Hubert Lubaczewski: explain.depesz.com – update

Wed, 17 Jun 2009 15:48:18 +0000

I just updated explain.depesz.com with 2 new interface features: When you put mouse cursor over node, it will mark direct child nodes with When you click on node, all child nodes (even indirect) will be hidden, and the node you clicked on will be marked with Both features are directed towards people who analyze longer plans [...]

Robert Treat: Got Team?

nospam@example.com (Robert Treat) — Tue, 16 Jun 2009 19:03:00 +0000

This blog post is just a test posting for some new functionality just added to Planet PostgreSQL. Since it revolves around teams, I thought I'd do some self promotion, or rather, team building promotion. You see, we're still looking for DBA's here at OmniTI. Our database management practice continues to grow, and we need smart people to come work on some of the biggest systems and highest transaction loads you can find on Postgres. We'll also make you work with things like Oracle, MySQL, Sqlite, and who knows what else, just enough to make you question your assumptions. It's not for the faint of heart, but it's certainly interesting. Sound good? See the website for details, and who knows, maybe you'll end up on Planet Postgres as part of "Team OmniTI"!

Magnus Hagander: Planet updates

nospam@example.com (Magnus Hagander) — Tue, 16 Jun 2009 18:43:00 +0000

I've just deployed a new version of the code that runs http://planet.postgresql.org. Most of this code was written by Selena and me during the initial days at PGCon. It just needed some minor polishing, which I didn't get around to until now. So, the new things are:

Support for Team blogs: This is just a grouping of existing blogs, not actually something new we parse. The idea is to give some exposure to a team someone works for - for example, a specific PostgreSQL support company.
Top posters list: The list of all subscriptions has been replaced with a list of top posters. The list was becoming a bit too large to manage, and didn't really fill a purpose. And it was hard to integrate nicely with the Team blogs feature.
There has also been a bunch of internal changes: Details available in the git repo on http://git.postgresql.org.

If you want to make use of the Team blogs feature, this has unfortunately not been implemented in the admin interface. We (well, me, really) were just a bit too lazy for that. So if you want to make use of it, please just send an email to planet@postgresql.org letting us know what name you want for the team, and which blogs to add to it (these blogs should already be subscribed to planet).

Bruce Momjian: Pg_Migrator Release Candidate 1

Tue, 16 Jun 2009 16:00:00 +0000

I have released pg_migrator release candidate 1 today. I was planning on doing the final release this week, but the backend API that pg_migrator uses was improved in Postgres 8.4 RC1 so pg_migrator now requires an 8.4 RC1 or later server and no longer works for Postgres 8.4 beta versions. Also, Tom Lane pointed out that pg_migrator shouldn't go final until PostgreSQL goes final, so I will continue in RC status until PostgreSQL 8.4 final is released.

Bruce Momjian: Professor Momjian

Mon, 15 Jun 2009 16:00:00 +0000

You might remember the PostgreSQL East conference that took place this past April at Drexel University in Philadelphia. As part of the event, I and other conference leaders had dinner with the Drexel Computer Science Department Head, Jeremy Johnson. One item we discussed was our frustration at the limited academic adoption of Postgres, even though Postgres is better for education than many other databases used in academia. Jeremy seemed to understand our plight, and I gave him my business card in case I could help Drexel, perhaps by doing a guest lecture about Postgres.

PostgreSQL Weekly News: PostgreSQL Weekly News June 14th 2009

Sun, 14 Jun 2009 08:00:00 +0000

PostgreSQL 8.4 RC1 will be out soon. Test! http://www.postgresql.org/developer/beta

Stefan Kaltenbrunner: Benchmarking 8.4 - Chapter 1/Read-Only workloads

nospam@example.com (Stefan Kaltenbrunner) — Fri, 12 Jun 2009 20:21:00 +0000

Computing platforms are constantly changing, evolving and improving. This holds true for both the hardware and the software running on that hardware.
With PostgreSQL 8.4 just around the corner and a 2U Nehalem based IBM x3650M2 at my hands I thought I would do some benchmarking and see how well PostgreSQL does on that kind of hardware under a range of different workloads. This is planned as a series of posts starting with read-only benchmarking, followed up by bulk load testing and maybe some OLTP benchmarks as well.

Continue reading "Benchmarking 8.4 - Chapter 1/Read-Only workloads"

Joshua D. Drake: The shortest path between two points

Fri, 12 Jun 2009 17:52:12 +0000

Recently I was doing some benchmarking on one of our machines. The benchmarking wasn't going so well due to bad batteries on the RAID controller. I had instructed one of our System Administrators to take care of the problem. Long story short, the Administrator went down a very long trail to an obvious solution. The trail was well mapped, thought out and precise. It however missed some important points. When I caught on to the long trail she was taking I asked, "What is the shortest path between two points?". She replied, "On a plane or sphere?". That is when I knew we were in trouble. Now most people would have just said, "Huh?". Luckily I have been blessed with at least a modicum of technical/mathematical knowledge and a general experience with Geeks for 18 years. Let's review. The System Administrators trail was:

Visit Colo to check cards physically after receiving BIOS message
Record all information about cards including BIOS versions
Research possible cause of batteries not charging, find that some versions of the BIOS can do that. Thus causing yet another trip to the colo for installation.
Go back to colo to update BIOS revisions to see if that resolves the problem
- If that doesn't solve the problem, research new batteries for order

My trail:

System reports batteries are bad
Known fact: Hardware was bought used
Buy new batteries
Upgrade BIOS during new battery installation

What is the difference? I run into this a lot with technical people. They become hyper focused and they are not able to abstract their problem solving skills to include the entirety of the problem. Now you say, "What the problem is the batteries don't work, fix them." It isn't that simple. Using the System Administrators path, the solution to the problem cost at least 1500.00-2000.00. Using my path, the cost is 550.00. My path is a single trip to the collocation facility with cards drop shipped, an hour to replace and upgrade BIOS. Same resolution, ~ 37% of the cost. But... But... what if that doesn't fix the problem? Then you know you have bad cards and it will still cost less to replace them that to perform multiple trips to the colo. Again it isn't that the System Administrators path was incorrect. In fact I would bet a lot of businesses would think it was the absolutely correct direction to take. I would rather just write off the cards and replace them. We can make far more money from the System Administrator if they are not focusing on long and winding roads to the same destination that can be reached by not taking a left at the fork.

Robert Treat: The Asynchronous Services Analogy

nospam@example.com (Robert Treat) — Fri, 12 Jun 2009 03:52:02 +0000

Today I had a chance to sit through a sneak preview of Theo Schlossnagle's new talk Scalable Internet Architectures, to be delivered next week at Velocity 2009 (Dev sessions are an underrated side benefit of working at OmniTI). As always Theo packs a lot of good information into his talks; I could probably do blog entries on half a dozen ideas I jotted down; but I wanted to highlight something that he mentioned with regards to scaling websites via asynchronous services.

Continue reading "The Asynchronous Services Analogy"

Josh Berkus: Tech Talk Tip #3: Talk Abstract

Thu, 11 Jun 2009 00:22:30 +0000

Ok, so you've investigated the conference and you have a title for your talk. Now they want a talk abstract and description, and maybe a bio. What do you put there?

Robert Hodges: Tungsten Development News - Lots of New Features!

noreply@blogger.com (Robert Hodges) — Wed, 10 Jun 2009 13:50:50 +0000

Articles on this blog have been pretty scanty of late for a simple reason--we have been 100% heads-down in Tungsten code since the recent MySQL Conference. The result has been a number of excellent improvements that are already in Subversion and will appear as open source builds over the next couple of weeks.

Tungsten has a simple goal: create highly available, performant database clusters using unaltered commodity databases that are simple to manage and look as close to a single database as possible for applications. Over the last two months we completed the integration of individual Tungsten components necessary to make this happen.

Full integration is a big step forward and finally gets us to the ease-of-use we were seeking. Imagine you want to add a slave database to the cluster. There's no management procedure any more--you just turn it on. Managers in the cluster automatically detect the new slave and add it as a data source. That's the way we want every component to work from top to bottom--either on or off, end of story. It was really nice to see it start to work a few weeks ago.

We are now ready to start pushing builds out to the Tungsten SourceForge.net project. Here is a selection of the features:

Tungsten Replicator -- API support for seamless failover, certification on Solaris, better Windows support, testing against MariaDB, and many other improvements like flush events for seamless failover. There are already 26 fixes in JIRA and I expect more before we post the build.

Tungsten SQL Router -- Pluggable load balancing with session consistency support. Session consistency means users see their own writes but can read changes by other users from a slave. It works using a single database connection, which is an important step toward eliminating application changes in order to scale on master/slave clusters.

Tungsten Manager -- Directory-based management model that allows you to view and manage both JMX-enabled services as well as regular operating system processes that follow the familiar LSB pattern of 'service name start/stop/restart'. The managers use group communications and can broadcast commands across multiple hosts, handle failures, and automatically detect new services as they come online.

Tungsten Monitor -- Improved monitoring of replicator status including slave latency, which is necessary to guide SQL Router load balancing features like session consistency.

There's a lot going on with Tungsten right now, in fact far too many things to mention even in a longish post like this one. One of my current code projects is to implement built-in backup and restore for Tungsten Replicator. I am planning on supporting slave auto-provisioning: a new slave comes up, restores the latest backup, and starts replicating. All you have to do is turn the slave on. (More of that on/off stuff--it's kind of an obsession for us at this point.)

Integrating backup/restore is the final big feature for Tungsten Replicator 1.0--after this we plan to turn attention to parallel replication and are already discussing how this might work with several potential customers. Feel free to contact me through this blog or better yet post on the community forums parallel replication topic to join the conversation.

One final bit of news, we are starting to work seriously on Tungsten PostgreSQL integration thanks to a new partnership between Continuent and 2nd Quadrant. This work is commercially focused for now but will lead to additional open source features in the not too distant future. Keep watching this space... :)

p.s., We also had a nice refit on the community website. Check it out.

Leo Hsu and Regina Obe: Planner Statistics

nospam@example.com (Leo Hsu and Regina Obe) — Tue, 09 Jun 2009 13:10:00 +0000

You'll often hear the term planner statistics thrown around by database geeks. Did you update your statistics. This lingo isn't even limited to PostgreSQL, but is part and parcel to how most decent databases work. For example in PostgreSQL you do a vacuum analyze to update your planner statistics in addition to cleaning up dead space. In SQL Server you do an UPDATE STATISTICS. In MySQL you do an ANALYZE TABLE or a more invasive OPTIMIZE TABLE.

Normally all this "update your stats so your planner can be happy" is usually unnecessary unless you just did a bulk load or a bulk delete or you are noticing your queries are suddenly slowing down. These stat things are generally updated behind the scenes by most databases on an as needed basis.

What makes SQL really interesting and a bit different from procedural languages is that it is declarative (like functional and logical programming languages) and relies on the database planner to come up with strategies for navigating the data. Its strategy is not fixed as it is in procedural languages. A big part of this strategy is decided on by the query planner which looks at distributions of data. Given different WHERE conditions for similar queries, it could come up with vastly different strategies if one value has a significantly higher distribution in a table than another. This is also the mystery of why it sometimes refuses to use an index on a field because it has decided a table scan is more efficient and also why some people consider HINTS evil because they pollute the imperative nature of the language.

Continue reading "Planner Statistics"

Francisco Figueiredo Jr: Another postcard has arrived!

noreply@blogger.com (Francisco Figueiredo Jr.) — Tue, 09 Jun 2009 13:07:12 +0000

Hi, all!

Another postcard has arrived. It is from Transylvania:

It is from Alin Hanghiuc (Update: Thanks for correction, Alin!) from Brasov, Romenia.

Thank you very much, Alin for sending this postcard. This give us a lot of motivation!

Andrew Dunstan: Software kinda guy

nospam@example.com (Andrew Dunstan) — Mon, 08 Jun 2009 17:49:53 +0000

Some people drool over hardware specs, They will talk animatedly about drives and CPUs and such.

I'm not one of these. In fact, I'm not really good with hardware at all. Discussions like the above leave me fairly bored. Elegant algorithms, and elegant expressions of algorithms, are things that interest and occasionally delight me. But not machine specs, and the like.

Now that might surprise some in a person who has made his living with computers for about 25 years. But it's always been so. I recall at one stage of my career working on some machines that I had never seen, and I wouldn't have known what they looked like if I fell over them. They lived in the data centre or computer room or whatever it was called back then, and all I needed to know was how to drive them from my terminal. I knew some basics about the hardware, but nothing terribly specific.

Perhaps this comes from the fact that I have never been very well coordinated physically (possibly a function of my bad vision, but possibly just genetic - it's been so from before high school), and so I tend to fumble things a lot. That doesn't matter too much when you can undo your typing in an editor, say, but it matters a lot when you're carrying things or plugging them in. I once saw someone drop a DASD pack, a rather expensive storage item back then for big iron machines, and maybe that too has frightened me a bit of getting too involved with hardware.

Anyway, I think it's probably a sort of Mars/Venus thing.

These thoughts have absolutely nothing to do with Postgres, except in the sense that they tell you something about why I spend my time working on it rather than on stuff closer to the metal.

All this popped into my head while I was wrestling with hardware, of course. I'm converting an oldish laptop to be dual boot, and moving data from an old 100Gb drive to a new 320Gb drive. In the course of exploring the wonders of Windows file systems and cloning them, I must have swapped the old and new drives in and out half a dozen times, dropping screws along the way, and finding them again.

The process is almost complete. I finally got it all working with a nice little package called SystemRescueCD which appears to do the job quite nicely. In particular, there's a package included there called FSArchiver which seems to work very nicely. Soon I will have my daughter happily dual booting this laptop. (And no, a virtual Windows machine would not have done - some of the software needs to work direct with the hardware.)

Now, back to work ...

David Wheeler: Need Help Naming Result Set Testing Functions

Mon, 08 Jun 2009 00:06:00 +0000

I've been thinking more since I posted about testing SQL result sets, and I think I've settled on two sets of functions for pgTAP: one that tests two SQL queries (though you will be encouraged to use a prepared statement), and one to test two cursors. I'm thinking of naming them:

PostgreSQL Weekly News: PostgreSQL Weekly News June 7th 2009

Sun, 07 Jun 2009 08:00:00 +0000

The CfP for PGCon Brazil is open! http://pgcon.postgresql.org.br/2009/chamadas.en.php

Bruce Momjian: Challenges of Pg_Migrator

Sat, 06 Jun 2009 16:00:00 +0000

I just put out pg_migrator beta2, with only small changes to the install file since beta1. The install file is critical to the success of pg_migrator. Unlike the Postgres backend or libpq, pg_migrator relies on many external tools and administrator actions, so detailed, accurate install instructions are essential for reliable migration. Pg_Migrator is really not usable without consulting the ten-step installation instructions.

David Fetter: Materialized Views Performance Tips

nospam@example.com (David Fetter) — Sat, 06 Jun 2009 04:08:03 +0000

So you don't feel your clients' time is best spent on scanning a multi-million row table, and you've decided to materialize some views. Here are some principles to keep in mind as you go forward...
Continue reading "Materialized Views Performance Tips"

PDXPUG: PDXPUG June Meeting

Fri, 05 Jun 2009 20:03:17 +0000

June's meeting will be at the usual time (7pm), but will be held at OSBridge. BoF format.

http://opensourcebridge.org/sessions/274

(You don't have to be a registered conference attendee for the BoF, so come on out!)

Magnus Hagander: Getting a range of entries centered around a point

nospam@example.com (Magnus Hagander) — Fri, 05 Jun 2009 13:26:00 +0000

I had a question yesterday on an internal IRC channel from one of my colleagues in Norway about a SQL query that would "for a given id value, return the 50 rows centered around the row with this id", where the id column can contain gaps (either because they were inserted with gaps, or because there are further WHERE restrictions in the query).

I came up with a reasonably working solution fairly quickly, but I made one mistake. For fun, I asked around a number of my PostgreSQL contacts on IM and IRC for their solutions, and it turns out that almost everybody made the exact same mistake at first. I'm pretty sure all of them, like me, would've found and fixed that issue within seconds if they were in front of a psql console. But I figured that was a good excuse to write a blog post about it.

The solution itself becomes pretty simple if you rephrase the problem as "for a given id value, return the 25 rows preceding and the 25 rows following the row with this id". That pretty much spells a UNION query. Thus, the solution to the problem is:


    SELECT * FROM (
        SELECT id,field1,field2 FROM mytable WHERE id >= 123456 ORDER BY id LIMIT 26
    ) AS a
UNION ALL
    SELECT * FROM (
        SELECT id,field1,field2 FROM mytable WHERE id < 123456 ORDER BY id DESC LIMIT 25
    ) AS b
ORDER BY id;

The mistake everybody made? Forgetting that you need a subselect in order to use LIMIT. Without subselects, you can't put ORDER BY or LIMIT inside the two separate parts of the query, only at the outer end of it. But we specifically need to apply the LIMIT individually, and the ORDER BY needs to be different for the two parts.

Another question I got around this was, why use UNION ALL. We know, after all, that there are no overlapping rows so the result should be the same as for UNION. And this is exactly the reason why UNION ALL should be used, rather than a plain UNION. We know it - the database doesn't. A UNION query will generate a plan that requires an extra unique node at the top, to make sure that there are no overlapping rows. So the tip here is - always use UNION ALL rather than UNION whenever you know that the results are not overlapping.

All things considered, this query produces a pretty quick plan even for large datasets, since it allows us to do two independent index scans, one backwards. Since there are LIMIT nodes on the scans, they will stop running as soon as they have produced the required number of rows, which is going to be very small compared to the size of the table. This is the query plan I got on my test data:


 Sort  (cost=54.60..54.73 rows=51 width=86)
   Sort KEY: id
   ->  Append  (cost=0.00..53.15 rows=51 width=86)
         ->  LIMIT  (cost=0.00..35.09 rows=26 width=51)
               ->  INDEX Scan USING mytable_pk ON mytable  (cost=0.00..55425.06 rows=41062 width=51)
                     INDEX Cond: (id >= 100000)
         ->  LIMIT  (cost=0.00..17.04 rows=25 width=51)
               ->  INDEX Scan Backward USING mytable_pk ON mytable  (cost=0.00..56090.47 rows=82306 width=51)
                     INDEX Cond: (id < 100000)

And yes, the final ORDER BY is still needed if we want the total result to come out in the correct order. With the default query plan, it will come out in the wrong order after the append node. But it's important to remember that by the specification the database is free to return the rows in any order it chooses unless there is an explicit ORDER BY in the query. The rows may otherwise be returned in a completely different order between different runs, depending on the size/width of the table and other parameters.

Josh Tolley: Inside PostgreSQL - Clause selectivity

noreply@blogger.com (Joshua Tolley) — Fri, 05 Jun 2009 10:55:02 +0000

One of the more valuable features of any conference is the so-called "hall track", or in other words, the opportunity to talk to all sorts of people about all sorts of things. PGCon was no exception, and I found the hall track particularly interesting because of suggestions I was able to gather regarding multi-column statistics, not all of which boiled down to "You're dreaming -- give it a rest". One of the problems I'd been trying to solve was where, precisely, to put the code that actually applies the statistics to a useful problem. There are several candidate locations, and certainly quite a few places where we could make use of such statistics. The lowest-hanging fruit, however, seems to be finding groups of query clauses that aren't as independent as we would normally assume. Between PGCon sessions one day, Tom Lane pointed me to a place where we already do something very similar: clausesel.c

"Clause selectivity" means much the same thing as any other selectivity: it's the proportion of rows from a relation that an operation will return. A "clause", in this case, is a filter on a relation, such as the "X = 1" and the "Y 10" in "WHERE X = 1 AND Y 10". PostgreSQL uses functions in clausesel.c to find clauses whose combined selectivity differs from the product of their individual selectivities. For instance, in "WHERE X 4 AND X 5", the "X 5" is redundant; the clauses' combined selectivity is simply that of "X 4". With "WHERE Y > 4 AND Y 10", clausesel.c can determine that we really want the selectivity of the clause "4 Y 10". It's also smart enough to recognize "pseudo-constants": values from non-volatile functions, or from the outer relation of a nested loop. Although these values aren't truly constants, they remain constant at the level of the query where the clause will be applied, and can be treated as constants.

With any luck, one day clausesel.c will also know enough to notice cases where, for instance, although "foo.x = 3" and "foo.y > 10" are individually true for much of table "foo", there are very few rows where both conditions are true.

Robert Treat: The First Rule of Postgres Club

nospam@example.com (Robert Treat) — Fri, 05 Jun 2009 04:04:00 +0000

I had to chuckle when reading the comments of Alan Snelsons blog entry talking about PostgreSQL's project stability in the face of recent MySQL unsteadiness. Of course it shouldn't be a surprise that mentioning Postgres around MySQL people you might get some type of negative reaction. And vice versa of course. And with other db communities too, not just these two. But I digress. However what I really liked about the comments was this quote:

"PostgreSQL vs. MySQL is a non-issue for most people because the decision is already made. I don’t know what’s the point in bringing that up every-time, it shines a bad light on the postgres community."

The beauty here is that I found this blog post via Planet MySQL, not Planet Postgres. Further I've never heard of Alan Snelson; the post's author; he hasn't spoken at a Postgres event, and isn't syndicated on Planet Postgres. As far as I know, he just some guy who uses MySQL a lot, and now is looking at Postgres. Apparently he likes it and wants to spread the word. And of course people who don't like hearing about Postgres use this as a mark against the Postgres community, even when the Postgres project has nothing to do with it. You know software is no good when people who use it want to tell you how awesome it is!

In truth, that's how a lot of Postgres advocacy works these days. When you see people posting on slashdot, most of the time it's not from people working directly with the project. For those of us who have been using Postgres for a long time, we've long since got the message that MySQL users don't want to hear about Postgres. When I sat in the MariaDB feature request round table at the Percona Performance conference, listening to people ask for feature after feature that Postgres already has; partial indexes, index fillfactor, online ddl, extensible data types, dtrace probes, PITR, the list goes on; I didn't wave the Postgres banner. The only time I spoke about Postgres was when Monty had questions about Postgres' implementation. Some people asked me why I wasn't more vocal, why didn't I tell everyone about all the wonderful things that Postgres can do? What can I say... it's the first rule of Postgres club.

Bruce Momjian: Pg_Migrator Beta 1 Released

Thu, 04 Jun 2009 16:00:00 +0000

I have released pg_migrator beta1 today. All outstanding issues have been addressed and I expect a brief beta period, perhaps a week, before pg_migrator 8.4 final is released.

Bruce Momjian: Back to Email

Thu, 04 Jun 2009 16:00:00 +0000

I have not read much email since the beginning of May --- that's when I started working on pg_migrator, then I attended PGCon, and then I did a week of training for EnterpriseDB. Only on May 30 was I able to devote full-time effort to reading Postgres email; by that time my unread email had grown to 3,000 messages. So for a week I only read email and have now caught up. You can actually see this in my graph of unread email.

Robert Treat: Guide to Postgres 8.4 video now available

nospam@example.com (Robert Treat) — Thu, 04 Jun 2009 03:39:00 +0000

For those looking for more information on the "Guide to Postgres 8.4" talk I have been giving recently, I thought I'd put up a few pointers to the information now available. First, there is now video available online from the presentation I gave at PGCon. It contains audio, video, and the slide stack to go with it. For some reason the audio feed doesn't work on my MacBook, though it does seem fine from my Linux box, so if you can get audio, it should be ok. I also noticed that some of the slides were cut in odd ways, so if you want the full slide stack from the talk, thats available on the PGCon website talk page. There's also a slightly longer slide stack from a 3 hour version of this talk I gave at PGEast which I made available at slideshare. There was supposed to be video of that session made available, but I haven't seen it yet, once it's up, I'll let you know. Hope this helps, and good luck with Postgres 8.4.

US PostgreSQL Association: DenverPUG/PgUS meeting rescheduled

Wed, 03 Jun 2009 17:31:44 +0000

JD wrote:

The June DenverPUG meeting that I was going to attend has been rescheduled due to summer conflicts. The current anticipated date is August. Stay tuned!