..::Planet PostgreSQL::..

15 days... and it all begins, PostgreSQL Conference East

2010-03-10T20:10:16+00:00

As we continue the countdown to the largest community and user conference in PostgreSQL history I am reminded of all the great content we have had in the past. Today while I was reviewing the curriculum for the PostgreSQL Performance and Maintenance class, I came across this great talk by Bruce Momjian as a further example of the high quality information you will receive not only from the various trainings but also all the other (over 50!) sessions at PostgreSQL Conference East!

Inside PostgreSQL Shared Memory: Bruce Momjian

PITR is not a PITA.

2010-03-10T18:30:07+00:00

I have a client who runs a PITR warm standby for their main production database. A couple of time recently it has fallen over, due to factors beyond the control of Postgres. The first time a data drive ran out of space, and the second time (today) a tablespace was created on the master that referred to a directory that didn't exist on the standby. Fixing this was really easy. Just remove the problem (clear space, create directory) and restart the standby. I'm really quite impressed by how robust the whole thing is. It's the sort of robustness that makes Postgres a joy to work with.

Free Your DBI-Link, and the Rest Will Follow

2010-03-10T17:46:57+00:00

DBI-Link is now on Github!

Here's how I did it:
Continue reading "Free Your DBI-Link, and the Rest Will Follow"

Temporal PostgreSQL Roadmap

2010-03-10T04:49:06+00:00

Why are temporal extensions in PostgreSQL important? Quite simply, managing time data is one of the most common requirements, and current general-purpose database systems don’t provide us with the basic tools to do it. Every general-purpose DBMS falls short both in terms of usability and performance when trying to manage temporal data.

What is already done?

PERIOD data type, which can represent anchored intervals of time; that is, a chunk of time with a definite beginning and a definite end (in contrast to a SQL INTERVAL, which is not anchored to any specific beginning or end time).
- Critical for usability because it acts as a set of time, so you can easily test for containment and other operations without using awkward constructs like BETWEEN or lots of comparisons (and keeping track of inclusivity/exclusivity of boundary points).
- Critical for performance because you can index the values for efficient “contains” and “overlaps” queries (among others).

Temporal Keys (called Exclusion Constraints, and will be available in the next release of PostgreSQL, 9.0), which can enforce the constraint that no two periods of time (usually for a given resource, like a person) overlap. See the documentation (look for the word “EXCLUDE”), and see my previous articles (part 1 and part 2) on the subject.
- Critical for usability to avoid procedural, error-prone hacks to enforce the constraint with triggers or by splitting time into big chunks.
- Critical for performance because it performs comparably to a UNIQUE index, unlike the other procedural hacks which are generally too slow to use for most real systems.

What needs to be done?

Range Types — Aside from PERIOD, which is based on TIMESTAMPTZ, it would also be useful to have very similar types based on, for example, DATE. It doesn’t stop there, so the natural conclusion is to generalize PERIOD into “range types” which could be based on almost any subtype.
Range Keys, Foreign Range Keys — If Range Types are known to the Postgres engine, that means that we can have syntactic sugar for range keys (like temporal keys, except for any range type), etc., that would internally use Exclusion Constraints.
Range Join — If Range Types are known to the Postgres engine, there could be syntactic sugar for a “range join,” that is, a join based on “overlaps” rather than “equals”. More importantly, there could be a new join type, a Range Merge Join, that could perform this join efficiently (without a Range Merge Join, a range join would always be a nested loop join).
Simple table logs — The ability to easily create an effective “audit log” or similar trigger-based table log, that can record changes and be efficiently queried for historical state or state changes.

I’ll be speaking on this subject (specifically, the new Exclusion Constraints feature) in the upcoming PostgreSQL Conference EAST 2010 (my talk description) in Philadelphia later this month and PGCon 2010 (my talk description) in Ottawa this May. In the past, these conferences and others have been a great place to get ideas and help me move the temporal features forward.

The existing features have been picking up a little steam lately. The temporal-general mailing list has some traffic now — fairly low, but enough that others contribute to the discussions, which is a great start. I’ve also received some great feedback from a number of people, including the folks at PGX. There’s still a ways to go before we have all the features we want, but progress is being made.

March meeting

2010-03-10T01:27:51+00:00

Next Thursday, for our regularly scheduled meeting, we will be having a 9.0 alpha testing party!

When: 7 pm, Thursday, March 19, 2010
Where: FreeGeek
Who: YOU!

If you want to test it on your own machine, please download it ahead of time:

http://www.postgresql.org/ftp/source/9.0alpha4/

If you want to work on one of the P4 Lab machines, contact Mark so he can get an account set up for you.

Read:
http://wiki.postgresql.org/wiki/HowToBetaTest
http://developer.postgresql.org/pgdocs/postgres/release-9-0.html

You don't need a laptop to participate - we can do pair-agile-eXtreme testing!

Afterwards, as usual, we will have beer/crash the PDX.pm Hackathon at the Lucky Lab.

Dynamic updates of fields in NEW in PL/pgSQL

2010-03-09T23:26:50+00:00

Dynamic updates of fields in NEW in PL/pgSQL Today, on #postgresql on IRC, strk asked about updating fields in NEW record, in plpgsql, but where name of the field is in variable. After some time, he sent his question to hackers mailing list. And he got prompt reply that it’s not possible. Well, I dare to disagree. First, big [...]

User Testimonial Video at PG East

2010-03-09T23:15:01+00:00

EnterpriseDB believes in the marketing potential of videos. You might remember them producing a video of me explaining pg_migrator last year.

This year, they will be interviewing PG East attendees to create a Postgres user testimonial video. The community has not used videos extensively for marketing, partly because, while it is easy to create a video, it is difficult to create a good video. (Consider the quality of the average YouTube video.) Anyway, my guess is that the video will be something like this customer video. The video will be shared with the community to help promote Postgres worldwide.

PostgreSQL UTF-8 Conversion

2010-03-09T19:47:01+00:00

It's becoming increasingly common for me to be involved in conversion of an old version of PostgreSQL to a new one, and at the same time, from an old "SQL_ASCII" encoding (that is, undeclared, unvalidated byte soup) to UTF-8.

Common ways to do this are to run pg_dumpall and then pipe the output through iconv or recode. When your source encoding is all pure ASCII, you don't need to do even that. When it's really all Windows-1252 (a superset of Latin-1 aka ISO-8859-1) it's easy.

But often, the data is stored in various unknown encodings from several sources over the course of years, including some that's already in UTF-8. When you convert with iconv, it dies with an error at the first problem, whereas recode will let you ignore encoding problems, but that leaves you with junk in your output.

The case I'm often encountering is fairly easy, but not perfect: Lots of ASCII, some Windows-1252, and some UTF-8. Since both pure ASCII and UTF-8 can be mechanistically detected, I put together this script to do the detection. It's Perl and uses the nice IsUTF8 module to do its character encoding detection:

Pipe input to the script. It handles one line at a time. When run with any arguments (such as --test) it will swallow pure ASCII lines, write lines it thinks are valid UTF-8 to stderr, and will convert the remaining presumed Windows-1252 lines to stdout, for manual examination.

If its guesses look correct, run it again with no arguments, and it will write all 3 types of encoding to stdout, ready for input to psql in your new UTF-8 encoded database.

(Don't forget to munge your pg_dump file to remove any hardcoded declarations of "SQL_ASCII" encoding from CREATE DATABASE statements, or otherwise make sure your database actually is created with UTF-8 encoding!)

PostgreSQL Conference East, Hotel Deadline!

2010-03-09T17:22:20+00:00

PostgreSQL Conference East, the largest PostgreSQL Conference for Users, Developers, Decision makers and anyone using PostgreSQL arranged for a hotel discount for attendees from the Radisson Warwick Hotel (the location of the conference).

The retail price of a double room is ~ 199.00. The discount rate is 132.00.

If you are attending PostgreSQL Conference East and you would like the discount you must register by the 11th of March. For more information:

Accommodations

Agenda

List of talks

Many thanks to our Premiere and Gold Sponsors:

OTG

PostgreSQL RPM sets for 9.0 Alpha4 released

2010-03-09T06:38:53+00:00

I released PostgreSQL RPM sets for 9.0 Alpha 4 yesterday. They are distributed through PostgreSQL FTP site & mirrors.

Also, they are available via the new PostgreSQL RPM repository: http://yum.pgrpms.org (more on this will come soon)

Please find more info about 9.0 Alpha 4 RPMs in here.

Countdown to pgCon

2010-03-08T20:49:21+00:00

As you may be aware, pgCon 2010, the PostgreSQL conference, is just a few months away. It is an important year for pgCon with the upcoming release of PostgreSQL 9.0. pgCon’s roots are in the PostgreSQL Anniversary Summit where hackers from all over the world flew in to celebrate PostgreSQL’s 10 years of open-source. pgCon has kept with the tradition of catering to the core of the PostgreSQL community including hackers and DBA’s alike.

As the largest PostgreSQL conference one can always expect to be exposed to new features, the latest in thoughts on PostgreSQL performance and scaling, not to mention the many talks aimed at all levels of PostgreSQL professionals and enthusiasts. To get ready for pgCon this year, I thought I might point out the talks not to miss.

One of the biggest talks will have to be Heikke Linnakangas introduction of the new built-in replication features in PostgreSQL, Hot Standby and Streaming Replication. If you’re not familiar with these features, they are truly game changing for Postgres. Building on the conceptual foundation of Warm-Standby and Point and Time Recovery log shipping, Hot Standby and Streaming Replication turn idle Warm Standby boxes in to active read-only slaves. Married with a HA solution, Hot Standby becomes an immediate failover solution with the ability to disconnect from the Hot Standby master server and turn on writes. In an effort to speed up the whole log-shipping paradigm, the streaming replication feature means no more external commands for copying log segments across the wire to Warm Standby servers. By combing these two features, one of the most sought-after features for PostgreSQL, native replication, becomes a reality. Heikke brings to this talk an intimate knowledge of the implementation of these two new features making this talk a must-see.

If you haven’t been to pgCon before and are a PostgreSQL user, DBA or enthusiast like myself, this is the year not to miss. As for the other significant talks for pgCon 2010, stay tuned as I will highlight my must-see list of talks in the coming weeks.

Actually, the Relational Model doesn't scale

2010-03-08T16:20:56+00:00

Before all my fellow DBAs' heads explode, let me just say that I am a relational guy. I like the relational model, think it's the best tool for the job, and think every programmer (not just DBA's) should aspire to be as familiar with it as they are with AJAX, MVC, or whatever other technology pattern you think is important. I'll even take that a step further; I think the NoSQL movement is mostly a re-hash of failed technologies from the last century. Object and document databases had their run in the market (some might say "they had their time"), and they were pretty thoroughly beaten by the RDBMS; that some people have reinvented that wheel doesn't change the game.

That said, I find the recent comments from Jeff Davis on the relational model and scalability to be overlooking some things. The state of computing tasks has changed over the past two decades, and what we know about computer engineering has also changed. Working on highly scalable systems like we do at OmniTI, you can't escape some of the inherent problems that you face when working in these types of environments. As much as I'd like the answer to every problem to be "just use an RDBMS", Brewer's CAP theorem just isn't something you can ignore.

When most people think about the relational model, they think of it in terms of parent-child relationships between tables. Without getting too deep in the details of it, I think it's pretty fair to say that Primary Keys and Foreign Keys are very large part of any relational implementation, and that pretty much all RDBMS strive to allow you to add these constraints to your model; it's what helps keep the data consistent. But there's the rub. CAP theorem points out that as we strive for tighter and tighter consistency, we are pulling away from availability, and sacrificing partition tolerance. Two theoretical systems that run smack dab into each other in the real world. This isn't really something new; if you have ever de-normalized, dropped a foreign key, or split data across multiple nodes, you've run into this before.

Now, where CAP theorem falls on it's face (imho) is that it also ignores another holy trinity of software development; Cheap, Fast, and Good. The size of your problem is dictated by the resources you have available; if you can afford decent tools (and let's be clear, decent is not your web dev throwing up MySQL on an EC2 instance) it is quite likely that the stressors of the relational model will never impact you in a way that most CAP folks are worried about. This is also one of the places the NoSQL movement fails; by throwing the baby out with the bath water. Giving up your data integrity before you have scalability issues is a form of premature optimization. The trick, as Theo would say, is having the experience to know when such optimizations are and aren't premature.

So what's the take away? I like to say that you use the relational model because it is best, and you use something else because it is necessary. Most SQL implementations can scale very well, and they should be your first choice when starting a new project. But we also can't pretend that there aren't inherent problems as these systems grow larger; let's understand the trade-offs and engineer appropriately.

PostgreSQL tip: arbitrary serialized rows

2010-03-08T12:05:17+00:00

Sometimes when using PostgreSQL, you want to deal with a record in its serialized form. If you're dealing with a specific table, you can accomplish this using the table name itself:

psql # CREATE TABLE foo (bar text, baz int);
CREATE TABLE

psql # INSERT INTO foo VALUES ('test 1', 1), ('test 2', 2);
INSERT 0 2

psql # SELECT foo FROM foo;
     foo      
--------------
 ("test 1",1)
 ("test 2",2)
(2 rows)

This works fine for defined tables, but how to go about this for arbitrary SELECTs? The answer is simple: wrap in a subselect and alias as so:

psql # SELECT q FROM (SELECT 1, 2) q;
   q   
-------
 (1,2)
(1 row)

Scalability and the Relational Model

2010-03-07T21:37:24+00:00

The relational model is just a way to represent reality. It happens to have some very useful properties, such as closure over many useful operations — but it’s a purely logical model of reality. You can implement relational operations using hash joins, MapReduce, or pen and paper.

So, right away, it’s meaningless to talk about the scalability of the relational model. Given a particular question, it might be difficult to break it down into bite-sized pieces and distribute it to N worker nodes. But going with MapReduce doesn’t solve that scalability problem — it just means that you will have a hard time defining a useful map or reduce operation, or you will have to change the question into something easier to answer.

There may exist scalability problems in:

SQL, which defines requirements outside the scope of the relational model, such as ACID properties and transactional semantics.
Traditional architectures and implementations of SQL, such as the “table is a file” equivalence, lack of sophisticated types, etc.
Particular implementations of SQL — e.g. “MySQL can’t do it, so the relational model doesn’t scale”.

Why are these distinctions important? As with many debates, terminology confusion is at the core, and prevents us from dealing with the problems directly. If SQL is defined in a way that causes scalability problems, we need to identify precisely those requirements that cause a problem, so that we can proceed forward without losing all that has been gained. If the traditional architectures are not suitable for some important use-cases, they need to be adapted. If some particular implementations are not suitable, developers need to switch or demand that it be made competitive.

The NoSQL movement (or at least the hype surrounding it) is far too disorganized to make real progress. Usually, incremental progress is best; and sometimes a fresh start is best, after drawing on years of lessons learned. But it’s never a good idea to start over with complete disregard for the past. For instance, an article from Digg starts off great:

The fundamental problem is endemic to the relational database mindset, which places the burden of computation on reads rather than writes.

That’s good because he blames it on the mindset not the model, and then identifies a specific problem. But then the article completely falls flat:

Computing the intersection with a JOIN is much too slow in MySQL, so we have to do it in PHP.

A join is faster in PHP than MySQL? Why bother even discussing SQL versus NoSQL if your particular implementation of SQL — MySQL — can’t even do a hash join, the exact operation that you need? Particularly when almost every other implementation can (including PostgreSQL)? That kind of reasoning won’t lead to solutions.

So, where do we go from here?

Separate the SQL model from the other requirements (some of which may limit scalability) when discussing improvements.
Improve the SQL model (my readers know that I’ve criticized SQL’s logical problems many times in the past).
Improve the implementations of SQL, particularly how tables are physically stored.
If you’re particularly ambitious, come up with a relational alternative to SQL that takes into account what’s been learned after decades of SQL and can become the next general-purpose DBMS language.

EDIT 2010-03-09: I should have cited Josh Berkus’s talk on Relational vs. Non-Relational (complete list of PGX talks), which was part of the inspiration for this post.

ERROR: invalid byte sequence for encoding

2010-03-07T19:29:25+00:00

One common problem that a lot of people seem to have is when they encounter error message like this: # \i test.sql psql:test.sql:1: ERROR: invalid byte sequence for encoding "UTF8": 0xb3 Why it happens? What can be done about it? Let’s see. First, we need some theory, so you will understand background. As you perhaps know, computers generally use [...]

PostgreSQL Weekly News March 7th 2010

2010-03-07T08:00:00+00:00

Bug fix updates for PostgreSQL 7.4 through 8.4 will be out soon. Get ready to upgrade!

9.0 Status Report

2010-03-06T17:00:00+00:00

Many people are waiting for Postgres 9.0, so here is a status report about the release. Two weeks ago we completed the last 9.0 CommitFest and released 9.0 Alpha 4. We are now working on a 9.0 open items list. There are relatively few open items compared to previous major releases because there are now more developers focused on closing open items.

Once we deal with all the items on that list, and any new items that appear, we can release the first 9.0 beta. Technically, we could release a beta before dealing with all open items as long as we are sure that none of the existing open items will require cluster reinitialization (initdb).

What is New in PostGIS Land

2010-03-05T07:09:00+00:00

This month we we will be giving two mini-tutorials at PgCon East 2010 on Saturday, March 27th. The topic of the talks will be, you guessed it, PostGIS. We have changed our Beyond talk to PostGIS: Adding spatial support to PostgreSQL to a beginner focus instead of an intermediate focus. Topic content will be more or less the same but focused more on people new to spatial database analysis. Our web applications talk will cater more to the web developer trying to integrate PostGIS in their web applications.

Marcus Rouhani of the Federal Aviation Administation will also be talking about the Airport GIS project and migration from Oracle to PostgreSQL.

On a somewhat related note, we also hope to be finished with all the chapters of our upcoming book this month. We just completed the first draft of our Chapter 10: PostgreSQL Add-ons and ancillary tools. After some back and forth with our editor, this will be up on MEAP, available for read and comments for early book buyers. Still two more chapters to finish after that before we get to the polishing of the text, images, layout and final print version.

Our publisher Manning is running a 50% off sale this Friday (tomorrow or is it today) on any MEAP book and they have a lot of interesting ones in the pipeline (including ours).

Waiting for PostGIS 2.0

The OSGEO just completed a recent coding sprint in New York. The New York sprint was a meeting of the minds of OSGEO people from various projects -- PostGIS, Mapserver, Geoserver, OpenLayers, GDAL, and some others were represented. Sadly we were not able to attend this one. A summary of the sprint with a PostGIS bent can be found on Olivier Courtin's New York sprint summary (Original French Version) and Olivier Courtin's New York sprint summary (Google English translation) and Paul's New York sprint summary.

Continue reading "What is New in PostGIS Land"

In Defense of varchar(x)

2010-03-05T00:23:00+00:00

This is a rebuttal to depesz's charx, varcharx, varchar, and text and David Fetter's varchar(n) considered harmful. I respect both depesz and David and in fact enjoy reading their blogs. We just have deferring opinions on the topic.

For starters, I am pretty tired of the following sentiments from some PostgreSQL people:

99% of the people who choose varchar(x) over text in PostgreSQL in most cases are just ignorant folk and don't realize that text is just as fast if not faster than varchar in PostgreSQL.
stuff your most despised database here compatibility is not high on my priority list.
It is unfortunate you have to work with the crappy tools you work with that can't see the beauty in PostgreSQL text implementation. Just get something better that treats PostgreSQL as the superior creature it is.

Continue reading "In Defense of varchar(x)"

XSLT working properly in Postgres at last

2010-03-04T19:21:48+00:00

Productive as ever, Tom Lane seems to have taken a break from making his head hurt on replication issues, and instead fixed some long-standing problems we've had with the xml2 contrib module's memory management. My small contribution was to get it building correctly on Windows.

This means (among other things) that we can now use xslt_process to transform XML in various useful ways. Examples could include generating HTML (or XHTML) from it, or preparing XSL-FO output as a step towards turning it into PDFs, or just pretty printing it. I've just spent a little time verifying that things that were previously broken now work.

Here for example is a simple XML pretty printer function (it can probably be improved on, but this will do for a demo):

create or replace function xml_pretty_print(input xml) 
    returns xml
    language sql as
$func$

select xslt_process($1::text,
$$<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
    version="1.0">
   <xsl:output method="xml" omit-xml-declaration="yes" indent="yes"/>
   <xsl:strip-space elements="*"/>
   <xsl:template match="@*|node()">
     <xsl:copy>
       <xsl:apply-templates select="@*|node()"/>
     </xsl:copy>
   </xsl:template>
 </xsl:stylesheet>
$$::text)::xml

$func$;

I have recently found myself doing a lot of XML for one reason or another, and the fact that Postgres just keeps getting better at it is good news.

PGCon 2010 - schedule

2010-03-04T14:17:49+00:00

The PGCon 2010 schedule has been announced. Have a read of our outstanding lineup, see where people come from to PGCon, and get your registration started.

Git + bash = win

2010-03-03T18:38:26+00:00

Modern git comes with a great way to improve your life: git-completion.bash. To use it, just add the following to your .bashrc:

if [ -f /etc/bash_completion ]; then

    . /etc/bash_completion

    export PS1='\h:\u:\w $(__git_ps1 "(%s) ")'

fi

Once you've done this, any time you're in a git subdirectory, the branch name will appear in your PS1.
Any time you're not, it won't.
Simple!

5 Steps to PostgreSQL Performance

2010-03-03T17:04:06+00:00

As we inch closer to PostgreSQL Conference East, I have been reviewing some of the great talks given at PostgreSQL Conference West. One of those talks was Josh Berkus's great, "5 Steps to PostgreSQL Performance". Here it is below:

If you would like to see more great talks on PostgreSQL, make sure and visit the website... and perhaps even attend PostgreSQL Conference East.

mongodb 1.3.3 (devel) with mongostat

2010-03-02T19:07:22+00:00

MongoDB 1.3.3 (devel) was released today. My tool called mongostat has been incorporated into mongo1.3.3 and changed from python to C++. So further development will come inside the mongodb distribution itself. Also of note is new slave lag for replication. Changelog here.

OmniTI is heading to PGEast 2010

2010-03-02T14:09:00+00:00

PGEast is the premiere Postgres conference held inside the U.S. each year, and this years conference, in Philadelphia, is now less than a month away. The organization and formatting have evolved a little from previous years, but one things still continues; a very strong presentation line up. We at OmniTI are very happy to be among that group of people, with four talks in this years conference lineup;

Know More Waiting, A Guide To PostgreSQL 9.0 by Robert Treat (hey, that's me), will give an overview of the upcoming PostgreSQL 9.0 release. While we're still a few months ahead of release, but we have a pretty good idea of what's coming in the next release, and this talk will help you start planning for how you will be able to take advantage of the new features coming our way.

PostgreSQL, meet AMQP, by Theo Schlossnagle, looks at pg_amqp, a "contrib" style module for Postgres that provides transaction style message queuing from inside of Postgres, using the AMQP standard.

Yet Another Replication Tool : RubyRep by Denish Patel, will delve into one of the newer Postgres replication solutions on the block. RubyRep is design for dead simple installation and setup, while still delivering advanced features like data comparing, synchronization between servers, and even master-master replication options.

Database Scalability Patterns by Robert Treat (me again), takes a look at the common patterns around scaling your database solution, and looks at some of the different options available to people scaling with Postgres.

But wait, there's more! While we at OmniTI are definitely excited to be participating in PGEast this year, there are a number of other good talks and speakers, including Magnus Hagander, Jeff Davis, Baron Schwartz, and many others. For full talk details, check out the conference talks page; I hope you'll journey out and say hi, it should be a pretty good time.

CHAR(x) vs. VARCHAR(x) vs. VARCHAR vs. TEXT – UPDATED 2010-03-03

2010-03-02T11:33:56+00:00

Fight! But more seriously – people tend to use various data types, and there have been some myths about them, so let’s see how it really boils down. First of all – All those data types are internally saved using the same C data structure – varlena. Thanks to this we can be nearly sure that there are [...]

Scripting psql for Fun and Profit

2010-03-01T21:09:22+00:00

Profit?

Yes!
Continue reading "Scripting psql for Fun and Profit"

On switching away from Core Data

2010-03-01T02:46:02+00:00

Brent Simmons has a very good piece about switching away from using Core Data to using SQLite directly in his iPhone app. Substituting “any common ORM” for “Core Data” (which, after all, is all Core Data is) and “any SQL database” for SQLite, he encounters the most common problems that plague those trying to develop scalable solutions on top of ORMs.

Some light hacking on selectivity functions, part 2

2010-02-28T14:17:58+00:00

So, I was getting bad estimates resulting from using a rule-of-thumb selectivity function designed for a different datatype and intended to force the use of an index regardless of the actual data. Query problem identified. But could I solve it?

PostgreSQL Weekly News February 28th 2010

2010-02-28T08:00:00+00:00

PostgreSQL 9.0alpha4 is out. Test! More info and download information below. http://www.postgresql.org/developer/alpha http://www.postgresql.org/ftp/source/9.0alpha4

28 Feb 2010

2010-02-28T02:56:18+00:00

So there was an earthquake here and it seems some people tried to get news from me. I can't reach my mail currently, so pardon me while I abuse this blog for a bit. Fortunately my family is all alive and well; no damage taken. The building we live on is still standing and has no damages either, though our appartment is now a mess of broken dishes and bottles.

I will be obviously unable to work much on patches, or anything really, for at least a week ...

Some light hacking on selectivity functions

2010-02-27T15:34:30+00:00

This is one of the things for which I love PostgreSQL. With Postgres, you always have the option of getting your hands dirty with some C code and fixing the problem yourself. And it's probably easier than you'd think.

More Philadelphia Action

2010-02-26T17:00:00+00:00

A month ago I mentioned a surprising number of Postgres activities in Philadelphia. One month later, those events are past but now there are more.

First, PG East is shaping up to be a big conference:

The conference agenda looks very full, with both developer and business-oriented talks. The business-oriented focus is rather new for Postgres conferences, and I think signals more mainstream adoption of Postgres.
It is at the Warwick Hotel. This lobby photo should give you a good idea of how nice the hotel is — we will simply have to adjust to having a Postgres conference in a fancy hotel.
Noel Yuhanna of Forrester Research will be speaking. He is the person who authored last year's Forrester Wave report that found Ingres and MySQL as the leading open source databases (news report). You can judge for yourself how much "research" went into that report, but we will be nice to him — a bodyguard will be unnecessary.

PostgreSQL 9.0 Alpha4

2010-02-26T13:25:53+00:00

Time to take PostgreSQL 9.0 Alpha4 for a spin! There's a bunch of new stuff to get to know about and try out.

Installing PostgreSQL

2010-02-26T01:35:13+00:00

First of all – I base information in this post on Linux, so if you are using another operating system – it’s quite likely that it will not help you. Second – if you’re using Linux, you have probably some package manager – dpkg, rpm, pacman, poldek, whatever. Right? So, it should be easy to install [...]

...and Devrim flies to...

2010-02-25T16:17:09+00:00

After quitting my old job, I got a few offers. Two of them were really good ones, but I had to pick up one.
Continue reading "...and Devrim flies to..."

FOSS STHML #1

2010-02-25T13:05:00+00:00

Yesterday was the first meeting for the FOSS STHLM "group" - a (very) loose group of FOSS interested people in the Stockholm region. We met in a lecture hall at the Stockholm University in Kista north of Stockholm, for a couple of hours of short presentations. The lineup was very nice: cool embedded stuff? yup, rockbox. General linux? Yup, upstart. Debian-specific? Of course. OpenSource Sweden? yeah. Curl? What else did you expect with Daniel as one of the organizers? PostgreSQL? Yeah, why else would I be blogging this?

Our allocated time was short, so the setup was many short talks. It actually worked a lot better than I thought it would, but it's still very hard to convey something useful in just 20 minutes.

I had a hard time figuring out what I should focus on, so I did a split into two parts (perfect strategy - if you have too little time to do one thing good, do two things in the same time instead...). I started with a section about "things to think about if you're switching to PostgreSQL or trying it out for the first time" - things like the very most basic config parameters that you always have to touch. And of course the classic - "ident authentication failed" issue that hits everybody on RedHat or Debian platforms at least (which is most of the users - definitely in this crowd). It's hard to do much in 12 minutes, hopefully it got some people interested.

I followed that with a very very very short version of "what's coming up in 9.0". Once again, I focused on one of my personal favorite features, which is Exclusion Constraints. While this is often listed as one of the cool things in 9.0, sometimes I feel that too much focus is on streaming replication and hot standby. Don't get me wrong, these are very good and very much needed features. But Exclusion Constraints is a real eye-opener. All databases (including PostgreSQL, of course) have replication - this is "just another way to do it". Yes, a very important and good way to do it, but it's still not something brand new. Exclusion Constraints is something that's fundamentally new. And it's a brilliant example of how PostgreSQL is moving the goalposts forward. Oh, and it's really useful and cool, of course! (and it'll be even better when we have the period datatype, or something similar, in 9.1!)

In summary, I think it was a great event. Big thanks to all those who helped make it happen! Hopefully we can follow it up with many more in similar ways - perhaps mixing these many-short-talks with some more focused discussions on specific areas or technologies. Time will tell...

*Update*: For those who asked, there were somewhere around 150 people in the sessions.

Trade-offs in Hot Standby Deployments

2010-02-24T18:46:44+00:00

The new Hot Standby feature in the upcoming PostgreSQL 9.0 allows running queries against standby nodes that previously did nothing but execute a recovery process. Two common expectations I've heard from users anticipating this feature is that it will allow either distributing short queries across both nodes, or allow running long reports against the standby without using resources on the master. These are both possible to do right now, but unless you understand the trade-offs involved in how Hot Standby works there can be some unanticipated behavior here.

Standard Long-running Queries

One of the traditional problems in a database using MVCC, like PostgreSQL, is that a long-running query has to keep open a resource--referred to as a snapshot in the current Postgres implementation--to prevent the database from removing data the query needs to operate. For example, just because another client has deleted a row and committed, if an already running query needs that row to complete you can't actually wipe the physical disk blocks related to that row out just yet. You have to wait until no open queries that expect that row to be visible are still around.

Hot Standby Limitations

If you have a long-running query you want Hot Standby to execute, there are a couple of types of bad things that can happen when the recovery process is applying updates. These are described in detail in the Hot Standby Documentation. Some of these bad things will cause queries running on the standby to be canceled for reasons that might not be intuitively obvious:

A HOT update or VACUUM related update arrives to delete something that query expects to be visible
A B-tree deletion appears
There is a locking issue between the query you're running and what locks are required for the update to be processed.

The lock situation is difficult to deal with, but not very likely to happen in practice for all that long if you're just running read-only queries on the standby, because those will be isolated via MVCC. The other two are not hard to run into. The basic thing to understand is that any UPDATE or DELETE on the master can lead to interrupting any query on the standby; doesn't matter if the changes even relate to what the query is doing.

Good, fast, cheap: pick two

Essentially, there are three things people might want to prioritize:

Avoid master limiting: Allow xids and associated snapshots to advance unbounded on the master, so that VACUUM and similar cleanup isn't held back by what the standby is doing
Unlimited queries: Run queries on the slave for any arbitrary period of time
Current recovery: Keep the recovery process on the standby up to date with what's happening on the master, allowing fast fail-over for HA

In any situation with Hot Standby, it's literally impossible to have all three at once. You can only pick your trade-off. The tunable parameters available already let you optimize a couple of ways:

Disabling all these delay/defer settings optimizes for always current recovery, but then you'll discover queries are more likely to be canceled than you might expect.
max_standby_delay optimizes for longer queries, at the expense of keeping recovery current. This delays applying updates to the standby once one that will cause a problem (HOT, VACUUM, B-tree delete, etc.) appears.
vacuum_defer_cleanup_age and some snapshot hacks can introduce some master limiting to improve on the other two issues, but with a weak UI to do that. vacuum_defer_cleanup_age is in units of transaction IDs. You need to have some idea the average amount of xid churn on your system per unit of time to turn the way people think about this problem ("defer by at least 1 hour so my reports will run") into a setting for this value. xid consumption rate just isn't a common or even reasonable thing to measure/predict. Alternately, you can open a snapshot on the primary before starting a long-running query on the standby. dblink is suggested in the Hot Standby documentation as a way to accomplish that. Theoretically a daemon on the standby could be written in user-land, living on the primary, to work around this problem too (Simon has a basic design for one). Basically, you start a series of processes that each acquire a snapshot and then sleep for a period before releasing it. By spacing out how long they each slept for you could ensure xid snapshots never advanced forward too quickly on the master. It should already sound obvious how much of a terrible hack this would be.

Potential Improvements

The only one of these you can really do something about cleanly is tightening up and improving the UI for the master limiting. That turns this into the traditional problem already present in the database: a long-running query holds open a snapshot (or at least limits the advance of visibility related transaction IDs) on the master, preventing the master from removing things needed for that query to complete. You might alternately think of this as an auto-tuning vacuum_defer_cleanup_age.

The question is how to make the primary respect the needs of long running queries on the standby. This might be possible if more information about the transaction visibility requirements of the standby were shared with the master. Doing that sort of exchange would really be something more appropriate for the new Streaming Replication implementation to share. The way a simple Hot Standby server is provisioned does not provide any feedback toward the master suitable for this data to be exchanged, besides approaches like the already mentioned dblink hack.

With PostgreSQL 9.0 just reaching a fourth alpha release, there may still be time to see some improvements in this area yet before the 9.0 release. It would be nice to see Hot Standby and Streaming Replication really integrated together in a way that accomplishes things that neither is fully capable of doing on their own before coding on this release completely freezes.

PgEast Talks posted

2010-02-23T18:03:21+00:00

I blogged last week about a small list of content being up for the conference. I am now pleased to announce that 99% of the topics are up. I would say 100% but I keep getting new talks that look good and am trying to accommodate them as the schedule allows. When I review the talks, I see a lot of good content. Of particular interest to me is the talk from the FAA (yes that FAA) as well as Kevin Grittner's talk on Transaction Isolation. I am also keen on seeing Chander's training class on HS/SR/PITR but unfortunately I will also be teaching on Sunday. I won't be giving the keynote this year. Instead Ed Boyajian, President and CEO of EnterpriseDB will be. This is probably a good choice considering my Keynote is always somewhat of an Un-Keynote. I am curious to see where Ed thinks things are moving and how quickly. I will be doing the Conference launch and closing session. This year we will continue the trend of having a raffle at the end of the conference. If you haven't registered yet, now is the time.

PostgreSQL EC2/EBS/RAID 0 snapshot backup

2010-02-23T15:58:55+00:00

One of our clients uses Amazon Web Services to host their production application and database servers on EC2 with EBS (Elastic Block Store) storage volumes. Their main database is PostgreSQL.

A big benefit of Amazon's cloud services is that you can easily add and remove virtual server instances, storage space, etc. and pay as you go. One known problem with Amazon's EBS storage is that it is much more I/O limited than, say, a nice SAN.

To partially mitigate the I/O limitations, they're using 4 EBS volumes to back a Linux software RAID 0 block device. On top of that is the xfs filesystem. This gives roughly 4x the I/O throughput and has been effective so far.

They ship WAL files to a secondary server that serves as warm standby in case the primary server fails. That's working fine.

They also do nightly backups using pg_dumpall on the master so that there's a separate portable (SQL) backup not dependent on the server architecture. The problem that led to this article is that extra I/O caused by pg_dumpall pushes the system beyond its I/O limits. It adds both reads (from the PostgreSQL database) and writes (to the SQL output file).

There are several solutions we are considering so that we can keep both binary backups of the database and SQL backups, since both types are valuable. In this article I'm not discussing all the options or trying to decide which is best in this case. Instead, I want to consider just one of the tried and true methods of backing up the binary database files on another host to offload the I/O:

Create an atomic snapshot of the block devices
Spin up another virtual server
Mount the backup volume
Start Postgres and allow it to recover from the apparent "crash" the server had (since there wasn't a clean shutdown of the database before the snapshot
Do whatever pg_dump or other backups are desired
Make throwaway copies of the snapshot for QA or other testing

The benefit of such snapshots is that you get an exact backup of the database, with whatever table bloat, indexes, statistics, etc. exactly as they are in production. That's a big difference from a freshly created database and import from pg_dump.

The difference here is that we're using 4 EBS volumes with RAID 0 striped across them, and there isn't currently a way to do an atomic snapshot of all 4 volumes at the same time. So it's no longer "atomic" and who knows what state the filesystem metadata and the file data itself would be in?

Well, why not try it anyway? Filesystem metadata doesn't change that often, especially in the controlled environment of a Postgres data volume. Snapshotting within a relatively short timeframe would be pretty close to atomic, and probably look to the software (operating system and database) like some kind of strange crash since some EBS volumes would have slightly newer writes than others. But aren't all crashes a little unpredictable? Why shouldn't the software be able to deal with that? Especially if we have Postgres make a checkpoint right before we snapshot.

I wanted to know if it was crazy or not, so I tried it on a new set of services in a separate AWS account. Here are the notes and some details of what I did:

Created one EC2 image:
Amazon EC2 Debian 5.0 lenny AMI built by Eric Hammond
Debian AMI ID ami-4ffe1926 (x86_64)
Instance Type: High-CPU Extra Large (c1.xlarge) - 7 GB RAM, 8 CPU cores
Created 4 x 10 GB EBS volumes
Attached volumes to the image

Created software RAID 0 device:

mdadm -C /dev/md0 -n 4 -l 0 -z max /dev/sdf /dev/sdg /dev/sdh /dev/sdi

Created XFS filesystem on top of RAID 0 device:
```
mkfs -t xfs -L /pgdata /dev/md0
```

Set up in /etc/fstab and mounted:

mkdir /pgdata
# edit /etc/fstab, with noatime
mount /pgdata

Installed PostgreSQL 8.3
Configured postgresql.conf to be similar to primary production database server
Created empty new database cluster with data directory in /pgdata
Started Postgres and imported a play database (from public domain census name data and Project Gutenberg texts), resulting in about 820 MB in data directory
Ran some bulk inserts to grow database to around 5 GB
Rebooted EC2 instance to confirm everything came back up correctly on its own
Set up two concurrent data-insertion processes:

50 million row insert based on another local table (INSERT INTO ... SELECT ...), in a single transaction (hits disk hard, but nothing should be visible in the snapshot because the transaction won't have committed before the snapshot is taken)
Repeated single inserts in autocommit mode (Python script writing INSERT statements using random data from /usr/share/dict/words piped into psql), to verify that new inserts made it into the snapshot, and no partial row garbage leaked through

Started those "beater" jobs, which mostly consumed 2-3 CPU cores
Manually inserted a known test row and created a known view that should appear in the snapshot
Started Postgres's backup mode that allows for copying binary data files in a non-atomic manner, which also does a CHECKPOINT and thus also a filesystem sync:
```
SELECT pg_start_backup('raid_backup');
```
Manually inserted a 2nd known test row & 2nd known test view that I don't want to appear in the snapshot after recovery
Ran snapshot script which calls ec2-create-snapshot on each of the 4 EBS volumes -- during first run, run serially quite slowly taking about 1 minute total; during second run, run in parallel such that the snapshot point was within 1 second for all 4 volumes
Tell Postgres the backup's over:
```
SELECT pg_stop_backup();
```
Ran script to create new EBS volumes derived from the 4 snapshots (which aren't directly usable and always go into S3), using ec2-create-volume --snapshot
Run script to attach new EBS volumes to devices on the new EC2 instance using ec2-attach-volume
Then, on the new EC2 instance for doing backups:

mdadm --assemble --scan
mount /pgdata
Start Postgres
Count rows on the 2 volatile tables; confirm that the table with the in-process transaction doesn't show any new rows, and that the table getting individual rows committed to reads correctly
VACUUM VERBOSE -- and confirm no errors or inconsistencies detected
pg_dumpall # confirmed no errors and data looks sound

It worked! No errors or problems, and pretty straightforward to do.

Actually before doing all the above I first did a simpler trial run with no active database writes happening, and didn't make any attempt for the 4 EBS snapshots to happen simultaneously. They were actually spread out over almost a minute, and it worked fine. With the confidence that the whole thing wasn't a fool's errand, I then put together the scripts to do lots of writes during the snapshot and made the snapshots run in parallel so they'd be close to atomic.

There are lots of caveats to note here:

This is an experiment in progress, not a how-to for the general public.
The data set that was snapshotted was fairly small.
Two successful runs, even with no failures, is not a very big sample set. :)
I didn't use Postgres's point-in-time recovery (PITR) here at all -- I just started up the database and let Postgres recover from an apparent crash. Shipping over the few WAL logs from the master collected during the pg_backup run after the snapshot copying is complete would allow a theoretically fully reliable recovery to be made, not just a practically non-failing recovery as I did above.

So there's more work to be done to prove this technique viable in production for a mission-critical database, but it's a promising start worth further investigation. It shows that there is a way to back up a database across multiple EBS volumes without adding noticeably to its I/O load by utilizing the Amazon EBS data store's snapshotting and letting a separate EC2 server offload the I/O of backups or anything else we want to do with the data.

Getting out of SQL_ASCII, part 2

2010-02-23T15:30:00+00:00

So, if you followed the previous blog entry, now you have a new database containing all the static tables encoded in UTF-8 rather than SQL_ASCII. Because if it was not yet the case, you now severely distrust this non-encoding.

Now is the time to have a look at properly encoding the live data, those stored in tables that continue to receive write traffic. The idea is to use the UPDATE facilities of PostgreSQL to tweak the data, and too fix the applications so as not to continue inserting badly encoded strings in there.

Finding non UTF-8 data

First you want to find out the badly encoded data. You can do that with this helper function that RhodiumToad gave me on IRC. I had a version from the archives before that, but the regexp was hard to maintain and quote into a PL function. This is avoided by two means, first one is to have a separate pure SQL function for the regexp checking (so that you can index it should you need to) and the other one is to apply the regexp to hex encoded data. Here we go:

create or replace function public.utf8hex_valid(str text)
 returns boolean
 language sql immutable
as $f$
   select $1 ~ $r$(?x)
                  ^(?:(?:[0-7][0-9a-f])
                     |(?:(?:c[2-9a-f]|d[0-9a-f])
                        |e0[ab][0-9a-f]
                        |ed[89][0-9a-f]
                        |(?:(?:e[1-9abcef])
                           |f0[9ab][0-9a-f]
                           |f[1-3][89ab][0-9a-f]
                           |f48[0-9a-f]
                          )[89ab][0-9a-f]
                       )[89ab][0-9a-f]
                    )*$
                $r$;
$f$;

Now some little scripting around it in order to skip intense manual and boring work (and see, some more catalog queries). Don't forget we will have to work on a per-column basis here...

create or replace function public.check_encoding_utf8
 (
   IN schemaname text,
   IN tablename  text,
  OUT relname    text,
  OUT attname    text,
  OUT count      bigint
 )
 returns setof record
 language plpgsql
as $f$
DECLARE
  v_sql text;
BEGIN
  FOR relname, attname
   IN SELECT c.relname, a.attname
        FROM pg_attribute a
             JOIN pg_class c on a.attrelid = c.oid
             JOIN pg_namespace s on s.oid = c.relnamespace
             JOIN pg_roles r on r.oid = c.relowner
       WHERE s.nspname = schemaname
         AND atttypid IN (25, 1043) -- text, varchar
         AND relkind = 'r'          -- ordinary table
         AND r.rolname = 'some_specific_role'
         AND CASE WHEN tablename IS NOT NULL
                  THEN c.relname ~ tablename
                  ELSE true
              END
  LOOP
    v_sql := 'SELECT count(*) '
          || '  FROM ONLY '|| schemaname || '.' || relname
          || ' WHERE NOT public.utf8hex_valid(encode(textsend('
          || attname
          || '), ''hex''))';

    -- RAISE NOTICE 'Checking: %.%', relname, attname;
    -- RAISE NOTICE 'SQL: %', v_sql;
    EXECUTE v_sql INTO count;
    RETURN NEXT;
  END LOOP;
END;
$f$;

Note that the tablename is compared using the ~ operator, so that's regexp matching there too. Also note that I wanted only to check those tables that are owned by a specific role, your case may vary.

The way I used this function was like this:

create table leon.check_utf8 as
 select *
   from public.check_encoding_utf8();

Then you need to take action on those lines in leon.check_utf8 table which have a count > 0. Rince and repeat, but you may soon realise building the table over and over again is costly.

Cleaning up the data

Up for some more helper tools? Unless you really want to manually fix this huge amount of columns where some data ain't UTF-8 compatible... here's some more:

create or replace function leon.nettoyeur
 (
  IN  action      text,
  IN  encoding    text,
  IN  tablename   text,
  IN  columname   text,

  OUT orig        text,
  OUT utf8        text
 )
 returns setof record
 language plpgsql
as $f$
DECLARE
  p_convert text;
BEGIN
  IF encoding IS NULL
  THEN
    p_convert := 'translate('
              || columname || ', '
              || $$'\211\203\202'$$
              || ', '
              || $$'   '$$
              || ') ';
  ELSE
    -- in 8.2, write convert using, in 8.3, the other expression
    -- p_convert := 'convert(' || columname || ' using ' || conversion || ') ';
    p_convert := 'convert(textsend(' || columname || '), '''|| encoding ||''', ''utf-8'' ) ';
  END IF;

  IF action = 'select'
  THEN
    FOR orig, utf8
     IN EXECUTE 'SELECT ' || columname || ', '
         || p_convert
         || '  FROM ONLY ' || tablename
         || ' WHERE not public.utf8hex_valid('
         || 'encode(textsend('|| columname ||'), ''hex''))'
    LOOP
      RETURN NEXT;
    END LOOP;

  ELSIF action = 'update'
  THEN
    EXECUTE 'UPDATE ONLY ' || tablename
         || ' SET ' || columname || ' = ' || p_convert
         || ' WHERE not public.utf8hex_valid('
         || 'encode(textsend('|| columname ||'), ''hex''))';

    FOR orig, utf8
     IN SELECT *
          FROM leon.nettoyeur('select', encoding, tablename, columname)
    LOOP
      RETURN NEXT;
    END LOOP;

  ELSE
    RAISE EXCEPTION 'Léon, Nettoyeur, veut de l''action.';

  END IF;
END;
$f$;

As you can see, this function allows to check the conversion process from a given supposed encoding before to actually convert the data in place. This is very useful as even when you're pretty sure the non-utf8 data is latin1, sometime you find it's windows-1252 or such. So double check before telling leon.nettoyeur() to update your precious data!

Also, there's a facility to use translate() when none of the encoding match your expectations. This is a skeleton just replacing invalid characters with a space, tweak it at will!

Conclusion

Enjoy your clean database now, even if it still accepts new data that will probably not pass the checks, so we still have to be careful about that and re-clean every day until the migration is effective. Or maybe add a CHECK clause that will reject badly encoded data...

In fact here we're using Londiste to replicate the live data from the old to the new server, and that means the replication will break each time there's new data written in non-utf8, as the new server is running 8.4, which by design ain't very forgiving. Our plan is to clean-up as we go (remove table from the subscriber, fix it, add it again) and migrate as soon as possible!

Bonus points to those of you getting the convoluted reference :)

HAST - High Availability Storage

2010-02-23T02:39:22+00:00

HAST has just been added to FreeBSD: HAST allows to transparently store data on two physically separated machines connected over the TCP/IP network. Those two machines together will be called a cluster and each machine is one cluster node. HAST works in Primary-Secondary (Master-Backup, Master-Slave) configuration, which means that only one of the cluster nodes can [...]

BWPUG March 10th, Falls Church take two.

2010-02-23T02:02:39+00:00

Barring a repeat of last months snowmageddon / snowpocalypse, we're going to take another stab at heading down to Falls Church, Va, for the March BWPUG meeting. If you haven't felt like trucking out to Columbia, then please try to make this one. Depending on response we may be able to do this more often.

When: March 10th, 6:30PM.
Where: 3150 Fairview Park Dr, Falls Church, VA
Host: Noblis, Inc. and the
Noblis Innovation and Collaboration Center (NICC)

It's basically at 495 and 50 in Northern Virginia. We'll be discussing the upcoming PostgreSQL 9.0 release, including a preview of my upcoming talk at PG East.

If you're planning to attend, please RSVP to Stephen Frost via sfrost at noblis.org. The host/facility we'll be meeting at does require a government issued photo ID (eg: driver's license or passport).

Hope to see you there!

PostgreSQL Europe opens for general membership

2010-02-22T19:44:00+00:00

After numerous emails, meetings in foreign lands and changes to the statutes, PostgreSQL Europe finally opened it's doors to new members today, of which I became the very first. Well, OK, technically I'm the fifth, as the elected board made up the first four, but I was the first non-elected, fee paying member.

So 'what's the point?' I hear you cry, and 'why should I part with my hard-earned cash?'. Well, joining PostgreSQL Europe costs €10 for 2 years membership, which is a fraction of what some other groups charge. You won't get a postgresql.eu email address for your money, but you do get the chance to vote for PostgreSQL Europe board members of your choosing, at the annual elections. Those four board members (who will become five after the upcoming elections) decide what events and activities the not-for-profit organisation will be involved in, and how to spend the money - what events should be sponsored, what swag should be purchased and so on.

Membership is open to anyone resident in geographical Europe (including all of Turkey and Russia), so if you want to join and play a part in the future of PostgreSQL, you can sign up on the postgresql.eu website. Don't leave it too long though - you need to be a member for at least a month to be eligible to vote in future elections.

PostgreSQL Weekly News February 21st 2010

2010-02-21T08:00:00+00:00

The final commitfest for 9.0 is winding down. Thanks to all the reviewers, etc., who are participating.

Why Blog?

2010-02-20T17:00:00+00:00

With so many ways of electronically communicating in the Postgres community, why blog? I have thought about that question and want to list the various electronically communication methods and their advantages:

Email: Email is great for task-oriented communication. It reaches a large audience and allows for deliberate, consensus-oriented solutions.

Python Driver Confusion

2010-02-20T17:00:00+00:00

A few weeks ago I learned there was serious confusion over the client-side Python drivers available for Postgres. This chart shows many Python drivers, but the recommended libpq-based driver, Psycopg, was listed on the chart as GPL licensed. This would require programs that use the driver to also be GPL licensed. However, the license is not actually GPL but a complicated hybrid license.

Fortunately, the Psycopg author has agreed to relicense his Python driver under the LGPL in the next release, and there is hope that Postgres developers will get involved to help add any missing features to the driver.

Summary of feedback from FOSDEM devroom

2010-02-20T14:55:00+00:00

I forgot completely to close up the feedback system for our talks at FOSDEM and post the feedback to the speakers. My apologies to all speakers and others who were interested! Thankfully, Marc Balmer reminded me today and the individual ratings for different speakers have been sent off to them.

When it comes to the aggregated statistics, they look fairly similar to what we saw at PGDay. The number of speakers were much lower, and unfortunately so were the number of people giving feedback. But in the end, the results were fairly clear. I am especially happy with the level of knowledge in the speakers we attract. Let's start with the graphs:

Breaking down the details, I think this shows a great quality of our speakers:

Name	Speaker Quality	Votes	Deviation
Simon Riggs	4.8	6	0.4
Magnus Hagander	4.6	5	0.5
David Fetter	4.5	4	1
Heikki Linnakangas	4.3	9	0.9
Dave Page	4.2	10	1.0
Marc Balmer	3.4	5	1.7
Peter Eisentraut	3.3	7	1.0

And the level of their knowledge is even higher:

Name	Speaker Knowledge	Votes	Deviation
Simon Riggs	5	6	0
Magnus Hagander	5	5	0
Heikki Linnakangas	5	9	0
David Fetter	4.8	4	0.5
Dave Page	4.3	10	1.1
Peter Eisentraut	4.3	7	1.1
Marc Balmer	3.6	5	1.7

Even at the bottom of this list, the numbers are very good! Unfortunately, the fairly low number of votes give a high deviation at the bottom of the chart.

For next time, we'd really like to get more people to leave their feedback. Does anybody have a good idea on how to make that happen?

CeBIT 2010

2010-02-20T01:22:00+00:00

... is coming up soon. CeBIT (March 2th to March 6th) is one of the biggest shows for IT and PostgreSQL will have a booth in the Open Source Project Lounge, organized by Andreas Scherbaum. I will staff the booth with Andreas on friday, 2010-03-05. If you are interested, feel free to meet us in hall 2, maybe we see us on friday!

Removal of Personal Copyrights

2010-02-19T17:00:00+00:00

Postgres does not require copyright assignment from contributors because it is BSD licensed. Unlike the GPL, there is no enforcement component in the BSD license. When code and documentation are contributed to the project, the authors give the project the right to distribute their contributions under the BSD license. However, technically, authors also retain full rights to their contributions. Of course, there is little practical difference between retaining full rights and obtaining contributions under the BSD license.

Anyway, while all work submitted to Postgres is covered under the BSD license, the existence of individual copyright lines in our source code, e.g. © Joe Coder, makes company lawyers nervous, so we have a policy of avoiding such content. Unfortunately, this policy has been active only during the past five years and was not always universally enforced. For this reason, we occasionally have to ask authors for permission to remove their copyright lines, or in drastic cases, rewrite trivial code segments when the author cannot be contacted. We perform such cleanups periodically as the personal copyrights are brought to our attention, e.g. 2006, 2006, 2007, 2009, 2010.

Part(ition)ing Is Such Sweet Sorrow

2010-02-19T04:21:41+00:00

There are excellent references on partitioning tables that depend on one table, but what happens when you need to partition the referenced table?

Let's find out!
Continue reading "Part(ition)ing Is Such Sweet Sorrow"

Postgres University Course Material

2010-02-18T17:00:00+00:00

Leonard Shapiro of Portland State University has created some excellent Postgres university course material.

His first course, Introduction to Databases, teaches databases from a Postgres perspective. His higher-level course, Relational DMBS/Database Internals, covers many aspects of Postgres in greater detail than is covered in the Postgres manuals.

Where Did My Error Come From?

2010-02-18T17:00:00+00:00

I am not sure people are aware that you can find the source code location of any generated server log message. When log_error_verbosity is set to 'verbose', additional details are output that can help diagnose errors. Specifically, I have added this to the Postgres 9.0 documentation:

VERBOSE output includes the SQLSTATE error code and the source code file name, function name, and line number that generated the error.

Got ISO? Create USB boot image

2010-02-18T16:26:30+00:00

The liveusb-creator is a cross-platform tool for easily installing live operating systems on to USB flash drives. It runs on various operating systems and allows you to create a bootable device from an ISO image. This could be useful for advocacy purposes.

PostgreSQL - now with added Open Source!

2010-02-18T16:19:39+00:00

Some of you may be aware that some time ago, Redhat's legal team changed the description of the PostgreSQL licence (yes, that is correct where I live - please don't email me to tell me I can't spell) from BSD to MIT. After doing some poking around (it helps when your boss's boss is an ex-Redhat exec) it transpired that they considered that our licence looked a lot more like the MIT licence than BSD.

It's true that ours doesn't look like the traditional BSD 3 or 4 clause licences - some people had already spotted that, but as it came from Berkeley we always just considered it to be a variant on the simplified BSD licence. The long and short of it was that lawyers said "no they're not the same", and we said "well it sure ain't MIT" and much discussion ensued.

The solution it seemed, was to try to get our licence recognised and approved by the Open Source Initiative. I originally tried to do it as an official BSD variant, but that wasn't accepted. I then resubmitted it as 'The PostgreSQL License" (yes, I know :-p), which was rejected as it wasn't a generic licence (OSI are trying to ensure that all new licences are resuable). Finally, I submitted it again with placeholders replacing the project name and copyright holder info, and nearly two months later, it's approved!

So there you have it ladies and gents. "The world's most advanced database with freely available source code" now actually is, officially, The world's most advanced Open Source database"!

Speaking at PG East

2010-02-18T13:54:42+00:00

If you've been involved in the PostgreSQL project at all in the last few years, you'll know that Josh Drake has been organising the PG East and PG West conference series since 2007. When we met in Paris at PGDay.EU, JD suggested that I submit a talk for East in March.

Which I did.

Which got accepted.

So now I'm travelling to Philadelphia in March :-)

Since we spoke though, the crack marketing team here at EnterpriseDB contacted JD and between them hatched a plan to make East the most impressive PostgreSQL conference to date. It'll be held over four days, at the Radisson Plaza - Warwick Hotel in Philadelphia, from March 25th to 28th and will include content for C-level execs and decision makers, as well as the regular crowd of database geeks like me. Of course, there will also be the traditional EnterpriseDB Party on the Friday night - if you haven't been to one of those yet, now's the perfect time to see what you've been missing out on!

So if you've been toying with the idea of going to a PostgreSQL conference, this is the perfect time to give into the urge and join us in Philly for database talk, beer and cheese-steak sandwiches. Mmmmm, cheese-steak....

Getting out of SQL_ASCII, part 1

2010-02-18T09:37:00+00:00

It happens that you have to manage databases designed by your predecessor, and it even happens that the team used to not have a DBA. Those histerical raisins can lead to having a SQL_ASCII database. The horror!

What SQL_ASCII means, if you're not already familiar with the consequences of such a choice, is that all the text and varchar data that you put in the database is accepted as-is. No checks. At all. It's pretty nice when you're lazy enough to not dealing with strange errors in your application, but if you think that t's a smart move, please go read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky now. I said now, I'm waiting for you to get back here. Yes, I'll wait.

The problem of course is not being able to read the data you just stored, which is seldom the use case anywhere you use a database solution such as PostgreSQL.

Now, it happens too that it's high time to get off of SQL_ASCII, the infamous. In our case we're lucky enough in that the data are all in fact latin1 or about that, and this comes from the fact that all the applications connecting to the database are sharing some common code and setup. Then we have some tables that can be tagged archives and some other live. This blog post will only deal with the former category.

For those tables that are not receiving changes anymore, we will migrate them by using a simple but time hungry method: COPY OUT|recode|COPY IN. I've tried to use iconv for recoding our data, but it failed to do so in lots of cases, so I've switched to using the GNU recode tool, which works just fine.

The fact that it takes so much time doing the conversion is not really a problem here, as you can do it offline, while the applications are still using the SQL_ASCII database. So, here's the program's help:

recode.sh [-npdf0TI] [-U user ] -s schema [-m mintable] pattern
        -d    debug
        -n    dry run, only print table names and expected files
        -s    schema
        -m    mintable, to skip already processed once
        -U    connect to PostgreSQL as user
        -f    force table loading even when export files do exist
        -0    only (re)load tables with zero-sized copy files
        -T    Truncate the tables before COPYing recoded data
        -I    Temporarily drop the indexes of the table while COPYing
   pattern    ^table_name_, e.g.

The -I option is neat enough to create the indexes in parallel, but with no upper limit on the number of index creation launched. In our case it worked well, so I didn't have to bother.

Take a look at the recode.sh script, and don't hesitate editing it for your purpose. It's missing some obvious options to get useful in the large, such as the recode request which is currently hardcoded to l1..utf8. If there's any demand about it, I'll setup a GitHub project for the little script.

We'll get back to the subject of this entry in part 2, dealing with how to recode your data in the database itself, thanks to some insane regexp based queries and helper functions. And thanks to a great deal of IRC based helping, too.

Using PostgreSQL to Decouple Applications, or, OO meets SQL (Part I)

2010-02-18T09:36:15+00:00

In the object oriented world, people like design patterns, or, elements of reusable software. One such pattern is the "Observer" pattern, where a piece of software, the "Observer", registers with another piece of software, the "Model", to be notified when the internal state of the "Model" changes. The "Observer" can then query the "Model" for the data changed. This design pattern is often applied in the so-called Model-View-Controller programming paradigm, where a "View" observes a "Model" and a "Controller" changes the state of the "Model".

What works at the object level in an object oriented programming language can be applied to distributed PostgreSQL applications as well, allowing for proper decoupling of applications or application parts. The tools needed for this are PostgreSQL's trigger procedures and the asynchronous notification mechanism. In this first installment of a small series of blog entries, I will talk about the basics needed to implement a distributed decoupled software solution.
Continue reading "Using PostgreSQL to Decouple Applications, or, OO meets SQL (Part I)"

Content, Content, Content... Oh My! (PgEast)

2010-02-17T17:54:00+00:00

When we moved PostgreSQL Conference East (register here)from a three day to a four day conference, I was concerned about our ability to pull it off. Primarily our conferences have been centered about the who's who of PostgreSQL. A nice mix of known contributors and avid users. A lot of the users, we would already knew as they contribute on the lists.
The migration to four days caused a need to expand our base. We actively starting soliciting from decision makers, educators, users and community contributors. I also know that several of our sponsors have been doing the same. So far it has paid off, we have more registrations at this point, than we have had at any other point historically for one of the PostgreSQL Conferences. The change seems to have been a true blessing.
The influx of talks has been amazing. We don't have them all up yet but you can get a taste here. What I find truly great, is the amount of diversity in the talks. We have case studies from Vonage, in depth security talks from Magnus Hagander, Advanced talks on Transaction Isolation; even Core member Dave Page is crossing the pond to talk about the PostgreSQL Infrastructure. There seems no better time to hit the PostgreSQL Conference series than now. There is going to be content for everyone.
We have also kept up our promise to integrate tertiary communities into the conference with content on PostGIS, PHP, Python, Ruby and Grails.
I was truly skeptical of the conference change. My hat is off to Platinum Sponsor EnterpriseDB for convincing me it was a good idea.

Waiting for 9.0 – extended frames for window functions

2010-02-17T14:14:47+00:00

On 12th of February Tom Lane http://archives.postgresql.org/pgsql-committers/2010-02/msg00142.php patch by Hitoshi Harada: Log Message: ----------- Extend the set of frame options supported for window functions. This patch allows the frame to start from CURRENT ROW (in either RANGE or ROWS mode), and it also adds support for ROWS n PRECEDING and ROWS n FOLLOWING start and end points. (RANGE value PRECEDING/FOLLOWING [...]