“We’re going to store data the way it’s stored naturally in the brain.”
Friend and colleague Bruce Momjian already shared in his presentation “Will Postgres live forever?” (https://attendee.gotowebinar.com/register/8793191803818201345) an overview of reasons why companies adopt open source software (OSS) in general. This overview was based on a survey done by Black Duck Software in 2016 (https://www.slideshare.net/mobile/blackducksoftware/2016-future-of-open-source-survey-results). The survey ranks databases on the second place of technologies that companies are moving towards OSS in.
In this series of short blogpost I want to highlight some of the practical cases of Postgres adoption in the Enterprise from our day-to-day practice, and see which of the general factors of OSS adoption we see with Postgres .
IBM DB2 to Postgres
The first of the cases deals with a semi-government organization that currently has an application landscape based on IBM DB2, Microsoft SQL-Server and Oracle. Our primary engagement is to support their move of their application landscape from IBM DB2 to Postgres.
The initial driving force behind this decision was the staggering renewal-cost this organization was facing for their DB2 licenses. These licenses included IBM’s Deep Compression option (https://www.ibm.com/developerworks/data/library/techarticle/dm-1205db210compression/index.html). This option reduces disk-space by as much as 50%. More importantly it reduces database IOPS to meet the performance of the applications. Moving from DB2 on Z/OS to DB2 on Linux, the company faced a performance deficit which they initially resolved by using this Deep Compression option.
This option was also a grounds for worry. Obviously an expensive database option, that was known to bring real benefits to DB2 database performance, would outperform Postgres, right? Would we be able to come near DB2 database performance when running on Postgres?
Discussion with our own Postgres experts could not take away these concerns. We would have to test and validate.
The project had already kicked into gear when EnterpriseDB was engaged. Some of the software had already been migrated and converted to fit on EDB Postgres Advanced Server (EPAS). For much of the data and schema migration, the software by Spectral Core, named Full Convert (https://www.spectralcore.com/fullconvert/).
Part of the extensive test-program of the migration project was performance testing, which also included a simulated full pressure practice test with load-generators.
The surprise was then also quite absolute, when the final results of the performance tests were released. EPAS was capable to keep up quite nicely with the original DB2-setup, and even improved performance in some specific tests.
Performance tests concluded, June 28 2018
|Test Number||Postgres Runtime|
|% of original DB2 runtime|
|Number 1||87.2 %|
|Number 2||101.0 %|
|Number 3||109.4 %|
|Number 4||79.3 %|
|Number 5||70.0 %|
|Number 6||109.5 %|
|Number 7||106.4 %|
In analysis of this particular project, we can conclude that the reasons for this project to choose Postgres are:
- Primarily #5, Cost reduction. Database running cost on an annual basis can be reduced in access of 80%
- Secondly #2, Freedom from vendor lock-in. Postgres is a community open source solution having no single company controlling the solution.
I am curious to learn what details you would want to read here in order to help you assess your specific situation! Let me know in the comments.
The cool thing about zooming out… is that your world appears to get bigger.
Being personally now no longer bound to Oracle, having the opportunity to work with PostgreSQL, also gives the opportunity to go new places and explore new possibilities. One of the cooler things is: participating in Postgres conferences.
Where Oracle conferences, although having some deep technical aspects, tend to lean towards the business aspects of technology, especially today with Cloud first / Cloud only… PostgreSQL conferences tend to lean towards engineering. What things are we – that same Postgres community – building in and around Postgres. What do we think about these developments and how can we improve them.
Postgres tends to have anywhere from 5 to 10 different directions in which the product is being developed. Lots of people check, test, improve, criticize and comment on all these developments.
A significant difference, somehow.
The atmosphere at PostgreSQL conferences, though is also simply super cool. New people to meet, new ways to incorporate.
I had the opportunity to join and participate in the Swiss pgday (find the program here) in the beautiful town of Rapperswil, at the university of applied science (HSR). Together with my friend and colleague, Postgres founding core team member, Mr. Bruce Momjian, I joined the event.
The Swiss Postgres community booked a nice result with a 30% higher number of participants. In two tracks, over 12 talks were delivered by local and international speakers on many aspects of Postgres, from a more business perspective on Postgres to the new things that come with Postgres 11, and can now be tested by anyone who wants to!
With all these shifting panels, with this second wave of Open Source rolling, that is now happening… More intricate systems, like relational database management systems, are now being offered and adopted.
It makes sense to zoom out as the opportunities increase so rapidly and in ways never foreseen.
I challenge and invite you all; come on board and ride this wave with us.
One of the attractive things of my job is this… Just a bit more often than every now and then, you get the opportunity to get out and meet people to talk about Postgres. I don’t mean the kind of talk I do every day, which has more of a commercial touch to it. – Don’t get me wrong, that is very important too! – But I mean, really talk about PostgreSQL, be part of the community and help spread the understanding of what open source database technology can do for companies. Running implementations, either small or large, trivial or mission critical…
This past week was one of those weeks.
I got to travel through Germany together with Mr. Bruce Momjian himself. Bruce is the one of the most established and senior community leaders for Postgres. Bruce is also my colleague and I would like to think I may consider him my friend. My employer, EnterpriseDB, gives us the opportunity to do this. To be an integral part of the PostgreSQL community, contribute, help expand the fame of Postgres, no strings attached. Support the success of the 30 to 40,000 engineers creating this most advanced open source RDBMS.
The week started with travel, and I got to Frankfurt. Frankfurt will be the proving ground for the idea of a pop-up meet-up. Not an EDB-marketing event or somewhere where we sell EnterpriseDB services, but allow anyone just to discuss PostgreSQL.
We will be in a city, in a public place, answering questions, discussion things or just relax with some coffee. Purpose is to show what the PostgreSQL community is all about, to anyone interested!
The first day in Frankfurt, we spent at the 25hrs hotel. We had some very interesting discussions on:
- Postgres vs. Oracle community
- Changing role of DBA:
- The demise of the Oracle DBA
- RDBMS DBA not so much
- Risk management
- “Data scientist”
- Significance of relational growing again
In the afternoon we took the Train to Munich, which was a quick and smooth experience. Munich would be the staging ground for a breakfast meeting, or a lunch… or just say hi.
Bruce and I spend the day discussing:
- How to go from using Postgres as replacement of peripheral Oracle to Postgres as replacement for all Oracle
- Using Postgres as polyglot data platform bringing new opportunities
After the meet-up we headed to Berlin training towards the final two events of this week.We spent Thursday teaching the EDB Postgres Bootcamp, having a lot of fun and absolutely not sticking to the program. With Bruce here, and very interesting questions from the participants, we were able to talk about the past and the future of Postgres and all the awesome stuff that is just around the corner.
Friday morning started with a brisk taxi-drive from Berlin to the Müggelsee Hotel. And, if you happen to talk to Bruce, you simply must ask him about this taxi-trip 😉
I got to meet a great number of the specialists that make up the Postgres community:
Andreas ‘Ads’ Scherbaum
I am already looking forward to the next Postgres events I get to attend… pgconf.de 2019 will in any case happen on the 10th of May in Leipzig.
It would be super cool to see you there, please submit your abstracts using the information from this page!
Weird title, right! Very slick and marketing talk…
It is also a deviation of the usual topics I discuss here, but at least (!)as important.
But it is true. And this is what we truly aim to do at JK-Consult. Not just professionally, but in everything we stand for.
Of course, as long as this is just words, what does it really mean? Not so much…
Therefor, starting in 2018, we are a sponsor for Senna Rodijk, up and coming Dutch karting-talent. Senna has chosen the bold path to be part of Formula 1 and will make this happen through her focus on karting with Chrono Karting.
Starting this year in a new class, the minimax karts, we have seen her racing today in the first race of the new season!
For the complete story, as she has some unique challenges, check out her website.
Not only is she trying to establish herself as a champion in a field with only guys. In a time that everybody wants to try their hand at karting, to be the second Max Verstappen! This alone is a challenge in it’s self…
Senna’s major sponsor and coach, her bonus-father, Carlo Hoevenaars, is diagnosed with terminal cancer and his treatments have been recently ended… this is not pity full -as they don’t desire pity-, it is an incredible challenge!
Deep, deep respect for Carlo and deep respect for Senna and Suzanne!!
They can always do with any (additional) support they can get.
Click here for more details and some words from Senna herself.
We have put our actions where our believes are! You only get traction through action.
We stepped up and JK-Consult is the proud sponsor for Senna’s racing tires for 2018!
At the time of this writing we have finished the first race of 2018, with an amazing result!! Senna finished 5th and 6th in the two heats at the circuit Horensbergdam in Genk!
This is the final episode of this short series of blog posts on some of my drivers for moving to Postgres from Oracle.
Please do read Part I and Part II of the series if you have not done so. It discussed the topics “History”, “More recently”, “The switch to Postgres”, “Realization”, “Pricing”, “Support” and “Extensibility”.
- Part one focused on “why not Oracle anymore, so much”
- Part two discussed on the comparison between PostgreSQL and Oracle
- Part three talks some more on what Postgres then actually is
One of the more important things to be really, really aware of is that Postgres is “not just open source“. Postgres is “community open source“.
Now, why would that be important, you might wonder.
We all know what open source stands for. There are many advantages to an open source system, and in our case, an open open source database.
A number of arguments are in this blog post series. If you take this one step further though, and realize that Postgres is a community open source project, what are extra advantages?
A community open source project is not limited, in any way, to any one specific group of developers (let’s call them a company). For example, let’s look at MongoDB. This is an open source database, but it is developed by MongoDB inc.
It is, in essence, controlled by MongoDB.
Postgres is developed by the Postgres Developer Community coached by the Postgres Core Team.
This makes Postgres incredibly open, independent and it enables its developers to truly focus on actual business problems that need to be solved. There is no ulterior drive to satisfy commercial goals or meet any non-essential requirements.
A very important discriminator, that only became this clearly and apparent to me, after I dove into Postgres some more, is the development…
The actual development of the database core-software is done by this community, we’ve just identified.
“Well, yes…” you might say, this is what open source stands for. But the impact of this extends well beyond support, as I mentioned in part 2 of this series. The ability to be part of where Postgres goes, to have actual influence on the development, is awesome, especially for a database platform in the current “world in flux”.
Postgres users don’t necessarily have to wait until “some company” decides to put something on the road-map or develop it at their discretion. These company-decisions will mostly be driven by the most viable commercial opportunity, not necessarily the most urgent technical need.
The development of Postgres is more focused on “getting it right”.
One nice example is the Postgres query optimizer. The Postgres community hates bugs. When bugs start to get discussed, it results in many emails within the community, which stand for a lot of reading!
Many bugs are fixed very quickly, so that this email storm stops!
For optimizer bugs therefore, turn-around times (from reporting to having a production-fix) can be as low as 72 hours, so even for mechanisms as complex as a query optimizer.
I would like to invite all of you in the Oracle community to take a look at the Postgres query optimizer and share your concerns, worries, bugs or praise with the Postgres community.
If you want to, you can share this with us at the https://www.postgresql.org/list/pgsql-hackers/ email list. We are looking forward to your contributions!
I can only speak from what I see. What I see is that Oracle is becoming an on-line services company. I see them moving away from core technology like the database and accompanying functions. Oracle is more an more and moving to highly specialized applications aimed at very big companies.
Chat-bots, social media interactions, integrated services and more, delivered from a tightly integrated but also tightly locked set of Oracle owned and operated data-centers, or rather, the Oracle cloud.
Is this useful? Of course, there will be targeted customers of Oracle who will continue to find this all extremely useful, and it will be, to them.
It this for me? No, not really.
In the beginning, Linux was not something anyone wanted for anything serious. I mean, who wanted to run anything mission critical on anything else than Solaris, HP-UX, VMS, IBM? No-one…
And that was just a few years ago. Imagine!
Today in any old data-center, if you would eliminate the Linux based servers, you would not have much left.
This same thing is now also happening in, what I guess is the second wave of open source. More complex engines are being replaced by open source and the ever present relational database engine is one of those.
Why? Price, extensibility, flexibility, focus, you name it. We have seen it before and we will see it again.
If you permit me just these few words.
I think EnterpriseDB is extremely important for PostgreSQL. We have been fighting on the forefront since the beginning, supporting PostgreSQL’s move to the Enterprise. EnterpriseDB has been and will continue to spend a large amount of our resources to PostgreSQL. We are a PostgreSQL support company. We just have been not very good at patting ourselves on the back…
As a company we are doing extremely well, simply because Postgres is rock solid in all facets and ready to take on the word, even the most daunting tasks – and beyond.
This will continue as Postgres will continue in this second wave of Open Source.
I thank you for your attention.
If you have addition questions or comments, please do not hesitate to contact me.
Continuing this short series of blog posts on some of my drivers for moving to Postgres from Oracle.
Please do read Part I of the series if you have not done so. It discussed the topics “History”, “More recently” and “The switch to Postgres”.
In the last months, discussing Postgres with my Oracle peers, playing with the software and the tooling, I actually quite quickly realized Postgres is a lot cooler, at least to me. Not so much of the overly complicated technology, but rather built to be super KISS. The elegance of simplicity and it still gets the job done.
Postgres handles a lot the more complex workloads than many (outsiders) might think. Some pretty serious mission-critical workloads are handled by Postgres today. Well, basically, it has been doing this for many, many years. This obviously is very little known, because who would want to spend good money on marketing for Open Source Software, right? You just spent your time building the stuff, let somebody else take care of that.
Well… we at EnterpriseDB do just those things, …too!
And, please, make no mistake, Postgres is everywhere, from your fridge and video camera, through TV set-top boxes up to major on-line banking software. Many other places you would not expect a database to (be able to) run. Postgres is installed in places that never get touched again. Because of the stability and the low to no-touch administrative character of Postgres, it is ideally suited for these specific implementations. Structured on some of the oldest design principles around Postgres, it doesn’t have to be easy to create the database engine, as long as it “just works” in the end.
Many years ago, an Oracle sales director also included such an overview in his pitch. All the places Oracle touches everybody’s lives, every day. This is no different for Postgres, it is just not pitched anywhere, by anyone, as much.
I have the fortunate opportunity to work closely together with (for instance) Bruce Momjian (PostgreSQL core team founding member and EnterpriseDB colleague). I also had the opportunity to learn from him some of the core principles on which Postgres was designed and built. This is fundamentally different from many other software projects I know and I feel it truly answers some of the core-requirements of database projects out there today! There is no real overview of these principles, so that’s on my to-do list.
Working with PostgreSQL
Postgres is open source… it is true open source. It is even a true community open source project, but more about that later in the next installment.
Open source software is free to use, it does not cost nothing!
But, wait! Open source does not mean for free! How…, why…, what do you mean??
Well… you need support, right!?
The community can and will help you, answer questions, solve some of your problems. But they will not come in to install, configure and run Postgres for you. You will need to select and integrate your specific selection out of the wealth of tools. You basically have a whole bunch of additional tasks to complete to get your Postgres platform sorted out.
Companies like EnterpriseDB can help you mitigate these tasks. This allows you to focus on the things you actually want to achieve, using Postgres
In comparison to traditional database vendors, the overall price of your solution will absolutely significantly reduce when using Postgres as your open source database engine.
A significant difference between Oracle (for instance) and Open Source support services is interchangeability.
In the end, Oracle support can only be given by Oracle. They are the only ones that have access to the software sources and can look up (and hopefully fix) issues. In the support of Postgres, or any true community open source product, different companies can provide support. If you don’t like the company you work with… you switch. This drives these companies to be really good at delivering support! How is that for an eye opener.
One of the superb advantages of Postgres is its native extensibility. I mean, think about it for a moment… having a relational database platform with the strength of Postgres, the strength of Oracle or Microsoft SQL Server for that matter. Postgres gives you more options to integrate a wealth of data sources, data types, custom operators and many more other extensions than you will ever need! The integration into Postgres is so solid, these extensions function like any other function in the core of Postgres.
And, rest assured, chances that you will ever be faced with having to built this yourself are extremely slim. There are 30 to 40 thousand developers working with larger and smaller pieces of code of Postgres. Chances are that if you find yourself challenged, somebody else faced and solved that challenge before you. That solution will then be available for you to take and use, solve your challenge and move on. That is also open source for you.
This capability is what makes Postgres ultimately suited to fit the central role in any polyglot environment, we see being built today.
Maximizing the amount of information from data available in multiple data silos in an organization. This is a challenge we see more and more often today. Integrating traditional applications as ERP, CRM, with data-warehousing results, again combined with Big-data analysis and event-data-capture aggregates. This generates additional decision-driving information out of the combination of these silos. Postgres, by design, is ultimately suited for this. It saves you for migrating YUGE amounts of data from one store to another, just to make good use of it.
The open source Dogma “Horses for courses” eliminates double investments, large data migrations or transformations, it just enables you to combine and learn from what you already have.
End of part II
A link to part three of this blog post will be placed here shortly.
As with many stories, if you have something to tell, it quickly takes up a lot of space. Therefore this will be a series of blog posts on Postgres and a bit of Oracle. It will be a short series, though…
After switching to a more novel system, we adopted DEC (Digital Equipment Corporation) VAX, VMS and Micro VAX systems! Arguably still the best operating system around… In any case, it brought us the ability to run, the only valid alternative for a database around, Oracle. With a shining Oracle version 6.2 soon to be replaced by version 7.3.4. Okay, truth be told, at that time I wasn’t really that deep into databases, so much of the significance was added later. My primary focus was on getting the job done, serving the business in making people better. Still working with SQL and analyzing data soon became one of my hobbies.
Some rain must fall
This disconnect was the first one. Moving forward I noticed and felt more and more of a disconnect between Oracle and, what I like to call, core technology. Call me what you will, I feel that if you want to bring a database to the market and want to stay on top of your game, your focus needs to be at least seriously fixed on that database.
Instead we saw ever more focus for “non-core” technology. Oracle Fusion, Oracle Applications (okay, Oracle Apps had been there always), and as time progressed, the dilution became ever greater. I grew more and more in the belief that Oracle didn’t want to be that Database Company anymore (which proved to be true in the), but it was tough for me to believe. Here I was, having spent most of my active career focused on this technology, and now it was derailing (as it felt to me).
We saw those final things, with the elimination of Oracle Standard Edition One, basically forcing an entire contingent of their customers either out (too expensive) or up (invest in Oracle Standard Edition Two, and deal with more cost for less functionality). What appeared to be a good thing, ended up leaving a bad taste in the mouth.
And, of course… the Oracle Cloud, I am not even going to discuss that in this blog post, sorry.
The switch to Postgres
For me the switch was in two stages. First, there was this situation that I was looking for something to do… I had completed my challenge and, through a good friend, ran into the kind people of EnterpriseDB. A company I only had little knowledge of doing stuff for PostgreSQL (or Postgres if you like, please, no Postgré or things alike please, find more about the project here), a database I had not so much more knowledge of. But, their challenge was very interesting! Grow and show Postgres and the good things it brings to the market.
Before I knew it, I was studying Postgres and all the things that Postgres brings. Which was easy enough in the end, as the internal workings and structures of Postgres and Oracle do not differ that much. I decided to do a presentation on the differences between Postgres and Oracle in Riga. I was kindly accepted by the committee even when I told them, my original submission had changed!
A very good experience, even today, but with an unaccepted consequence. -> The second part of the switch was Oracle’s decision to cut me out of the Oracle ACE program.
It does free me up, somehow, to help database users across Europe, re-evaluate their Oracle buy-in and lock-in. Look at smarter and (much) more (cost)-effective ways to handle their database workloads. This finalized “the switch”, so to speak.
Meanwhile more and more people are realizing that there actually are valid alternatives to the Oracle database. After the adoption of the Oracle database as the only serious solution back in the early 1990’s, the world has changed, also for serious database applications!
End of Part I
Please follow this link to the second part of this blog post.
On a rainy Tuesday morning we set off on a Polish Airlines flight to Warsaw.
Our target: PGConf.EU with some of the biggest crew EnterpriseDB ever sent off.
Our goal: spread the love of EDB within the PostgreSQL community, where EDB is such an intrinsic part of.
The evening before the conference promises to be an interesting one. We have rented off the Hard rock Cafe in Downtown Warsaw for the kickoff of our European Tour. During this tour we will visit Hard rock Cafes throughout Europe to talk about the future of Postgres.
With 80 people present, our space at the Cafe was a full house! With food and drinks it was a perfect place for networking and sharing stories about Postgres. After the drinks, Warsaw city tour-guides took us on a walking sightseeing tour through the city. This inevitably led us to Vodka and Herring, which formed the crown on the day.
For me personally it was the perfect opportunity to submerge myself in the Postgres community, feel the energy and meet some of the leading names like Bruce Momjian, Dave Page, Robert Haas, Andreas Scherbaum, (“Blame”) Magnus Hagander and many, many more.
Second day in Warsaw, first day of PGConf.EU. The EDB crew slept well and is ready to roll!
At the EDB stand we are hosting a quiz where participants can contest to actually win an electric guitar and free access to one year of on-line Postgres training. The challenge: by means of speed and correctness answer 11 questions on PostgreSQL!
In a busy and energetic there are loads of very good sessions going down, making hard to single out the sessions to follow…
The second day of PGConf.EU continues the flow of excellent talks and fierce competition at our Postgres Rocks Quiz. EDB Postgres Experts are sharing their knowledge abundantly at this conference.
Day two also is my presenting debut at a Postgres-conference. Together with friend and partner in crime Daniel Westermann of DBI-services. Our talk was well received and we were able to share some of our experiences of entering the PostgreSQL world, coming from Oracle.
EDB concluded the second day of PGConf.EU with a team-dinner in the heart of Warsaw. A unique opportunity to bring all the facets of EnterpriseDB togehter. This ranges from our community foundation to our strategic business vision, all together in one place to exchange ideas and enjoy good company over a very good meal.
As with all good things, we inevitably reach the closing day, no difference there for PGConf.EU!
More excellent talks on general Postgres development and the influence of various projects on the future of the leading Open Source database project.
We also had a very exciting final to our EDB Postgres Quiz, with a surprise victory for Rafal Hawrylak of TomTom and an excellent runner up Matthijs v/d Vleuten of Hendrikx ITC.
On behalf of the entire EnterpriseDB, I would like to thank the PostgreSQL conference board for an excellent experience in Warsaw!
Live free or die!!
If you like to relive pgconfeu, simply review the comprehensive tweet-timeline.
Since over half a year I have made an adjustment in my course… nothing too dramatic, but still it has had some impact.
I have chosen the path of the more pure technique again. Not in a sense that I don’t manage anymore, though I don’t, but that’s more of a side effect
I have chosen the path of the more pure technique in a sense that the change from Closed Source software to Open Source software allows you to actually work with and solve things by creating solutions rather than trying to figure out how something, someone created for some issue, can reconfigured so it resembles a solution for your actual problem.
Okay… okay… this of course is exaggerated, but it serves to help think about the issue.
“No one ever got fired for buying Oracle” is one of the phrases I have heard numerous times over the past period.
Well, no… but it’s also no free pass to –sorry for the phrase– waste money on technology you either never going to use.
Over 80% of the installed base uses less than
20% of the technology they are paying for!
I have followed a number of the brightest mind in the industry (our industry, the database industry) for many years investing vast amounts of time in reverse-engineering pieces of technology that have been built, in order to explain certain behavior.
Of course, very necessary, no argument there, but wouldn’t it be so much more cool if this overwhelming amount of brain-power could be used to actually create stuff??
Open Source in stead of Closed Source…
The answer, I think!
Sure, I am raised with vendor created solutions, that was the default MO when I got trained. VMS, MS-DOS, HP-UX… (are you _that_ old, yes, I am _that_ old) and a number of applications that did the work.
Well, those days are gone… operating systems in data centers have (nearly) all been replaced by Linux distribution installs. And I mean like, as good as all of them.
Sufficiently stable, cost effective and they get the job done.
With the current explosive growth of Open Source databases, brace yourselves. Or rather, embrace!
All the exact same arguments that are there for Operating Systems apply. There is no difference, and you, the industry, chose! And you will choose this again. Simply because “it is good enough”, it is much more cost effective and it gets the job done.
The extensibility, the agility of Open Source database software gives you the ability to let your database, be it OLTP, OLAP, Big Data, Polyglot, or whatever we come up with, do what needs to be done.
The current leader of the Open Source relational database systems is PostgreSQL. A platform developed in over 30 years to become an absolutely stable data processing engine for a fraction of the costs of the Closed Source players in this market.
- We have seen wave 1 of Open Source where we all choose to replace the operating system standard with Linux.
- We will now see wave 2 of Open Source where we will choose to replace the database management system standard with… PostgreSQL (or in specific cases one of the other, more specialized systems, depending on the need).