This blog post is a written version of my presentation given at the ITx conference in Wellington on 13 July 2016.
Introduction
For the last 20 years or so I have been building new business tools at Radio New Zealand, mainly using Free and Open Source Software. These have ranged from a database for our music library, to the centralised system for managing contact information that is at the core of our news operation, and of course, the RNZ website.
In this talk I’ll be focussing on that.
In late 2004 I was asked to lead a project to establish a new website for RNZ and that was launched in October 2005.
This is what it looked like back then.
At the time RNZ had no tools to publish to the web apart from the one copy of Microsoft FrontPage was used to update the very basic site.
Building a new site from scratch was both a technical and business challenge. A lot of technical design and business process design was required.
So, how did Free Software allow us to build the site, innovate, and save money?
Firstly, what do I mean by Free Software?
- The code can used for any purpose.
- You can give away the code you receive.
- The code can be studied and changed as you wish.
- You can give way your own modifications to the code.
The third of these is at the core of enabling innovation, closely followed by the network effects that all four enable in terms of creating an ecosystem of continual improvement.
Before looking at how we’ve used Free Software, I want to dive into some business theory to show why I frequently choose Free Software to help solve business problems.
Some Business Theory
A business is a system. W. Edwards Deming defined a system as a network of interdependent components that work together to try to accomplish the aim of the system.
This diagram from the book Out Of The Crisis shows a system.
Deming obviously had manufacturing in mind when he drew this, although it also applies to businesses producing non-physical goods.
Using RNZ as an example think of a journalist recording an interview. This is edited, and along with other items assembled into a programme. That is inspected by the programme’s executive producer, and then distributed, or as we say broadcast, to our consumers, or listeners.
Supporting this are a whole range of tools. Recording equipment, editing tools, writing tools, sound mixing tools. External tools such as the telephone system and the internet are also used to help the system accomplish its aim.
For most businesses today ‘tools’ means IT systems, including software. These tools are generally chosen to either support the existing business or to support some finite future change to the business.
An important part of the diagram is a feedback loop, showing that the system is improved based on feedback gained from research. Where most businesses go wrong is that they buy tools for today, or for tomorrow, but without any plan for on-going improvement. They only know about current requirements, or the change to be made tomorrow. There is no research, no real strategy. So many projects fail because of this – requirements change during the course of a project, sometimes to the point that the chosen tool is no longer suitable.
Any vendor can promise to deliver a tool today that meets you current needs. But what of tomorrow. <sarcasm>Oh right, just extend the project budget and timeline and they can customise it for you.</sarcasm>
My key point here is that the capability and flexibility of your business processes are dependant on the tools you choose to support them.
When I started at RNZ recording was done on tape, editing was done by transferring tape-to-tape, writing was done on typewriters and our mixing and broadcast equipment used a mix of germanium transistors and valves. Four track recording was available for commercial production, 24 track a luxury only afforded to music production.
Today, our field recorders are solid-state, with no moving parts, our audio, video and writing tools are all based entirely in software, our mixing desks are just big banks of Digital Signal Processors (DSP), and oddly, valves are back in fashion. Most producers have a 24 track digital production workstation on their desktop.
While the actual tools have changed dramatically in the last 30 years, the basic processes used to create radio programming had not changed that much. They have been streamlined of course, although the system was highly optimised for one purpose – radio broadcast.
Other broadcasters had created entire new divisions to handle this medium alongside their existing business. But my aim was to integrate any new web publishing processes into the existing business, and leverage off existing tools, skills, and people as much as possible.
This is why systems thinking became so important. Process redesign became the key, supported with flexible tools that are not dictated by someone else’s roadmap.
The key words are autonomy, independence, and self-determination.
How???
Has Free software allowed us to innovate and save money where proprietary software would not have? That is the 40 million dollar question (for future readers, the week before this was written there was a large Lotto jackpot).
For the record RNZ uses lots of proprietary software. I was responsible for choosing some of it – for example the SADiE audio editor we use for feature audio production. Yes, there are open source audio editing tools. But the system we chose 15 years ago was fully featured, supported on-going process change and improvements, and is still in use today.
The best tools are sometimes proprietary, but having said that, I still believe that in most cases use of free software is a better long-term business choice.
The money
OK, what about the cost differences?
The cost of running any system includes the license costs, the costs of upgrading versions, and the cost of support. Everything has a cost, Free or not.
The total cost of ownership also includes losses incurred through NOT being able to adapt to a changing market, lost customers, employees who leave because of the tools, and the we’ve-got-you-over-a-barrel-now cost of migration when you finally do have to change.
The unseen cost is often the loss of knowledge capital about how your business works to the vendor to whom you’ve outsourced.
I think that flexible software that you can modify and adapt is cheaper than buying for today only, or replacing for tomorrow.
Innovation
In order to demonstrate how the software has allowed us to innovate I’m going to show how our web systems have evolved over the last 10 years.
Our first CMS was ‘open-core’ – a core of free (of cost) functionality was available under a GPL license. Free to download, free to use, free to modify, free to give those changes away.
Purchasing 24/7 support gave us access to commercial modules such as advanced search, and upgrade services, but we gave up the freedom to modify the code. This is often a reasonable trade-off when system stability is a primary concern. In our case, at that time, it was.
At around the four year mark we started to feel the pain of growth. While the system had allowed us to quickly grow and add new features like podcasts, it was starting to limit innovation due to divergence between the vendor’s roadmap and our own. With 10,000 assets, managing content in the system was becoming a problem, as was the performance of the system under load. The vendor were focussing more on functionality for corporate and government use, while we needed simplicity and speed.
We evaluated a number of commercial systems and free software projects looking to see if something off-the-shelf or nearly of-the-shelf would suit. A key criteria was the need for performance – fast page delivery and fast publishing of content. Another was the ability to make fast and frequent changes to the code and to deploy that code frequently. A third was to have the system support key business objects – a station, a programme, a news story, a piece of audio. We did consider projects such as Drupal and Silverstripe, but in the end chose to build our own CMS using the Ruby On Rails framework.
The migration from the old CMS to the new took two years, as we built out sections in the new system, and maintained APIs between the two for any content that was required in both places. You can read posts about this process in my blog archive.
One downside of building our own CMS is that in the last few years the business has changed faster than we’ve been able to adapt and improve admin section, but this is a resource issue, not a problem with the technology. One thing it still is though is fast.
I’ll explain four areas where we leverage the flexibility of Free Software to innovate.
News Publishing
For news, speed is the most important feature of any editing and publishing system. At RNZ we use a tool called iNews to create and edit news copy, and for a range of other broadcast related tasks. It is very fast, easy to learn, and is tightly integrated into the news operation.
The browser-based editor and workflow we had available to us in 2005 was vastly inferior to this desktop tool, so I made the decision to adapt iNews to handle publishing to the site.
There was one problem though – the only way to get content out of iNews was FTP, one story at a time.
This is where Free Software was invaluable, allowing me to built an application that received these stories, processed them, and updated the CMS via an API. The CMS vendor wrote an API script for us, and both my application and the API were later adapted when we added news categories to the site. The first versions of the interface only allowed for new stories to be created, which these days would be considered pretty useless. Later versions allowed one way updates via the push-only API.
In our first CMS, the time to live for a story in the top 5 was up to 5 minutes, other stories tools up to 20 minutes to update, these times being dictated by cache times and performance tradeoffs in the CMS.
Because the code was open, we were able to work closely with our sysadmin to build some intelligence into the cache expiration – i.e. hack around it – that was more closely aligned with our publishing cycle.
When we migrated to Rails I enhanced this system of working, created a more powerful API, rewrote the publishing engine in Ruby, and we’ve been extending and enhancing it ever since.
News staff order stories in a queue and publish all stories (and their order), or just one story (updating just the content) at the press of one key. They can assign categories, tags, and can also remove stories if needed.
At first glance this seems odd and looks clunky, but when people use it they soon find out how fast it is. Any number of new stories can be added, the order of stories changed, and these changes are then published together in one bulk operation. It takes about 30 seconds to update 60 stories in what we call a ‘full publish’.
Updating a single story from iNews takes about 5 seconds. This is great for fast changing news, as editors can update as frequently as they want with no penalty.
At some stage in the future we will move editing of news into the CMS, but this will require a lot of work to ensure that speed and the ability to collaborate are not compromised. Some of the new editors based around React look promising.
Audio Outbox
Our audio is edited and produced on a system called CoSTAR. It shipped with a very basic script written in Visual Basic that exported audio dropped into a folder as an MP3. For mass publishing of audio and its associated metadata this was quite inadequate, so we extended that script to extract metadata from the system into an XML file. Sadly, the script used to crash and we were unable to debug it beyond ascertaining that the crash always happened just before the final End statement.
The other issue was that Windows did not support SCP or SSH to transfer the files to the CMS or media servers.
The script was ported to Perl, which has a Win32::OLE module for accessing the Windows COM interface, we installed windows ports of SSH and SCP, and we were away laughing. The Perl version of the script ran in a loop, same as the VB one, but would run without crashing on its own.
This outbox has been extended over time to allow the addition of images to our audio files, transcoding to Ogg Vorbis, and it was eventually ported to Ruby.
The publishing process this application supports is now so streamlined we can typically get a broadcast interview on the site within 5 minutes of broadcast. It is so fast that some programmes are published to the site before they have finished on air! (This is possible because some content is pre-recorded, and goes live on the site at the start of broadcast.)
Because it’s our own software bugs can be fixed quickly and improvements added as needed. When we started tagging our audio, it took a coupe of hours to add this capability.
ELF Marks
One innovation we added to the CMS is the concept of ELFmarks. (Our CMS is called ELF). These marks allow editors to reference content in the system without having to know any HTML.
To add a half-sized image with its default caption:
[image:12345:half]
It renders like this:
Or an audio player in a news story (this is all on one line):
http://www.radionz.co.nz/national/programmes/morningreport/audio/201807917/grey-power-chapter-petitions-for-medical-marijuana “We want to grow it in our gardens” – Beverley Adridge on Morning Report
Renders thus:
Want a pullquote style? Use this:
[pullquote] The whare is more to us than wood and nails. It’s an ancestral house. -Pare Sannyasi
And you get this:
This is distinct from an inline quote, which are very common in news copy and are supported with the standard HTML <blockquote> element.
The system can also render external media such as Tweets, YouTube and Facebook video, and Instagram. For example:
This removes the need to for editors to cut and paste embed codes and it gives us control over how and when any third party javascript is loaded. For example, we only load the Facebook code when there is a Facebook video on a page, something we do to limit leakage about our visitors’ personal data.
Most CMSs allow raw HTML or links to be added manually. This is error prone and everyone likes to have a play to make it look how they think it should. ELFmarks eliminate these problems as an invalid mark just does not render, and the HTML that is used is consistent across the whole site.
One by-product of our approach is that links based on ELFmarks can never break because the system checks to see if that content is still live. If the original audio or news headline is updated, that change is reflected everywhere because the mark is a reference to the original content object.
As I mentioned, ELFmarks also give us complete control over the HTML rendering of all media on the site. So what, you say?
Well, we are currently redesigning the site, and in most of our content we have no legacy markup to deal with. We just change the HTML generated by the ELFmark, style it, and we are done.
Performance
I care a lot about performance. Fast pages, served quickly, make for happy visitors. Free software has allowed us to continue to fine-tune the system to remain performant, even though traffic has increased 5-fold in five years, and the number of assets served by the system (images mostly) has increased four times.
This increase has been on the same server hardware we’ve had since 2011. In that time we’ve upgraded OS versions 5 times, database versions 3 times, including a move to a MySQL fork, Percona, update the SOLR-based search, and decreased the page delivery times by about 50%.
Summary
There is a lot more that we have done, and plan to do, with the site and the CMS that drives it. Free Software has allowed us to set our own course and focus on the the things that matter to our business.
Richard Hulse 