The Cost of Open Source

As a bit of background, I have composed a number of musical pieces in recent years, two of which have been submitted to competitions where selected works were performed by an affiliated orchestra. Obviously when submitting works to an orchestra, the score and its presentation is what matters most - not a MIDI file (which is limited to 16 channels to divvy up among the instrument parts), or even an MP3 rendering of a MIDI file.

If anyone has been involved with composing for an ensemble, they would find that there is also a standard to adhere to and conventions to follow when writing the score. Instruments have to be in the correct logical order, tempi must appear above both the top line in the score and the First Violins, full instrument names should appear on the first page and abbreviations on subsequent pages, etc. It should be obvious that to do this properly, you need decent software to help achieve the end results. (Who'd have the time and patience to write it by hand?) As you'd probably guess from the title, this post is more about how the software used fits into the process, rather than the music itself or any competition.

Ask anyone academically or professionally involved with music what software they use for scorewriting, and you'll usually get one of two responses. Sibelius or Finale. Usually the former. The only down-side to these packages is that the functionality needed for the intricacies of some pieces (be they multiple time signatures throughout a piece, divisi on separate staves, tuplets, single- and double-line staves for percussion, etc.) does not come cheap, with the market leader selling for over $1,000NZ (unless you're eligible for an academic discount). Don't get me wrong - Sibelius (version 5 anyway) is a great package - but there is an obvious financial cost involved. Note that I'm not considering cheaper packages - in many cases they are typically marketed as 'virtual studio' software or sequencers - designed for the audio (or MIDI) output. Others simply lack the capabilities simply needed for the demands of the score and impose limits such as 16 staves in a score (to coincide with the number of available MIDI channels - one of which is always for percussion).

But surely there must be cheaper or even free software out there that does what is needed? Especially given the commitment of some to developing Open-Source Software.

In many cases, there is the economic principle, 'you get what you pay for', but there are some notable software packages that go against this, and are gaining traction in their respective communities. Blender is becoming a popular competitor to other 3D modelling and animation software; many would have discovered Linux (e.g. Ubuntu) and its applications; and around half reading this are probably reading this inside Firefox.

Naturally, it shouldn't come as too much of a surprise that there is open-source scorewriting software out there. Rosegarden is pretty capable, but is designed more as a sequencer than a dedicated notation package - and is only available on Linux. The other option (that I know of) is LilyPond - software dedicated to engraving music with aesthetics of the output being a top priority. Because of this focus, not much attention is paid to the user interface. In fact, there is no user interface - if you want a WYSIWYG (what you see is what you get) interface in the vein of Sibelius or Rosegarden, you're out of luck.

So how does LilyPond work?

Music is written in a text editor, in a markup language (similar in concept to LaTeX for documents) devised by the people behind LilyPond, and saved as a LilyPond file (*.ly). LilyPond is run as a command-line process with the LilyPond file as a parameter. In Windows, there is a shortcut in the right-click menu (of Windows Explorer) for LilyPond files, which allows you to run LilyPond on the selected file. In either case, the engraved music is generated music in PostScript and PDF format. If there are errors in the input, these will be listed in a log file that is also generated. It is also possible to generate MIDI files from LilyPond input, so you can hear what the piece sounds like. Think of this as Computer Programming Meets Music.

Unfortunately, when composing for a competition (or a commission), there is a deadline imposed. A limit on how much a piece can be crafted to perfection. I simply don't have time to work on a piece, learn the syntax of the LilyPond markup language, rewrite the piece in a text editor, and enter into the loop of producing output and debugging. I'm sure many other composers would feel the same - even if they have a similar programming / Computer Science background.

Fortunately, there are tools to help improve efficiency. How many composers would really compose in a text editor and wait for LilyPond to do its thing before being able to preview what it sounds like in a MIDI file? Especially when there are potentially many cycles to get that phrase right? For me (and no doubt many others), I'd prefer to work in a WYSIWYG environment, where I can preview my changes immediately. The good thing is that Rosegarden provides such an environment, and is able to export music to LilyPond format. However, last time I tried this I had to correct parts of the output, since the LilyPond version was different to what was expected by LilyPond. It wasn't just changing the version number at the top - there were numerous markup extensions that were new, and others that became obsolete. So that was one all-nighter finishing off "Perpetual Restlessness" before submission. (Oh, the irony.)

Lilypond also provides utilities to convert MIDI and MusicXML files into LilyPond format. Since I have experience with the MIDI to LilyPond converter, I'll walk through that using my more recent piece "Maui" as an example. In the knowledge that the true cost of using the free LilyPond was time (and the learning curve), I allocated much more time for getting the LilyPond output right. Here's the process I used to produce the score for "Maui" (first revision):

Compose the piece in Notation Package 'A'. This is the package that limits the score to 16 staves, meaning that I had to add both harp parts later. With this show-stopper, it was moot that notes didn't line up properly between staves or even voices (also a show-stopper - I don't have time to drag all the notes to where they should be).
Export to MIDI file #1.
Import MIDI file into Sequencer 'B' and add the harp parts. This sequencer lacked a score editor, so I had to create the notes in the piano roll view. Not the best interface for composing, unless you're working on a 12-tone piece and need to visualise the relationships.
Export to MIDI file #2.
Run 'midi2ly' on the second MIDI file to convert it to LilyPond format.

Here's the sort of output that was produced when LilyPond itself was run on the generated Maui.ly file:

Note that all notes (with a few exceptions) are rendered as crotchets! The markup file is not much better, with 2,456 lines of this sort of stuff:

...
  % 73
  e dis d cis c b4*128/960 s4*32/960 ais4*128/960 s4*32/960 a4*128/960 
  s4*32/960 g4*128/960 s4*32/960 a4*128/960 s4*32/960 ais4*128/960 
  s4*32/960 cis4*128/960 s4*32/960 d4*128/960 s4*32/960 a4*128/960 
  s4*32/960 d4*128/960 s4*32/960 a'4*128/960 s4*32/960 f4*128/960 
  s4*32/960 a4*128/960 s4*32/960 d4*128/960 s4*32/960 a4*128/960 
  s4*32/960 f4*128/960 s4*32/960 a4*128/960 s4*32/960 f4*128/960 
  s4*32/960 d4*128/960 s4*32/960 |
  % 74
  a4*384/960 s4*2256/960 e''4*192/960 s4*48/960 f4*128/960 s4*32/960 cis4*128/960 
  s4*32/960 c4*128/960 s4*32/960 <c gis >4*128/960 s4*32/960 ais4*128/960 
  s4*32/960 gis4*128/960 s4*32/960 |
  % 75
  <b g >4*128/960 s4*32/960 a4*128/960 s4*32/960 g4*128/960 s4*32/960 fis4*128/960 
  s4*32/960 e4*128/960 s4*32/960 d'4*128/960 s4*32/960 e4*128/960 
  s4*32/960 b4*128/960 s4*32/960 g4*128/960 s4*352/960 cis4*128/960 
  s4*32/960 a,16 s16 a4*1152/960 s4*288/960 |
  % 76
  d,16 s16 d4*1152/960 s4*288/960 <a'' dis >4*256/960 s4*224/960 c,,4*1152/960 
  s4*288/960 |
  % 77
  <dis'' a >4*256/960 s4*704/960 <fis, c' >4*256/960 s4*384/960 <fis c' >4*256/960 
  s4*64/960 <a d >4*256/960 s4*224/960 d,,4*1152/960 s4*288/960 |
  % 78
  <e'' g >4*256/960 s4*704/960 <e g >4*256/960 s4*8384/960 d,,4*384/960 
  s4*576/960 <d f >4*384/960 s4*576/960 |
  % 81
  <d f gis >4*384/960 s4*576/960 <gis f b >4*384/960 s4*576/960 <f a d >4*768/960 
  s4*192/960 <e g d' >4*384/960 s4*96/960 <d' g, e >4*384/960 s4*96/960 |
  % 82
  <d g, e >4*384/960 s4*96/960 <e, g d' >4*384/960 s4*96/960 d'4*128/960 
  s4*32/960 d4*128/960 s4*32/960 d4*128/960 s4*32/960 d4*128/960 
  s4*32/960 d4*128/960 s4*32/960 d4*128/960 s4*32/960 <f, a d >8 
  s8 <c' e e, >8 s8 |
  % 83
...

Split all instrumental parts into separate LilyPond files and include them (similar to #include in C/C++) in the master file, to make the score easier to work with.
Go through each file, correcting time signatures, note durations and removing unnecessary "s4"'s. Find & Replace worked to an extent, but each part took a while.
Convert chords into multiple-voice passages, keeping in mind that the order of notes in each chord is arbitrary from one chord to the next. Also keep in mind that changing the order of notes in a chord may disrupt the octave in 'relative pitch' mode, causing the rest of the music in that part to jump up or down an octave.
Fix octave pitch indicators where the music jumps up or down an octave. Note that this is an iterative process within each part.
Research and investigate the appropriate markup extensions to get the remaining layout 'good enough'. Very tricky in some cases.
Add dynamics and articulation marks to each part, and tempo indicators to the relevant parts (usually Flutes 1-2 and 1st Violins).

At the end of this two-week process (I learnt the lesson of allocating more time from "Perpetual Restlessness") and yet another all-nighter, the LilyPond source looked a bit more meaningful (for those well-versed in LilyPond syntax):

...

% 20 //Main melody 1/2 starts here
	\clef tenor
	<< {f'4( e f g | f2) r4 e8( d |
	c4 a'8. bes16 a8 g f e | f4 g f e} \\
	{d4( c d e | d2.) r4 |
	a1( | d2) (d4 c} >> |

% 24 //Main melody 2/2
	<< {f4 e f) f( | g2. r4 |
	r2. a,4( | f2.) r4} \\
	{d'4 c d) d( | b2.) r4 |
	\clef bass
	r4 d( a e8. d16 | d2.) r4 } >> |

% 28 //Variation A (p)
	R1*3 | r2. a'4(\( |

% 32 //Variation A (ff-f)
	8( <c' e,> |
	<d, bes'>2) r4  |
	<a' cis,>1)\) |

% 36 //Variation B -> 'anchor drop' (End of Section I)
	a2 r8 a( bes g | a4) r4. bes8( a g |
	d4. e8 f g <d' bes>  |
	<< {r2) \times 2/3 {f16[ g f} \times 2/3 {e d cis]}
	\times 2/3 {d[ e d} \times 2/3 {c bes a]} |
	r2 \times 2/3 {a16[ bes a} \times 2/3 {g a g]}
	\times 2/3 {f16[ g f} \times 2/3 {e f e]} | d1}
	\\
	{ r1 | \times 2/3 {bes'16[ c bes} \times 2/3 {a g f]}
	\times 2/3 {g[ a g} \times 2/3 {f e d]}
	\times 2/3 {f[ g f} \times 2/3 {e d c]}
	\times 2/3 {bes[ a g} \times 2/3 {a g e]} | d1 } >> |

% 42 //Secondary theme (p) (Section II - 'The Catch')
	\time 5/4
...

It probably comes as little surprise from this (for those who know a bit of Maori mythology) that this piece is based on the legend of Maui fishing up what became known as the North Island of New Zealand. Here's a part of the (somewhat more tolerable) output:

Excerpt of Maui score, with quavers rendered correctly

So what have I learned from all this?

Know the true cost of what you're buying into.

Cost isn't just measured in terms of money; it can also be measured in terms of other factors such as time and effort.

So at the end of all this, I'm now the proud owner of a copy of Sibelius 5. Yes, it's expensive, but I've found that it is well worth it. Especially version 5, since the introduction of the Panorama view. Don't get me wrong - LilyPond itself has its merits, and it is admirable that a group of people have taken the time and effort to create this software, let alone for free. Rather than bashing LilyPond, I'm demonstrating that it is not ideal for composing or notating compositions when there are tight time pressures involved.

After all, if you were composing a piece under similar circumstances, where would you want to spend your effort: composing or text-editing?