Chapter27

Humdrum provides three pre-defined representations pertinent to text or lyrics. The **text representation can be used to represent words; the **silbe representation can be used to represent syllables; and the **IPA representation can be used to represent phonemes (via the International Phonetic Alphabet). Discussion of the **IPA representation will be delayed until Chapter 33. In this chapter we will look at various representational and processing issues related to the manipulation of words and syllables.

In the **silbe representation tokens represent individual syllables. In **silbe the hyphen (-) is used explicitly to signify syllable boundaries and the tilde (~) is used to signify boundaries between hyphenated words (necessarily also a syllable boundary). In other words, four types of syllables are distinguished by **silbe: (1) a single-syllable word, (2) a word-initiating syllable, (3) a word-completing syllable, and (4) a mid-word syllable. The following table illustrates how these signifiers are used:

Table 26.1.

text	a single-syllable word
text-	a word-initiating syllable
-text	a word-completing syllable
-text-	a mid-word syllable
text~	a single-syllable word beginning a hyphenated multi-word
~text	a single-syllable word completing a hyphenated multi-word
~text~	a single-syllabe word continuing a hyphenated multi-word
~text-	a word-initating syllable continuing a hyphenated multi-word
-text~	a word-completing syllable -- part of a hyphenated multi-word

A-Z	upper-case letters A to Z
a-z	lower-case letters a-z
(	open parenthesis
)	closed parenthesis
{	beginning of phrase
}	end of phrase
%	silence (rest) token (character by itself)
M	humming voice (character by itself)
[	beginning of spoken voice
[[	beginning of whisper
]	end of spoken voice
]]	end of whisper
<	beginning of Sprechstimme
>	end of Sprechstimme
#	beginning of laughing voice
##	end of laughing voice
@	laughter (no text)
&	sob or cry (no text)
$	beginning of emotional voice
$$	end of emotional voice
*	follows stressed word (**test) or stressed syllable (**silbe)

The text Command

A simple text-related task might be looking for occurrences of a particular word, such as the German "Liebe" (love). If the lyrics are encoded in the **text representation, then a simple grep will suffice:

grep -n 'Liebe' schubert

Recall that the -n option gives the line number of any occurrences found. If the input is encoded in the **silbe representation, then the output of text can be piped to grep:

extract -i '**silbe' schubert | text | grep -n 'Liebe'

Frequently, it is useful to search for a group of words rather than individual words. Suppose we are looking for the phrase "white Pangur." The context command can be used to amalgamate words as multiple stops. If we are looking for a phrase consisting of just two words, we might use the -n 2 option for context:

text barber | context -n 2 | grep -i 'white Pangur'

Another common task is simply to provide a readable text of the text or lyrics of a work. Given a **text representation, we can use the rid command to eliminate all records except non-null data records. This will result in a list of words -- one word per line. UNIX provides a simple text formatter called fmt that will assemble words or lines into a block text where all output lines are roughly the same width. Consider the Gregorian chant A Solis Ortus from the Liber Usualis (shown in Example 27.2.)

Example 27.2. Beginning of chant A Solis Ortus.

Another useful output would have the text arranged with one sentence or phrase on each line. As before we can use the context command with the -e option to amalgamate words, where each amalgamated line ends with a punctuation mark:

extract -i '**silbe' chant12 | text | context -e '[.,;:?!]' \

| rid -GLId

The corresponding output is:

A solis ortus cardine ad usque terrae limitem,
Christum canamus principem,
natum Maria Virgine.
Beatus auctor saeculi servile corpus induit:
ut carne carnem liberans,
ne perderet quos condidit.
Castae parentis viscera cae lestis intratgratia:
venter puellae bajulat secreta,
quae non noverat.
Domus pudici pectoris tem plum repente fit Dei:
intacta nesciens virum,
concepit alvo filium.

Yet another way of arranging the text output would be to parse the text according to explicit phrase marks in the **kern data. This will require a little more work, but it's worth going through the steps since the same process can be applied to any representation. First, we will need to transfer the end-of-phrase signifier (`}') from the **kern spine to the **silbe spine. This transfer entails four steps. (1) Extract the monophonic **kern spine and eliminate all data signifiers except closing curly braces (`}'). Store the result in a temporary file:

extract -i '**kern' chant12 | humsed 's/[^}]*//; s/^$/./' \

> temp1

Notice that humsed has been given two substitution commands. The first eliminates all data signifiers except the close curly brace. The second substitution transforms empty output lines to null data records by adding a single period.

(2) Extract the **silbe spine, translate it to **text and store the result in another temporary file:

extract -i '**silbe' chant12 | text > temp2

(3) Assemble the two temporary files together and use the cleave command to join the end-of-phrase marker to the syllable representation.

assemble temp1 temp2 | cleave -i '**kern,**text' \

-o '**text' > temp3

With this cleaved data we can now use the context command to amalgamate phrase-related text. Finally, rid is used to eliminate everything but non-null data records.

context -o = -e } temp3 | rid -GLId

The result is as follows:

A solis ortus cardine }
ad usque terrae limitem, }
Christum canamus principem, }
natum Maria Virgine. }
Beatus auctor saeculi }
servile corpus induit: }
ut carne carnem liberans, }
ne perderet quos condidit. }
Castae parentis viscera }
cae lestis intratgratia: }
venter puellae bajulat }
secreta, quae non noverat. }
Domus pudici pectoris }
tem plum repente fit Dei: }
intacta nesciens virum, }
concepit alvo filium. }

We could clean up the output by using the sed command to remove the trailing closed curly brace. We simple add the following to the pipeline:

. . . | sed 's/}//'

You might have noticed that each of the above phrases seems to consist of eight syllables. We can confirm this by returning to the syllabic rather than word-oriented output. For the above command sequence, simply omit the text command and replace **text with **silbe. The revised script becomes:

extract -i '**kern' chant12 | humsed 's/[^}]*//; s/^$/./' \

> temp1

extract -i '**silbe' chant12 > temp2
assemble temp1 temp2 | cleave -i '**kern,**silbe' \

-o '**silbe' > temp3

context -o = -e } temp3 | rid -GLId | sed 's/}//'

The corresponding output is:

A so/- -lis or/- -tus car/- -di- -ne
ad us- -que ter/- -rae li/- -mi- -tem,
Chri/- -stum ca- -na/- -mus prin/- -ci- -pem,
na/- -tum Ma- -ri/- -a Vir/- -gi- -ne.
Be- -a/- -tus au/- -ctor sae/- -cu- -li
ser- -vi/- -le cor/- -pus in/- -du- -it:
ut car/- -ne car/- -nem li/- -be- -rans,
ne per/- -de- -ret quos con/- -di- -dit.
Ca/- -stae pa- -ren/- -tis vis/- -ce- -ra
cae/ le/- -stis in/- -trat- -gra/- -ti- -a:
ven/- -ter pu- -el/- -lae ba/- -ju- -lat
se- -cre/- -ta, quae non no/- -ve- -rat.
Do/- -mus pu- -di- -ci pe/- -cto- -ris
tem/ plum re- -pen/- -te fit De/- -i:
in- -ta/- -cta ne/- -sci- -ens vi/- -rum,
con- -ce/- -pit al/- -vo fi/- -li- -um.

The above output was generated using the humsed command. Any syllable containing a trailing asterisk (*) is re-written as a `1', otherwise as a `0'.

. . . | humsed 's/[^ ][^ ]*\*/1/g; s/[^1][^1]*$/0/g'

With the above output, we can generate an inventory of phrase-related text-rhythms.

. . . | sort | uniq -c | sort

With the following results:

5 0 1 0 1 0 1 0 0
4 1 0 0 1 0 1 0 0
2 0 1 0 0 0 1 0 0
1 0 1 0 1 0 0 1 0
1 1 1 0 1 0 1 0 0
1 1 0 0 0 0 1 0 0
1 1 0 0 1 0 0 1 0
1 0 0 0 1 0 1 0 0

We can create a summary rhythmic pattern by adding together the values in each column -- that is, counting the number of accented syllables that occur in each syllable position within the phrase. We can isolate each column using the UNIX cut command; cut is analogous to the Humdrum extract command. Fields are delineated by white space (tabs or spaces). For example, cut -f 1 will isolate the first column of numbers. We can then pipe the results to the stats utility in order to calculate the numerical total. For example,

. . . | cut -f 1 | stats | grep 'total'
. . . | cut -f 2 | stats | grep 'total'
. . . | cut -f 3 | stats | grep 'total'
etc

For the chant O Solis Ortus the results are as follows:

7 9 0 13 0 14 2 0

This means that there are seven stressed syllables in the first syllable position of the phrase, nine stressed syllables in the second syllable position, and so on. These results suggest the following rhythmic structure: medium-strong-weak-strong-weak-strong-weak-weak. By way of conclusion, it appears that this work has a strongly rhythmic text structure -- implying that this `chant' might have been sung rhythmically.

The following script will generate our concordance file. For each file specified in the input, we extract the **silbe spine and store it. We then process this spine no less than three times. In the first pass, we translate from the **silbe to the **text representation, and generate a context of 5 words (-n 5) making sure to omit barlines (-o =). We also pad the amalgamated line with three null tokens (-p 3) so the context is centered near the third word in the sequence. In the second pass, we generate a new spine (**nums) that contains only bar numbers. The ditto command is used to ensure that every data record contains a bar number. To ensure that pick-up bars are numbered with the value 0, we've used humsed to replace any leading null-tokens with the number 0. In the third pass, we replace every data token with the name of the file. Finally, we assemble all three of these spines, eliminate everything but data records, and also eliminate lines that don't contain any text. All of this processing is carried out in a while-loop that cycles through all of the files provided when the command is invoked.

while [ $# -ne 0 ]
do

extract -i '**silbe' $1 > temp1
text temp1 | context -o = -n 5 -p 3 > temp2
num -n = -a '**nums' temp1 | extract -i '**nums' \

| ditto | humsed 's/\./0/' > temp3

humsed "s/.*/$1/" temp1 > temp4
assemble temp4 temp3 temp2 | rid -GLId | sed '/.* \.$/d'
shift;

done
rm temp[1-4]

Having generated our concordance file, we can now create a simple tool that allows us to search for keywords. Suppose we kept our concordance information in a file called ~/home/concord/master. In essence, we'd like to create a command akin to grep -- but one that searches this file solely according to the third word in the in the context. We cannot use grep directly since it will find all occurrences of a word no matter where it occurs in the context. We need to tell grep to ignore all other data. The filename, bar number, and context fields are separated by tabs. We can ignore the first two fields by eliminating everything up to the last tab in the line. Since words are separated by blank space, the expression [^ ]+ will match a word not containing spaces. In short, the regular expression "^.*tab[^ ]+ [^ ]+ " will match everything up to the first tab, followed by two additional words. All we need to do is paste our keyword to the end of this expression.

Below is a simple one-line script for a command called keyword. The user simply types the command keyword followed by a regular expression that will allow him/her to search for a given word in context. Note that since we've used the extended regular expression character `+' -- we must invoke egrep rather than grep in our script:

# KEYWORD - A script for searching a master concordance file
#
# Usage: keyword
#
egrep "^.* [^ ]+ [^ ]+ $1" ~/home/concord/master

Concordances can be used for a number of applications. One might use a concordance to help identify metaphor or image related words (such as "light," "darkness," etc.)

First we transform and isolate the text data using the text and extract commands:

text inputfile | extract -i '**text'

Since the input may contain multiple-stops, we might consider the precaution of ensuring no more than one word per data record. For this we can use humsed. Specifically, we can replace any spaces by a carriage return. Since the carriage return is interpreted by the shell as the instruction to begin executing a command, we need to escape it. Depending on the shell, the carriage return can be escaped in various ways. One way is to precede the carriage return by control-V (meaning "verbatim"). Another way is to type control-M rather than a carriage return. In the following command we have used the backslash to escape a control-M character:

text inputfile | extract -i '**text' | humsed 's/ */\^M/g' \

| egrep -4 '^|(like)|(as)$'

Having ensured that there is no more than one word per line we can now search for a line contain just "like" or "as." The -4 option for egrep causes any matched lines to be output with four preceding and four following lines of context. In addition, an extra line is added consisting of two dashes (--) to segregate each pattern output. That is, for each match, ten lines of output are typically given. In order to generate our final output, we need to transform the linear list of words into a horizontal list where each line represents a single match for "like" or "as."

The context command would enable us to do this. Unfortunately, however, the output from egrep fails to conform to the Humdrum syntax. In particular, adding ^\* to the regular expression will fail to ensure a proper Humdrum output since preceding and following contextual lines will also be output.

The hum command is a special command that takes non-Humdrum input and adds sufficient interpretation records so as to make the input conform to the Humdrum syntax. Typically, this means simply adding a generic initial exclusive interpretation (**A) and a spine-path terminator (*-). If the input contains tabs, then appropriate spines will be added. If the input contains empty lines, then they will be changed to null data records.

text inputfile | extract -i '**text' | humsed 's/ */\^M/g' \

| egrep -4 '^|(like)|(as)$' | hum

Now we can make use of the context command. Each context ends with the double-dash delimiters generated by egrep. The rid command can be used to eliminate the interpretations added by hum.

text inputfile | extract -i '**text' | humsed 's/ */\^M/g' \

| egrep -4 '^|(like)|(as)$' | hum | context -e '--' \
| rid -Id

First we translate any pitch data to **semits and any **silbe data to **text. We will also filter the outputs to ensure that only **semits and **text are present.

semits * | text | extract -i '**semits,**text'

Since a word may be sustained through more than one pitch, and a pitch may be intoned for more than one word, we should use the ditto command to ensure that null tokens are filled-in.

semits * | text | extract -i '**semits,**text' | ditto -s =

Next, we can use egrep to search for the words of interest:

semits * | text | extract -i '**semits,**text' | ditto -s = \

| egrep -i '^\*|high|hoch|haut'

Notice the addition of the expression ^\* in the search pattern. This expression will match any Humdrum interpretation records and so ensures that the output conforms to the Humdrum syntax. We can now isolate the **semits data and pass the output to stats in order to determine the average pitch for the words coinciding with the words high/hoch/haut:

semits * | text | extract -i '**semits,**text' | ditto -s = \

| egrep -i '^\*|high|hoch|haut' | extract -i '**semits' | stats

The average pitch for the entire work can be determined as follows:

semits * | extract -i '**semits' | ditto -s = | rid -GLI \

| stats

The humsed command provides an appropriate place to start. In effect, we would take a table (such as Table 26.4) and use it to create a series of substitutions. Emotionally-charged words would be replaced by a numerical rating. Our humsed script would have the following form. Notice that the first substitution is used to eliminate punctuation marks.

s/[.,;:'`"!?]//g

s/begin/+3.8/

s/river/+4.2/

s/friend/+5.2/

s/love/+8.6/

s/hate/-9.7/

s/detest/-9.8/

/[^0-9+-]/s/.*/./

In order to process our input, any syllabic text would first be translated to the **text representation, and all other spines discarded using extract -i.

text inputfile | extract -i '**text' ...

Then we would translate the words using our "emotionality" script, eliminate everything other than data records, and calculate the numerical statistics:

text inputfile | extract -i '**text' | humsed -f emotion \

| rid -GLId | stats

In general, works whose lyrics express predominantly positive emotions ought to exhibit positive emotionality estimates. Similarly, works expressing predominantly negative emotions ought to exhibit negative emotionality estimates. Of course the process of averaging may be deceptive. Two sorts of problems may arise. First, a large number of fairly neutral words will tend to dilute an otherwise large positive or negative score. It may be preferable to observe the maximum positive and negative values. Alternatively, it may be appropriate to limit the average to (say) the ten most emotionally charged words. We can do this by sorting the numerical values and using the head and tail commands to select the highest or lowest values. In our revised processing, we use sort -n to sort the values in numerical order -- placing the output in a temporary file. The UNIX head command allows us to access a specified number of lines at the beginning of a file: the option -5 specifies the first five lines. Similarly, the UNIX tail command allows us to access a specified number of lines at the end of a file. The ten highest and lowest values are then concatenated together and piped to the stats command:

text inputfile | extract -i '**text' | humsed -f emotion \

| rid -GLId | sort -n > temp

head -5 temp > lowest
tail -5 temp > highest
cat highest lowest | stats

A second problem with averaging together emotion rating values is that an emotionally-charged work might include a rough balance of passionate words expressing both positive and negative emotions. This might result in an average near zero and be mistaken for lyrics that exhibit little emotionality. The stats command outputs a variance measure that can be used to gauge the spread of the data. However, another way to address this problem is by ignoring the plus and minus signs in the input. That is, a rough index of emotionality -- independent of whether the emotion is predominantly negative or positive would simply focus on the most emotionally charged words.

The plus and minus signs can be eliminated using a simple humsed substitution prior to numerical sorting:

humsed 's/[+-]//g'

Once again, we could use the head command to isolate the 10 or 20 most emotionally charged words.

Another variant of this approach might be to identify those words in a text which are most emotionally charged. Suppose we wanted to determine the location of the most emotionally charged word. A combination of sort and grep can be applied to this task. First we generate a spine containing the emotional-charge values taking care to eliminate the signs:

text inputfile | extract -i '**text' | humsed -f emotion \

| humsed 's/[+-]//g' > charges

Next we assemble this new spine with the original input:

assemble charges inputfile

We can isolate data records using rid and then use sort -n to sort according to the numbers present in the first column. The most emotional charged word will be at the end of the file (largest number) so we can use tail -1 to identify the word:

assemble charges inputfile | rid -GLId | sort -n | tail -1

Having established what word has been estimated as having the highest emotional-charge, we can then use grep -n to establish the location(s) of this word in the original input file.

Apart from emotionality, language tends to be used differently in different musical genres. The contrast between aria and recitative provides a classic example. The aria is intended to be a poetic reflection of a certain emotional state or reaction whereas the recitative moves the action along by focusing on concrete circumstances. Any number of variants on our "emotionality" processing can be conceived. For example, we might create another language index related to the degree of abstraction/concreteness for words. Words such as Verona, knife and Montague are comparatively concrete, whereas words such as feud, love and tragedy are more conceptual or abstract. We might expect to be able to observe such differences in recitative versus aria texts.

In this chapter we have introduced two text-related representations: **text and **silbe. We have examined the text command (which translates from **silbe to **text). We have also been exposed to the UNIX fmt command (a simple text formatter), the cut command (similar to extract -f), and the head and tail commands.

In Chapter 33 we will examine further representations and processes related to phonetic data.

• Next Chapter
• Previous Chapter
• Table of Contents
• Detailed Contents
  
  © Copyright 1999 David Huron