KCL • CCH • Minor programme • AV1000

AV1000
Fundamentals of the digital humanities
Cleaning up your corpus: an exercise in thinking with rules

The following is an exercise in applying a specialised “language” within Microsoft Word automatically to identify character-strings in a text file and replace them with other character-strings. You should also make use of Word's internal documentation.

Devising and following rules for transforming data

In manipulating alphanumeric data one occasionally needs to transform a large number of identical data-elements in the same way. It is very tedious to do this manually, one instance at a time, so automating the process is quite a good idea. The standard mechanism is called “search-and-replace” or simply “replace”. Wordprocessors and text-editors are usually used for the job, although there are programs specifically written for the task.

Finding the right sequence of commands or rule can be tricky.

The first caution applies to computing in general: the replace command will do exactly what you tell it to do, which is often rather different from what you want it to do. You are therefore well advised to try out a replace operation one instance at a time until you are convinced that there are none which conform to the specifications you have given but are not instances of the thing you want changed. (Word, for example, gives you two buttons in the Replace dialogue-box, “Replace” and “Replace All”; the former changes only the next instance, the later all of them.) Also be sure to save a copy of a file you are changing before you make each series of changes; the Undo command is a great help here, but in a complex series of changes the number of Undos can lead to confusion.

The second caution follows from the first. This is that the way you actually go about transforming your data will often have to be rather different from the way you would ordinarily describe the change. For example, let's say I have manually numbered some items, 1, 2, 3, … 9, 10, then I want to insert a new item 4. Ordinarily I might think first, so as to make room for the new 4, to change the old 4 into 5, the 5 into 6 and so forth up to the 10, which would become 11. But if I do this with the computer exactly as described, the following will happen:

The example given below for you to apply is considerably more complex, but it follows the same principle. The computer scientist Edsger Dijkstra used to tell a story that illustrates it much more memorably; see “A Parable”.

Replace in Word

The simplest replace command in Word specifies the entity to be replaced explicitly, e.g. in English, “change ‘4’ into ‘5’”. Often, however, what you need to do is of the form e.g. “change ‘4’ followed by any character whatever into ‘5: ’”. A symbol devised to represent that “any character whatever” is called a “wildcard”. Wildcards also may specify a more limited set of characters, e.g. “any numerical character”, “any alphabetic character between a and m or “any mark of punctuation”. The wildcards for Word are given below, under “Wildcards in Word”.

Especially with text files you often want to affect the formatting, as in the exercise given below. Then you require special symbols to represent the formatting effects, e.g. for paragraphing, tabs, spaces, page-breaks. These are given under “Wildcards in Word”, items 2 and 3.

Exercise: cleaning up text copied from the Web

The case in point is transforming (i.e. cleaning up) text copied by cut-and-paste from the Web so that you can concord it effectively. The most common problems with such text are multiple contiguous spaces and multiple contiguous EOL (end-of-line) characters, perhaps with one or more spaces intermingled.

The following find-and-replace commands will eliminate these. In Word select Replace under the Edit menu, tick “Use wildcards”, then enter the following in the order given. Note that the symbol • is used here to represent a blank space; you should type the space character. Word will show you the hidden formatting characters if you select Tools: Options: View and then check the right boxes under “Formatting marks”; this will make it easier for you to follow what is happening.

1. Change all multiple, contiguous spaces into a single space:

   Find what: •(•@)

   Replace with: •

2. Change all groups of EOLs intermingled with spaces (an EOL followed by one or more spaces followed by an EOL) into two contiguous EOLs:

   Find what: (^13) •@(^13)

   Replace with: ^p^p

3. Change all multiple, contiguous EOLs into two EOLs:

   Find what: (^13)(^13)@

   Replace with: ^p^p

Can you follow what these symbol-sequences are describing? You may have to experiment and adapt the above to suit the copied text you are attempting to transform. Again, if you turn on display of the ordinarily hidden formatting characters it will be easier to figure out exactly what needs doing.

Wildcards in Word

1. Type wildcards for items you want to find

You can fine-tune a search by using any of the following wildcards. On the Edit menu, click Find or Replace. If you don't see the Use wildcards check box, click More. Then select the Use wildcards check box and type the wildcard character and any other text in the Find what box.

Notes

• You can use parentheses to group the wildcard characters and text and to indicate the order of evaluation. For example, type <(pre)*(ed)> to find "presorted" and "prevented".
• You can use the \n wildcard to search for an expression and then replace it with the rearranged expression. For example, type (Newton) (Christie) in the Find what box and \2 \1 in the Replace with box. Word will find "Newton Christie" and replace it with "Christie Newton".

2. Type codes for items you want to find and replace

You can find and replace the following special characters and document elements. Just click Find or Replace on the Edit menu, and then type codes for the items in the Find what and Replace with boxes. As specified, some codes work only if the Use wildcards option is on or off.

Codes that work in the Find what or Replace with box

Codes that work in the Find what box only

Codes that work in the Find what box only (when wildcards are off)

Codes that work in the Replace with box only

Notes

• When wildcards are on, you can also search for footnote and endnote marks, fields, paragraph marks, section breaks and manual page breaks, or white space. Just type substitute codes in the Find what box.
• If you include the optional hyphen code, Word finds only text with optional hyphens in the position you specified. If you omit the optional hyphen code, Word finds all matching text, including text with optional hyphens.
• If you're searching for fields, you must display field codes. To switch between displaying field codes and field results, click the field, and press SHIFT+F9. To show or hide field codes for all fields in the document, press ALT+F9.
• If you're searching for graphics, Word finds only inline graphics; Word does not find floating graphics. By default, Word inserts imported graphics as inline graphics. How to change floating graphics to inline graphics.

3. When I use wildcard characters, Word can't search for certain items.

To use wildcards, click Find or Replace on the Edit menu, click More, and then select the Use wildcards check box. When this check box is selected, Word doesn't recognize the codes you enter in the Find what box for the following items: endnote and footnote marks, fields, paragraph marks, section breaks, or white space. To search for these items, you can type the following substitute codes in the Find what box. (Note that no substitute code is available for fields.)

revised October 2007