Python Standard Library "re – Regular Expressions" 筆記整理
來源 http://pymotw.com/2/re/index.html
https://docs.python.org/2/howto/regex.html#regex-howto
https://docs.python.org/2/library/re.html#re.MatchObject
re – Regular Expressions¶
| 功能: | 整理文字後變成所要的格式作利用 |
|---|---|
| Python 版本: | 1.5 之後都支援 |
Regular expressions are text matching patterns described with a formal syntax. The patterns are interpreted as a set of instructions, which are then executed with a string as input to produce a matching subset or modified version of the original. The term “regular expressions” is frequently shortened to as “regex” or “regexp” in conversation. Expressions can include literal text matching, repetition, pattern-composition, branching, and other sophisticated rules. A large number of parsing problems are easier to solve with a regular expression than by creating a special-purpose lexer and parser.
Regular expressions are typically used in applications that involve a lot of text processing. For example, they are commonly used as search patterns in text editing programs used by developers, including vi, emacs, and modern IDEs. They are also an integral part of Unix command line utilities such as sed, grep, and awk. Many programming languages include support for regular expressions in the language syntax (Perl, Ruby, Awk, and Tcl). Other languages, such as C, C++, and Python supports regular expressions through extension libraries.
There are multiple open source implementations of regular expressions, each sharing a common core syntax but with different extensions or modifications to their advanced features. The syntax used in Python’s re module is based on the syntax used for regular expressions in Perl, with a few Python-specific enhancements.
Note
Although the formal definition of “regular expression” is limited to expressions that describe regular languages, some of the extensions supported by re go beyond describing regular languages. The term “regular expression” is used here in a more general sense to mean any expression that can be evaluated by Python’s re module.
Finding Patterns in Text[尋找文字作比對]
The most common use for re is to search for patterns in text. This example looks for two literal strings, 'this' and 'that', in a text string.
search() takes the pattern and text to scan, and returns a Match object when the pattern is found. If the pattern is not found, search() returns None.
The Match object returned by search() holds information about the nature of the match, including the original input string, the regular expression used, and the location within the original string where the pattern occurs.
The start() and end() methods give the integer indexes into the string showing where the text matched by the pattern occurs.
Compiling Expressions
re includes module-level functions for working with regular expressions as text strings, but it is usually more efficient to compile the expressions your program uses frequently. The compile() function converts an expression string into aRegexObject.
The module-level functions maintain a cache of compiled expressions, but the size of the cache is limited and using compiled expressions directly means you can avoid the cache lookup overhead. By pre-compiling any expressions your module uses when the module is loaded you shift the compilation work to application startup time, instead of a point where the program is responding to a user action.
Multiple Matches
So far the example patterns have all used search() to look for single instances of literal text strings. The findall() function returns all of the substrings of the input that match the pattern without overlapping.
There are two instances of ab in the input string.
finditer() returns an iterator that produces Match instances instead of the strings returned by findall().
This example finds the same two occurrences of ab, and the Match instance shows where they are in the original input.
Pattern Syntax
Regular expressions support more powerful patterns than simple literal text strings. Patterns can repeat, can be anchored to different logical locations within the input, and can be expressed in compact forms that don’t require every literal character be present in the pattern. All of these features are used by combining literal text values with metacharacters that are part of the regular expression pattern syntax implemented by re. The following examples will use this test program to explore variations in patterns.
The output of test_patterns() shows the input text, including the character positions, as well as the substring range from each portion of the input that matches the pattern.
Repetition
There are five ways to express repetition in a pattern. A pattern followed by the metacharacter * is repeated zero or more times (allowing a pattern to repeat zero times means it does not need to appear at all to match). Replace the * with +and the pattern must appear at least once. Using ? means the pattern appears zero or one time. For a specific number of occurrences, use {m} after the pattern, where m is replaced with the number of times the pattern should repeat. And finally, to allow a variable but limited number of repetitions, use {m,n} where m is the minimum number of repetitions and n is the maximum. Leaving out n ({m,}) means the value appears at least m times, with no maximum.
Notice how many more matches there are for ab* and ab? than ab+.
The normal processing for a repetition instruction is to consume as much of the input as possible while matching the pattern. This so-called greedy behavior may result in fewer individual matches, or the matches may include more of the input text than intended. Greediness can be turned off by following the repetition instruction with ?.
Disabling greedy consumption of the input for any of the patterns where zero occurences of b are allowed means the matched substring does not include any b characters.
Character Sets
A character set is a group of characters, any one of which can match at that point in the pattern. For example, [ab] would match either a or b.
The greedy form of the expression, a[ab]+, consumes the entire string because the first letter is a and every subsequent character is either a or b.
A character set can also be used to exclude specific characters. The special marker ^ means to look for characters not in the set following.
This pattern finds all of the substrings that do not contain the characters -, ., or a space.
As character sets grow larger, typing every character that should (or should not) match becomes tedious. A more compact format using character ranges lets you define a character set to include all of the contiguous characters between a start and stop point.
Here the range a-z includes the lower case ASCII letters, and the range A-Z includes the upper case ASCII letters. The ranges can also be combined into a single character set.
As a special case of a character set the metacharacter dot, or period (.), indicates that the pattern should match any single character in that position.
Combining dot with repetition can result in very long matches, unless the non-greedy form is used.
Escape Codes
An even more compact representation uses escape codes for several pre-defined character sets. The escape codes recognized by re are:
| Code | Meaning |
|---|---|
| \d | a digit |
| \D | a non-digit |
| \s | whitespace (tab, space, newline, etc.) |
| \S | non-whitespace |
| \w | alphanumeric |
| \W | non-alphanumeric |
Note
Escapes are indicated by prefixing the character with a backslash (\). Unfortunately, a backslash must itself be escaped in normal Python strings, and that results in expressions that are difficult to read. Using raw strings, created by prefixing the literal value with r, for creating regular expressions eliminates this problem and maintains readability.
These sample expressions combine escape codes with repetition to find sequences of like characters in the input string.
To match the characters that are part of the regular expression syntax, escape the characters in the search pattern.
These patterns escape the backslash and plus characters, since as metacharacters both have special meaning in a regular expression.
Anchoring
In addition to describing the content of a pattern to match, you can also specify the relative location in the input text where the pattern should appear using anchoring instructions.
| Code | Meaning |
|---|---|
| ^ | start of string, or line |
| $ | end of string, or line |
| \A | start of string |
| \Z | end of string |
| \b | empty string at the beginning or end of a word |
| \B | empty string not at the beginning or end of a word |
The patterns in the example for matching words at the beginning and end of the string are different because the word at the end of the string is followed by punctuation to terminate the sentence. The pattern \w+$ would not match, since . is not considered an alphanumeric character.
Constraining the Search
In situations where you know in advance that only a subset of the full input should be searched, you can further constrain the regular expression match by telling re to limit the search range. For example, if your pattern must appear at the front of the input, then using match() instead of search() will anchor the search without having to explicitly include an anchor in the search pattern.
Since the literal text is does not appear at the start of the input text, it is not found using match(). The sequence appears two other times in the text, though, so search() finds it.
The search() method of a compiled regular expression accepts optional start and end position parameters to limit the search to a substring of the input.
This example implements a less efficient form of iterall(). Each time a match is found, the end position of that match is used for the next search.
Dissecting Matches with Groups
Searching for pattern matches is the basis of the powerful capabilities provided by regular expressions. Adding groups to a pattern lets you isolate parts of the matching text, expanding those capabilities to create a parser. Groups are defined by enclosing patterns in parentheses (( and )).
Any complete regular expression can be converted to a group and nested within a larger expression. All of the repetition modifiers can be applied to a group as a whole, requiring the entire group pattern to repeat.
To access the substrings matched by the individual groups within a pattern, use the groups() method of the Match object.
Match.groups() returns a sequence of strings in the order of the group within the expression that matches the string.
If you are using grouping to find parts of the string, but you don’t need all of the parts matched by groups, you can ask for the match of only a single group with group().
Group 0 represents the string matched by the entire expression, and sub-groups are numbered starting with 1 in the order their left parenthesis appears in the expression.
Python extends the basic grouping syntax to add named groups. Using names to refer to groups makes it easier to modify the pattern over time, without having to also modify the code using the match results. To set the name of a group, use the syntax (P?<name>pattern).
Use groupdict() to retrieve the dictionary mapping group names to substrings from the match. Named patterns are included in the ordered sequence returned by groups(), as well.
An updated version of test_patterns() that shows the numbered and named groups matched by a pattern will make the following examples easier to follow.
Since a group is itself a complete regular expression, groups can be nested within other groups to build even more complicated expressions.
In this case, the group (a*) matches an empty string, so the return value from groups() includes that empty string as the matched value.
Groups are also useful for specifying alternative patterns. Use | to indicate that one pattern or another should match. Consider the placement of the | carefully, though. The first expression in this example matches a sequence of a followed by a sequence consisting entirely of a single letter, a or b. The second pattern matches a followed by a sequence that may include either a or b. The patterns are similar, but the resulting matches are completely different.
When an alternative group is not matched, but the entire pattern does match, the return value of groups() includes a None value at the point in the sequence where the alternative group should appear.
Defining a group containing a sub-pattern is also useful in cases where the string matching the sub-pattern is not part of what you want to extract from the full text. These groups are called non-capturing. To create a non-capturing group, use the syntax (?:pattern).
Compare the groups returned for the capturing and non-capturing forms of a pattern that matches the same results.
Search Options
You can change the way the matching engine processes an expression using option flags. The flags can be combined using a bitwise or operation, and passed to compile(), search(), match(), and other functions that accept a pattern for searching.
Case-insensitive Matching
IGNORECASE causes literal characters and character ranges in the pattern to match both upper and lower case characters.
Since the pattern includes the literal T, without setting IGNORECASE the only match is the word This. When case is ignored, text also matches.
Input with Multiple Lines
There are two flags that effect how searching in multi-line input works. The MULTILINE flag controls how the pattern matching code processes anchoring instructions for text containing newline characters. When multiline mode is turned on, the anchor rules for ^ and $ apply at the beginning and end of each line, in addition to the entire string.
The pattern in the example matches the first or last word of the input. It matches line. at the end of the string, even though there is no newline.
DOTALL is the other flag related to multiline text. Normally the dot character . matches everything in the input text except a newline character. The flag allows dot to match newlines as well.
Without the flag, each line of the input text matches the pattern separately. Adding the flag causes the entire string to be consumed.
Unicode
Under Python 2, str objects use the ASCII character set, and regular expression processing assumes that the pattern and input text are both ASCII. The escape codes described earlier are defined in terms of ASCII by default. Those assumptions mean that the pattern \w+ will match the word “French” but not “Français”, since the ç is not part of the ASCII character set. To enable Unicode matching in Python 2, add the UNICODE flag when compiling the pattern.
The other escape sequences (\W, \b, \B, \d, \D, \s, and \S) are also processed differently for Unicode text. Instead of assuming the members of the character set identified by the escape sequence, the regular expression engine consults the Unicode database to find the properties of each character.
Note
Python 3 uses Unicode for all strings by default, so the flag is not necessary.
Verbose Expression Syntax
The compact format of regular expression syntax can become a hindrance as expressions grow more complicated. As the number of groups in your expression increases, you will have trouble keeping track of why each element is needed and how exactly the parts of the expression interact. Using named groups helps mitigate these issues, but a better solution is to use verbose mode expressions, which allow you to add comments and extra whitespace.
A pattern to validate email addresses will illustrate how verbose mode makes working with regular expressions easier. The first version recognizes addresses that end in one of three top-level domains, .com, .org, and .edu.
This expression is already complex. There are several character classes, groups, and repetition expressions.
Converting the expression to a more verbose format will make it easier to extend.
The expression matches the same inputs, but in this extended format it is easier to read. The comments also help identify different parts of the pattern so that it can be expanded to match more inputs.
This expanded version parses inputs that include a person’s name and email address, as might appear in an email header. The name comes first and stands on its own, and the email address follows surrounded by angle brackets (< and >).
As with other programming languages, the ability to insert comments into verbose regular expressions helps with their maintainability. This final version includes implementation notes to future maintainers and whitespace to separate the groups from each other and highlight their nesting level.
Embedding Flags in Patterns
In situations where you cannot add flags when compiling an expression, such as when you are passing a pattern to a library function that will compile it later, you can embed the flags inside the expression string itself. For example, to turn case-insensitive matching on, add (?i) to the beginning of the expression.
Because the options control the way the entire expression is evaluated or parsed, they should always come at the beginning of the expression.
The abbreviations for all of the flags are:
| Flag | Abbreviation |
|---|---|
| IGNORECASE | i |
| MULTILINE | m |
| DOTALL | s |
| UNICODE | u |
| VERBOSE | x |
Embedded flags can be combined by placing them within the same group. For example, (?imu) turns on case-insensitive matching for multiline Unicode strings.
Looking Ahead, or Behind
There are many cases where it is useful to match a part of a pattern only if some other part will also match. For example, in the email parsing expression the angle brackets were each marked as optional. Really, though, the brackets should be paired, and the expression should only match if both are present, or neither are. This modified version of the expression uses a positive look ahead assertion to match the pair. The look ahead assertion syntax is (?=pattern).
There are several important changes in this version of the expression. First, the name portion is no longer optional. That means stand-alone addresses do not match, but it also prevents improperly formatted name/address combinations from matching. The positive look ahead rule after the “name” group asserts that the remainder of the string is either wrapped with a pair of angle brackets, or there is not a mismatched bracket; the brackets are either both present or neither is. The look ahead is expressed as a group, but the match for a look ahead group does not consume any of the input text, so the rest of the pattern picks up from the same spot after the look ahead matches.
A negative look ahead assertion ((?!pattern)) says that the pattern does not match the text following the current point. For example, the email recognition pattern could be modified to ignore noreply mailing addresses commonly used by automated systems.
The address starting noreply does not match the pattern, since the look ahead assertion fails.
Instead of looking ahead for noreply in the username portion of the email address, the pattern can also be written using a negative look behind assertion after the username is matched using the syntax (?<!pattern).
Looking backwards works a little differently than looking ahead, in that the expression must use a fixed length pattern. Repetitions are allowed, as long as there is a fixed number (no wildcards or ranges).
A positive look behind assertion can be used to find text following a pattern using the syntax (?<=pattern). For example, this expression finds Twitter handles.
The pattern matches sequences of characters that can make up a Twitter handle, as long as they are preceded by an @.
Self-referencing Expressions
Matched values can be used in later parts of an expression. For example, the email example can be updated to match only addresses composed of the first and last name of the person by including back-references to those groups. The easiest way to achieve this is by referring to the previously matched group by id number, using \num.
Although the syntax is simple, creating back-references by numerical id has a couple of disadvantages. From a practical standpoint, as the expression changes, you must count the groups again and possibly update every reference. The other disadvantage is that only 99 references can be made this way, because if the id number is three digits long it will be interpreted as an octal character value instead of a group reference. On the other hand, if you have more than 99 groups in your expression you will have more serious maintenance challenges than not being able to refer to some of the groups in the expression.
Python’s expression parser includes an extension that uses (?P=name) to refer to the value of a named group matched earlier in the expression.
The address expression is compiled with the IGNORECASE flag on, since proper names are normally capitalized but email addresses are not.
The other mechanism for using back-references in expressions lets you choose a different pattern based on whether or not a previous group matched. The email pattern can be corrected so that the angle brackets are required if a name is present, and not if the email address is by itself. The syntax for testing to see if a group has matched is (?(id)yes-expression|no-expression), where id is the group name or number, yes-expression is the pattern to use if the group has a value and no-expression is the pattern to use otherwise.
This version of the email address parser uses two tests. If the name group matches, then the look ahead assertion requires both angle brackets and sets up the brackets group. If name is not matched, the assertion requires the rest of the text not have angle brackets around it. Later, if the brackets group is set, the actual pattern matching code consumes the brackets in the input using literal patterns, otherwise it consumes any blank space.
Modifying Strings with Patterns
In addition to searching through text, re also supports modifying text using regular expressions as the search mechanism, and the replacements can reference groups matched in the regex as part of the substitution text. Use sub() to replace all occurances of a pattern with another string.
References to the text matched by the pattern can be inserted using the \num syntax used for back-references above.
To use named groups in the substitution, use the syntax \g<name>.
The \g<name> syntax also works with numbered references, and using it eliminates any ambiguity between group numbers and surrounding literal digits.
Pass a value to count to limit the number of substitutions performed.
Only the first substitution is made because count is 1.
subn() works just like sub() except that it returns both the modified string and the count of substitutions made.
The search pattern matches twice in the example.
Splitting with Patterns
str.split() is one of the most frequently used methods for breaking apart strings to parse them. It only supports using literal values as separators, though, and sometimes a regular expression is necessary if the input is not consistently formatted. For example, many plain text markup languages define paragraph separators as two or more newline (\n) characters. In this case, str.split() cannot be used because of the “or more” part of the definition.
A strategy for identifying paragraphs using findall() would use a pattern like (.+?)\n{2,}.
That pattern fails for paragraphs at the end of the input text, as illustrated by the fact that “Paragraph three.” is not part of the output.
Extending the pattern to say that a paragraph ends with two or more newlines, or the end of input, fixes the problem but makes the pattern more complicated. Converting to re.split() instead of re.findall() handles the boundary condition automatically and keeps the pattern simple.
The pattern argument to split() expresses the markup specification more precisely: Two or more newline characters mark a separator point between paragraphs in the input string.
Enclosing the expression in parentheses to define a group causes split() to work more like str.partition(), so it returns the separator values as well as the other parts of the string.
The output now includes each paragraph, as well as the sequence of newlines separating them.
