Difference between revisions of "Dedupe 2.0"

Revision as of 13:36, 29 March 2013

Dedupe 2.0: Group 1

Section 2.0 of the dedupe profile will guide you through the verification and parameters used for hitting on the 010/020/022 fields.

Group 1 Hit Fields

Grouping allows the user to have different parameters for different potential matches. Since numeric fields such as the LCCN, ISBN, and ISSN are fairly reliable in most cases, they are grouped together and will have the same verification parameters.

Library of Congress Control Number (LCCN/010)

The Library of Congress control number assigned to a catalogued item is recorded in the 010 tag. This number is used to to distinguish each record from every other record in the database Library of Congress database.

The LCCN has three parts: the prefix, a year (represented by two or four digits) and a serial number (six digits) followed by another space in the case of pre-2001 LCCNs. Suffixes and revision dates following some printed LCCNs may or may-not be keyed into MARC record.

The following subfields are valid in the 010 tag:

• a - LC control number
• b - NUCMC control number -- This subfield is used only in archival/manuscripts format
• z - Cancelled/invalid LC control number

 010 $a___91001938_
 010 $a__2001012884
 010 $a___08000123_$z___80000123_

International Standard Book Number (ISBN/020)

This field records the International Standard Book Number(s) assigned to a catalogued item. Each valid ISBN is entered in a separate 020 tag; two or more invalid or cancelled ISBNs may be recorded in a single 020 tag.

The following subfields are valid in the 020 tag:

• a - International Standard Book Number
• c - Terms of availability
• z - Cancelled/invalid ISBN

Valid ISBNs are always ten or thirteen digits long; all ISBNs are assumed valid unless they have too many or too few digits, or unless a shelflist card specifically identifies an ISBN as cancelled or invalid:

 020 $a0049812187
 020 $a9780049853217

The only letter that is ever part of an ISBN is X (roman numeral 10); it must always be capitalized:

 020 $a012817409X

Subfield a may contain qualifying information (publisher, binding, format, volume numbers). This information is usually entered within parentheses; separate pieces of information with space-colon-space:

 020 $a001281947X (pbk.)
 020 $a0018942113 (Bally Bros. : pbk.)
 020 $a0137183911 (large print)

Prices appearing after ISBNs are catalogued in subfield c:

 020 $a0174620684 :$c$21.95
 020 $a0049812187 (pbk.) :$c$17.40

International Standard Serial Number (ISSN/022)

An International Standard Serial Number (ISSN) is an identification number assigned to a serial (the entire serial, not just a particular issue). The ISSN is similar in function to the ISBN assigned to books. The 022 tag is repeated whenever a serial has two or more valid ISSNs. This sometimes happens when a serial changes its title and a new ISSN is assigned; when a record is created for the new title, both the new and the old ISSNs (both still valid) are entered in the new record.

The following subfields are valid in the 022 tag:

• a - International Standard Serial Number
• l - ISSN-L
• m - Cancelled ISSN-L
• y - Incorrect ISSN
• z - Cancelled ISSN

This field only contains digits except the last digit may be X (roman numeral 10):

 022 $a1234-5678
 022 $a9876-123X

Subfield l links together various media versions of a continuing resource:

 022 $a1234-5678$▼l1234-1231

Group 1 - Verification Criteria and Terms

The Verify criteria is used to reduce the number of initial matches found using the Hit criteria. This allows you to further define what constitutes a good match based on the existence of other fields within your records. Fields should be selected on the basis of the match-rate you expect to see with the deduplication. Selecting fewer fields will result in more matches; selecting more fields will result in fewer, but better matches. Verify criteria should also be selected in conjunction with Hit criteria.

Verification Terms - Method

There are a few methods for verifying: FULL, PARTIAL, and WITHIN. These methods will be used for comparing data found in a specific field against the same field in a potential match record.

• FULL - Compares the full verify string up to the verify length.
• PARTIAL - Truncates the compare strings to the shortest string, then does a full compare:

 "The fox in and the hound" in one record, "The fox" on the other record: Both truncate to "The fox" and compared.

• WITHIN - Searches each compare string truncated at verify length against the full un-truncated string of the other field:

 "Cat" will verify against "The cat in the hat."

Verification Terms - Normalization

Normalization refers to how the string will be presented when compared to another string. Note that any normalization will not change anything in the record, but is only used when the program compares the strings.

Types of normalization are:

• NACO/CJK - retains spaces and subfield delimiters:

 245 $aDaniel Boone :$ba pioneer. would be normalized as $adaniel boone :$ba pioneer

• FULL - NACO normalization with all spaces and subfield delimiters removed:

 245 $aDaniel Boone :$ba pioneer. would be normalized as danielbooneapioneer

Verification Terms - LENGTH and WORDS

This refers to how much of a given string the program will present for potential matches:

• LENGTH - Refers to the number of characters for the verify field. The number of characters to be used is 1-2048, or all:

 Using 10 for LENGTH would truncate 245 $aDaniel Boone :$ba pioneer. to Daniel Boon if FULL method was used.

• WORDS - Refers to a count of words to match within a given string:

 Using 2 words for 245 $Daniel Boone :$ba pioneer returns any of the words as keywords for match possibilities.

NOTE: using WORDS will not include non-filers.

Verification Terms - Must Verify

If this option is used for any given field, then that verify has to verify or it is not considered a match. This is almost always used for the 245verification and is common in other fields as well

Verification Terms - Only if Both

This only does a verify comparison if both records have a specified field; verifies as true if only one of the records has the field. If this option was used on the 1xx field, the following would be true: Example 1:

 Record A has 100$aTwain, Mark.
 -- 245$aAdventures of Huckleberry Finn.
 Record B has no 100
 -- 245$aAdventures of Huckleberry Finn.

RESULT: Example 1 would be a match because the 100 exists in one but not the other.

Example 2:

 Record A has 100$aTwain, Mark.
 -- 245$aAdventures of Huckleberry Finn.
 Record B has 100$Clemens, Samuel.
 -- 245$aAdventures of Huckleberry Finn.

RESULT: Example 2 would not be a match because the 100 differs.

links