User:Fæ/LOC maps
Introduction edit
Map uploads from the Library of Congress "portal" for maps at https://www.loc.gov/maps. Requested by Jheald in April 2018.
All maps appear in Category:Maps in the Library of Congress, which pre-batch upload started with 602 images. A report of files in this batch upload project is Petscan, this may lag the live data by a day. This search shows only jpegs with LOC in the title in the bucket category, which gives a more relevant total count of uploads than both jpegs and tiffs.
GLAM dashboard reports are available:
The methods applied for the upload are similar to those discussed at the LOC Prints project page.
Useful searches of the maps collection edit
- 18th Century or earlier maps
- 2005 or more recent maps
- National Park Service maps, including many infographics and posters
- Political and administrative maps
- Central Intelligence Agency maps
Technical edit
Formats edit
Images are pulled using iiif url requests. In theory these can return many formats including jp2, png, gif and tiff. In practice the requests for full size png formats may indefinitely return server 503 errors, presumably because the pngs are not held natively and are probably transcoded on the fly from a jp2 source. Consequently tiff is requested, which unfortunately also frequently gives 503 errors.
As tiffs are rendered by Wikimedia Commons poorly, a jpeg format is created in parallel and cross-linked by a gallery on the image page.
Known errors edit
- 503 - Due to the potential indefinite nature of 503 errors from the LOC iiif service, which may be related to file size or resolution, after a dozen upload failures there is a final attempt to upload the jpeg format alone. Example LOC 2009582412
- empty-file - This is a regular error flagged by the Commons API, most often this is corrected by a repeated attempt to upload. Cases where the failure persists may be due to large image sizes, making the tif creation time-out, though the repeated attempts means this takes 20 minutes of processing time before the tif is skipped. In those cases a jpg will still be uploaded, example jpg @ 24,596 × 9,033 pixels where the identical tif is shown as being a 635MB file at LoC. A default work-around of skipping tif uploads for images greater than 162 megapixels has been added for multisheet maps as there appears to be a 100% failure rate above that size
- Poor truncation File names are deduced from the map title metadata. Where over 200 characters, the string is truncated by a word at a time until it is realistically short. Unfortunately it may be broken mid-sentence such as Jamaica, particulars and conditions of sale of valuable Sugar Estate - known as "Trinity Plantation," "Roslyn Pen," "Cromwell Plantation," and Cromwell Mountain," all sutuate in the Parish of St. where the original title is over 600 characters and the automatic truncation breaks "St. Mary". In the cases of multiple sheet documents the automatic category may have a poorly broken name, and where confusing will need a manual fix
Upload configuration edit
Copyright edit
Based on the LOC description of the collection, the maps either date from before 1923, or are US Government works. Consequently the basic licenses are chosen as:
if year<1918:
license = "PD-Art|PD-old-100-1923"
elif year<1923:
license = "PD-Art|PD-old-80-1923"
else:
license = "PD-USGov"
The generic PD-USGov may be refined automatically by the pre-existing maintenance task User:Fæ/code/PD-USGov.
Categorisation edit
There are two forms of category checked, place and date:
- Old maps of <place>
- <year> maps
If these exist as categories, they are added to the file. <place> alternatives are found by checking for location_city, location_county, location_state and location_country. For multiple sheet maps, the metadata has a 'location' array, which unfortunately appears unpredictably sorted, consequently the place category may not be the most local possible.
A separate housekeeping task is picking up on categories which are generated from U.S. location names but accidentally match to existing categories for non-U.S. locations. The mappings are:
mapping = [
["Birmingham|Alabama", "Old maps of Alabama"],
["Birmingham|Connecticut", "Old maps of Connecticut"],
["Bristol|Connecticut", "Old maps of cities in Connecticut"],
["Derby|Connecticut","Old maps of cities in Connecticut"],
["Norwich|Connecticut", "Old maps of Connecticut"],
["Salisbury|Connecticut", "Old maps of Connecticut"],
["Athens|Georgia", "Old maps of Georgia (U.S. state)"],
["Georgia", "Old maps of Georgia (U.S. state)"],
["Lincoln|Illinois", "Old maps of Illinois"],
["Attica|Indiana", "Old maps of Indiana"],
# M
["Cumberland|Maryland", "Old maps of cities in Maryland"],
["North East England|Maryland", "Old maps of cities in Maryland"],
["Belfast|Maine", "Old maps of Maine"],
["Cambridge|Maryland", "Old maps of Maryland"],
["Bedford|Massachusetts", "Old maps of Massachusetts"],
["Cambridge|Massachusetts", "Old maps of Cambridge, Massachusetts"],
["Chester|Massachusetts", "Old maps of Massachusetts"],
["Gloucester|Massachusetts", "Old maps of Massachusetts"],
["Ipswich|Massachusetts","Old maps of Massachusetts"],
["Worcester|Massachusetts", "Old maps of Massachusetts"],
["Versailles|Missouri", "Old maps of Missouri"],
# N
["Bristol|New Hampshire", "Old maps of New Hampshire"],
["Canterbury|New Hampshire", "Old maps of New Hampshire"],
["Exeter|New Hampshire", "Old maps of New Hampshire"],
["Portsmouth|New Hampshire", "Old maps of New Hampshire"],
["Lincoln|Nebraska", "Old maps of Nebraska"],
["Antwerp|New York", "Old maps of New York (state)"],
["Cambridge|New York", "Old maps of New York (state)"],
["Dundee|New York", "Old maps of New York (state)"],
["Rome|New York", "Old maps of New York"],
["Salamanca|New York", "Old maps of New York"],
["Syracuse|New York", "Old maps of New York (state)"],
["Cambridge|Ohio", "Old maps of cities in Ohio"],
["Lancaster|Ohio", "Old maps of cities in Ohio"],
["Portsmouth|Ohio", "Old maps of cities in Ohio"],
# P
["Alexandria|Pennsylvania", "Old maps of Pennsylvania"],
["Berlin|Pennsylvania", "Old maps of Pennsylvania"],
["Bradford|Pennsylvania", "Old maps of Pennsylvania"],
["Bristol|Pennsylvania", "Old maps of Pennsylvania"],
["Carlisle|Pennsylvania", "Old maps of Pennsylvania"],
["Cornwall|Pennsylvania", "Old maps of Pennsylvania"],
["Derry|Pennsylvania", "Old maps of Pennsylvania"],
["Hamburg|Pennsylvania", "Old maps of Pennsylvania"],
["Oxford|Pennsylvania", "Old maps of Pennsylvania"],
["Somerset|Pennsylvania", "Old maps of Pennsylvania"],
["Bristol|Rhode Island", "Old maps of Rhode Island"],
# S
["Bristol|Vermont", "Old maps of Vermont"],
["Essex|Vermont", "Old maps of Vermont"],
["Rutland|Vermont", "Old maps of Vermont"],
["Suffolk|Virginia", "Old maps of Virginia"],
["Alexandria|Virginia", "Old maps of Virginia"],
["Bristol|Virginia", "Old maps of Virginia"],
["Portsmouth|Virginia", "Old maps of Virginia"],
["Winchester|Virginia", "Old maps of Virginia"],
["Berlin|Wisconsin", "Old maps of cities in Wisconsin"],
]
Where the mapping is <mismatched category, without "Old maps of " prefix>|<optional wikitext conditional match> → <correct U.S. location category>. The extra conditional match is needed for locations like "Bristol" where the default matched to the U.K. location and Bristol exists as a (realistic LOC metadata) place in Virginia and Connecticut. This test only applies to LOC map files where the Location metadata, added to the wikitext image page, includes "United States".
Singletons edit
Maps are separated in to those with multiple pages and those displayed as a gallery with the flag "hassegments" in the json data. The first run of uploads are solely with singletons to avoid the complexity of categorisation.
During 2018, once the uploads appeared stable, the uploading was put out to an always connected Raspberry Pi device, as an experiment with low-power headless loading, and to reduce demand on a primary desktop.
Multiple sheet maps edit
Multiple sheet maps are added to an automatically created atlas category based on <title>. Example Category:Sanborn Fire Insurance Map from Newberry, Newberry County, South Carolina.
As the way the information template is used varies significantly from singletons, a separate batch upload script is used. Year and place categories are added to the generated atlas category.
As there are a lot of Sanborn maps, the categorization attempts to find a sub-category of Category:Sanborn maps, if there is a relevant match to place metadata.