Wikimedia Commons

Commons:Batch uploading/Weather maps

< Commons:Batch uploading

Weather maps edit

The Hydrometeorological Prediction Center provides daily weather maps of the United States from 2003 to the present. These are high-quality and educational, and are able to be used in galleries, articles and other content pages on several projects. All strictly {{PD-USGov}}. This is just a proposal for now, rather than an actual request, to see what folks think. –Juliancolton | Talk 16:54, 19 March 2010 (UTC)[reply]

Opinions edit

I don't bite! :) –Juliancolton | Talk 00:22, 24 May 2010 (UTC)[reply]
If you feel like the images would indeed be useful, then I am happy to upload them to Commons. There are exactly 18490 images to be downloaded, counting from September 1, 2002 until today, October 15, 2012. I'll let you know as soon as I have them on my HDD. odder (talk) 14:20, 15 October 2012 (UTC)[reply]
Looks like the Hydrometeorological Prediction Center doesn't publish its maps in advance, so I only managed to get the maps until October 14, 2012. There are 14,841 .gif files to be uploaded, and they're around 650 MiB in total. odder (talk) 07:50, 19 October 2012 (UTC)[reply]
Taking a look. It would be interesting to do this jointly, we can layout the process as we go along. No hurry on this one, so let's consider this a slow and careful project.  . --Fæ (talk) 14:15, 21 May 2013 (UTC)[reply]
I think there are five different files here, for (almost) every day since September 1, 2002: maxmin, colormaxmin, dwm500_test, precip and stnplot. (Two files from August 10, 2005 are missing, though: 1, 2). odder (talk) 15:27, 21 May 2013 (UTC)[reply]

Coordination edit

  1. Identify key fields. I'm not sure what can be drawn from the source. Source file names look like "colormaxmin_20020901.gif" and there appear to be 4 different types of graphic that can be uploaded. So I guess the variables than can be pulled into a data ingestion template on upload are just:
    1. {date} the day the graph represents (inferred from the file name)
    2. {type} the graph type (also inferred from the file name), this may be transcluded with a standard plain English explanation of one of the five file types:
      1. maxmin - Max-min Temperature Map
      2. colormaxmin - Color Max-min Temperature Map
      3. dwm500_test - 500-Millibar Height Contour Map
      4. precip - 24-hr Precipitation Map
      5. stnplot - Surface Weather Map
      Static text can include the permission statement from source, the standard source URL where the images can be looked up by date, the equivalent of author and any credits that are appropriate.
    Consequently if I upload from odder's cache of images, then there is no need to data-mine the source website. Other suggestions? --Fæ (talk) 14:46, 21 May 2013 (UTC)[reply]
  2. Set up ingestion template. Draft started at User:Fæ/dailywxmap. Moved to Template:NOAA-dailywxmap
  3. Agree standard filename structure. Perhaps "Weather map <ISOdate> <typename> NOAA.png"?
    How about <ISO date> <typename> NOAA.png? For instance, 2013-05-19 Surface Weather Map NOAA.png (or Min-Max Temp Map, 500-Milibar Height Contour Map, 24-hr Precipitation Map)? odder (talk) 16:18, 21 May 2013 (UTC)[reply]
    Looks good, let's go with that so long as it is accurate and navigable (i.e. we might want to give links to next-map, last-map in the template or in parent categories (one category per month might be useful)  . --Fæ (talk) 16:21, 21 May 2013 (UTC)[reply]
  4. Convert GIFs to PNGs.
    Done, with the exception of dwm500_test_20031009, which appears corrupted. (I couldn't even open the file on my computer.) odder (talk) 17:07, 21 May 2013 (UTC)[reply]
  5. Run test sample upload.
    Done - first 3 days worth (15 maps) available at Category:NCEP 2002 weather maps.
    Extended sample of the first 3 months of images uploaded, over 500 files, using Noaabot.
  6. Set up special bot account.
    User:Noaabot created. Flag requested. Full upload will start once the bot flag is given.
    (Optional) Integrate the gif->png conversion into the Python script (in OSX this can probably be a simple sips command) to make future regular updates easy, perhaps an automated weekly or monthly run.
  7. Extend to pdfs
    There are a number of recent pdf summaries at pdffiles. This appears a limited backlog, being weekly summaries from week 49 of 2012 through to week 20 of 2012 and then some recent daily summaries for the remaining days (the files are named as DWM<2 digit week><2 digit year>.pdf and they are in black and white as well as colour). There is no archive of past daily summaries, they appear to get bundled into the weekly ones. These are potentially valuable as the maps embedded in the pdfs are vector maps rather than gifs, so worth uploading and potentially these might be un-embedded, or the NCEP might release these online and we can upload directly.
    Naming:
    Weekly pdf summaries: <4 digit year> week <week number> Daily Weather Map <color> summary NOAA.pdf
    "color" will only be included if in color.
    (Parked for the moment, these effectively duplicate the pages in the weeklies) Daily pdf summaries: <ISO date> Daily Weather Map summary NOAA.pdf
    Noaabot is uploading these to Category:NCEP weekly weather maps.
Assigned to Progress Bot name Category
Fæ and odder Beta test - first month of images Noaabot Category:Images from NOAA uploaded by Noaabot
Fæ Bot approval   Done| User:Noaabot
Retrieved from "https://commons.wikimedia.org/w/index.php?title=Commons:Batch_uploading/Weather_maps&oldid=108952476"