Commons:Spacemedia

Spacemedia is a tool written by Don-vip that continuously harvests media libraries of various space agencies in order to find free media not yet uploaded to Wikimedia Commons.

The tool runs on Toolforge (Kubernetes/jdk17 cluster) and uses OptimusPrimeBot account to upload automatically new discovered media.

The discussion to authorize this tool takes place here.

As of 2024-05-05 the tool has already imported more than 200.000 files.

Links edit

Useful discussions / projects edit

Monitored repositories and status edit

Repositories providing free media with compatible licences
Agency / repository Licence(s) Remark Status
US Air Force / Space Force
https://www.flickr.com/photos/airforcespacecommand Cc-by-2.0, Cc-zero Should be PD-USGov-Military-Air Force   Done
https://www.flickr.com/people/129133022@N07/ PD Should be PD-USGov-Military-Air Force   Done
https://www.dvidshub.net/search/?q=&filter%5Btype%5D=image&filter%5Bbranch%5D=Space+Force&view=grid&sort=publishdate PD-USGov-Military-Space Force   Done
https://www.spaceforce.mil/Multimedia/Photos/ PD-USGov-Military-Space Force   Done
DLR (Deutsches Zentrum für Luft- und Raumfahrt, German Aerospace Center)
https://www.flickr.com/photos/dlr_de Cc-by-2.0, Cc-by-sa-2.0   Done
https://www.dlr.de/EN/organisation-dlr/news/all-news.html Cc-by-sa-3.0 Many news images in cc-by-sa, need to check individually though   Not done
ESA (European Space Agency)
http://www.esa.int/spaceinimages/Images Cc-by-sa-3.0-IGO   Done
https://www.flickr.com/photos/europeanspaceagency/ Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark   Done
https://www.flickr.com/photos/esa_events Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark   Done
https://www.youtube.com/channel/UClB8L8TJEQfZ41Ii0gJRTSQ Cc-by-sa-3.0-IGO   Not done
EU (European Union)
https://emergency.copernicus.eu/mapping/list-of-activations-rapid - https://emergency.copernicus.eu/mapping/list-of-activations-risk-and-recovery Attribution-Copernicus required to inform about the source of information and/or data by using provided reference   Not done
https://www.copernicus.eu/en/media/image-day Attribution-Copernicus   Done
https://erccportal.jrc.ec.europa.eu/ECHO-Products/Maps#/maps/latest PD-European-Commission Category:ECHO Daily Map of 2023   Done
Observatories, Space telescopes, International Astronomical Union
https://www.eso.org/public/images/ ESO (CC-by-4.0)   Half done
https://www.iau.org/public/images/ CC-by-4.0   Half done
https://www.spacetelescope.org/images/ ESA-Hubble   Half done
https://hubblesite.org/resource-gallery/images PD-Hubble   Half done
https://webbtelescope.org/ PD-Webb   Done
https://esawebb.org/ ESA-Webb   Done
https://gallery.lsst.org Cc-by-4.0 Webdam (old)   Not done
https://rubin.canto.com/v/gallery/ Cc-by-4.0 Canto (new)   Not done
https://www.almaobservatory.org/en/copyright-notice/ Cc-by-4.0   Not done
https://noirlab.edu/public/images/ NOIRLab   Done
https://nso.edu/about/image-use-policy/ Cc-By 4.0   Not done
INPE (Instituto Nacional de Pesquisas Espaciais) - Brazil
http://www.dpi.inpe.br/galeria/ INPE, Cc-By-Sa 4.0   Done
https://www.flickr.com/photos/observacao-da-terra/ INPE, Cc-By-Sa 2.0   Done
ISRO (Indian Space Research Organisation)
http://pib.gov.in/PhotoCategories.aspx?MenuId=8 GODL-India (through PIB) https://gitlab.com/Gazoth/pib-upload/issues/1   Not done
KARI (Korean Aerospace Research Institute)
https://www.kari.re.kr/kor/kariimg/list.do?img_gbn=PHO KOGL   Done
NASA (National Aeronautics and Space Administration) - live
https://images.nasa.gov/ PD-USGov-NASA   Half done
https://www.nasa.gov/image-article/ PD-USGov-NASA   Half done
https://www.nasa.gov/gallery/ PD-USGov-NASA   Half done
https://photojournal.jpl.nasa.gov/ PD-USGov-NASA Use Template:NASA Photojournal   Half done
https://earthobservatory.nasa.gov/images PD-USGov-NASA Use Template:ISS Crew Earth Observations   Not done
https://eol.jsc.nasa.gov/Collections/ PD-USGov-NASA To coordinate with User_talk:Askeuhd/Archive_2#Usefulness_of_ISS_image_dumps

Use Template:ISS Crew Earth Observations

  Not done
https://www.nasa.gov/multimedia/imagegallery/iotd.html PD-USGov-NASA To do. https://kaijento.github.io/2017/05/06/web-scraping-nasa-image-of-the-day/   Not done
https://solarsystem.nasa.gov/resources/ PD-USGov-NASA   Not done
https://www.sti.nasa.gov/harvesting-data-from-ntrs/ PD-USGov-NASA To do. Run requests between 8PM-8AM U.S. ET. No more than one request every 3 seconds.   Not done
https://svs.gsfc.nasa.gov/ PD-USGov-NASA   Half done
https://www.uahirise.org/ PD-NASA-HiRISE https://www.uahirise.org/media/usage.php   Not done
https://visibleearth.nasa.gov/ PD-USGov-NASA   Not done
https://asterweb.jpl.nasa.gov/ PD-USGov-NASA   Done
https://modis.gsfc.nasa.gov/gallery/showall.php PD-USGov-NASA   Half done
https://sdo.gsfc.nasa.gov/ PD-USGov-NASA   Done
https://epic.gsfc.nasa.gov/galleries PD-USGov-NASA   Not done
https://ails.arc.nasa.gov/ PD-USGov-NASA   Not done
https://chandra.si.edu/photo/ PD-USGov-NASA   Half done
https://www.dfrc.nasa.gov/Gallery/Photo/index.html PD-USGov-NASA   Not done
https://www.flickr.com/photos/atmospheric-infrared-sounder Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/earthrightnow Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/gsfc Cc-by-2.0, Cc-zero Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/morpheuslander Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasa2explore Cc-by-2.0, CC-PD-Mark Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasa_appel Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasa_goddard Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasa_ice Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasablueshift Cc-by-2.0, Cc-by-sa-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasacommons Flickr-no known copyright restrictions Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasaearthobservatory Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasafo CC-PD-Mark Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasahubble Cc-by-2.0, CC-PD-Mark Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasakennedy Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasarobonaut Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/nasawebbtelescope Cc-by-2.0 Should be PD-USGov-NASA   Half done
https://www.flickr.com/photos/uahirise-mars Cc-zero, CC-PD-Mark Should be PD-USGov-NASA   Half done
https://video.ibm.com/channel/nasa-tv-wallops TBC Should be PD-USGov-NASA   Not done
https://nasa-external-ocomm.app.box.com/s/onrtmdvofqluv5ei5kfu5u1pf8v4xqtl/folder/109925769802 PD-USGov-NASA   Done
https://www.missionjuno.swri.edu/junocam/processing?source=public&ob_from=&ob_to=&perpage=100 Cc-By Be careful, the licence must be checked for each image, some are cc-by-nc   Not done
https://lroc.sese.asu.edu/posts PD-USGov-NASA   Done
https://www.shadowcam.asu.edu/images PD-USGov-NASA   Done
https://tothemoon.ser.asu.edu/ PD-USGov-NASA   Not done
NASA (National Aeronautics and Space Administration) - dead
http://grin.hq.nasa.gov PD-USGov-NASA Category:Great Images in NASA - Commons:Batch uploading/Great Images in NASA   Done
https://www.ssc.nasa.gov/sirs/scripts/xmlWelcome.pl PD-USGov-NASA Stennis Image Retrieval System (SIRS) - shut down around 2020   Not done
http://lisar.larc.nasa.gov/UTILS/info.cgi? PD-USGov-NASA Langley Image Scanning, Archival and Retrieval   Not done
https://mix.msfc.nasa.gov/ PD-USGov-NASA Marshall image eXchange   Not done
NOAA (National Oceanic and Atmospheric Administration)
https://www.flickr.com/photos/noaasatellites/ CC-PD-Mark Should be PD-USGov-NOAA or PD-USGov-NASA   Half done
https://photolib.noaa.gov/Collections/NOAA-In-Space PD-USGov-NOAA   Not done
https://www.nesdis.noaa.gov/real-time-imagery/imagery-collections PD-USGov-NOAA   Not done
https://www.ngdc.noaa.gov/stp/satellite/goes-r.html PD-USGov-NOAA Curated SUVI images and animations processed at the National Centers for Environmental Information (NCEI)   Not done
http://sxi.ngdc.noaa.gov/ PD-USGov-NOAA Older SXI image data from GOES 12-15   Not done
https://www.ncei.noaa.gov/data/goes-r-series-l2-operational-space-weather-products/ PD-USGov-NOAA Original operational versions of SUVI images are also available through NCEI   Not done
USGS (United States Geological Survey)
https://lima.usgs.gov/access.php PD-USGov-USGS   Not done
https://astrogeology.usgs.gov/site/annex PD-USGov-USGS + PD-USGov-NASA   Not done
Individuals
https://www.flickr.com/photos/charlesandmaggie/ Cc-by-2.0   Done
https://www.flickr.com/photos/geckzilla Cc-by-2.0   Done
https://www.flickr.com/photos/harrystrangerphotographyhttps://www.flickr.com/photos/194849271@N04 PD   Done
https://www.flickr.com/photos/192271236@N03 Cc-by-2.0   Done
https://www.flickr.com/people/kevinmgill Cc-by-2.0   Done
https://www.flickr.com/photos/pierre_markuse Cc-by-2.0   Done
Universities
https://www.cfa.harvard.edu/resources/image-gallery PD https://www.cfa.harvard.edu/news/image-use-policy   Not done
https://www.cassis.unibe.ch/ Cc-By-Sa 3.0 IGO   Not done
https://cimss.ssec.wisc.edu/satellite-blog/ PD https://cimss.ssec.wisc.edu/satellite-blog/about   Not done
Private companies
https://www.youtube.com/@arianespace Videos no longer free :(   Done
https://www.planet.com/gallery/ Formerly Cc-by-4.0. No longer :(   Done
https://www.flickr.com/photos/sentinelhub Cc-By 2.0   Done
https://www.flickr.com/photos/spacex Formerly Cc-zero-SpaceX. No longer :(   Done
https://registry.opendata.aws/umbra-open-data/ Cc-By 4.0 https://umbra.space/open-license   Done
https://registry.opendata.aws/capella_opendata/ Cc-By 4.0   Done

Technical description edit

The tool is developed using Maven, Spring Boot, Spring Data JPA, Spring Cache, Jsoup, Dozer, Hibernate Search, Flickr4Java and Apache Commons libraries.

The tool connects to the Commons database replica to search for existing medias (by their base36 SHA-1) in these three tables:

  • image: current version of images
  • oldimage: old versions of images
  • filearchive: deleted images (when another version with a different SHA-1 exists)

The result of the analysis is persisted in a dedicated user database.

Most of the configuration is in application.properties. Sensitive information are not stored in the git repository and must be provided at runtime, on the command line.

Launch tool on local machine edit

java -Dcommons.datasource.username=$user -Dcommons.datasource.password=$password -Dflickr.api.key=$flickr_api_key -Dflickr.secret=$flickr_secret -jar spacemedia.jar --spring.profiles.active=dev

Launch tool on Toolforge edit

java -Xmx4G -Ddomain.datasource.url=jdbc:mariadb://tools.db.svc.eqiad.wmflabs:3306/${user}__spacemedia -Ddomain.datasource.username=$user -Ddomain.datasource.password=$password -Dcommons.datasource.username=$user -Dcommons.datasource.password=$password -Dflickr.api.key=$flickr_api_key -Dflickr.secret=$flickr_secret -jar spacemedia.jar --spring.profiles.active=toolforge

Resource usage edit

The production database takes around 300Mb.

The tool can be quite memory hungry when checking if ESA files are valid (to avoid uploading corrupted files to Wikimedia Commons), especially for large TIFF files.

Future edit