Wikimedia Commons

Commons:Batch uploading/Bibliothèque Nationale de France

< Commons:Batch uploading
(Redirected from User talk:BnF import)

French National Library edit

Thanks to the partnership of Wikimédia France with the French National Library (BnF/Bibliothèque Nationale de France) [1] [2] [3], we will have to upload 1416 DjVu files on Commons. The volume is about 100 Go.

We have a server where we must create the DjVu before (the BnF gave us images files + OCR files). I'm not sure, but perhaps there are files whose the size is more than 100 Mo (limit for the HTTP upload). Would it be possible that someone upload these DjVu files from our server May 15 or slightly after? Thanks a lot ~ Seb35 [^_^] [mail] 18:38, 2 May 2010 (UTC)[reply]

UPDATE: the files should be available in some days (Wednesday?), is it possible to have a contact/answer quickly because we must return the server (we rent) Monday 24, May (else we could rent the server one another month but... we would prefer stop the 24). ~ Seb35 [^_^] 20:20, 16 May 2010 (UTC)[reply]

Opinions edit

We hope we can make a test upload very soon. Early feedback on the sample Vigneron pointed would be much appreciated (for obvious things we would have missed). Jean-Fred (talk) 23:27, 17 May 2010 (UTC)[reply]

Sound like a nice project. You need to end up with 1416 pairs of files. Each pair would consist of <some file>.djvu, the djvu file, and <some file>.txt, a UTF-8 encoded text file containing the description of the file in wikitext format. These files need to be available on a publicly accessible website (you can keep the address private) so that one of the sysadmins can download the files and import these files to Commons.
So how to get there?
You probably have the metadata in some format. You have to use this to generate descriptions. You can find some programs I wrote in the past here, maybe that helps. Is it possible to make this metadata available somewhere so I can comment on it?
You should create a bot account for the import, user:BNFbot is probably a nice username. If you create the account while logged in it's attached to your account (examples).
You should probably add a tracker category to {{BNF cooperation project}}
Far from complete, but I hope this can get you going. Feel free to contact met on irc (freenode) if you have any questions. Multichill (talk) 07:08, 18 May 2010 (UTC)[reply]
Thanks for your comments.
  • Thanks to the hints you gave us on your talk page, our process already set up a tar file with one text file with wikitext and the DjVu. We also have a dedicated SSH entry point to the server for the sysadmins (Plyd has the details).
  • We have two sources of metadata :
  • We already wrote some code in Python that parses the XMLs for the following uses :
    • The Commons description page (metadata — as described above) ;
    • The WikiSource Book namespace page (metadata) ;
    • Every Wikisource Page namespace pages (text)
    • The DjVu (text and images)
At the moment, we still have some issues with the DjVu, but hopefully this will get resolved today. The rest of the process is okay.
Jean-Fred (talk) 10:45, 18 May 2010 (UTC)[reply]
Looks good! It's probably nice to create {{Creator}} templates for the authors like for example Creator:Émile Zola and also add a category for the author. Could you publish the code you're written so other projects can benefit from it? Multichill (talk) 18:14, 18 May 2010 (UTC)[reply]
Thanks :-)
  • The authors : we have too poor metadata about them, unfortunately : only names, and sometimes mispelled. VIGNERON has made the huge work of checking manually the authors list at s:fr:Wikisource:Dialogue BnF/Auteurs. With this, we will probably be able to generate proper author names, but we do not have enough metadata to generate decent Creator templates. We're looking into testing their existence and adding the Creator template in this case.
  • Yeah, we will defintely publish the code when it is completed.
Jean-Fred (talk) 22:12, 19 May 2010 (UTC)[reply]
I like creator templates so I hacked something together to generate them based on info from wikipedia. See Commons:Batch uploading/Bibliothèque Nationale de France/Auteurs for the progress. Each creator template should probably be checked and maybe improved, but that doesn't stop the upload from already using them. Also corresponding categories should be created, I could probably do that based on the info from Wikipedia too. Multichill (talk) 16:19, 23 May 2010 (UTC)[reply]
I'm done, I used en, fr & de Wikipedia as sources. Multichill (talk) 18:08, 23 May 2010 (UTC)[reply]
Wow, that is great. Thanks! Jean-Fred (talk) 00:22, 24 May 2010 (UTC)[reply]
I got to this page by investigating why do we suddenly have over 150 creator templates in Category:Creator templates without home category after I recently managed to resolve all the creator pages in that category by creating proper categories and adding them to the existing category tree (see File:Creator Category Schema.png). Creation of proper categories would be greatly helped if the script which generated those pages run by User:Multichill would update them by adding birth/death locations and full dates of death/birth which are often present in the articles. Also the wiki-links to the wikipedia articles ( in the NAME field) and descriptions (like french writer, etc.) would be great. I realize that what I am asking for would be much more than "something hacked together", but I believe it could be used over and over for creators related to other mass uploads. I recently manually fixed over 200 of creator pages created by user:BrooklynMuseumBot and it is a lot of work. --Jarekt (talk) 17:06, 14 June 2010 (UTC)[reply]
I can create the categories, but I was waiting for the checking effort to finish. Just say when I should run and I'll create the categories. Multichill (talk) 17:33, 14 June 2010 (UTC)[reply]
I am a newcomer to this discussion. Is there any checking/improving effort going on? --Jarekt (talk) 17:59, 14 June 2010 (UTC)[reply]
Afaik User:VIGNERON and User:Yann are working on this (linkie). Multichill (talk) 19:06, 14 June 2010 (UTC)[reply]
Welcome Jarekt! The process was delayed because the Wikisource folks wanted to re-check the list (book titles and authors names). The checking is now finished, and we plan to launch the generation of the files like, tomorrow :-). I will upload the file File:Frédéric II de Prusse - Correspondance avec Voltaire, tome 5.djvu later tonight with the new descriptions and metadata. Let us know your opinions on it, but as soon as possible (if you do not mind). It is quite urgent, because it takes about six days to generate the files, and we are behind schedule. We would like to launch the generation tommorrow evening. Jean-Fred (talk) 19:46, 16 June 2010 (UTC)[reply]
Gasp. That book is too big, could not upload :-/. I updated File:Paris, Gaston - Le roman du comte de Toulouse.djvu so you can see how the description page is now. This file is a pretty good example of what metadata we have.
Note: If the book is anonymous, the script uses {{Anonymous}} as author and Category:Anonymous books
Jean-Fred (talk) 23:14, 16 June 2010 (UTC)[reply]
I do not want to be nagging here but are there any plans to create categories (and categorize them) for creator pages which do not have them? They can be found in Category:Creator templates without home category. I could easily run a bot to create those categories and add category:people by name and {{creator:{{PAGENAME}}}} to them but my bot can not add death/birth year categories or categories like category:Writers from France. That would have to be done by hand unless some other bot can do it. --Jarekt (talk) 13:53, 8 July 2010 (UTC)[reply]
Yes, I have been doing that, but manually it takes a lot of time. If you could run a bot, you are more than welcome. Yann (talk) 16:34, 8 July 2010 (UTC)[reply]
I started my bot the new categories will be temporarily added to Category:Creator templates to fix, until they are checked and categories like category:Writers from France are added. --Jarekt (talk) 03:42, 9 July 2010 (UTC)[reply]

Test upload edit

Just came out today.

Reviews edit

Multichill already raised the following issues, :

  • Manual "<year> books" category (done by template = bad) : ✓ Done
  • Creator templates and Author categories : ✓ Done as for coding the description pages ; but the basis is the table on Wikisource which needs to be updated/checked beforehand. Follow-up on the French Scriptorium. Jean-Fred (talk) 22:29, 25 May 2010 (UTC)[reply]

First batch ready edit

Our of the ~1400 books, we already have 777 ready. The books size is in average 180 Mo, ranging from 250Mo to 300Mo, so as we suspected we will have to rely on a server-side import. We have everything ready (file+description text all in a tar file), could you please point us to the relevant people ? Jean-Fred (talk) 20:52, 26 June 2010 (UTC)[reply]

Please could you wait some time before downloading the files on our server. The big sizes (one book weights 1Go and all are x00Mo) makes it would be better amha to review our process, since there is perhaps an alternative to drastically reduce the size (use bitonal images instead of multicolor images, since all are in black-and-white). ~ Seb35 [^_^] 21:47, 27 June 2010 (UTC)[reply]
Yes, 1 Go seems unreasonable for a DJVU file. Yann (talk) 04:42, 28 June 2010 (UTC)[reply]

We revised our process, DjVus are much lighter now (average is 11 Mio, only three beyond 100 Mio). Much, much better :-)

We just requested a server-side upload to Tim Starling (though the books are small, there are quite a lot of them). Expect 1416 books to hit Commons after the week-end.

Jean-Fred (talk) 23:34, 2 July 2010 (UTC)[reply]

I created the category Category:Books provided by the BNF. Yann (talk) 13:26, 8 July 2010 (UTC)[reply]
We need a list of files without a OCR text. Yann (talk) 17:15, 8 July 2010 (UTC)[reply]
I suppose you changed the configuration of the conversion to DjVu. What about sharing it (on s:en:Help:DjVu files or whatever)? Thank you, Nemo 18:23, 20 July 2010 (UTC)[reply]
Nothing really fancy, we basically changed switched from multicolor to shades of gray (short story, Seb35 has the details).
Our current planning is to finish next week the technical report of the project, which will explain all our process, decisions and all, and will probably be made public soon after. After that, we will clean up our code and put it on FishEye, GitHub or similar, so that others can reuse and improve it. Our idea is to make a central place to build some solid code which could create any DjVu from any standard source (daydreaming a bit ;-)
Jean-Fred (talk) 23:05, 20 July 2010 (UTC)[reply]
Great, I love this daydream. :-) --Nemo 19:23, 26 July 2010 (UTC)[reply]

Categories edit

Is there any way the upload bot dealing with these files could check on existing categories before adding new ones? For example, File:Frédéric II de Prusse - Correspondance avec Voltaire, tome 2.djvu, together with 19 other files belonging to the same batch, is tagged Category:Frédéric II de Prusse. This category doesn't exist, but the correct category already does, i.e. Category:Friedrich II of Prussia. I have corrected a few cases manually, such as Category:Sophie Rostopchine, Comtesse de Ségur (instead of Category:Comtesse de Ségur), but after a while it seems like a huge waste of time and I doubt I'll be able to keep up with the bot for very long :-) Mu (talk) 01:28, 14 July 2010 (UTC)[reply]

Yes, this should be corrected. I think the upload is complete. Yann (talk) 05:07, 14 July 2010 (UTC)[reply]
Thanks for the info - too bad I missed the bus. But maybe the point could be noted in case of future uploads. - Mu (talk) 10:48, 14 July 2010 (UTC)[reply]
Most of those bot created categories have to be checked by hand and merged with existing category structure multiple names for the same person are quite common. Right now over 100 categories wait in (a temporary category I used before with no longer correct name) for check if they should be merged with other categories and for inclusion into profession by country category tree. --Jarekt (talk) 12:55, 14 July 2010 (UTC)[reply]

TIFFs edit

Hei, we intend to upload the TIFF files (the original files given by the BnF), it would permit to re-work on these files if needed. The TIFF handler is not already functionnal, but it seems it will be in some time [4]. Jean-Fred has already upload one test file, probably with the same metadata/wikitext we put with the DjVu files, it should be a good and easy way for that. ~ Seb35 [^_^] 20:40, 2 August 2010 (UTC)[reply]

Don't forget to add links between the file's in the other versions field. Multichill (talk) 20:50, 2 August 2010 (UTC)[reply]
Seb is right, I used the very same wikitext than the DjVu.
@Multichill : Yeah, but {{Book}} does not have such a field. Should be trivial to add though.
Question : Should we dump the files in the same tracking category, or set up a different one ?
Jean-Fred (talk) 21:01, 2 August 2010 (UTC)[reply]
I added the Other versions parameter in the program. For the category, is Category:Scanned French books in TIFF correct? And I think we can use the same Category:Books provided by the BNF with the same template. ~ Seb35 [^_^] 07:50, 16 August 2010 (UTC)[reply]
Do we keep the category "Scanned French books in TIFF" or not? Since we have not a lot of time before to return the server, I propose to replace the template "BnF cooperation project" by "BnF cooperation project/TIFF" (which would be simply a redirect) so that we can do some adjustements after the upload and possibly subst/replace with some other stuff after. I intend to launch the computation of the notices this evening to upload in the next days. ~ Seb35 [^_^] 19:10, 19 August 2010 (UTC)[reply]
I intend to upload the files tomorrow evening (25/08/2010 17:00 UTC). If you see any objection to the first upload File:Sand - La dernière Aldini. Simon.tif, please say it before. One small point we have not discussed is the extension .tif or .tiff, I prefer the first for better compatibility with old systems Windows 98 or so. ~ Seb35 [^_^] 22:03, 24 August 2010 (UTC)[reply]
OK for me. I created the category. Yann (talk) 03:49, 25 August 2010 (UTC)[reply]
The upload is finished for 1397 books. The 19 others are bigger than 100 Mio and cannot be uploaded via an external bot, they are currently on the toolserver since we have no more our server. Don't know if it is worthwhile to try to upload them. Any suggestion? ~ Seb35 [^_^] 18:22, 30 August 2010 (UTC)[reply]

Just added |Other versions= to {{Book}}.

See File:Vinet - Boutmy - Quelques idées sur la création d'une faculté libre d'enseignement supérieur.tif. Should we display the link as a link or as a thumb ? If the latter, should we set up a template to i18n the caption « DjVu version » ?

Jean-Fred (talk) 16:30, 14 August 2010 (UTC)[reply]

A link for me: the two versions are pretty similar so we don't need to get the image of the thumb. ~ Seb35 [^_^] 07:52, 16 August 2010 (UTC)[reply]

File not public domain edit

Hello, Please add your opinion here: Commons:Deletion requests/File:Allais - À se tordre.djvu. Illustrations by Pierre Delarue-Nouvelliere (1889-1973). Yann (talk) 07:02, 27 August 2010 (UTC)[reply]


Copyright status: File:Clergeac - Cartulaire de l’abbaye de Gimont.tif edit

bahasa melayu ∙ català ∙ čeština ∙ dansk ∙ deutsch (Sie-Form) ∙ deutsch ∙ english ∙ español ∙ français ∙ galego ∙ hrvatski ∙ italiano ∙ magyar ∙ nederlands ∙ norsk ∙ norsk bokmål ∙ norsk nynorsk ∙ português ∙ polski ∙ português do Brasil ∙ română ∙ sicilianu ∙ slovenčina ∙ slovenščina ∙ suomi ∙ svenska ∙ türkçe ∙ беларуская ∙ беларуская (тарашкевіца) ∙ български ∙ македонски ∙ русский ∙ українська ∙ ಕನ್ನಡ ∙ ತುಳು ∙ മലയാളം ∙ 한국어 ∙ 日本語 ∙ 中文（简体） ∙ 中文（繁體） ∙ עברית ∙ العربيَّة ∙ فارسی ∙ +/−
 
This media may be deleted.
Thanks for uploading File:Clergeac - Cartulaire de l’abbaye de Gimont.tif. I notice that the file page either doesn't contain enough information about the license or it contains contradictory information about the license, so the copyright status is unclear.

If you created this file yourself, then you must provide a valid copyright tag. For example, you can tag it with {{self|GFDL|cc-by-sa-all}} to release it under the multi-license GFDL plus Creative Commons Attribution-ShareAlike All-version license or you can tag it with {{PD-self}} to release it into the public domain. (See Commons:Copyright tags for the full list of license tags that you can use.)

If you did not create the file yourself or if it is a derivative of another work that is possibly subject to copyright protection, then you must specify where you found it (e.g. usually a link to the web page where you got it), you must provide proof that it has a license that is acceptable for Commons (e.g. usually a link to the terms of use for content from that page), and you must add an appropriate license tag. If you did not create the file yourself and the specific source and license information is not available on the web, you must obtain permission through the VRT system and follow the procedure described there.

Note that any unsourced or improperly licensed files will be deleted one week after they have been marked as lacking proper information, as described in criteria for deletion. If you have uploaded other files, please confirm that you have provided the proper information for those files, too. If you have any questions about licenses please ask at Commons:Village pump/Copyright or see our help pages. Thank you.

I removed {{PD-old-70}}, {{PD-old}} or similar templates from the files whose authors who died less then 70 years ago. Now the files have no license and unless it is fixed will be deleted in a week. If you know of other reason why those files are in public domain please feel free to add a new license template and alert me and I will remove {{No license}} tag. If you have any questions please check my FAQ or contact me on my talk page. Yours sincerely, Jarekt (talk) 12:18, 9 October 2015 (UTC)[reply]

File:Du Camp - Souvenirs d’un demi-siècle, tome 1.djvu edit

 
File:Du Camp - Souvenirs d’un demi-siècle, tome 1.djvu has been listed at Commons:Deletion requests so that the community can discuss whether it should be kept or not. We would appreciate it if you could go to voice your opinion about this at its entry.

If you created this file, please note that the fact that it has been proposed for deletion does not necessarily mean that we do not value your kind contribution. It simply means that one person believes that there is some specific problem with it, such as a copyright issue. Please see Commons:But it's my own work! for a guide on how to address these issues.

Please remember to respond to and – if appropriate – contradict the arguments supporting deletion. Arguments which focus on the nominator will not affect the result of the nomination. Thank you!
Afrikaans ∙ Bahasa Indonesia ∙ bosanski ∙ català ∙ čeština ∙ dansk ∙ Deutsch ∙ Deutsch (Sie-Form)‎ ∙ eesti ∙ English ∙ español ∙ Esperanto ∙ français ∙ galego ∙ hrvatski ∙ íslenska ∙ italiano ∙ magyar ∙ Nederlands ∙ norsk bokmål ∙ norsk nynorsk ∙ occitan ∙ Plattdüütsch ∙ polski ∙ português ∙ português do Brasil ∙ română ∙ shqip ∙ sicilianu ∙ slovenčina ∙ slovenščina ∙ suomi ∙ svenska ∙ Tiếng Việt ∙ Türkçe ∙ Zazaki ∙ Ελληνικά ∙ беларуская ∙ беларуская (тарашкевіца)‎ ∙ български ∙ македонски ∙ русский ∙ српски / srpski ∙ українська ∙ հայերեն ∙ বাংলা ∙ മലയാളം ∙ ပအိုဝ်ႏဘာႏသာႏ ∙ မြန်မာဘာသာ ∙ ไทย ∙ 조선말 ∙ 한국어 ∙ 日本語 ∙ 简体中文‎ ∙ 繁體中文‎ ∙ עברית ∙ العربية ∙ پښتو ∙ فارسی ∙ ދިވެހިބަސް ∙ +/−

ShakespeareFan00 (talk) 17:29, 29 August 2023 (UTC)[reply]

Retrieved from "https://commons.wikimedia.org/w/index.php?title=Commons:Batch_uploading/Bibliothèque_Nationale_de_France&oldid=796940582"