OpenStreetMap/Panoramax

Da Wikimedia Italia.
Vai alla navigazione Vai alla ricerca

Panoramax è un software FOSS federato per la raccolta di immagini libere a livello stradale. Si pone come alternativa libera a soluzioni completamente proprietarie come Google Street View o Bing Streetside e a soluzioni collaborative per immagini libere ma basate su software proprietario come Mapillary e KartaView.

Le immagini a livello stradale sono estremamente utili per la mappatura di OpenStreetMap da remoto, e viceversa i dati geografici di OpenStreetMap sono estremamente utili per le piattaforme che raccolgono queste immagini. Pertanto molte di queste piattaforme hanno messo a disposizione i propri dati su OSM (una lista completa si trova in OSM Wiki) e la comunità OSM si è mossa per raccogliere immagini sulle piattaforme collaborative.

Rispetto alle alternative proprietarie citate sopra, Panoramax ha vari vantaggi:

  • Se le immagini raccolte sono in mano a una organizzazione privata e questa non fosse più in grado di usarle (es. causa fallimento) non ha nessun interesse che queste vengano usate e i dati rischiano di andare persi. In caso un servizio Panoramax gestito dalla comunità venga terminato c'è tutto l'interesse per salvare i dati: questo è un incentivo per i contributori.
  • Alcuni volontari sono restii a affidare le proprie immagini alle grandi multinazionali che gestiscono gli attuali servizi. Per essi l'uso di un servizio Panoramax è un incentivo.
  • La struttura federata di Panoramax previene un Single Point Of Failure
  • Uso di uno stack completamente open source

Istanze

Le istanze esistenti e la loro area geografica di riferimento sono documentate su osm:Panoramax#Services.

Attualmente (novembre 2025) alcuni volontari italiani usano l'istanza francese ma sarebbe corretto creare un'istanza italiana in base a quanto scritto qui e ribadito sul forum OSM: 

In the short term, OSM FR can host pictures from the outside of France, but worldwide coverage is not sustainable in the long term, due to storage limits (and costs). Therefore, it is preferred that other local communities set up their own federated instances

Vari membri della comunità italiana ha già chiesto più volte a Wikimedia Italia di creare un'istanza italiana (qui, qui qui, ...)

Aspetti legali

Nuvola apps xmag.png Per approfondire, vedi anche OpenStreetMap/Panoramax/Aspetti legali

Costi di gestione

Deploying a Panoramax instance: the pre-flight check list su Geocommuns

Indicazioni di Christian Quest, il manutentore di Panoramax

Financial investment

 Panoramax is mostly a storage challenge.
 That’s why we opted for decentralization and instances, but it remains a challenge for each instance.
 For example, a rough estimation for a unique dense coverage of France gives something like 100M pictures. With redundancy and a backup we easily reach 1PB of storage.
 In most cases, cloud base hosting is very expensive especially on storage and not an option to consider in the long term (except it if is offered… in the long term too !).
 OSM-FR choice has been to self-host our own server and storage bay in an sponsored colocation halk-rack we have near Paris. All hardware except the SSDs is donated or second-life (ebay), including the HDDs. That allowed to lower the cost.
 The server is a Dell R7910, with 2 GTX 1070 (for the blurring API), 256GB of RAM, 2 4TB NVMe SSDs, and mostly 8TB HDDs (around 40000h old). Total cost: 3500€.
 We compensate the risk of old disks failure by using ZFS soft-raid redundancy.
 The last disk we bought where less than 10€/TB.
 To give an idea of the storage need, we currently have 20M pictures with around 100TB of storage used.
 Distributing the storage is something to consider, one way to do it and to investigate/test is using Garage which is a distributed S3 storage. The drawback it that more storage is needed to guarantee redundancy.
 GPU are not mandatory to start (see below).

Se raggiungeranno effettivamente 1PB di storage e la stima di 10€/TB (per hardware usato) è realistica, lo storage raggiungerebbe 10k€. OSM FR ha stimato che per loro il massimo investimento totale in Hardware e sistemistica che potrebbero raggiungere per Panoramax è di circa 20k€.

Human investment

 Setting up a server and administering it takes some time and means someone has to do it.
 Upgrading the software stack takes some time especially when the code base is moving fast.
 A few hours a week is what you should dedicate to maintain an instance (see below).
 We can help for the setup, do not hesitate to ask !
 Administering a Panoramax instance means administering at least :
 - the storage space
 - the Panoramax backend API (python based)
 - the underlying postgresql database
 - the Panoramax frontend with the viewer (js)
 The blurring API is an additional part you can deploy or you can start by using the one setup by OSM-FR (we have 2 servers, with a total of 4 GPU and they are far from being overloaded).

Per il setup si può valutare se qualche volontario è disposto a occuparsene. A prescindere DOBBIAMO mantenere allineato il consulente pagato che segue la sistemistica generale di Wikimedia Italia perchè si occupi della manutenzione in modo che il progetto sia altamente disponibile indipendentemente dalla disponibilità dei volontari.

OSM FR numbers

 A few numbers from the OSM-FR instance as of mid-december 2024:
 - 22.7M pictures
 - 48.5 TB storage for originals (2.2TB / 1M pictures)
 - 22.5 TB for the derivates (1 TB/ 1M picture)
 - 157 GB for the postgresql DB (7GB/1M pictures, 114GB once compressed by ZFS)

A Panoramax UK instance? su OSM Community - commento 9

OSM-FR uses around 100TB EXCLUDING backups

When I proposed the project on the OSM-FR forum, I did a calculation to have an idea for one dense coverage all over France. With redundancy and backup, the total was around 1 PB.
Storage is really the #1 issue and that’s why we chose to have a decentralized sharing. We chose to self-host our instance to control the costs and we use mostly second life hardware, the only exception are SSDs.
Most of our HDD are 8TB now (we started with 3TB ones), and we can get very good deals on this capacity (around or even less than 10€/TB). They are not new, so we have redundancy (2 disks in each set of 6 disks, all managed using ZFS).
Our hardware investment for the server is around 3500€ so far. Server hosting (colocation) is offered by a Paris area datacenter.
Do not hesitate if you have question, need for advices or help to setup an instance.
The core dev team is working only in english in the gitlab repo and documentations are all available in english.
Do not hesitate to show up on the geocommons forum, english posts are ok even if most of past post are in french.

Link a geocommons forum

OSM-FR Panoramax server “only for testing if outside of France” su OSM Community - commento 20

On OSM-FR instance, we started 3 years ago with the following hardware:

- Dell R7910 (dual Xeon E5-2690v4)
- 256GB of RAM
- 2 x 4TB NVMe SSD (new !)
- 3TB used HDDs in a 60 SAS/SATA LFF IBM storage bay
- 2 x GTX 1070 (used by the blurring API)

Then we replaced the 3TB HDD with 8TB (currently 30 of them).
Then we added 12 x 16 TB HDD… totalling around 400TB.

A Panoramax UK instance? su OSM Community - commento 15

Wales’s instance only just went live. User murein on the osm discord is the owner, i dont know if he active on here. I’ve been talking to him regularly about it recently as i’m currently dumping my entire archive of 360 footage on to it. He is currently self funding it and his response when i previously asked about the cost was:

> a server cost of about €4-5 per month, S3 storage varies by size (currently €0) about €5.5 (6usd) for 0.5TB of photo storage. The domain was €5-10 per year I think.

We are quickly discovering that 8k(27MP) 360 images every minimum ~4m adds up very quickly so this biggest cost will always be the S3 storage, the size of which would very much be dependant of the popularity of the instance. Atm, I believe i’m the only one contributing Wales other than murein themself but i’m also contributing far more images than the average user would and the instance is still young.

Christian Quest risponde: 

That’s why OSM-FR have its storage on premise, instead of costly S3.
We BUY 1TB around 10€ (used drives) which is roughtly the monthly RENT of 1TB on S3.
Even with redundancy and backups this is way cheaper… but yes, you need to take care of your storage yourself (currently 400TB in the server excluding backup).

Storage needs for a Panoramax instance su Geocommuns

For 360° pictures, Panoramax is also generating a tiled version for faster display by the viewer. It allows to download only the visible part, usually 1/4 of the total. The tiled version for 360° pictures is more or less doubling the storage need. 
In our example above with imajbox 72 Mpx 360°, this means we have around 18TB used by both the originals and the derivates.

The above storage is a raw storage needed for the pictures. You don’t need fast (SSD) storage for the pictures, HDD are enough, but SSD it is much better for the postgresql database unless you have enough RAM cache.
The database itself is roughly 4GB for million of pictures. Some maintenance process may temporarily need twice this space.
Disks can fail and redundancy is the way to avoid the consequences of failures.
We cannot give specific advice on this, only share what we do.
On OSM-FR instance, we use ZFS to take care of all storage:

Aspetti tecnici

Deploying a Panoramax instance: the pre-flight check list su Geocommuns

Indicazioni di Christian Quest, il manutentore di Panoramax con i requisiti

Setting up OSM-HR Panoramax instance sul diario OSM di Matija Nalis

Istruzioni step by step per l'installazione di un'istanza

Possibili strade per la realizzazione di un'istanza

Post installazione

Possibili attività:

  • Importare le immagini esistenti dall'istanza francese in territorio italiano
    • Valutare se vale la pena, considerando la struttura federata di Panoramax
    • L'istanza francese usa la licenza CC-BY-SA-4.0
    • Valutare fattibilità tecnica: l'API offre un modo per esportare in bulk in base all'area geografica? Chiedere a ai creatori di Panoramax se possibile e ai gestori dell'istanza francese se preferiscono esportarlo loro da amministratori invece che pesare sull'API

Link utili

Soci interessati

Se sei interessato a collaborare attivamente nell'avvio di una istanza che copra l'Italia, anche solo come tester, segnati qui sotto (con ~~~)

Estratto da "https://wiki.wikimedia.it/index.php?title=OpenStreetMap/Panoramax&oldid=1299350"