Japan OSM improvement

[planemad]

Based on observations from #116. There are multiple data issues in japan which requires a comprehensive cleanup strategy with close involvement of the Japanese OSM community.

This ticket will track all the high level conversations and strategy on improving OSM data for Japan.

Existing data issues

• Most of Japan road network has been imported from Yahoo Japan in 2011
• The road data has positional errors (5-30m) when compared to Strava GPS and includes incorrect topology that does not match satellite imagery.
  
• Roads in metropolitan areas have been realigned to the correct position, but large parts of the country are still untouched since 2011.
  
• The Bing imagery coverage for Japan is comprehensive but does not match the OSM data or Strava GPS data. There is both an offset and orthorectification errors that varies throughout Japan.
  
• There is high resolution orthorectified imagery for Japan from GSI which perfectly matches Strava GPS data and is the best imagery source available. The coverage of this imagery is not complete.
  

Data improvement strategy

1. Select project area
   1. Priority: Urban areas > Highway corridors > Rural areas
   2. Check the accuracy of orthorectified imagery and Bing imagery by comparing with GPS data. Publish offsets if required.
   3. Evaluate extent of data offset and data freshness for the area
   4. Publish findings and communicate with local mappers to initiate a mapping task
2. Road realignment
   1. Create a road improvement task using the tasking manager
   2. Verify imagery alignment using GPS data
   3. For gridded networks, attempt to offset data to match imagery. If not successful delete imported unnamed roads and retrace area
   4. Delete imported roads with no match to imagery
   5. Realign all other roads and mark as finished
3. Road tracing
   1. Go through realigned tasks and invalidate to commence road tracing
   2. Trace missing roads and tracks from imagery and mark as finished
4. Validation

Next steps

• Get feedback from Japan OSM community on above process
• @planemad to set up trial map improvement task: http://tasks.openstreetmap.in/project/38
• @planemad and @Rub21 to run task with team and closely monitor issues

cc @mapbox/team-data

[planemad]

Realignment trial report 09/16

• Today @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 15,194 nodes in the Mutsu area
• Initial pass was to realign all roads in the task area, but due to inconsistencies in the reference sources and lack of GSI satellite imagery coverage, we could not verify that our work was accurate.
• Our revised strategy was to realign only highways (motorway-secondary) based on Strava GPS which was more accurate, but this created inconsistencies wrt to the existing data
• In the interest of keeping the improvement process simple for the community to contribute to, the conclusion was to sacrifice small positional accuracies to maintain large scale data consistency.

Revised data improvement strategy

1. Focus on high priority areas based on @Rub21 tweet map
   
2. Use the primary tracing source as GSI Japan Satellite (ort) where available and fallback on Bing in other areas. Most high priority areas have GSI Satellite coverage.

3. Realign road network only in urban areas to `GSI Japan Satellite (ort)`, ignore offsets (<10m) wrt `Bing` 4. Trace missing roads based on tracing source

From tomorrow team-blr will try the above approach and report findings. A log of observations is maintained here

cc @mapbox/team-data

[lxbarth]

Wow, great progress.

In the interest of keeping the improvement process simple for the community to contribute to, the conclusion was to sacrifice small positional accuracies to maintain large scale data consistency.

@planemad - what does this concretely mean? Can you give an example for that?

[planemad]

@lxbarth to be as accurate as possible we were using Strava and OSM GPS to offset the Bing/GSI imagery for the best match, this is never perfect and results in an offset in another area. These continuous localized corrections which cannot be replicated exactly be someone else and adds to the data inconsistency since no source perfectly will match the data everywhere.

To make it simple, we're just going to use GSI Satellite or Bing as is without offset correction. We will skip realigning data with offsets under 10m from satellite imagery such as these and focus on adding missing roads.

[mapconcierge]

@planemad

I understand all steps.
Yes, GSI ortho image layer (ort) has good accuracy.

電子国土基本図（オルソ画像）MaxZoomLv18
http://cyberjapandata.gsi.go.jp/xyz/ort/{z}/{x}/{y}.jpg
source=GSImaps/ort

We have started special project team on Facebook Group with Asian addicted mappers, OSM teachers in Japan. My students will join.
https://www.facebook.com/groups/697237443754399/

[mapconcierge]

@planemad

Mutsu city area doesn't have good GSI ort images and Strava GPS.
We should choice

std 標準地図 MaxZoomLv18
http://cyberjapandata.gsi.go.jp/xyz/std/{z}/{x}/{y}.png
source=GSImaps/std

[pratikyadav]

Realignment trial report 09/17

• Today @planemad @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 6437 nodes in the Aomori region as suggested by @mapconcierge .
• The focus was only on merging/aligning highways (secondary+) to GSI Satellite imagery for pass 1.
• Tomorrow in pass 2 the team will focus on realigning the minor roads and compare the time taken with pass 1.

[unsungNovelty]

@mapconcierge, what kind of highway will the below road be? a service road, residential road or a living-street road?

https://www.openstreetmap.org/#map=17/40.785891873025996/140.7407283726662

[pratikyadav]

Realignment trial report 09/21

Today @planemad @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 3201 nodes in the Aomori region.

The focus under pass 1A and 2 was -

• to create dual carriageways on major highways,
• realign and merge tertiary
• Adding missing tertiary+ highways
• Adding missing bridges/tunnels on tertiary+ highways;

Today under pass 3, the team will realign, merge and add missing unclassified, residential roads, living streets.

[planemad]

Realignment Summary

• Team of @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy conducted a multi phase data correction exercise in Aomori
• We synced daily to break the data correction in multiple steps to prioritize effort to the most critical data first to least important and improve the process from our observations
• General observations:
  • Time taken to realign an object is proportional to the number of ways and nodes
  • By merging consecutive ways and removing unnecessary nodes in straight sections, the effort can be reduced
  • In areas with dense road network, it is faster to delete roads without extra properties (name, ref) and retrace than realign
  • Focusing a task on a limited set of features allows the use of filters to remove distracting objects
  • Time per user per task should be limited to under 60 mins, else there is a feeling of stagnation
  • Retagging roads should not be done without valid reason
  • At all points the larger objective should be to simplify the contribution of any newbie to OSM in Japan
  • Tasks must be selected away from active ones to avoid conflicts

Realignment proposal for Japan
Based on these finding this is the proposal for a large scale road network correction for Japan:

• Select a project area with GSI imagery coverage and create a mapping task
• Notify OSM-JP of the task
• Make multiple passes of each area and make improvements in steps:
  • Pass 1: Realign and merge major highways (motorway-secondary); Create dual carriageways and link roads with correct oneways
  • Pass 2: Realign and merge tertiary; Adding missing tertiary+ highways and link roads; Adding missing bridges/tunnels on tertiary+ highways;
  • Pass 3: Realign, merge and add missing unclassified, residential roads, living streets
  • Pass 4: Realign, merge and add missing service roads, tracks and paths
  • Pass 5: Correct natural features polygons and waterways, add missing bridges, tunnels and viaducts
• The @mapbox/team-data will focus on completion upto pass 2 throughout Japan and support the community for further improvements

cc @mapconcierge @lxbarth @mapbox/team-data

[planemad]

Since we have Aomori completed till pass 2, the @mapbox/team-data is going to follow the similar procedure in the more dense Beppu-Oita region on Kyushu island from today http://tasks.openstreetmap.in/project/44

@mapconcierge would be great to have the OSM-JP community help with the validation of our work and feedback on the process.

[planemad]

While realigning highway junctions, make sure to fix any kinks on branching roads.

Fix any incorrect highway junctions where there is no crossing

.

[unsungNovelty]

@mapbox/team-data - Also merge and align the 4 way junctions which are split like this:

[samely]

We already finished http://tasks.openstreetmap.in/project/31#, which is a pilot task for realignment and adding roads in Chiba.

Those are the number of editions per user

User	Num OSM Objects	Num Changeset	Num nodes V1	Num nodes Vx	Num ways V1	Num ways Vx	Num relation V1	Num relation Vx
samely	77,862	333	45,220	18,416	5,856	8,370	0	0
RichRico	65,217	441	31,035	28,070	2,511	3,601	0	0
karitotp	64,290	215	32,538	21,848	4,531	5,370	0	3
Luis36995	53,561	429	24,981	21,864	2,751	3,965	0	0
calfarome	40,985	309	19,079	16,561	2,624	2,721	0	0
dannykath	38,549	653	20,531	11,481	3,563	2,973	0	1
ediyes	32,781	540	15,149	12,291	2,467	2,869	0	5
andygol	10,087	24	2,007	7,382	240	455	2	1
abel801	31,879	289	9,912	19,519	638	1,810	0	0
Rub21	1,815	31	759	711	131	214	0	0
pratikyadav	986	26	693	76	158	59	0	0
ruthmaben	767	18	236	405	58	68	0	0
jinalfoflia	311	7	240	5	52	14	0	0
nikhilprabhakar	310	5	254	9	34	13	0	0
srividya_c	294	17	167	14	75	38	0	0
ramyaragupathy	266	13	158	29	45	34	0	0
Chetan_Gowda	265	7	154	94	5	12	0	0
Total	420,225	3,357	203,113	158,775	25,739	32,586	2	10

[calfarome]

Team progress in Chiba Prefecture, Japan.