Fix core packages that have validation errors

[zelima]

Almost half of the core packages are not published cause they do not pass datapackage validation

Why do we care that data is valid? (This is worth asking as sometimes we can get obsessed with validation and forget why we do it). We want valid data packages so that:

• Page rendering works (we don't get 500 errors because license is an array rather than string or vice versa)
• Graph rendering works (e.g. date is actually a date)

At the same time, we want to make it as easy as possible for users to publish -- be liberal in what you accept and strict in what you emit!

Acceptance criteria

• all core packages are live
• we have normalization script for "normalizing" (fixing) common errors (including upgrade from old spec)
• Analysis of what the problems were and any actions arising for upstream libraries or for validator library so that we can improve the tooling ecosystem

Tasks

Prep

• Clone repos locally using dpm-py scripts
• Research key specs so we understand how things change

Analyse the problems

• Create a spreadsheet with a list of all data packages that are not working with columns like the following: Current Error Message, Better Error Message, Notes on details of the error, Fix type (manual or automatic), Details of fix (if needed), Resolution Status (FIXED or empty)
  • In progress here:
    https://docs.google.com/spreadsheets/d/14kJluhePaMOx6vYBic0poVjDYK3I8-v_xF3sv4Focac/edit#gid=0
• [x ] Summarize resolution plan as a set of tasks below
• Write up of what is not working so well re validation error messages and upstream libraries and what we could automatically normalize

Resolve

• We are going to automate in a script the following:
  • Fix datapackage
    • Add in a date field "format" = "any", if there are other formats, replace with "any"
    • Repalce unknown type "decimal" to "number"
    • Change sources to array
• Fixes needed by upstream implementors
  • Geopoint type error in DPM-py
• Items for manual fixes can be found in the following spreadsheet from row 17-28
  https://docs.google.com/spreadsheets/d/14kJluhePaMOx6vYBic0poVjDYK3I8-v_xF3sv4Focac/edit#gid=0&range=17:28

Summary

We need a proper documentation about properties of data packages. Detailed specification on data types and formats, especially for date type since it is very common issue. And make a list of common errors, description and explanation for them and suggestions how to fix them.

Old Analysis

Possible solutions

• [N1] Change type to new year (or other) type in JS library (in upcoming release - but not supported in current js lib!)
• [N2] Write the small scripts that deals with common problems Eg changes format in datapackage.json and modifies csv file to have correct field values [8]
  • write tests [1]
  • write script [3]
  • run against all packages that have that type of problem one by one; [1]
  • publish [1]
  • rise the PR [1]
  • fix the packages that can not be fixed manually; publish; rise PR [1]
• [N3] standard code / algorithm to do conforming in dpm-py [>7]
  • write test [1]
  • come up with generic function (if possible at all) [5]
  • run dpm publish against broken packages [1]
  • fix the remaining ones [??]

[zelima]

Moving to backlog for now as no clear decisions how to fix this