New data format by micbar · Pull Request #89 · owncloud/data_exporter

micbar · 2019-07-25T14:44:41Z

Description

First attempt on new Data Format

Related Issue

#77 (comment)

Motivation and Context

Introducing streamed json files (*.jsonl)
Better Handling of large instances.

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

Current Format

php occ instance:export:user someUser /tmp

# tree someuser
someuser
├── files
│   ├── files
│   │   ├── someFile.txt
│   │   └── someFolder
│   │       └── anotherFile.txt
│   ├── files_trashbin
│   │   ├── files
│   │   │   └── deleted.txt.d1565074252
│   │   ├── keys
│   │   └── versions
│   └── files_versions
│       └── someFolder
│           ├── anotherFile.txt.v1565074194
│           ├── anotherFile.txt.v1565074215
│           ├── anotherFile.txt.v1565074216
│           ├── anotherFile.txt.v1565074218
│           ├── anotherFile.txt.v1565074219
│           ├── anotherFile.txt.v1565074220
│           ├── anotherFile.txt.v1565074221
│           ├── anotherFile.txt.v1565074222
│           ├── anotherFile.txt.v1565074223
│           └── anotherFile.txt.v1565074224
├── files.jsonl
├── shares.jsonl
└── user.json

9 directories, 16 files

files.jsonl (prettyfied)

{
  "type": "folder",
  "path": "/files",
  "eTag": "5d49234cc58d8",
  "permissions": 31,
  "mtime": 1565074252
}
{
  "type": "file",
  "path": "/files/someFile.txt",
  "eTag": "2ecd8be94415b7330c34cf3b7598006d",
  "permissions": 27,
  "mtime": 1565074170
}
{
  "type": "folder",
  "path": "/files/someFolder",
  "eTag": "5d492330a970f",
  "permissions": 31,
  "mtime": 1565074224
}
{
  "type": "file",
  "path": "/files/someFolder/anotherFile.txt",
  "eTag": "e147c69432d1373ffc943c70a6dc6d01",
  "permissions": 27,
  "mtime": 1565074224
}
{
  "type": "folder",
  "path": "/files_trashbin",
  "eTag": "5d49234cc4422",
  "permissions": 31,
  "mtime": 1565074252
}
{
  "type": "folder",
  "path": "/files_trashbin/files",
  "eTag": "5d49234cc4422",
  "permissions": 31,
  "mtime": 1565074252
}
{
  "type": "file",
  "path": "/files_trashbin/files/deleted.txt.d1565074252",
  "eTag": "325681baa0768a0129414f8cd3982d61",
  "permissions": 27,
  "mtime": 1565074247
}
{
  "type": "folder",
  "path": "/files_trashbin/keys",
  "eTag": "5d4922e6d3dbf",
  "permissions": 31,
  "mtime": 1565074150
}
{
  "type": "folder",
  "path": "/files_trashbin/versions",
  "eTag": "5d4922e6d25a1",
  "permissions": 31,
  "mtime": 1565074150
}
{
  "type": "folder",
  "path": "/files_versions",
  "eTag": "5d492330a5bd4",
  "permissions": 31,
  "mtime": 1565074224
}
{
  "type": "folder",
  "path": "/files_versions/someFolder",
  "eTag": "5d492330a5bd4",
  "permissions": 31,
  "mtime": 1565074224
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074194",
  "eTag": "b19c5543bf50070bb158071dbc6ece45",
  "permissions": 27,
  "mtime": 1565074215
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074215",
  "eTag": "898d7dae7e034e9d18fc965e3e12ee83",
  "permissions": 27,
  "mtime": 1565074216
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074216",
  "eTag": "f5ef07922ed8dd24fe9c813633a30ced",
  "permissions": 27,
  "mtime": 1565074218
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074218",
  "eTag": "82f64b1e65a384a32e54723d22625cc6",
  "permissions": 27,
  "mtime": 1565074219
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074219",
  "eTag": "204194d5841e32b6f81ca17df7a3acc0",
  "permissions": 27,
  "mtime": 1565074220
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074220",
  "eTag": "14b912e5a2bca2374c8ee1d4a8cc9bcc",
  "permissions": 27,
  "mtime": 1565074221
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074221",
  "eTag": "dc6d66f6ba87f697e5680fde2ef7b73f",
  "permissions": 27,
  "mtime": 1565074222
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074222",
  "eTag": "d673e1c66c64b0660be6eafac31cd21d",
  "permissions": 27,
  "mtime": 1565074223
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074223",
  "eTag": "d5077261a68176e5fef6f69d1cc46bad",
  "permissions": 27,
  "mtime": 1565074224
}
{
  "type": "file",
  "path": "/files_versions/someFolder/anotherFile.txt.v1565074224",
  "eTag": "1c0ec99c238496d24252c88bb0b64b63",
  "permissions": 27,
  "mtime": 1565074224
}

shares.jsonl (prettyfied)

{
  "path": "/someFile.txt",
  "shareType": "user",
  "owner": "someUser",
  "sharedBy": "someUser",
  "type": "file",
  "sharedWith": "admin",
  "permissions": 19,
  "expirationDate": null,
  "password": null,
  "name": null,
  "token": null
}
{
  "path": "/someFolder",
  "shareType": "group",
  "owner": "someUser",
  "sharedBy": "someUser",
  "type": "folder",
  "sharedWith": "admin",
  "permissions": 31,
  "expirationDate": null,
  "password": null,
  "name": null,
  "token": null
}
{
  "path": "/someFile.txt",
  "shareType": "link",
  "owner": "someUser",
  "sharedBy": "someUser",
  "type": "file",
  "sharedWith": null,
  "permissions": 1,
  "expirationDate": 1567123200,
  "password": "1|$2y$10$Tu5Z7i5SK3Ky3Jn6BR.TheC1tdtwboAyYPzVFKWsznyTZXUBGgjbS",
  "name": "Public Link",
  "token": "j8n1uZ2DaDZJjl8"
}

user.json (prettyfied)

{
  "date": "2019-08-06T06:56:15+00:00",
  "originServer": "http://localhost/",
  "user": {
    "userId": "someUser",
    "displayName": "someUser",
    "email": null,
    "quota": "default",
    "backend": "Database",
    "enabled": true,
    "groups": [
      "people"
    ],
    "preferences": [
      {
        "appId": "core",
        "configKey": "lang",
        "configValue": "de"
      },
      {
        "appId": "core",
        "configKey": "timezone",
        "configValue": "Europe/Berlin"
      }
    ]
  }
}

IljaN · 2019-07-25T15:15:44Z

We should also include an .md in docs which has examples of the jsonl files

IljaN · 2019-07-29T06:08:50Z

@micbar I have an somewhat similar solution prototyped which moves the responsibility of encoding jsonl-streams in to the serializer. Will provide PR for discussion shortly. So we might merge our ideas.

felixboehm · 2019-07-29T07:37:14Z

We should also include an .md in docs which has examples of the jsonl files

yes, please.

CLAassistant · 2019-08-01T15:04:11Z

All committers have signed the CLA.

micbar · 2019-08-06T08:49:54Z

@butonic @IljaN
During testing I noticed, that etags of folders change.

We need to consider this topic in the context of OCIS

We need a follow-up issue to export the mtime.
Skip writing files on import

codecov · 2019-08-06T19:46:19Z

Codecov Report

Merging #89 into feature/next will decrease coverage by 2.02%.
The diff coverage is 85.58%.

@@               Coverage Diff                @@
##           feature/next      #89      +/-   ##
================================================
- Coverage         87.88%   85.85%   -2.03%     
================================================
  Files                36       31       -5     
  Lines               999      884     -115     
================================================
- Hits                878      759     -119     
- Misses              121      125       +4

Impacted Files	Coverage Δ
lib/Importer.php	`0% <0%> (ø)`	⬆️
lib/Extractor/MetadataExtractor.php	`100% <100%> (ø)`	⬆️
lib/Exporter.php	`100% <100%> (ø)`	⬆️
...actor/MetadataExtractor/FilesMetadataExtractor.php	`100% <100%> (ø)`	⬆️
...ib/Extractor/MetadataExtractor/SharesExtractor.php	`100% <100%> (ø)`	⬆️
lib/Importer/MetadataImporter/ShareImporter.php	`75.5% <79.41%> (-2.28%)`	⬇️
lib/Importer/FilesImporter.php	`87.5% <86.11%> (-0.5%)`	⬇️
lib/Utilities/StreamHelper.php	`95.65% <95.65%> (ø)`
lib/Importer/MetadataImporter/UserImporter.php	`88.23% <0%> (+8.82%)`	⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54a853c...acc1140. Read the comment docs.

IljaN · 2019-08-27T14:13:14Z

@micbar Please squash away "Fix Tests" commits before merging

Fix Importer Unit Tests

micbar added the 2 - Developing label Jul 25, 2019

micbar self-assigned this Jul 25, 2019

micbar force-pushed the feature/new-data-format branch from 08eef5b to d49e1da Compare July 25, 2019 18:48

micbar changed the base branch from feature/next to master July 25, 2019 18:53

micbar changed the base branch from master to feature/next July 25, 2019 18:53

micbar force-pushed the feature/new-data-format branch 3 times, most recently from dca6664 to 6ee859b Compare July 25, 2019 19:15

IljaN mentioned this pull request Jul 29, 2019

Streaming Serializer [WIP][Concept] #90

Open

9 tasks

micbar force-pushed the feature/new-data-format branch 2 times, most recently from 58569e9 to a285c7f Compare August 1, 2019 15:04

micbar force-pushed the feature/new-data-format branch 6 times, most recently from 1471520 to 11c4773 Compare August 6, 2019 06:06

micbar force-pushed the feature/new-data-format branch 2 times, most recently from 5302f00 to fb191e8 Compare August 6, 2019 20:46

micbar mentioned this pull request Aug 6, 2019

Don't read whole file in to memory on importing/exporting #28

Closed

micbar changed the title ~~[WIP] New data format~~ New data format Aug 27, 2019

micbar added 3 - To review and removed 2 - Developing labels Aug 27, 2019

micbar added this to the ready for reva milestone Aug 27, 2019

IljaN self-requested a review August 27, 2019 13:49

IljaN approved these changes Aug 27, 2019

View reviewed changes

micbar force-pushed the feature/new-data-format branch 6 times, most recently from aa4241f to 0eb07f0 Compare August 28, 2019 09:04

micbar and others added 4 commits August 28, 2019 14:36

New data format

f4ba415

Fix Importer Unit Tests

Remove stable10 test jobs

7a0e0d1

do not have multiple when in steps

75da278

Modify drone pipeline

7eee03a

IljaN force-pushed the feature/new-data-format branch from acc1140 to 7eee03a Compare August 28, 2019 12:36

micbar merged commit ba90718 into feature/next Aug 28, 2019

delete-merged-branch Bot deleted the feature/new-data-format branch August 28, 2019 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

New data format#89

New data format#89
micbar merged 4 commits into
feature/nextfrom
feature/new-data-format

micbar commented Jul 25, 2019 •

edited

Loading

Uh oh!

IljaN commented Jul 25, 2019

Uh oh!

IljaN commented Jul 29, 2019 •

edited

Loading

Uh oh!

felixboehm commented Jul 29, 2019

Uh oh!

CLAassistant commented Aug 1, 2019 •

edited

Loading

Uh oh!

micbar commented Aug 6, 2019

Uh oh!

codecov Bot commented Aug 6, 2019 •

edited

Loading

Uh oh!

IljaN commented Aug 27, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

micbar commented Jul 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Current Format

files.jsonl (prettyfied)

shares.jsonl (prettyfied)

user.json (prettyfied)

Uh oh!

IljaN commented Jul 25, 2019

Uh oh!

IljaN commented Jul 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixboehm commented Jul 29, 2019

Uh oh!

CLAassistant commented Aug 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

micbar commented Aug 6, 2019

Uh oh!

codecov Bot commented Aug 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

IljaN commented Aug 27, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

micbar commented Jul 25, 2019 •

edited

Loading

IljaN commented Jul 29, 2019 •

edited

Loading

CLAassistant commented Aug 1, 2019 •

edited

Loading

codecov Bot commented Aug 6, 2019 •

edited

Loading