Migrate ID columns to bigint by nickvergessen · Pull Request #6563 · nextcloud/server

nickvergessen · 2017-09-19T10:54:51Z

For better scaling

mention-bot · 2017-09-19T10:54:53Z

@nickvergessen, thanks for your PR! By analyzing the history of the files in this pull request, we identified @rullzer, @MorrisJobke and @LukasReschke to be potential reviewers.

rullzer

Fine by me

codecov · 2017-09-19T12:26:30Z

Codecov Report

Merging #6563 into master will increase coverage by 3.19%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #6563      +/-   ##
============================================
+ Coverage     53.06%   56.25%   +3.19%     
============================================
  Files          1417       61    -1356     
  Lines         87809     7574   -80235     
  Branches       1340     1340              
============================================
- Hits          46596     4261   -42335     
+ Misses        41213     3313   -37900

Impacted Files	Coverage Δ	Complexity Δ
apps/files_external/lib/Command/Delete.php
settings/ajax/setquota.php
apps/files_external/lib/Lib/Backend/DAV.php
apps/theming/lib/ThemingDefaults.php
lib/private/Preview/Image.php
apps/dav/lib/CardDAV/Converter.php
lib/private/Files/Cache/StorageGlobal.php
...s/workflowengine/lib/Controller/FlowOperations.php
core/Command/Maintenance/UpdateTheme.php
apps/workflowengine/lib/Check/RequestTime.php
... and 1345 more

LukasReschke

Merge conflict

nickvergessen · 2017-09-19T18:10:07Z

Yeah, as to be expected. I will rebase after #6564 and #6566

nickvergessen · 2017-09-20T07:11:33Z

Rebased, ready for review and merge

nickvergessen · 2017-09-22T07:10:21Z

@LukasReschke please review, so this is available early in dailies and master for testing

MorrisJobke · 2017-09-26T12:13:51Z

+/**
+ * Auto-generated migration step: Please modify to your needs!
+ */
+class Version1004Date20170919103422 extends BigIntMigration {


This one is not executed at all. :/

My migrations table for dav looks like this:

dav | 1004Date20170825134824 dav | 1004Date20170919104507 dav | 1004Date20170924124212

Ah I bet that is ebcause we merged the other migration to dav.

Yeah need to rebase and change the timestamp to be newer...
#JustDevProblems....

Will have a look separate if we can detect not executed migrations and run them individually

Will have a look separate if we can detect not executed migrations and run them individually

isn't that the idea of migrations? xD I thought it will just be checked what was already done and what not and then execute all not executed ones.

No the issue is you need to do it sequentially. So you can't just insert a new one in the middle. Because that could lead to inconsistencies.

Exactly, I would also only allow this for debug,...

Okay actually it should have worked, the problem is another PR bumped the version number for the dav app on master already, therefor no update was done at all.

Fixed it now.

MorrisJobke · 2017-09-27T08:25:37Z

+		return [
+			'addressbooks' => ['id'],
+			'addressbookchanges' => ['id', 'addressbookid'],
+			'calendars' => ['id', 'calendarid'],


Doctrine\DBAL\Schema\SchemaException: There is no column with name 'calendarid' on table 'oc_calendars'.

Stupid copy and paste, but I checked the dav tables and it looks good now.

codecov-io · 2017-09-27T13:19:54Z

Codecov Report

Merging #6563 into master will decrease coverage by 0.01%.
The diff coverage is 34.14%.

@@             Coverage Diff              @@
##             master    #6563      +/-   ##
============================================
- Coverage     53.03%   53.01%   -0.02%     
- Complexity    22583    22606      +23     
============================================
  Files          1417     1422       +5     
  Lines         87873    87954      +81     
  Branches       1341     1341              
============================================
+ Hits          46601    46629      +28     
- Misses        41272    41325      +53

Impacted Files	Coverage Δ	Complexity Δ
...av/lib/Migration/Version1004Date20170926103422.php	`0% <0%> (ø)`	`1 <1> (?)`
version.php	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
...es/lib/Migration/Version1002Date20170926101419.php	`0% <0%> (ø)`	`1 <1> (?)`
core/Migrations/Version13000Date20170926101637.php	`0% <0%> (ø)`	`1 <1> (?)`
lib/public/Migration/BigIntMigration.php	`0% <0%> (ø)`	`4 <4> (?)`
lib/private/DB/ConnectionFactory.php	`58.02% <33.33%> (-0.95%)`	`24 <0> (+1)`
lib/private/DB/OCPostgreSqlPlatform.php	`96.42% <96.42%> (ø)`	`15 <15> (?)`
... and 2 more

MorrisJobke

Tested and works 👍

Fixed.

MorrisJobke · 2017-09-27T13:47:07Z

I also fixed the autoloader.

nickvergessen · 2017-09-29T08:55:21Z

What, postgres fails?

Error while trying to create admin user: An exception occurred while executing 'ALTER TABLE oc_personal_settings ALTER id TYPE BIGSERIAL':

SQLSTATE[42704]: Undefined object: 7 ERROR: type "bigserial" does not exist

Signed-off-by: Joas Schilling <coding@schilljs.com>

nickvergessen · 2017-09-29T09:11:51Z

Copied owncloud/core#28364

MorrisJobke

Still working fine 👍

nickvergessen added 3. to review Waiting for reviews enhancement labels Sep 19, 2017

nickvergessen added this to the Nextcloud 13 milestone Sep 19, 2017

nickvergessen requested review from MorrisJobke and rullzer September 19, 2017 10:54

rullzer approved these changes Sep 19, 2017

View reviewed changes

nickvergessen mentioned this pull request Sep 19, 2017

Fix database structure of twofactor backupcodes app to match 12 #6566

Merged

nickvergessen added the downstream label Sep 19, 2017

LukasReschke previously requested changes Sep 19, 2017

View reviewed changes

nickvergessen force-pushed the bigint-ids branch from 4df2a51 to 58b307f Compare September 20, 2017 07:11

nickvergessen force-pushed the bigint-ids branch 2 times, most recently from 4ef49be to beea34b Compare September 20, 2017 07:16

MorrisJobke requested changes Sep 26, 2017

View reviewed changes

nickvergessen force-pushed the bigint-ids branch 2 times, most recently from 53d0734 to 4243f40 Compare September 26, 2017 14:14

MorrisJobke requested changes Sep 27, 2017

View reviewed changes

MorrisJobke approved these changes Sep 27, 2017

View reviewed changes

MorrisJobke added 4. to release Ready to be released and/or waiting for tests to finish and removed 3. to review Waiting for reviews labels Sep 27, 2017

nickvergessen added 2 commits September 29, 2017 11:10

Add a base migration for easy updating to bigints

124b437

Signed-off-by: Joas Schilling <coding@schilljs.com>

Migrate all core id columns to bigint

ce3bcf7

Signed-off-by: Joas Schilling <coding@schilljs.com>

nickvergessen added 6 commits September 29, 2017 11:10

Migrate the twofactor_backupcodes app to bigint

c58ff29

Signed-off-by: Joas Schilling <coding@schilljs.com>

Migrate ids to bigint in the dav app

740d820

Signed-off-by: Joas Schilling <coding@schilljs.com>

Restore the sequencial order

c65e591

Signed-off-by: Joas Schilling <coding@schilljs.com>

Fix copy paste errors

155d451

Signed-off-by: Joas Schilling <coding@schilljs.com>

Fix bigint handling on postgres

a3bdb31

Signed-off-by: Joas Schilling <coding@schilljs.com>

Update autoloading

f205412

Signed-off-by: Joas Schilling <coding@schilljs.com>

nickvergessen force-pushed the bigint-ids branch from dcea970 to f205412 Compare September 29, 2017 09:11

MorrisJobke approved these changes Sep 29, 2017

View reviewed changes

MorrisJobke merged commit 870d716 into master Sep 29, 2017

MorrisJobke deleted the bigint-ids branch September 29, 2017 10:23

MorrisJobke mentioned this pull request Sep 29, 2017

Numeric value out of range: 167 Out of range value for column 'fileid' #2775

Closed

BernhardPosselt mentioned this pull request Jan 19, 2018

Updating Nextcloud 12.0.5 to 13.0.0.rc2 fails on Postgres #7960

Closed

Uh oh!

Uh oh!

Conversation

nickvergessen commented Sep 19, 2017

Uh oh!

mention-bot commented Sep 19, 2017

Uh oh!

rullzer left a comment

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Sep 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

LukasReschke left a comment

Choose a reason for hiding this comment

Uh oh!

nickvergessen commented Sep 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickvergessen commented Sep 20, 2017

Uh oh!

nickvergessen commented Sep 22, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Sep 27, 2017 • edited by codecov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MorrisJobke left a comment

Choose a reason for hiding this comment

Uh oh!

MorrisJobke commented Sep 27, 2017

Uh oh!

nickvergessen commented Sep 29, 2017

Uh oh!

nickvergessen commented Sep 29, 2017

Uh oh!

MorrisJobke left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Sep 19, 2017 •

edited

Loading

nickvergessen commented Sep 19, 2017 •

edited

Loading

codecov-io commented Sep 27, 2017 •

edited by codecov Bot

Loading