Read big JSON files?

[sdegutis]

Bug Report

🔎 Search Terms

"large json files"

Problem

None of the predefined sections are relevant here.

I'm trying to get types from a JSON file using resolveJsonModule. I'm using VS Code to do everything here. My JSON file is 4.9 MB and 200k lines. When I try to inspect the result of require('./big-file.json), it says {}. But it works great for another smaller file, albeit a bit slow.

Not sure if this is relevant or not, but it also fails for this big file in quicktype and works for the slightly smaller file. I assumed they're using TypeScript's parser/compiler under the hood, which would explain it. If they're not, then maybe it's a Node.js limitation that it's coming across.

To be clear, the require call itself works fine and returns a valid JSON object. So it might not be a Node.js memory limitation issue. But it might be one if TypeScript is more than doubling the memory by all its type analysis, or by duplicating substrings, etc.

Just some thoughts. I have no idea what's going on. All I know is that I was hoping VS Code would analyze my JSON files for me, but it's only working on some files.

[RyanCavanaugh]

Not sure how quicktype works.

Our JSON support involves using the normal type system mechanics and is going to not do well on huge huge huge files. I'd really recommend generating a .d.ts file from a smaller subset of it or writing a manual definition if your file is that big; there's not much we can do to improve performance in this particular case.

[sdegutis]

That's pretty much what I did, manually figured out the types at runtime and wrote them down in a .d.ts file. So never mind I suppose. Thanks.

[jzxchiang1]

Same error here.

[alanscandrett]

So I have a large GeoJSON object as a file and it runs into this same issue. All the worlds nations outlines are represented in coordinate pairs.

I have however included a context.d.ts file which is read successfully, but the GeoJSON object is not and it seems to only appear as an empty object (error below):
Property 'type' is missing in type '{}' but required in type 'CountryJson'.ts(2741)

the context.d.ts file:

declare namespace SomeNamespace {
    export interface CountryJson {
        features?: Array<T>;
        type: string;
    }
}

After deleting countries from the array, starting by removing half and doing so until it goes, the error eventually disappears. I got it down to the difference of one additional country. I tried deleting the same country and random countries and it has the same effect.

It seems like TS taps out with larger objects, it would be good if the error message was clear about this as it caused quite a bit of time to be wasted figuring this out. I also am not sure how to proceed as it makes no sense to split the file when it is within the JSON and GeoJSON spec, but the error is never going to go away..

[kdthomas2121]

@alanscandrett did you get anywhere with this? I'm trying to do the same with a geojson file

[alanscandrett]

@alanscandrett did you get anywhere with this? I'm trying to do the same with a geojson file

Unfortunately not, I just split the dataset :/

[MrScriptX]

Same problem, that is an issue. My file isn't that big (33k lines) and it's just a bunch of arrays. No complex nesting or anything.

[1valdis]

So why is this closed? It's not fixed and there's no best practice recommended on dealing with big JSON files if static typing is needed. I believe this should be reopened.

[snarbles2]

#42761 (comment)

Our JSON support involves using the normal type system mechanics and is going to not do well on huge huge huge files. I'd really recommend generating a .d.ts file from a smaller subset of it or writing a manual definition if your file is that big; there's not much we can do to improve performance in this particular case.

[1valdis]

This is useful but raises further questions and issues. Following this issue and that very approach, I scanned the whole Declaration Files section of the handbook, and found no mentions of doing declaration files for .json files. So I tried a naïve approach. I have a JSON file with an array of objects. So for this code

import data from './omm.json' with { type: 'json' };

I created an omm.d.ts file:

declare type Items = Array<{ key: "value" }> // simplified

export default Items;

Which Typescript doesn't recognize.

Then elsewhere I saw people had success with *.d.json.ts files, so I tried that; this also forced me to enable allowArbitraryExtensions for the omm.d.json.ts to be recognized by Typescript. This did something (apparently now at least Typescript reads the .d.json.ts file), but now this code

data.map(item => console.log(item))

errors on .map with

Property 'map' does not exist on type 'typeof import("[redacted]/omm.d.json", { with: { "resolution-mode": "import" } })'.

which doesn't explain anything and no idea where to proceed here, because Typescript kind of read the file, but still didn't consider that the data should be of type Array.

So the approach is actually not documented and anyone trying this is basically on their own, could be ending up like me. Unless we find out what's going on, fix/work around this, and people could use the info from this issue.