SAX Process an extremely large JSON file and also pull out chunks of the JSON?

[jdredd87]

I have A JSON file from eBay that they supply.

This file is 375mb big and only getting bigger!

And no. They won't change this.

I can currently read and process the data out of this file using the SAX Style JSON parsing.

Even in A 32 bit program it runs, and runs very quickly even.

One problem I face and can't find much about, is how can I get JSON chunks of data out of the SAX parsing, if even possible?

Included here is a short version of the data file they provide. Just a few items out of thousands.

aspects.clean.short.zip

{
"categoryTreeId": "100",
"categoryTreeVersion": "80",
"categoryAspects": [
{
"category": {
"categoryId": "183523",
"categoryName": "Accessory Mounts"
},
"aspects": [
{
"localizedAspectName": "Brand",
"aspectConstraint": {
"aspectDataType": "STRING",
"itemToAspectCardinality": "SINGLE",
..
..
..

In my case, categoryAspects is my target here to work with.
This is an array. Thousands of elements in it. Each element itself has data/arrays/blah inside it.

In the SAX processing if possible, is in each of these categoryAspects array, I want to extract the raw JSON for that array element.

I have A requirement to break this file up and write individual records to A database for quick look ups by some other applications.

So they could in this case, look up table for categoryId , 183523 , and there would be a field of JSON data. The JSON data for this individual categoryAspects array element.

So in the end, thousands of thousands of rows of table data. 1 row per categoryId.

I can do this with XSuperJSON/XSuperObject but I have to use 64bit AND it takes like 6gb or more of memory to load. And it takes forever to load.

var
j: isuperobject;
t: tstringlist;
a: isuperarray;
idx: integer;
begin
t := tstringlist.Create;
t.LoadFromFile('aspects.clean.short.json');
j := so(t.text);
t.Free;
a := j.a['categoryAspects'];
for idx := 0 to a.Length - 1 do
begin
Memo1.Lines.Add(a.O[idx].AsJSON(false));
Memo1.Lines.Add('----------------------------------------------------');
end;
end;