DB Friendly mods

[user31415]

after Line # 267: SIAttributeSizeNT = 48
add delimiter = "|" # CSV definable delimiter because the data contains commas

modified line # 636
FROM: if ( p == 0 ) or ( p == 5 ):
TO: if ( p == 0 ) or ( p >4 ): # probably unnecessary but I was looking for a spurious generated "1" when 4 filenames

modified Line # 663-677
Make headers DB friendly: Replace spaces with underscores, replace objectionable symbols (# & /), prefix FN1...4 to make headers unique.

after line # 716: # If this goes above four FN attributes ...
add: # Limit to 4
add: if MFTR['fncnt'] > 4: MFTR['fncnt'] = 4

after line # 723: # Pad out the remaining FN columns
add: tmpBuffer = [] # Initialize to empty. *** This was the cause of the spurious "1"

modified line # 890:
FROM: output_file = csv.writer(open(options.output, 'wb'), dialect=csv.excel,quoting=1)
TO: output_file = csv.writer(open(options.output, 'wb'), delimiter = delimiter, quoting=1) # Defined delimiter

The above allowed me to successfully import a little under 1/2 million records into SQLite.

[user31415]

One more change to the modified line # 890 above:
FROM: output_file = csv.writer(open(options.output, 'wb'), dialect=csv.excel,quoting=1)
TO: output_file = csv.writer(open(options.output, 'wb'), delimiter = delimiter, quoting=csv.QUOTE_MINIMAL) # Defined delimiter and remove most quotes for SQLite.

[dkovar]

Greetings,

I failed to include this awhile back. Could we get these updates included in the current version? They're definitely valuable.

Thanks.

-David