FDM Bulk import exceptions

I’m trying to import https://www.dropbox.com/s/fiaxbqvjib4chdl/LaunchSites.json.zip?dl=0 but am seeing this exception with FDM: https://pastebin.com/GLWGNGai

Help appreciated or troubleshooting appreciated :slight_smile:.

I was able to import this file successfully. Did you change .props file ?

./fdm -source path=./data/LaunchSites.json -dest key=KKKK
2020-06-11.14.02.57
===================

FDM   1.14
2020-04-10 16:30
Loaded property file path: /Users/jayreddy/customer_support/fdm/fdm-1.14/fdm.props

Field Formatting
================
<<None>>

Config File
===========
   load:                      data.json
   host:                      cloud
   policy.data:               COPY

Connecting to: CLOUD
Connection was successful.
Starting Load at 2020-06-11 14:03:00.
File LaunchSites.json -> LaunchSites loaded 3517  execution time:    4.996 sec  ReadQ   0% - 0  ConvertQ   6% - 17
File LaunchSites.json -> LaunchSites loaded 6901  execution time:    9.997 sec  ReadQ   0% - 0  ConvertQ   0% - 1
File LaunchSites.json -> LaunchSites loaded 10375  execution time:   14.995 sec  ReadQ   0% - 0  ConvertQ   0% - 0
File LaunchSites.json -> LaunchSites loaded 14077  execution time:   19.996 sec  ReadQ   0% - 0  ConvertQ   0% - 0
File LaunchSites.json -> LaunchSites loaded 18199  execution time:   24.995 sec  ReadQ   1% - 2  ConvertQ   0% - 0
File LaunchSites.json -> LaunchSites loaded 22845  execution time:   29.995 sec  ReadQ   0% - 0  ConvertQ   0% - 0
File LaunchSites.json -> LaunchSites documents 27672 size 68362 KB  execution time:   34.386 sec  1.941 MB/sec  804.7 docs/sec
FDM finished at 2020-06-11 14:03:35.

Not touched the props file. Let me try again.

still getting errors, and no way of knowing what 258 documents were skipped :frowning:

Auto discovery of file types.
File LaunchSites.json is of type JSON Line.
File .DS_Store is of type Unknown.
Connecting to: CLOUD
Connection was successful.
Starting Load at 2020-06-11 11:35:07.
File LaunchSites.json -> LaunchSites loaded 3412  execution time:    4.994 sec  ReadQ   0% - 0  ConvertQ   4% - 13
File LaunchSites.json -> LaunchSites loaded 6373  execution time:    9.996 sec  ReadQ   0% - 0  ConvertQ   0% - 0
File LaunchSites.json -> LaunchSites loaded 10105  execution time:   14.996 sec  ReadQ   0% - 0  ConvertQ   2% - 5
File LaunchSites.json -> LaunchSites loaded 14112  execution time:   19.998 sec  ReadQ   3% - 8  ConvertQ   1% - 3
File LaunchSites.json -> LaunchSites loaded 18394  execution time:   24.999 sec  ReadQ   0% - 0  ConvertQ  31% - 94
File LaunchSites.json -> LaunchSites loaded 23856  execution time:   29.998 sec  ReadQ  85% - 255  ConvertQ 100% - 300
File LaunchSites.json -> LaunchSites documents 27672 size 68362 KB  execution time:   33.640 sec  1.985 MB/sec  822.6 docs/sec
File .DS_Store ->  skipped
Failed to drain insert queue.java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
ERROR Shutdown Insert Queue 258 left
FDM finished at 2020-06-11 11:37:51.

~/Downloads/fdm-1.14 took 2m 51s

Can you please retry updating transactionSize=100 in .props file to 10 ?

It looks like even though it’s reporting “258 left”, all were imported fine. Odd