database performance - how to load data faster with talend and sql server -
i use talend load data sql-server database.
it appears weakest point of job not dataprocessing, effective load in database, not faster 17 rows/sec.
the funny point can launch 5 jobs in same time, , they'll load @ 17rows/sec .
what explain slowness , how improve speed?
thanks
new informations:
the transfer speed between desktop , server 1mbyte
my job commits every 10 000
i use sql server 2008 r2
and schema use jobs this:
database insert or update
methods incredibly costly database cannot batch of commits @ once , must them line line (acid transactions force because if attempted insert , failed of other records in commit fail).
instead, large bulk operations best predetermine whether record inserted or updated before passing commit database , sending 2 transactions database.
a typical job needed functionality assemble data insert or updated
, query database table existing primary keys. if primary key exists can send update
, otherwise insert
. logic can done in tmap
component.
in job have data wish insert or update
database table contains pre-existing data:
and wish add following data it:
the job works throwing new data thashoutput
component can used multiple times in same job (it puts memory or in large instances can cache disk).
following on 1 lot of data read out of thashinput
component , directly tmap
. thashinput
component utilised run parameterised query against table:
you may find this guide talend , parameterised queries useful. here returned records (so ones inside database already) used lookup tmap
.
this configured inner join
find records need updated
rejects inner join
inserted:
these outputs flow separate tmysqloutput
components update
or insert
necessary. , when main subjob complete commit
changes.
Comments
Post a Comment