database performance - how to load data faster with talend and sql server -


i use talend load data sql-server database.

it appears weakest point of job not dataprocessing, effective load in database, not faster 17 rows/sec.

the funny point can launch 5 jobs in same time, , they'll load @ 17rows/sec .

what explain slowness , how improve speed?

thanks

new informations:

the transfer speed between desktop , server 1mbyte

my job commits every 10 000

i use sql server 2008 r2

and schema use jobs this:

enter image description here

database insert or update methods incredibly costly database cannot batch of commits @ once , must them line line (acid transactions force because if attempted insert , failed of other records in commit fail).

instead, large bulk operations best predetermine whether record inserted or updated before passing commit database , sending 2 transactions database.

a typical job needed functionality assemble data insert or updated , query database table existing primary keys. if primary key exists can send update, otherwise insert. logic can done in tmap component.

insert or update job example

in job have data wish insert or update database table contains pre-existing data:

initially loaded data

and wish add following data it:

insert or update data

the job works throwing new data thashoutput component can used multiple times in same job (it puts memory or in large instances can cache disk).

following on 1 lot of data read out of thashinput component , directly tmap. thashinput component utilised run parameterised query against table:

parameterised query parameter config

you may find this guide talend , parameterised queries useful. here returned records (so ones inside database already) used lookup tmap.

this configured inner join find records need updated rejects inner join inserted:

tmap configuration

these outputs flow separate tmysqloutput components update or insert necessary. , when main subjob complete commit changes.


Comments

Popular posts from this blog

windows - Single EXE to Install Python Standalone Executable for Easy Distribution -

c# - Access objects in UserControl from MainWindow in WPF -

javascript - How to name a jQuery function to make a browser's back button work? -