Recently, we skillfully handled an industry project to import a great amount
of data from file to Oracle in comparatively short time.
At the beginning, we tried to import the data with the sqlldr of Oracle, only
to find it is surprisingly time-consuming to load a great amount of data: Too
slow if 2.5 hours have to be spent to import a table containing 80 million
Later, by adopting the parallel import of sqlldr, the time is shortened to
0.8 hour. Here are the full details:
Train of thoughts
Split the data file to import into 10 shares. Then, with the multi-task
parallelism, execute the sqlldr command for the corresponding shares.
Needless to say, prepare the control files of the same amount. Then, multiple
clients will start to import data to database all at the same time.
Please note these two things: 1. The way to generate multiple sqlldr commands
and corr... (more)
Recently, a performance issue occurs in a big detail report, where the
original data volume in the two data tables as it involved has exceeded 3
million rows; the data volume that the report can retrieve after filtering
remains at round 2 million rows. As we expected before, by using paged
retrieve method, this report would present a much higher performance.
However, for the fact that the user also needs to export the data from the
report, the only option is that the data retrieve can be achieved from Oracle
at a time. As the JDBC of Oracle runs quite slowly, it takes least 6-7 ... (more)
The data computation layer in between the data persistent layer and the
application layer is responsible for computing the data from data persistence
layer, and returning the result to the application layer. The data
computation layer of Java aims to reduce the coupling between these two
layers and shift the computational workload from them. The typical
computation layer is characterized with below features:
Ability to compute on the data from arbitrary data persistence layers, not
only databases, but also the non-database Excel, Txt, or XML files. Of all
these computations, the... (more)
In Java development, the typical data computation problems are characterized
Long computation procedure requiring a great deal of debugging Data may from
database, or Excel/Txt Data may from multiple databases, instead of just one.
Some computation goals are complex, such as relative position computation,
and set-related computation
Just suppose a sales department needs to make statistics on the top 3
outstanding salesmen ranking by their monthly sales in every month from Jan
to the previous month, based on the order data.
Java alone is difficult to handle such computations... (more)
Based on the generic data type, esProc provides the sequence and the Table
Sequence for implementing the complete set-lizing and the much more
convenient relational queries.
The relation between the department and the employee is one-to-many and that
between the employee and the SSN (Social Security Number) is one-to-one.
Everything is related to everything else in the world. The relational query
is the access to relational dataset with the mathematical linguistics. Thanks
to the associated query, the relational database (RDBMS) is extensively
I Case and Comparison... (more)