JDBC Performance Benchmark Comparsion & Validation

Hi,
Are there any published benchmarks available on comparing Microsoft's
JDBC Driver for SQL Server with other JDBC drivers so that I can
validate some results acquired in our testlab.Surprisingly ISVs did
much better than Microsoft in this area. The following table summarises
the results we have derived by comparins 4 drivers, i-net's Opta 2000,
JNetDirect JSQL Connect, Microsoft and the Open Source jTDS driver.All
timings are in milliseconds.
__________________________________________________ _____________________
Test| i-net OPTA 2000| JNetDirect JSQL Connect | Microsoft Driver
|jTDS
UnpreparedInsert 64775454955601055219
UnpreparedUpdate30803247552633028639
UnpreparedDelete62988485564427443452
PreparedInsert25509239491659818093
PreparedUpdate28793248112690832201
PreparedSelects3656475261465880
PreparedDelete21872177012528019729
PreparedInsertMultipleStatements113303801868770291821
CursorInsert48260389973469932554
CursorUpdate37004427485982339310
CursorDelete53801364996215642082
ResultSetReadAllRows1535704955454
CursorReadAllRows10365481746352
BatchUnpreparedInsert35216239182981424357
BatchUnpreparedUpdate32052197532109119635
BatchPreparedInsert29456232452607922704
BatchPreparedUpdate11931101771232310037
BlobInsert6975808073757273
ClobInsert10309115871033410029
BlobRetrieve795767634884
ClobRetrieve849103313941464
InvokeStoredProcedure1785181637031824
__________________________________________________ __________________
This was done in a lab connecting 2 Windows 2000 boxes over gigabit
ethernet.
Let me know if missed out in providing other details.
Thank You,
paddy
The JNetDirect license prohibits publishing of any benchmark results;
so does the DataDirect license, I think. That's why you won't find any
benchmark results including these drivers. Some older results (without
the two mentioned drivers) can be found on the jTDS webpage:
http://jtds.sourceforge.net/benchTest.html
And it's not surprising at all the MS driver is the worst of all the
bunch. It's a 4 year old JDBC 2.0 driver from DataDirect, with only a
few bugs fixed since. Plus Microsoft probably has no intention of
really supporting Java or making it seem fast enough for any serious
use.
Alin,
The jTDS Project.
|||Alin,
Thank You for the clarification on that. I have just removed my
previous message lest I should incur any problems with licensing.
The older results are not exactly in agreement with my tests.Is there
anyway you could ask your project folks to update the benchTest so that
it helps me compare and validate our own internal decision in NOT
choosing MS SQL driver.
Thanks
Paddy
|||Paddy,
It was I who benchmarked the drivers and posted those results on the
site. I would do it again, but there are a number reasons for not doing
it:
(i) as you can see, it's not possible to list all drivers and comparing
jTDS to the bridge and the MS driver is pretty useless
(ii) the benchmarks themselves are pretty irrelevant, they don't match
real world performance and are built in such a way as to show the
strengths of the respective driver
(iii) results can vary a lot depending on setup and the fact that e.g.
most of these drivers don't actually implement batching is not visible
in the tests (the differences are minor, but some drivers send a single
request and process a single response while others send one request per
executed statement -- a huge difference over a network with even a tiny
latency)
(iv) performance is not the ultimate goal; all the drivers have issues
and in a server environment stability is essential; so is memory usage,
for example -- being able to control buffering and caching could prove
useful; there are a lot of features that you could look at before
deciding speed is important.
My advice would be to build a benchmark of your own that tests what you
actually need (e.g. if you use Hibernate -- or a framework that behaves
similarly -- then you would probably want a driver that's really fast
at prepare-execute-close and maybe has statement caching and meta data
caching to help with this). Then, based on that benchmark and some
research regarding the features and stability of the various choices,
make a decision. If you'll care to share the results with us, that's
ok. Anyway, they will probably be very particular to your case.
And my second advice would be not to go with the MS driver. I would not
waste my time on jTDS if the MS driver was any good and had some decent
support beyond "this is a known issue, but it's not our decision to fix
it".
Alin,
The jTDS Project.
|||Paddy,
Before you write off a Microsoft JDBC driver you should take a look at this:
http://www.microsoft.com/sql/downloads/2005/jdbc.mspx
What it is:
A new (unrelated to our previous offering) type four JDBC 3.0 compliant
driver for Sql Server 2000 and Sql Server 2005. It is built using the
1.4.2jvm and it is currently in Beta 1. While we can't guarantee a release
date at this point it is our intention to ship "within a reasonable window"
of Sql Server 2005.
Unlike our previous driver we are going to be fully supporting this with an
in-house product group and PSS. We want to ship as many betas as we can cram
in the reduced ammount of time that we have left and we could use any and
all feedback on this product even if you decide to go with a different
vendor.
Alin,
"I would not waste my time on jTDS if the MS driver was any good and had
some decent support beyond "this is a known issue, but it's not our decision
to fix it".
I can definitelly guarantee that the new Microsoft driver will have decent
support, the "any good" part is of course still to be determined. We hope to
make it as good as possible before it ships. That said I hope that your
statement is somewhat retorical, it is awesome to see a free Sql Server
driver with the performance of jTDS and I certainly hope that you continue
to "waste your time" on this provider. I have learned a lot by reading your
posts (I am somewhat of a newsgroup junkie) and hope to be in a lot more
threads with you.
Angel Saenz-Badillos [MS] Dataworks
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging about ADO.NET: http://weblogs.asp.net/angelsb/
"Alin Sinpalean" <alin@.earthling.net> wrote in message
news:1119618439.274894.102390@.f14g2000cwb.googlegr oups.com...
> Paddy,
> It was I who benchmarked the drivers and posted those results on the
> site. I would do it again, but there are a number reasons for not doing
> it:
> (i) as you can see, it's not possible to list all drivers and comparing
> jTDS to the bridge and the MS driver is pretty useless
> (ii) the benchmarks themselves are pretty irrelevant, they don't match
> real world performance and are built in such a way as to show the
> strengths of the respective driver
> (iii) results can vary a lot depending on setup and the fact that e.g.
> most of these drivers don't actually implement batching is not visible
> in the tests (the differences are minor, but some drivers send a single
> request and process a single response while others send one request per
> executed statement -- a huge difference over a network with even a tiny
> latency)
> (iv) performance is not the ultimate goal; all the drivers have issues
> and in a server environment stability is essential; so is memory usage,
> for example -- being able to control buffering and caching could prove
> useful; there are a lot of features that you could look at before
> deciding speed is important.
> My advice would be to build a benchmark of your own that tests what you
> actually need (e.g. if you use Hibernate -- or a framework that behaves
> similarly -- then you would probably want a driver that's really fast
> at prepare-execute-close and maybe has statement caching and meta data
> caching to help with this). Then, based on that benchmark and some
> research regarding the features and stability of the various choices,
> make a decision. If you'll care to share the results with us, that's
> ok. Anyway, they will probably be very particular to your case.
> And my second advice would be not to go with the MS driver. I would not
> waste my time on jTDS if the MS driver was any good and had some decent
> support beyond "this is a known issue, but it's not our decision to fix
> it".
> Alin,
> The jTDS Project.
>
|||Angel,
I am definitely going to download and take a look at the new driver. I
certainly hope it's better than the previous one, which I think you
agree was far from what users needed, not only in terms of
compatibility and missing features but also in stability and support (I
have honestly never seen a bug report end in "Thanks that fixed it for
me").
There's one more huge difference between jTDS and Microsoft's offering:
jTDS is completely free; you may do with it whatever you want/need
(including selling it!) as long as it remains free (as in speech).
Reading the new MS driver's license I see that it now also prohibits
publishing of any benchmark results. That's a big pity. Even though I
said performance isn't everything when deciding for a JDBC driver, it's
certainly an issue and not being able make benchmark results public
definitely doesn't help anyone.
Alin,
The jTDS Project.
|||Alin,
I am a big supporter of your work, I really like the fact that Sql Server
has had a free JDBC driver and have always hoped that we could stay out of
this space since it is not our core competency. As it became time to ship
Sql Server 2005 we became worried by two important issues.
First, we need to make sure that we expose the new features that we have
worked so hard on over the last five years to java users. Second, a number
of important Sql Server customers no longer saw our previous JDBC solution
as a viable solution going forward and they are not willing to consider
third party alternatives.
We have started late and I really expect to be playing catch up for a long
time, but we _will_ be seeing the "Thanks that fixed it for me" bugs, we
will be trying to do the right thing in terms of features (from past
experience this may still mean that we will have missing features) and most
importantly we will do our best in terms of stability and support. We are
already taking bugs through the microsoft product feedback center:
Entering a bug:
Go to http://lab.msdn.microsoft.com/produc...k/default.aspx
Product/Technology:
SQL Server 2005
Version:
SQL Server 2005 Community Technology Preview June 2005 - Developer
Edition
Product Language:
English
Category:
JDBC Driver
I honestly believe that for this driver to be successfull we are going to
need a lot of help from the Java community, we are specially going to need
help from a set of people with whom we may not have seen eye to eye in the
past.
Angel Saenz-Badillos [MS] DataWorks
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging : http://weblogs.asp.net/angelsb/
"Alin Sinpalean" <alin@.earthling.net> wrote in message
news:1119950668.919962.202060@.g47g2000cwa.googlegr oups.com...
> Angel,
> I am definitely going to download and take a look at the new driver. I
> certainly hope it's better than the previous one, which I think you
> agree was far from what users needed, not only in terms of
> compatibility and missing features but also in stability and support (I
> have honestly never seen a bug report end in "Thanks that fixed it for
> me").
> There's one more huge difference between jTDS and Microsoft's offering:
> jTDS is completely free; you may do with it whatever you want/need
> (including selling it!) as long as it remains free (as in speech).
> Reading the new MS driver's license I see that it now also prohibits
> publishing of any benchmark results. That's a big pity. Even though I
> said performance isn't everything when deciding for a JDBC driver, it's
> certainly an issue and not being able make benchmark results public
> definitely doesn't help anyone.
> Alin,
> The jTDS Project.
>
|||Angel,
Good luck with your work, then. As for help, you can try out the jTDS
test suite. I have tried to put the driver through it and there are a
lot of starting points for you. I know this is only a beta, but you
will need to put in quite some effort to bring it up to the level your
customers expect.
Alin,
The jTDS Project.
|||Thanks for your comments, as far as trying out the test suite I was not
aware that you had a jTDS specific test suite available for download. We
have run this driver through both the JNetDirect's JDBC Performance
Benchmark and the i-net Software's BenchTest 2.1 for MS SQL Server ,aside
from a minor coercion issue the results were encouraging.
Angel Saenz-Badillos [MS] DataWorks
This posting is provided "AS IS", with no warranties, and confers no
rights.Please do not send email directly to this alias.
This alias is for newsgroup purposes only.
I am now blogging: http://weblogs.asp.net/angelsb/
"Alin Sinpalean" <alin@.earthling.net> wrote in message
news:1119977698.367755.199710@.g14g2000cwa.googlegr oups.com...
> Angel,
> Good luck with your work, then. As for help, you can try out the jTDS
> test suite. I have tried to put the driver through it and there are a
> lot of starting points for you. I know this is only a beta, but you
> will need to put in quite some effort to bring it up to the level your
> customers expect.
> Alin,
> The jTDS Project.
>
|||Angel,
The jTDS test suite is not a performance test, it's a (rather large,
~500 tests) collection of unit tests created to ensure any regressions
in jTDS are caught as soon as possible. Very few of these are jTDS
specific and will fail with other drivers, but most are testing clearly
defined JDBC behaviors.
Regarding something you said in an earlier post:

> First, we need to make sure that we expose the new features that we have
> worked so hard on over the last five years to java users. Second, a number
> of important Sql Server customers no longer saw our previous JDBC solution
> as a viable solution going forward and they are not willing to consider
> third party alternatives.
Have you (not personally, but Microsoft) ever considered supporting
jTDS for example in order to obtain the same results, faster? No
offence, I'm just curious; if you did consider this alternative were
there any technical reasons for not doing it?
Alin,
The jTDS Project.