SELECT involving CLOB column is 15X slow than without CLOB column

[gyut]

Steps to reproduce

User.role is a CLOB. Takes 80ms.
User.select(User.column_names).all.to_a
D, [2018-08-13T12:03:31.270377 #56195] DEBUG -- : User Load (78.8ms)

Without User.role it takes 5ms.
User.select(User.column_names - %w(roles)).all.to_a
D, [2018-08-13T12:03:23.412199 #56195] DEBUG -- : User Load (4.5ms)

Expected behavior

Running a SELECT query that includes a CLOB column should be roughly as fast as running a SELECT query without the CLOB column.

Actual behavior

Running a SELECT query that includes a CLOB column is about 15x slower than without the CLOB column.

System configuration

Rails version: 5.2.1

Oracle enhanced adapter version: 5.2.3

Ruby version: 2.4.2p198

Oracle Database version: Oracle Database 11g Release 11.2.0.4.0 - 64bit Production

[kubo]

I guess that it is due to the number of network round trips.
When a CLOB column is not included in a query, rows are fetched up to the number of prefetch rows (default 100) in a network round trip. However when a CLOB column is included, CLOB column values are fetched as LOB locators. Then contents in a CLOB are retrieved from LOB locators. It requires more than one network round trip for each CLOB value.

When OCI_ATTR_DEFAULT_LOBPREFETCH_SIZE (the default value seems zero) is changed, the performance will be improved.

class OCI8
  def default_lobprefetch_size
    # OCI_ATTR_DEFAULT_LOBPREFETCH_SIZE = 438 in oci.h
    @session_handle.send(:attr_get_ub4, 438)
  end
  def default_lobprefetch_size=(newsize)
    @session_handle.send(:attr_set_ub4, 438, newsize)
  end
end

...
conn = ... get ruby-oci8 connection from oracle-enhanced connection ...
conn.default_lobprefetch_size = 8192 # determine proper size by yourself.
...
User.select(User.column_names).all.to_a

[gyut]

Thanks for the quick reply, @kubo. That looks like it might help. Can you expand on the details of how to get the connection and set the default_lobprefetch_size in a Rails app?

[kubo]

ActiveRecord::Base.connection.raw_connection?
I'm not familiar with Rails specific features.

I think it is better to set the lobprefetch size just after ruby-oci8 connection is established as prefetch_rows.

[kubo]

@gyut
As far as I checked packets captured by wireshark, lobprefetch size may not help. When lobprefech size is set, LOB contents are sent with other column values. However same contents are sent again when OCI8::CLOB#read is called. Moreover it may decrease performance because same contents are sent twice.

I'll add OCI8#lobprefetch_size (renamed from default_lobprefetch_size) in the next ruby-oci8 release. But I'll hide it in documents because I'm not sure whether it is usable for users.

[gyut]

Thanks, @kubo. That's exactly what we saw last week. We were able to increase and vary the lobprefetch, but no setting improved performance, and some settings degraded performance.

We'd love to hear other ideas for improving the performance of tables with CLOB columns.

[kubo]

There are likely two issues.

1. Lob prefetching doesn't work as explained at my previous comment.

   This will be solved by changing ruby-oci8 code to fetch LOBs as strings, not as LOB locators.

   If you know the maximum size of CLOB data, change ruby-oci8 code as follows.

   • Add the following code here to make OCI8::BindType::String handle CLOB data

        when :clob
          param = {:length => 1024 * 1024}
        when :nclob
          param = {:length => 1024 * 1024, :nchar => true}

     It allocates 1024 * 1024 bytes for each CLOB even when the actual size is small.
     When the actual size is larger than the allocated size, fetching fails with ORA-01406: fetched column value was truncated.
   • Add the following code somewhere to make ruby-oci8 use OCI8::BindType::String for CLOB columns

     OCI8::BindType::Mapping[:clob] = OCI8::BindType::String
     OCI8::BindType::Mapping[:nclob] = OCI8::BindType::String

   If you don't know the maximum size of CLOB data, it will be fixed by using piecewise defines for LOBs using the data interface. It will need some time to implement it in ruby-oci8. It will be at the weekend of the next at earliest.

2. Rows prefetching doesn't work when a CLOB column is included in a query.

   When a CLOB column is not included in a query, rows are fetched up to the number of prefetch rows (default 100) in a network round trip as noted before. However when a CLOB column is included in a query, rows are fetched one by one no matter whether the column is fetched as LOB locater or as string (using OCI8::BindType::String). (I checked it on the latest Oracle version - 18.3.0.0.0.)

   This issue is fixed by using array fetching - allocating buffer for rows in advance and fetching more than one row at once in ruby-oci8 layer. (Prefetching caches rows in Oracle library layer.) Ruby-oci8 doesn't support array fetching now. However it may be implemented relatively easily. I cannot say when. I may overlook some pitfalls to implement it.

[gyut]

Thanks for the super-details analysis, @kubo! If you have time in the next few weeks to make improvements, we'd be happy to test them.

[kubo]

@gyut
I commited kubo/ruby-oci8@669e29f. Use the latest code as follows.

$ git clone https://github.com/kubo/ruby-oci8.git
$ cd ruby-oci8
$ gem build ruby-oci8.gemspec
$ gem install ./ruby-oci8-2.2.6.gem

LOB contents are retrieved as strings, not as LOB locators, by the following code.

OCI8::BindType::Mapping[:clob] = OCI8::BindType::Long
OCI8::BindType::Mapping[:nclob] = OCI8::BindType::Long
OCI8::BindType::Mapping[:blob] = OCI8::BindType::LongRaw
OCI8::BindType::Mapping[:bfile] = OCI8::BindType::LongRaw

Note that when LOB columns are fetched as LOB locators, empty lobs and NULLs are distinguishable. However when LOB columns are fetched as string, both empty lobs and NULLs are fetched as nil.

OCI8::BindType::Long and OCI8::BindType::LongRaw allocates buffers internally as follows.

They initially allocate a 32-kilobyte buffer for each column. When column data are longer than allocated buffers, they allocate an additional buffer whose size is twice. The size doesn't increase infinitely. The maximum size of one buffer is 8M.

For example

• Initial buffer size is 32K.
• When column data are longer than 32K, 64K buffer is added and total size become 96K.
• When column data are longer than 96K, 128K buffer is added and total size become 224K.
• When column data are longer than 224K, 256K ...
• ..., 512K ...
• ..., 1M ...
• ..., 2M ...
• ..., 4M ...
• ..., 8M ...
• ..., 8M ...
• ..., 8M ...

When a LOB size is 100K, the initial, second and third buffers contain 32K, 64K, 4K respectively.

The initial and maximum sizes are customizable by the following code.

OCI8::BindType::Base.initial_chunk_size = 4 * 1024  # default: 32 * 1024
OCI8::BindType::Base.max_chunk_size = 128 * 1024 * 1024 # default: 8 * 1024 * 1024

I may change the default sizes later.

Array fetching is working locally and has not been committed yet. I'm now writing tests about it.

[kubo]

@gyut
I tested the commit in the previous comment by using ruby-oci8 test cases.
I'm not sure whether it passes oracle-enhanced test cases...

[yahonda]

@kubo Thanks for the commit. I have not taken a look at the commit in detail yet, let me inform that Oracle enhanced adapter CI uses the master branch of ruby-oci8. The latest CI https://travis-ci.org/rsim/oracle-enhanced/jobs/423702697#L672 works fine using kubo/ruby-oci8@669e29f

[kubo]

@yahonda Thanks for letting me know the CI.

I didn't explain my concern enough. I guess that the oracle-enhanced test works well when OCI8::BindType::Mapping[:clob] = OCI8::BindType::Long. However I haven't checked it.

[yahonda]

Understood your concern.

[kubo]

Unfortunately, oracle-enhanced tests failed when LOBs are fetched as strings.
See https://travis-ci.org/kubo/oracle-enhanced/jobs/424338945

Failures are categorized into two types.

1. Empty CLOB data are fetched as nil.
   This is similar to varchar2 data type, which treats empty strings as NULL.
   I have no idea to fix it.
2. write_lob expects a LOB locator but a string are fetched.
   This disappeared by commenting out this line.
   See https://travis-ci.org/kubo/oracle-enhanced/jobs/424355106

[kubo]

@gyut
I did all what I could. Could you use the latest code as follows.

$ git clone https://github.com/kubo/ruby-oci8.git
$ cd ruby-oci8
$ gem build ruby-oci8.gemspec
$ gem install ./ruby-oci8-2.2.6.gem

It reduces number of network round trips even when LOB data are fetched as LOB locators.

Suppose the following cases.

• A: No CLOB in a query
• B: CLOBs in a query & ruby-oci8 2.2.6 or before
• C: CLOBs in a query & latest ruby-oci8 in the github.
• D: CLOBs in a query & latest ruby-oci8 in the github & fetching LOB as string

C is the case just using the latest code in your application.
D is the case that (1) OCI8::BindType::Mapping[:clob] = OCI8::BindType::Long is set, (2) one line of oracle-enhanced is commented out as the previous comment and (3) your application doesn't use empty CLOB.

Case	number of network round trips
A	((total_rows + 1).to_f / prefetch_rows).ceil
B	1 + (total_rows + 1) + total_rows * number_of_clob_columns * 2
C	1 + ((total_rows + 1).to_f / prefetch_rows).ceil + total_rows * number_of_clob_columns
D	1 + ((total_rows + 1).to_f / prefetch_rows).ceil

The following table is the number of network round trips for each query step when one CLOB column is included in a query which returns 5 rows and the setting of prefetch rows is 3.
Emoji symbols 1️⃣, 2️⃣ and 3️⃣ indicate number of rows in the response of a network round trip. 🔚 is no-data-found in the response. a in a+b is the number of network round trips by fetch and b is by lob.read.

Case	A	B	C	D
exec	1 3️⃣	1	1	1
1st fetch + lob.read	0	1+2 1️⃣	1+1 3️⃣	1 3️⃣
2nd fetch + lob.read	0	1+2 1️⃣	0+1	0
3rd fetch + lob.read	0	1+2 1️⃣	0+1	0
4th fetch + lob.read	1 2️⃣ 🔚	1+2 1️⃣	1+1 2️⃣ 🔚	1 2️⃣ 🔚
5th fetch + lob.read	0	1+2 1️⃣	0+1	0
6th fetch returning nil	0	1 🔚	0	0
total	2	17	8	3

[cjbj]

Now that @kubo has done this really nifty rewrite (and informative tables!), you could also look at tuning the Oracle Net SDU, depending on how your LOBs are stored.