#50329 Possible Security Issue: DOS due to ioblocktimeout not applying to TLS
Closed: fixed 4 months ago by tbordaz. Opened 6 months ago by firstyear.

Issue Description

It has been reported that ioblocktimeout may not apply to some connections. We should investigate this report and determine it's accuracy and if a relevant fix is possible.

https://bugzilla.suse.com/show_bug.cgi?id=1132385
https://bugzilla.redhat.com/show_bug.cgi?id=1693612


Hi,

I do not believe there is a security risk or bug here. I think that perhaps there is a usability and communication issue over our settings. Some settings could be improved with review. Additionally, I believe there are insufficent details in the original report.

SUMMARY: Unless a proof of concept or detailed reproduction steps are provided, then I think this may not be a real issue. I think the server will always recover after the blocks pass.

Read Path analysis:

All connections are by default set to an idletime of 0 (unlimited) which as a default, we could consider changing. Connections are checked in the connection table, and they are polled with a timeout of SLAPD_WAKEUP_TIMER which is 250 milliseconds as a timeout for polling.

When a connection is set to be read from, it's activity is handled in connection_threadmain. In turn, this uses connection_read_operation to read from the socket. There is a small blocking wait here, defined by ioblocktimeout_waits = maxbersize / CONN_TURBO_TIMEOUT_INTERVAL (1000), which is to allow a connection to rapidly fill a buffer if delays are being experienced. Note that if this number ever becomes 0 (for some odd division reason), the timeout will be interpreted as PR_INTERVAL_NO_WAIT, meaning no blocking or delay is allowed. If the connection does not have sufficent data in the series of loops, timed by CONN_TURBO_TIMEOUT_INTERVAL (1000) defined by ioblocktimeout_waits, the connection is put back to the table, and the thread yields for new work. IE the connection may only block for 1 second iterations until eventually it yields.

Write Path Analysis:

Most ldap results are written via write_function, as part of flush_ber from result.c. Inside of this we use the conn->c_sb to allow plug-ability of the different possible io layers. The write itself is conducted by a callback series through ber_flush, which then passes to openldap_write_function, which calls the write_function.

In write_function, slapd_poll is called to determine the readiness of a socket to recieve writes. slapd_poll uses ioblocktimeout to block until the write is able to proceed.

By default ioblocktimeout is 5 minutes - which is certainly long enough to cause threads to starve on waiting on a client that delays the reading process when a write is available.

There is no difference between TLS/Unencrypted write paths. It appears they use the same.

Recommendations:

  • CONN_TURBO_TIMEOUT_INTERVAL should be shorter, and should not be defined by maxbersize.
  • idletimeout should have a default value != 0 to prevent fd starvation.
  • ioblocktimeout's default value should be greatly reduced to prevent the appearance of stalls. As a result will only be sent when the client requests it from the result ldap operation, it should be in a state where it should be nearly ready to recieve. I would recommend a value of 1 second at the greatest, but to achieve this we may need to do testing and step down values over releases to determine if issues arise.

Metadata Update from @firstyear:
- Custom field origin adjusted to None
- Custom field reviewstatus adjusted to None

6 months ago

@mreynolds See recommendations section: what do you think?

Metadata Update from @tbordaz:
- Custom field rhbz adjusted to https://bugzilla.redhat.com/show_bug.cgi?id=1668457

6 months ago

A secure socket is set in block mode. The mechanism to enforced the ioblocktimeout does not apply as the read does not return. So tuning ioblocktimeout will not help.
https://pagure.io/389-ds-base/pull-request/50331

8ca1420..4d9cc24 master
2c583a9..fcf2b5d 389-ds-base-1.4.0
a6112a4..dd4b69b 389-ds-base-1.3.9

Metadata Update from @mreynolds:
- Issue close_status updated to: fixed
- Issue set to the milestone: 1.3.9
- Issue status updated to: Closed (was: Open)

6 months ago

Metadata Update from @tbordaz:
- Issue status updated to: Open (was: Closed)

5 months ago

The fix https://pagure.io/389-ds-base/issue/50329#comment-566117 trigger a regression BZ 1705125

It requires more investigation to know why polling(read_flag) the secure connection sometime timeout.

468b8a8..f35ad37 master
dd14c53..4950a3e 389-ds-base-1.4.0
37f4f70..9a89e77 389-ds-base-1.3.9

73cdeb7..f20e982 master
ff5b4fd..7b0e7f6 389-ds-base-1.4.0
5f40c02..33ac4f5 389-ds-base-1.3.9

Metadata Update from @tbordaz:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

4 months ago

Login to comment on this ticket.

Metadata