Add file buffering capability to TFileCacheRead by bbockelm · Pull Request #146 · root-project/root

bbockelm · 2016-03-22T15:01:20Z

This pull request adds an intermediate buffering mode between "normal ROOT IO" and the prefetching system. When enabled, it will cache a remote file to the local disk (uses the same logic as prefetching to determine what is "remote") for as long as it is opened and automatically cleans up afterward.

This is useful in cases where you want to hide the effects of network latency (for various use cases which work poorly with TTreeCache, such as when an unpredictable set of branches are used or non-sequential scans) but do not want to set aside a directory to use as a persistent cache or have a cache-unfriendly workflow.

The approach has been ported from CMSSW (there, it is called lazy-download) where it has been in use for several years.

bbockelm · 2016-09-11T19:47:23Z

@pcanal - This is another one ready to review.

pcanal · 2016-09-12T08:36:50Z

+}
+
+
+bool TFileBufferRead::cache(off_t start, off_t end) {


function name need to be camel case.

all functions must have a doxygen documentation

Can you replace off_t by a ROOT type?

pcanal · 2016-09-12T08:42:00Z

Rather, the focus of this buffer is to hide the effects of network latency.

This is also the focus of the TTreeCache, we must extent the documentation to explain in which use case this is yet-another-caching/prefetch layer is beneficial (compare to the other ones include the TTreeCache, the one the remote protocol might provide (xrootd) or even the XNet.ReadCacheSize or the one behind TFile::OpenFromCache).

bbockelm · 2016-09-28T20:22:06Z

Hi Philippe,

I think this technique is useful in the following cases (both assuming the file is larger than the TTC):

We have a job reading the file's events out-of-order (i.e., randomly selecting events) and reading many events. This occurs when we have two open files and have to match events in file A with events in file B (and B's event order is very different than A).
Cases where there is high latency and the events are read more than once.

(Forgot to mention - have unit tests ready to be posted once this is merged.)

Brian

pcanal · 2017-01-12T20:39:21Z

+   *
+   * On error, -1 is returned and errno is set appropriately.
+   */
+  ssize_t pread(char *into, size_t n, off_t pos);


Could you update the file for naming convention (camel case for funciton name) spacing (indention 3 spaces)?

Thanks.

pcanal · 2017-01-12T20:49:03Z

+
+
+bool TFileBufferRead::tmpfile(const std::string &tmpdir) {
+  std::string pattern(tmpdir);


Why not using TSystem::TempFileName ?

pcanal · 2017-01-12T20:54:00Z

+    size_t len = std::min(static_cast<off_t>(fSize - start), static_cast<off_t>(CHUNK_SIZE));
+    if (!fPresent[index]) {
+
+      void *window = mmap(0, len, PROT_READ | PROT_WRITE, MAP_SHARED, fFd, start);


This wont compiles using the Microsoft compiles.
See interpreter/llvm/src/lib/Support/Unix/Memory.inc
vs interpreter/llvm/src/lib/Support/Windows/Memory.inc

For inspiration (maybe).

vgvassilev · 2017-06-24T09:10:57Z

@bbockelm, could we add a gtest for this feature?

bbockelm · 2017-07-03T20:00:03Z

@vgvassilev - is there a short guide on writing gtests within ROOT? I just barely got caught up with doing roottest stuff...

vgvassilev · 2017-07-20T15:25:40Z

@bbockelm, it looks like your question was snowed under by our bot.

You could look at io/io/test/. It is just adding a file. A test I recently added could be found here

pcanal · 2017-08-23T15:38:13Z

How does this relates and/or superseed the local caching from TFilePrefetch?

bbockelm · 2017-09-06T03:41:13Z

@pcanal - the local caching mechanism in TFilePrefetch is permanent cache -- it lives on the local disk beyond the lifetime of the actual process.

This is a buffer -- it uses the system memory (i.e., the page cache and, eventually, the filesystem) to pull the data locally in large, storage-friendly reads. This allows us to do latency hiding for the duration of the running process.

It does not cache the data beyond the lifetime of the running process. It is much simpler than the TFilePrefetch mechanism as the buffering is only done on demand.

Overall, it's a pretty useful improvement - a modest improvement, admittedly - that we've used for many years in CMS.

pcanal · 2017-09-10T22:52:09Z

This is a buffer -- it uses the system memory (i.e., the page cache and, eventually, the filesystem) to pull the data locally in large, storage-friendly reads.

This sounds like functionally it overlaps with the TTreeCache. How it is different? How are they playing along?

bbockelm · 2017-09-11T19:43:32Z

This captures a wider set of use cases than TTreeCache, but doesn't perform as well as TTreeCache for the cases TTC is designed for.

TTC works best when most of the IO is walking sequentially through the tree, reading from non-local storage, with approximately the same branches read for each event. TFileCacheRead will handle this case OK - but not as well as TTC.
TFileCacheRead will perform better when the event access is non-sequential, when branch access patterns are not predictable, or when non-TTree data is used.

pcanal · 2017-09-27T19:40:56Z


   Bool_t         fAsyncReading;
   Bool_t         fEnablePrefetching;///< reading by prefetching asynchronously
+   bool           fEnableBuffering {false}; ///< buffer remote files on local disk


Keep the initialization style consistent.

pcanal · 2017-09-27T19:41:49Z

+  if (pattern.empty()) {
+    pattern = "/tmp";
+  }
+  pattern += "/cmssw-shadow-XXXXXX";


pcanal · 2017-09-27T19:43:30Z

   //initialise the prefetch object and set the cache directory
   // start the thread only if the file is not local
   fEnablePrefetching = gEnv->GetValue("TFile.AsyncPrefetching", 0);
+   fEnableBuffering   = gEnv->GetValue("TFile.LocalBuffer", 0);


Please add doc/example in config/rootrc.in

xvallspl · 2017-12-07T13:21:24Z

@bbockelm Hi, Brian. This should be an easy error to fix :)

Or is this ongoing work? I will tag it as donotmerge if so.

etejedor · 2018-02-20T12:43:08Z

Hi @bbockelm, do you still want to go for this?

etejedor · 2018-04-26T08:22:44Z

Hi @bbockelm , do you want to close this one or is it ongoing work?

TFileBufferRead is meant to provide a local-disk buffer for a remote file. It aims to provide protection against low performance on high-latency links while being simpler than various caching and prefetching approaches. This version is a WIP as it is not yet integrated with TFileCacheRead.

Now, when TFile.LocalBuffer is set in the root environment, we will buffer the remote file on local disk.

guitargeek · 2025-05-29T07:14:45Z

Hi @bbockelm! I'm closing this PR because of inactivity. Please feel free to reopen if this is still a relevant development for you, solve the merge conflicts, and move the test from root-project/roottest#243 to this main PR, since the roottest repository has become a subdirectory of the main repo.

If you reopen this PR and do these updates, we'll make sure we'll wrap it up quickly. Thanks for your understanding!

Axel-Naumann assigned pcanal Jun 27, 2016

pcanal reviewed Sep 12, 2016
View reviewed changes

pcanal reviewed Jan 12, 2017

View reviewed changes

Comment thread io/io/inc/TFileBufferRead.h

pcanal reviewed Jan 12, 2017

View reviewed changes

pcanal reviewed Sep 27, 2017

View reviewed changes

bbockelm force-pushed the filebuffer branch from 13607c9 to abc28f5 Compare October 3, 2017 20:59

oshadura added the do not merge label Mar 21, 2018

bbockelm added 4 commits October 27, 2018 07:12

Minor bugfixes for TFileBufferRead.

02b10d2

Integrate TFileBufferRead with TFileCacheRead.

5f67f93

Now, when TFile.LocalBuffer is set in the root environment, we will buffer the remote file on local disk.

Update TFileBufferRead to match ROOT coding conventions.

28b44d8

bbockelm mentioned this pull request Oct 27, 2018

Add test for file buffering capability root-project/roottest#243

Closed

root-project deleted a comment from phsft-bot May 29, 2025

guitargeek marked this pull request as ready for review May 29, 2025 07:10

guitargeek requested review from bellenot and dpiparo as code owners May 29, 2025 07:10

guitargeek marked this pull request as draft May 29, 2025 07:11

guitargeek closed this May 29, 2025



		bool TFileBufferRead::tmpfile(const std::string &tmpdir) {
		std::string pattern(tmpdir);

Conversation

bbockelm commented Mar 22, 2016

Uh oh!

bbockelm commented Sep 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pcanal commented Sep 12, 2016

Uh oh!

bbockelm commented Sep 28, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vgvassilev commented Jun 24, 2017

Uh oh!

bbockelm commented Jul 3, 2017

Uh oh!

vgvassilev commented Jul 20, 2017

Uh oh!

pcanal commented Aug 23, 2017

Uh oh!

bbockelm commented Sep 6, 2017

Uh oh!

pcanal commented Sep 10, 2017

Uh oh!

bbockelm commented Sep 11, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xvallspl commented Dec 7, 2017

Uh oh!

etejedor commented Feb 20, 2018

Uh oh!

etejedor commented Apr 26, 2018

Uh oh!

guitargeek commented May 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants