Support CopyFile with streaming (#12658)
authorYong Tang <yong.tang.github@outlook.com>
Thu, 8 Feb 2018 00:46:31 +0000 (16:46 -0800)
committerJonathan Hseu <vomjom@vomjom.net>
Thu, 8 Feb 2018 00:46:31 +0000 (16:46 -0800)
commitb1f5f433959406c7aad634c05e85ccd62fd06e87
tree964aacc34e0d92a2198510ad689b9049dff8de20
parent17ffceb08910e78f186fd932c5d4175febc9d402
Support CopyFile with streaming (#12658)

* Support CopyFile with streaming

This fix tries to address the issue raised in 12641 where
it was not possible to have CopyFile with streaming. The original
implementation copies the whole content of the file to a string
buffer and write to the file. This could be an issue
if the file size is large (than memory of the host).

This fix streams the CopyFile operation.

This fix fixes 12641.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Use sendfile for CopyFile implementation in Linux

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Merge CopyFile for same fs and different fs

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* `sendfile64` -> `sendfile` to fix Android build

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Add sendfile processing for Darwin

This commit adds sendfile processing for OSX Darwin.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Not using sendfile in MacOSX

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Address review feedback

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Remove the size check and test OUT_OF_RANGE instead.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Small fixes

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
* Rename CopyFile to FileSystemCopyFile to fix Windows build errors

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
tensorflow/core/platform/env.cc
tensorflow/core/platform/env.h
tensorflow/core/platform/file_system.cc
tensorflow/core/platform/file_system.h
tensorflow/core/platform/posix/posix_file_system.cc
tensorflow/core/platform/posix/posix_file_system.h
tensorflow/python/lib/io/file_io.i