1Filesystem 2========== 3 4Simple filesystem read/write is achieved using the ``uv_fs_*`` functions and the 5``uv_fs_t`` struct. 6 7.. note:: 8 9 The libuv filesystem operations are different from :doc:`socket operations 10 <networking>`. Socket operations use the non-blocking operations provided 11 by the operating system. Filesystem operations use blocking functions 12 internally, but invoke these functions in a `thread pool`_ and notify 13 watchers registered with the event loop when application interaction is 14 required. 15 16.. _thread pool: http://docs.libuv.org/en/v1.x/threadpool.html#thread-pool-work-scheduling 17 18All filesystem functions have two forms - *synchronous* and *asynchronous*. 19 20The *synchronous* forms automatically get called (and **block**) if the 21callback is null. The return value of functions is a :ref:`libuv error code 22<libuv-error-handling>`. This is usually only useful for synchronous calls. 23The *asynchronous* form is called when a callback is passed and the return 24value is 0. 25 26Reading/Writing files 27--------------------- 28 29A file descriptor is obtained using 30 31.. code-block:: c 32 33 int uv_fs_open(uv_loop_t* loop, uv_fs_t* req, const char* path, int flags, int mode, uv_fs_cb cb) 34 35``flags`` and ``mode`` are standard 36`Unix flags <http://man7.org/linux/man-pages/man2/open.2.html>`_. 37libuv takes care of converting to the appropriate Windows flags. 38 39File descriptors are closed using 40 41.. code-block:: c 42 43 int uv_fs_close(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb) 44 45 46Filesystem operation callbacks have the signature: 47 48.. code-block:: c 49 50 void callback(uv_fs_t* req); 51 52Let's see a simple implementation of ``cat``. We start with registering 53a callback for when the file is opened: 54 55.. rubric:: uvcat/main.c - opening a file 56.. literalinclude:: ../../code/uvcat/main.c 57 :linenos: 58 :lines: 41-53 59 :emphasize-lines: 4, 6-7 60 61The ``result`` field of a ``uv_fs_t`` is the file descriptor in case of the 62``uv_fs_open`` callback. If the file is successfully opened, we start reading it. 63 64.. rubric:: uvcat/main.c - read callback 65.. literalinclude:: ../../code/uvcat/main.c 66 :linenos: 67 :lines: 26-40 68 :emphasize-lines: 2,8,12 69 70In the case of a read call, you should pass an *initialized* buffer which will 71be filled with data before the read callback is triggered. The ``uv_fs_*`` 72operations map almost directly to certain POSIX functions, so EOF is indicated 73in this case by ``result`` being 0. In the case of streams or pipes, the 74``UV_EOF`` constant would have been passed as a status instead. 75 76Here you see a common pattern when writing asynchronous programs. The 77``uv_fs_close()`` call is performed synchronously. *Usually tasks which are 78one-off, or are done as part of the startup or shutdown stage are performed 79synchronously, since we are interested in fast I/O when the program is going 80about its primary task and dealing with multiple I/O sources*. For solo tasks 81the performance difference usually is negligible and may lead to simpler code. 82 83Filesystem writing is similarly simple using ``uv_fs_write()``. *Your callback 84will be triggered after the write is complete*. In our case the callback 85simply drives the next read. Thus read and write proceed in lockstep via 86callbacks. 87 88.. rubric:: uvcat/main.c - write callback 89.. literalinclude:: ../../code/uvcat/main.c 90 :linenos: 91 :lines: 16-24 92 :emphasize-lines: 6 93 94.. warning:: 95 96 Due to the way filesystems and disk drives are configured for performance, 97 a write that 'succeeds' may not be committed to disk yet. 98 99We set the dominos rolling in ``main()``: 100 101.. rubric:: uvcat/main.c 102.. literalinclude:: ../../code/uvcat/main.c 103 :linenos: 104 :lines: 55- 105 :emphasize-lines: 2 106 107.. warning:: 108 109 The ``uv_fs_req_cleanup()`` function must always be called on filesystem 110 requests to free internal memory allocations in libuv. 111 112Filesystem operations 113--------------------- 114 115All the standard filesystem operations like ``unlink``, ``rmdir``, ``stat`` are 116supported asynchronously and have intuitive argument order. They follow the 117same patterns as the read/write/open calls, returning the result in the 118``uv_fs_t.result`` field. The full list: 119 120.. rubric:: Filesystem operations 121.. code-block:: c 122 123 int uv_fs_close(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb); 124 int uv_fs_open(uv_loop_t* loop, uv_fs_t* req, const char* path, int flags, int mode, uv_fs_cb cb); 125 int uv_fs_read(uv_loop_t* loop, uv_fs_t* req, uv_file file, const uv_buf_t bufs[], unsigned int nbufs, int64_t offset, uv_fs_cb cb); 126 int uv_fs_unlink(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb); 127 int uv_fs_write(uv_loop_t* loop, uv_fs_t* req, uv_file file, const uv_buf_t bufs[], unsigned int nbufs, int64_t offset, uv_fs_cb cb); 128 int uv_fs_copyfile(uv_loop_t* loop, uv_fs_t* req, const char* path, const char* new_path, int flags, uv_fs_cb cb); 129 int uv_fs_mkdir(uv_loop_t* loop, uv_fs_t* req, const char* path, int mode, uv_fs_cb cb); 130 int uv_fs_mkdtemp(uv_loop_t* loop, uv_fs_t* req, const char* tpl, uv_fs_cb cb); 131 int uv_fs_rmdir(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb); 132 int uv_fs_scandir(uv_loop_t* loop, uv_fs_t* req, const char* path, int flags, uv_fs_cb cb); 133 int uv_fs_scandir_next(uv_fs_t* req, uv_dirent_t* ent); 134 int uv_fs_opendir(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb); 135 int uv_fs_readdir(uv_loop_t* loop, uv_fs_t* req, uv_dir_t* dir, uv_fs_cb cb); 136 int uv_fs_closedir(uv_loop_t* loop, uv_fs_t* req, uv_dir_t* dir, uv_fs_cb cb); 137 int uv_fs_stat(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb); 138 int uv_fs_fstat(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb); 139 int uv_fs_rename(uv_loop_t* loop, uv_fs_t* req, const char* path, const char* new_path, uv_fs_cb cb); 140 int uv_fs_fsync(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb); 141 int uv_fs_fdatasync(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_fs_cb cb); 142 int uv_fs_ftruncate(uv_loop_t* loop, uv_fs_t* req, uv_file file, int64_t offset, uv_fs_cb cb); 143 int uv_fs_sendfile(uv_loop_t* loop, uv_fs_t* req, uv_file out_fd, uv_file in_fd, int64_t in_offset, size_t length, uv_fs_cb cb); 144 int uv_fs_access(uv_loop_t* loop, uv_fs_t* req, const char* path, int mode, uv_fs_cb cb); 145 int uv_fs_chmod(uv_loop_t* loop, uv_fs_t* req, const char* path, int mode, uv_fs_cb cb); 146 int uv_fs_utime(uv_loop_t* loop, uv_fs_t* req, const char* path, double atime, double mtime, uv_fs_cb cb); 147 int uv_fs_futime(uv_loop_t* loop, uv_fs_t* req, uv_file file, double atime, double mtime, uv_fs_cb cb); 148 int uv_fs_lstat(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb); 149 int uv_fs_link(uv_loop_t* loop, uv_fs_t* req, const char* path, const char* new_path, uv_fs_cb cb); 150 int uv_fs_symlink(uv_loop_t* loop, uv_fs_t* req, const char* path, const char* new_path, int flags, uv_fs_cb cb); 151 int uv_fs_readlink(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb); 152 int uv_fs_realpath(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_fs_cb cb); 153 int uv_fs_fchmod(uv_loop_t* loop, uv_fs_t* req, uv_file file, int mode, uv_fs_cb cb); 154 int uv_fs_chown(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_uid_t uid, uv_gid_t gid, uv_fs_cb cb); 155 int uv_fs_fchown(uv_loop_t* loop, uv_fs_t* req, uv_file file, uv_uid_t uid, uv_gid_t gid, uv_fs_cb cb); 156 int uv_fs_lchown(uv_loop_t* loop, uv_fs_t* req, const char* path, uv_uid_t uid, uv_gid_t gid, uv_fs_cb cb); 157 158 159.. _buffers-and-streams: 160 161Buffers and Streams 162------------------- 163 164The basic I/O handle in libuv is the stream (``uv_stream_t``). TCP sockets, UDP 165sockets, and pipes for file I/O and IPC are all treated as stream subclasses. 166 167Streams are initialized using custom functions for each subclass, then operated 168upon using 169 170.. code-block:: c 171 172 int uv_read_start(uv_stream_t*, uv_alloc_cb alloc_cb, uv_read_cb read_cb); 173 int uv_read_stop(uv_stream_t*); 174 int uv_write(uv_write_t* req, uv_stream_t* handle, 175 const uv_buf_t bufs[], unsigned int nbufs, uv_write_cb cb); 176 177The stream based functions are simpler to use than the filesystem ones and 178libuv will automatically keep reading from a stream when ``uv_read_start()`` is 179called once, until ``uv_read_stop()`` is called. 180 181The discrete unit of data is the buffer -- ``uv_buf_t``. This is simply 182a collection of a pointer to bytes (``uv_buf_t.base``) and the length 183(``uv_buf_t.len``). The ``uv_buf_t`` is lightweight and passed around by value. 184What does require management is the actual bytes, which have to be allocated 185and freed by the application. 186 187.. ERROR:: 188 189 THIS PROGRAM DOES NOT ALWAYS WORK, NEED SOMETHING BETTER** 190 191To demonstrate streams we will need to use ``uv_pipe_t``. This allows streaming 192local files [#]_. Here is a simple tee utility using libuv. Doing all operations 193asynchronously shows the power of evented I/O. The two writes won't block each 194other, but we have to be careful to copy over the buffer data to ensure we don't 195free a buffer until it has been written. 196 197The program is to be executed as:: 198 199 ./uvtee <output_file> 200 201We start off opening pipes on the files we require. libuv pipes to a file are 202opened as bidirectional by default. 203 204.. rubric:: uvtee/main.c - read on pipes 205.. literalinclude:: ../../code/uvtee/main.c 206 :linenos: 207 :lines: 61-80 208 :emphasize-lines: 4,5,15 209 210The third argument of ``uv_pipe_init()`` should be set to 1 for IPC using named 211pipes. This is covered in :doc:`processes`. The ``uv_pipe_open()`` call 212associates the pipe with the file descriptor, in this case ``0`` (standard 213input). 214 215We start monitoring ``stdin``. The ``alloc_buffer`` callback is invoked as new 216buffers are required to hold incoming data. ``read_stdin`` will be called with 217these buffers. 218 219.. rubric:: uvtee/main.c - reading buffers 220.. literalinclude:: ../../code/uvtee/main.c 221 :linenos: 222 :lines: 19-22,44-60 223 224The standard ``malloc`` is sufficient here, but you can use any memory allocation 225scheme. For example, node.js uses its own slab allocator which associates 226buffers with V8 objects. 227 228The read callback ``nread`` parameter is less than 0 on any error. This error 229might be EOF, in which case we close all the streams, using the generic close 230function ``uv_close()`` which deals with the handle based on its internal type. 231Otherwise ``nread`` is a non-negative number and we can attempt to write that 232many bytes to the output streams. Finally remember that buffer allocation and 233deallocation is application responsibility, so we free the data. 234 235The allocation callback may return a buffer with length zero if it fails to 236allocate memory. In this case, the read callback is invoked with error 237UV_ENOBUFS. libuv will continue to attempt to read the stream though, so you 238must explicitly call ``uv_close()`` if you want to stop when allocation fails. 239 240The read callback may be called with ``nread = 0``, indicating that at this 241point there is nothing to be read. Most applications will just ignore this. 242 243.. rubric:: uvtee/main.c - Write to pipe 244.. literalinclude:: ../../code/uvtee/main.c 245 :linenos: 246 :lines: 9-13,23-42 247 248``write_data()`` makes a copy of the buffer obtained from read. This buffer 249does not get passed through to the write callback trigged on write completion. To 250get around this we wrap a write request and a buffer in ``write_req_t`` and 251unwrap it in the callbacks. We make a copy so we can free the two buffers from 252the two calls to ``write_data`` independently of each other. While acceptable 253for a demo program like this, you'll probably want smarter memory management, 254like reference counted buffers or a pool of buffers in any major application. 255 256.. WARNING:: 257 258 If your program is meant to be used with other programs it may knowingly or 259 unknowingly be writing to a pipe. This makes it susceptible to `aborting on 260 receiving a SIGPIPE`_. It is a good idea to insert:: 261 262 signal(SIGPIPE, SIG_IGN) 263 264 in the initialization stages of your application. 265 266.. _aborting on receiving a SIGPIPE: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#The_special_problem_of_SIGPIPE 267 268File change events 269------------------ 270 271All modern operating systems provide APIs to put watches on individual files or 272directories and be informed when the files are modified. libuv wraps common 273file change notification libraries [#fsnotify]_. This is one of the more 274inconsistent parts of libuv. File change notification systems are themselves 275extremely varied across platforms so getting everything working everywhere is 276difficult. To demonstrate, I'm going to build a simple utility which runs 277a command whenever any of the watched files change:: 278 279 ./onchange <command> <file1> [file2] ... 280 281The file change notification is started using ``uv_fs_event_init()``: 282 283.. rubric:: onchange/main.c - The setup 284.. literalinclude:: ../../code/onchange/main.c 285 :linenos: 286 :lines: 26- 287 :emphasize-lines: 15 288 289The third argument is the actual file or directory to monitor. The last 290argument, ``flags``, can be: 291 292.. code-block:: c 293 294 /* 295 * Flags to be passed to uv_fs_event_start(). 296 */ 297 enum uv_fs_event_flags { 298 UV_FS_EVENT_WATCH_ENTRY = 1, 299 UV_FS_EVENT_STAT = 2, 300 UV_FS_EVENT_RECURSIVE = 4 301 }; 302 303``UV_FS_EVENT_WATCH_ENTRY`` and ``UV_FS_EVENT_STAT`` don't do anything (yet). 304``UV_FS_EVENT_RECURSIVE`` will start watching subdirectories as well on 305supported platforms. 306 307The callback will receive the following arguments: 308 309 #. ``uv_fs_event_t *handle`` - The handle. The ``path`` field of the handle 310 is the file on which the watch was set. 311 #. ``const char *filename`` - If a directory is being monitored, this is the 312 file which was changed. Only non-``null`` on Linux and Windows. May be ``null`` 313 even on those platforms. 314 #. ``int flags`` - one of ``UV_RENAME`` or ``UV_CHANGE``, or a bitwise OR of 315 both. 316 #. ``int status`` - Currently 0. 317 318In our example we simply print the arguments and run the command using 319``system()``. 320 321.. rubric:: onchange/main.c - file change notification callback 322.. literalinclude:: ../../code/onchange/main.c 323 :linenos: 324 :lines: 9-24 325 326---- 327 328.. [#fsnotify] inotify on Linux, FSEvents on Darwin, kqueue on BSDs, 329 ReadDirectoryChangesW on Windows, event ports on Solaris, unsupported on Cygwin 330.. [#] see :ref:`pipes` 331