就性能而言,用 BufferedOutputStream 包装 FileOutputStream 在什么时候有意义?

我有一个模块,负责读取,处理和写入字节到磁盘。字节通过 UDP 传入,在组装各个数据报后,处理并写入磁盘的最终字节数组通常介于 200 字节和 500,000 字节之间。有时,会有字节数组在组装后超过500,000字节,但这些相对罕见。

我当前正在使用 FileOutputStreamwrite(byte\[\]) 方法。我还在尝试将 包装在 BufferedOutputStream 中,包括使用接受缓冲区大小作为参数的构造函数FileOutputStream

使用 似乎倾向于稍微好一点的性能,但我才刚刚开始尝试不同的缓冲区大小。我只有一组有限的示例数据可供使用(来自示例运行的两个数据集,我可以通过管道通过应用程序)。是否有一般的经验法则,我可以应用它来尝试计算最佳缓冲区大小,以减少磁盘写入,并根据我所知道的有关我正在写入的数据的信息最大限度地提高磁盘写入的性能?BufferedOutputStream


答案 1

BufferedOutputStream 在写入小于缓冲区大小(例如 8 KB)时提供帮助。对于较大的写入,它无济于事,也不会使情况变得更糟。如果所有写入都大于缓冲区大小,或者每次写入后您总是 flush(),我不会使用缓冲区。但是,如果很大一部分写入小于缓冲区大小,并且您不是每次都使用 flush(),则值得拥有。

您可能会发现将缓冲区大小增加到 32 KB 或更大会带来边际改进,或者使其变得更糟。断续器


您可能会发现 BufferedOutputStream.write 的代码很有用

/**
 * Writes <code>len</code> bytes from the specified byte array
 * starting at offset <code>off</code> to this buffered output stream.
 *
 * <p> Ordinarily this method stores bytes from the given array into this
 * stream's buffer, flushing the buffer to the underlying output stream as
 * needed.  If the requested length is at least as large as this stream's
 * buffer, however, then this method will flush the buffer and write the
 * bytes directly to the underlying output stream.  Thus redundant
 * <code>BufferedOutputStream</code>s will not copy data unnecessarily.
 *
 * @param      b     the data.
 * @param      off   the start offset in the data.
 * @param      len   the number of bytes to write.
 * @exception  IOException  if an I/O error occurs.
 */
public synchronized void write(byte b[], int off, int len) throws IOException {
    if (len >= buf.length) {
        /* If the request length exceeds the size of the output buffer,
           flush the output buffer and then write the data directly.
           In this way buffered streams will cascade harmlessly. */
        flushBuffer();
        out.write(b, off, len);
        return;
    }
    if (len > buf.length - count) {
        flushBuffer();
    }
    System.arraycopy(b, off, buf, count, len);
    count += len;
}

答案 2

我最近一直在尝试探索 IO 性能。从我所观察到的,直接写到a已经导致了更好的结果;我将其归因于 的本地调用。此外,我还观察到,当 的延迟开始收敛于直接延迟时,它会波动得多,即它可以突然加倍(我还没有能够找出原因)。FileOutputStreamFileOutputStreamwrite(byte[], int, int)BufferedOutputStreamFileOutputStream

附言:我正在使用Java 8,现在无法评论我的观察结果是否适用于以前的Java版本。

这是我测试的代码,其中我的输入是一个大约10KB的文件

public class WriteCombinationsOutputStreamComparison {
    private static final Logger LOG = LogManager.getLogger(WriteCombinationsOutputStreamComparison.class);

public static void main(String[] args) throws IOException {

    final BufferedInputStream input = new BufferedInputStream(new FileInputStream("src/main/resources/inputStream1.txt"), 4*1024);
    final ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    int data = input.read();
    while (data != -1) {
        byteArrayOutputStream.write(data); // everything comes in memory
        data = input.read();
    }
    final byte[] bytesRead = byteArrayOutputStream.toByteArray();
    input.close();

    /*
     * 1. WRITE USING A STREAM DIRECTLY with entire byte array --> FileOutputStream directly uses a native call and writes
     */
    try (OutputStream outputStream = new FileOutputStream("src/main/resources/outputStream1.txt")) {
        final long begin = System.nanoTime();
        outputStream.write(bytesRead);
        outputStream.flush();
        final long end = System.nanoTime();
        LOG.info("Total time taken for file write, writing entire array [nanos=" + (end - begin) + "], [bytesWritten=" + bytesRead.length + "]");
        if (LOG.isDebugEnabled()) {
            LOG.debug("File reading result was: \n" + new String(bytesRead, Charset.forName("UTF-8")));
        }
    }

    /*
     * 2. WRITE USING A BUFFERED STREAM, write entire array
     */

    // changed the buffer size to different combinations --> write latency fluctuates a lot for same buffer size over multiple runs
    try (BufferedOutputStream outputStream = new BufferedOutputStream(new FileOutputStream("src/main/resources/outputStream1.txt"), 16*1024)) {
        final long begin = System.nanoTime();
        outputStream.write(bytesRead);
        outputStream.flush();
        final long end = System.nanoTime();
        LOG.info("Total time taken for buffered file write, writing entire array [nanos=" + (end - begin) + "], [bytesWritten=" + bytesRead.length + "]");
        if (LOG.isDebugEnabled()) {
            LOG.debug("File reading result was: \n" + new String(bytesRead, Charset.forName("UTF-8")));
        }
    }
}
}

输出:

2017-01-30 23:38:59.064 [INFO] [main] [WriteCombinationsOutputStream] - Total time taken for file write, writing entire array [nanos=100990], [bytesWritten=11059]

2017-01-30 23:38:59.086 [INFO] [main] [WriteCombinationsOutputStream] - Total time taken for buffered file write, writing entire array [nanos=142454], [bytesWritten=11059]