Discussion:
SPARSE files considered harmful - please revert
Martin Buchholz
2003-05-18 22:04:44 UTC
Permalink
This patch is a bad idea.

2003-02-18 Vaclav Haisman <***@sh.cvut.cz>
* fhandler_disk_file.cc: Include winioctl.h for DeviceIoControl.
(fhandler_disk_file::open): Set newly created and truncated files as
sparse on platforms that support it.

As someone on the mailing list asked, "If making every file sparse is
such a good idea, why isn't it the default?".

My experience has been that for me, sparse files take up much more
disk space than non-sparse files, and are also signicantly slower.

I build software. My build trees have 50000 files, average size 8k.
When I copied build trees to a Win2000 NTFS disk using Cygwin tools
(either cp or tar or rsync) the actual space used on the disk (as
reported by df, not du) quintupled.

Here's what I think is happening. Sparse files are implemented like
compressed files, using 16 clusters. See this web page:

http://www.storageadmin.com/Articles/Index.cfm?ArticleID=15900&pg=1&show=654

As a result, a non-empty but small sparse file takes up a minimum of
16*clustersize bytes on the disk. My measurements suggest an overhead
of 32kb per file with a cluster size of 4kb.

Here are some experiments to support my results:
MKS's commands creates files 5 times smaller than Cygwin commands.

----------------------------------------------------------------
In 1.3.22:
cpdir is a trivial script that does basically
(cd $dir1; tar cf - .) | (cd $dir2; tar xf -)
`cp -pr' works the same way.

# Use Cygwin commands to create a huge file tree
#
$ df .; cpdir dev2 copy-of-dev2; df .
Filesystem Type 1M-blocks Used Available Use% Mounted on
d: system 11492 6001 5491 53% /d
==> mkdir -p copy-of-dev2
cpdir dev2 copy-of-dev2 17.46s user 53.72s system 18% cpu 6:33.99 total
Filesystem Type 1M-blocks Used Available Use% Mounted on
d: system 11492 8438 3054 74% /d
$ du -sm dev2 copy-of-dev2
419 dev2
419 copy-of-dev2
du -h -sm dev2 copy-of-dev2 5.64s user 16.36s system 76% cpu 28.784 total


----------------------------------------------------------------

After reverting to 1.3.20, or patching latest CVS:
I used this method to reclaim disk space that was eaten up by the
SPARSE file disk hog.

$ df .; mv ws ws-old; cpdir ws-old ws; df .
Filesystem Type 1M-blocks Used Available Use% Mounted on
d: system 11492 6910 4582 61% /d
==> mkdir -p ws
cpdir ws-old ws 58.68s user 225.50s system 19% cpu 23:44.30 total
Filesystem Type 1M-blocks Used Available Use% Mounted on
d: system 11492 9085 2407 80% /d
$ df .; rm -rf ws-old; df .
Filesystem Type 1M-blocks Used Available Use% Mounted on
d: system 11492 9085 2407 80% /d
rm -rf ws-old 21.86s user 71.33s system 38% cpu 4:01.85 total
Filesystem Type 1M-blocks Used Available Use% Mounted on
d: system 11492 3689 7803 33% /d


----------------------------------------------------------------

I'm sure if you do the experiments yourself, you will see this for
yourself. To reproduce this problem, you need NTFS 5.0 on Windows
2000. Sparse files are a recent NTFS feature.

The patch is obvious, but I'll send it to cygwin-patches anyways.

Without this patch, Cygwin is unusable for me.

Martin
Martin Buchholz
2003-05-18 22:20:45 UTC
Permalink
cygwin-patches wants me to be a subscriber to post a patch, but I'm
not ready to become a regular contributor (yet) and you'll likely want
to make this change in a different way, so I'm posting my (obvious)
patch here instead of cygwin-patches.

--------- Message bounced from cygwin-patches follows. -----------

I expect that you can do a better job of undoing Vaclav's patch than
this, but this patch is the one that Worked For Me.

2003-05-18 Martin Buchholz <***@xemacs.org>

* fhandler_disk_file.cc (open): Don't make files sparse.



Index: fhandler_disk_file.cc
===================================================================
RCS file: /cvs/src/src/winsup/cygwin/fhandler_disk_file.cc,v
retrieving revision 1.50
diff -u -c -r1.50 fhandler_disk_file.cc
*** fhandler_disk_file.cc 11 May 2003 21:52:09 -0000 1.50
--- fhandler_disk_file.cc 18 May 2003 21:58:48 -0000
***************
*** 408,426 ****
&& !allow_ntsec && allow_ntea)
set_file_attribute (has_acls (), get_win32_name (), mode);

- /* Set newly created and truncated files as sparse files. */
- if ((real_path->fs_flags () & FILE_SUPPORTS_SPARSE_FILES)
- && (get_access () & GENERIC_WRITE) == GENERIC_WRITE
- && (get_flags () & (O_CREAT | O_TRUNC)))
- {
- DWORD dw;
- HANDLE h = get_handle ();
- BOOL r = DeviceIoControl (h , FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &dw,
- NULL);
- syscall_printf ("%d = DeviceIoControl(0x%x, FSCTL_SET_SPARSE, NULL, 0, "
- "NULL, 0, &dw, NULL)", r, h);
- }
-
set_symlink_p (real_path->issymlink ());
set_execable_p (real_path->exec_state ());
set_socket_p (real_path->issocket ());
--- 408,413 ----
Max Bowsher
2003-05-18 22:25:48 UTC
Permalink
Post by Martin Buchholz
This patch is a bad idea.
* fhandler_disk_file.cc: Include winioctl.h for DeviceIoControl.
(fhandler_disk_file::open): Set newly created and truncated files as
sparse on platforms that support it.
As someone on the mailing list asked, "If making every file sparse is
such a good idea, why isn't it the default?".
Me, I think.

I agree that making *all* files sparse is a bad idea, but let me just say
that I consider the usage of the words "considered harmful" harmful to
effective discussion. It sets the scene for antagonistic flamewars. Ditto
"revert".
Post by Martin Buchholz
My experience has been that for me, sparse files take up much more
disk space than non-sparse files, and are also signicantly slower.
I build software. My build trees have 50000 files, average size 8k.
When I copied build trees to a Win2000 NTFS disk using Cygwin tools
(either cp or tar or rsync) the actual space used on the disk (as
reported by df, not du) quintupled.
Here's what I think is happening. Sparse files are implemented like
http://www.storageadmin.com/Articles/Index.cfm?ArticleID=15900&pg=1&show=654
Post by Martin Buchholz
As a result, a non-empty but small sparse file takes up a minimum of
16*clustersize bytes on the disk. My measurements suggest an overhead
of 32kb per file with a cluster size of 4kb.
MKS's commands creates files 5 times smaller than Cygwin commands.
...
Post by Martin Buchholz
I'm sure if you do the experiments yourself, you will see this for
yourself. To reproduce this problem, you need NTFS 5.0 on Windows
2000. Sparse files are a recent NTFS feature.
The patch is obvious, but I'll send it to cygwin-patches anyways.
Without this patch, Cygwin is unusable for me.
May I suggest a middle road? Why not let sparse files be configurable as a
$CYGWIN option? This would allow those users who actually want them to
enable them with minimal effort, but keep them off for most users.

Max.
Martin Buchholz
2003-05-18 23:40:58 UTC
Permalink
Max> May I suggest a middle road? Why not let sparse files be configurable as a
Max> $CYGWIN option? This would allow those users who actually want them to
Max> enable them with minimal effort, but keep them off for most users.

I suspect that SPARSE files are genuinely useful, when storing large
files that have holes in them. But I can't imagine one ever wanting
to use SPARSE for all files, because most files aren't like that. So
I don't think sparseness is a good candidate for being put into
$CYGWIN.

We could have a much cleverer implementation of sparseness, if we kept
statistics on the number and size of zero bytes in a file while it was
being written. When we did the close(), we could automatically
transform it into a sparse file. But I don't think even that should
be the default behavior, because it would make all IO slower.

A program I might actually use myself is one that examines a file on
disk to see if it could be stored more compactly as a sparse file, and
transform it if that were the case. Give it a -r option, and you
would have a "disk optimizer". You can do something similar on Unix.

Martin
Max Bowsher
2003-05-18 23:52:02 UTC
Permalink
Post by Martin Buchholz
Max> May I suggest a middle road? Why not let sparse files be configurable
as a Max> $CYGWIN option? This would allow those users who actually want
them to Max> enable them with minimal effort, but keep them off for most
users.
I suspect that SPARSE files are genuinely useful, when storing large
files that have holes in them. But I can't imagine one ever wanting
to use SPARSE for all files, because most files aren't like that. So
I don't think sparseness is a good candidate for being put into
$CYGWIN.
Agreed. I was just trying to find some simple compromise. Have you reviewed
the long conversation that went on in cygwin-patches in February? Based on
the ease with which this patch was accepted, I'm conjecturing that the core
developers won't want a simple reversion.
Post by Martin Buchholz
We could have a much cleverer implementation of sparseness, if we kept
statistics on the number and size of zero bytes in a file while it was
being written. When we did the close(), we could automatically
transform it into a sparse file. But I don't think even that should
be the default behavior, because it would make all IO slower.
And it wouldn't achieve Vaclav Heisman's original goal, either - he wanted
to avoid the delay caused by Windows zero-filling a file when it was
initially writted to at a large offset.


Max.
Martin Buchholz
2003-05-19 00:21:46 UTC
Permalink
Max> May I suggest a middle road? Why not let sparse files be configurable
Post by Martin Buchholz
as a Max> $CYGWIN option? This would allow those users who actually want
them to Max> enable them with minimal effort, but keep them off for most
users.
I suspect that SPARSE files are genuinely useful, when storing large
files that have holes in them. But I can't imagine one ever wanting
to use SPARSE for all files, because most files aren't like that. So
I don't think sparseness is a good candidate for being put into
$CYGWIN.
Max> Agreed. I was just trying to find some simple compromise. Have you reviewed
Max> the long conversation that went on in cygwin-patches in February? Based on
Max> the ease with which this patch was accepted, I'm conjecturing that the core
Max> developers won't want a simple reversion.

Certainly using $CYGWIN is a solution that works. When you want to
create your sparse file, you do

env CYGWIN="$CYGWIN sparse" make-hole

But it's not very elegant.
Post by Martin Buchholz
We could have a much cleverer implementation of sparseness, if we kept
statistics on the number and size of zero bytes in a file while it was
being written. When we did the close(), we could automatically
transform it into a sparse file. But I don't think even that should
be the default behavior, because it would make all IO slower.
Max> And it wouldn't achieve Vaclav Heisman's original goal, either - he wanted
Max> to avoid the delay caused by Windows zero-filling a file when it was
Max> initially writted to at a large offset.

I see.

Here's another idea...

Whenever a program does a seek on a file descriptor that has O_CREAT set,
we can check if the new position is more than, say 32kb from the current
position. If so, we can call DeviceIoControl(FSCTL_SET_SPARSE).
This is cheap to implement and might just make everyone happy.
If it works.

Martin
Bill C. Riemers
2003-05-19 14:27:17 UTC
Permalink
Only executable files should be sparse files... Under Linux, the executable
bit is checked by cp. (Or at least this was true several years ago, when I
last looked at the code.) Only if the file is marked as executable is it
possibly made sparse. Unfortunately, under CYGWIN, virtually everything is
marked as executable, even though most are not.

There are two ways to correct this problem:
1. Change all your permissions recursively with a find command like:
find / -type f -perm +111 \( -not \( -name \*.exe -or -name
\*.dll -or -name \*.sys -or -name \*.pm -or -name \*.pl -or -name
\*.tcl -or -name \*.pyc -or -name \*.py -or -name \*.sh -or -name
\*.done -or -name \*.bat -or -name \*.el -or -name \*.elc -or -name \*.cgi
\) \) -exec checkx \{\} \; -exec chmod ugo-x \{\} \;

Where the checkx script is:

#!/bin/bash
f=`file "$1" |grep -vi '.*:.*script'`
if [ -n "$f" ] ; then
echo "$f"
exit 0
fi
exit 1

or
2: Change code to check for *.dll or *.exe extensions instead
of checking for the executable bit.

Granted sometimes you do have a data file which can be made sparse.
However, unless you are familiar with the code used to read the file, this
is very dangerous. So this should be decided on a case by case basis.

Bill



----- Original Message -----
From: "Martin Buchholz" <***@xemacs.org>
To: "Max Bowsher" <***@ukf.net>
Cc: <***@cygwin.com>
Sent: Sunday, May 18, 2003 7:40 PM
Subject: Re: SPARSE files considered harmful - please revert
Post by Martin Buchholz
Max> May I suggest a middle road? Why not let sparse files be configurable as a
Max> $CYGWIN option? This would allow those users who actually want them to
Max> enable them with minimal effort, but keep them off for most users.
I suspect that SPARSE files are genuinely useful, when storing large
files that have holes in them. But I can't imagine one ever wanting
to use SPARSE for all files, because most files aren't like that. So
I don't think sparseness is a good candidate for being put into
$CYGWIN.
We could have a much cleverer implementation of sparseness, if we kept
statistics on the number and size of zero bytes in a file while it was
being written. When we did the close(), we could automatically
transform it into a sparse file. But I don't think even that should
be the default behavior, because it would make all IO slower.
A program I might actually use myself is one that examines a file on
disk to see if it could be stored more compactly as a sparse file, and
transform it if that were the case. Give it a -r option, and you
would have a "disk optimizer". You can do something similar on Unix.
Martin
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Problem reports: http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
Max Bowsher
2003-05-19 14:47:32 UTC
Permalink
Post by Bill C. Riemers
Only executable files should be sparse files... Under Linux, the executable
bit is checked by cp. (Or at least this was true several years ago, when I
last looked at the code.) Only if the file is marked as executable is it
possibly made sparse.
I haven't looked in the code, but there is absolutely no mention of this in
the man page.
Post by Bill C. Riemers
Granted sometimes you do have a data file which can be made sparse.
However, unless you are familiar with the code used to read the file, this
is very dangerous. So this should be decided on a case by case basis.
Um? By my understanding, making a file sparse can never be dangerous. It can
cause sub-optimal performance, but code reading the file doesn't have to be
aware of anything special - the OS takes care of it.



Max.
Pierre A. Humblet
2003-05-19 15:53:31 UTC
Permalink
Post by Bill C. Riemers
Post by Bill C. Riemers
Only executable files should be sparse files... Under Linux, the
executable
Post by Bill C. Riemers
bit is checked by cp. (Or at least this was true several years ago, when
I
Post by Bill C. Riemers
last looked at the code.) Only if the file is marked as executable is it
possibly made sparse.
I haven't looked in the code, but there is absolutely no mention of this in
the man page.
... and the non-executable file /var/log/lastlog can cause big trouble if it
isn't sparse.
Post by Bill C. Riemers
Post by Bill C. Riemers
Granted sometimes you do have a data file which can be made sparse.
However, unless you are familiar with the code used to read the file, this
is very dangerous. So this should be decided on a case by case basis.
Um? By my understanding, making a file sparse can never be dangerous. It can
cause sub-optimal performance, but code reading the file doesn't have to be
aware of anything special - the OS takes care of it.
Agree. Something along the lines of Martin Buchholz's suggestion makes sense:
make the file sparse when writing after a seek past the end of file.
There is already support to detect that condition, to handle the win95 lseek
bug.

Pierre
Bill C. Riemers
2003-05-19 16:46:19 UTC
Permalink
Post by Max Bowsher
Um? By my understanding, making a file sparse can never be dangerous. It can
cause sub-optimal performance, but code reading the file doesn't have to be
aware of anything special - the OS takes care of it.
I remember there use to be a warning the man page that file system holes can
cause seek offsets to be wrong for programs that do seek's across or into
hole boundries. The original Linux code only handled when code was loaded
as a result of an mmap which happens for exec's and dlopen. It could be by
now these restrictions have been eliminated. I notice the latest GNU cp
info page lists sparse files a filesystem capacity not a kernel capability,
and the logic for --sparse=auto simply copies a file as-is, nothing smart
like checking for the execution bit.

I looked through the NTFS document on MSDN. It seems at least for NTFS
there is no restriction on what types of files can be sparse. Of course, if
you make files sparse that are going to be accessed with RW operations, you
are going to fragment the file.

Bill
Christopher Faylor
2003-05-19 17:59:13 UTC
Permalink
Post by Bill C. Riemers
Post by Max Bowsher
Um? By my understanding, making a file sparse can never be dangerous. It
can
Post by Max Bowsher
cause sub-optimal performance, but code reading the file doesn't have to
be
Post by Max Bowsher
aware of anything special - the OS takes care of it.
I remember there use to be a warning the man page that file system holes can
cause seek offsets to be wrong for programs that do seek's across or into
hole boundries. The original Linux code only handled when code was loaded
as a result of an mmap which happens for exec's and dlopen. It could be by
now these restrictions have been eliminated. I notice the latest GNU cp
info page lists sparse files a filesystem capacity not a kernel capability,
and the logic for --sparse=auto simply copies a file as-is, nothing smart
like checking for the execution bit.
I looked through the NTFS document on MSDN. It seems at least for NTFS
there is no restriction on what types of files can be sparse. Of course, if
you make files sparse that are going to be accessed with RW operations, you
are going to fragment the file.
I think you need to read the documentation a little more closely. Either that
or provide references to the parts of the documentation that says that normal
RW operations would fragment a sparse file.
Bill C. Riemers
2003-05-19 23:27:06 UTC
Permalink
Post by Christopher Faylor
I think you need to read the documentation a little more closely. Either that
or provide references to the parts of the documentation that says that normal
RW operations would fragment a sparse file.
It is rather obvious. Let say you have three blocks worth of data, and
is written into a file with a physical block followed by a sparse
block followed by a physical block. No disk space is reserved for the
sparse block. Why should it be, as it would defeat the whole purpose of
using sparse files? So physically on disk you have two consecutive physical
blocks. What then happens if you open the file in RW mode, seek to the
sparse
block and write some data? You now have a file with three physics blocks.
However, unless the operating system is going to rewrite the whole file to
disk
again, the new block can not be written in consecutive order. It is either
written
after the last block of the file, or somewhere else on disk.

Before:
... [BLOCK 1][BLOCK 3] ...

After:
... [BLOCK 1][BLOCK 3] ... [BLOCK 2] ...

Of course, if you run the defragmenter regularly, you can fix these
fragmented files. But if it is a file you regularly modify, it is best not
to make
it a sparse file to begin with.


Bill
Christopher Faylor
2003-05-20 02:41:51 UTC
Permalink
Post by Bill C. Riemers
Post by Christopher Faylor
I think you need to read the documentation a little more closely. Either that
or provide references to the parts of the documentation that says that
normal RW operations would fragment a sparse file.
It is rather obvious. Let say you have three blocks worth of data, and
is written into a file with a physical block followed by a sparse block
followed by a physical block. No disk space is reserved for the sparse
block. Why should it be, as it would defeat the whole purpose of using
sparse files? So physically on disk you have two consecutive physical
blocks. What then happens if you open the file in RW mode, seek to the
sparse block and write some data?
1) You are assuming behavior that isn't documented. I can imagine that
the first block could occupy, say 16 blocks and depending on the size of
the hole, there could be no fragmentation.

2) Normal read/write behavior would not result in a file that has a
sparse block. I think it is a rare program which writes beyond EOF. So
this would normally be a non-issue.

3) What no one seems to be mentioning is that we are trying to emulate
UNIX behavior here. If the above is an issue for Windows then it could
also be an issue for UNIX.

cgf
Rolf Campbell
2003-05-20 15:01:08 UTC
Permalink
Post by Christopher Faylor
Post by Bill C. Riemers
Post by Christopher Faylor
I think you need to read the documentation a little more closely. Either that
or provide references to the parts of the documentation that says that
normal RW operations would fragment a sparse file.
It is rather obvious. Let say you have three blocks worth of data, and
is written into a file with a physical block followed by a sparse block
followed by a physical block. No disk space is reserved for the sparse
block. Why should it be, as it would defeat the whole purpose of using
sparse files? So physically on disk you have two consecutive physical
blocks. What then happens if you open the file in RW mode, seek to the
sparse block and write some data?
1) You are assuming behavior that isn't documented. I can imagine that
the first block could occupy, say 16 blocks and depending on the size of
the hole, there could be no fragmentation.
A agree that he is making an assumption, but he is probably right. Even
if 16 blocks are reserved for adding intermediate blocks, you would
still end up with out-of-order blocks in the file; which isn't as bad as
real fragmentation, but isn't as good as all blocks in order.
Post by Christopher Faylor
3) What no one seems to be mentioning is that we are trying to emulate
UNIX behavior here. If the above is an issue for Windows then it could
also be an issue for UNIX.
cgf
And it is.
Christopher Faylor
2003-05-20 17:14:46 UTC
Permalink
Post by Rolf Campbell
Post by Christopher Faylor
Post by Bill C. Riemers
Post by Christopher Faylor
I think you need to read the documentation a little more closely.
Either that
or provide references to the parts of the documentation that says that
normal RW operations would fragment a sparse file.
It is rather obvious. Let say you have three blocks worth of data, and
is written into a file with a physical block followed by a sparse block
followed by a physical block. No disk space is reserved for the sparse
block. Why should it be, as it would defeat the whole purpose of using
sparse files? So physically on disk you have two consecutive physical
blocks. What then happens if you open the file in RW mode, seek to the
sparse block and write some data?
1) You are assuming behavior that isn't documented. I can imagine that
the first block could occupy, say 16 blocks and depending on the size of
the hole, there could be no fragmentation.
A agree that he is making an assumption, but he is probably right. Even
if 16 blocks are reserved for adding intermediate blocks, you would
still end up with out-of-order blocks in the file; which isn't as bad as
real fragmentation, but isn't as good as all blocks in order.
No, you wouldn't. If you allocate a block, skip 4 blocks, write a
block, go back to the begining, skip a block and write a new block, I
would not be surprised to see that there is no fragmentation.

You would end up with a potential gap in the middle of the file but that
isn't necessarily even a bad thing given rotational latency. It's not as
simple as saying "there's a gap so it must be bad".

As usual, however, this discussion is really not worth the effort that
goes into it given the number of sparse files out there.
Post by Rolf Campbell
Post by Christopher Faylor
3) What no one seems to be mentioning is that we are trying to emulate
UNIX behavior here. If the above is an issue for Windows then it could
also be an issue for UNIX.
And it is.
So there goes your argument. You can't argue that you want cygwin to
behave differently from unix and really win.

cgf
Lapo Luchini
2003-05-24 07:58:13 UTC
Permalink
Post by Rolf Campbell
Post by Christopher Faylor
3) What no one seems to be mentioning is that we are trying to emulate
UNIX behavior here. If the above is an issue for Windows then it could
also be an issue for UNIX.
cgf
And it is.
e.g. from FreeBSD 4.8's "man mmap":

WARNING! Extending a file with ftruncate(2), thus cre-
ating a big hole, and then filling the hole by modify-
ing a shared mmap() can lead to severe file fragmenta-
tion. In order to avoid such fragmentation you should
always pre-allocate the file's backing store by
write()ing zero's into the newly extended area prior to
modifying the area via your mmap(). The fragmentation
problem is especially sensitive to MAP_NOSYNC pages,
because pages may be flushed to disk in a totally ran-
dom order.
--
Lapo 'Raist' Luchini
***@lapo.it (PGP & X.509 keys available)
http://www.lapo.it (ICQ UIN: 529796)
Christopher Faylor
2003-05-24 17:38:11 UTC
Permalink
Post by Lapo Luchini
Post by Rolf Campbell
Post by Christopher Faylor
3) What no one seems to be mentioning is that we are trying to emulate
UNIX behavior here. If the above is an issue for Windows then it could
also be an issue for UNIX.
And it is.
WARNING! Extending a file with ftruncate(2), thus cre-
ating a big hole, and then filling the hole by modify-
ing a shared mmap() can lead to severe file fragmenta-
tion. In order to avoid such fragmentation you should
always pre-allocate the file's backing store by
write()ing zero's into the newly extended area prior to
modifying the area via your mmap(). The fragmentation
problem is especially sensitive to MAP_NOSYNC pages,
because pages may be flushed to disk in a totally ran-
dom order.
And so, my point is proved.

Thanks. The feature stays.

cgf
Gary R. Van Sickle
2003-05-25 02:26:06 UTC
Permalink
Post by Christopher Faylor
Post by Lapo Luchini
Post by Rolf Campbell
Post by Christopher Faylor
3) What no one seems to be mentioning is that we are trying to emulate
UNIX behavior here. If the above is an issue for Windows then it could
also be an issue for UNIX.
And it is.
WARNING! Extending a file with ftruncate(2), thus cre-
ating a big hole, and then filling the hole by modify-
ing a shared mmap() can lead to severe file fragmenta-
tion. In order to avoid such fragmentation you should
always pre-allocate the file's backing store by
write()ing zero's into the newly extended area prior to
modifying the area via your mmap(). The fragmentation
problem is especially sensitive to MAP_NOSYNC pages,
because pages may be flushed to disk in a totally ran-
dom order.
And so, my point is proved.
Your point is what exactly, Chris? That Cygwin should duplicate any and all
dubious "features" of any and all Unii? Why? What benefit is anybody gaining
by essentially bypassing NTFS's "1 cluster==1 sector" abilities and going back
to the wonderous days of FAT16 on 99.9999999944% of all files created by Cygwin?
The only proof here, from the experiments of two people, is that this benefits
almost nobody while adversely affecting everybody.

Hell, why not just create every file as compressed? At least everybody would
gain something from that, and it'd do pretty much the same thing with that one
file that benefits from being marked sparse.

--
Gary R. Van Sickle
Brewer. Patriot.

Bill C. Riemers
2003-05-20 14:50:56 UTC
Permalink
Post by Christopher Faylor
1) You are assuming behavior that isn't documented. I can imagine that
the first block could occupy, say 16 blocks and depending on the size of
the hole, there could be no fragmentation.
You are assuming an optimization that may or may not exist. In my example,
there is certainly no reason why the first block would occupy 16 blocks. I
already specified the hole is exactly one block size. At most the file
system may allocate 3 blocks, so the middle one could be filled later. But
even in that case you would still get fragmentation as a result. However,
the fragmentation would more likely result from a one block file being
written into the "reserved" space, before it is needed for the updated
sparse file. Either way use of a sparse file for a file that is regularly
accessed in RW mode will result in fragmentation. The only question is how
fast it will fragment. That behavior depends on the filesystem, and how the
drivers are implemented.

Really sophisticated drivers might even do things like rewrite the file if
it is below
a threshold size, just to fix fragmentation on the fly. I can definitely
say NTFS is
not that sophisticated. Even on disks with a large amount of free space
NTFS
fragments at an alarmingly fast rate. I defragment Linux partition once
every few
years at most (by repartitioning and copying). Any more frequent and there
is no
noticeable improvement in performance. For NTFS I find I need to run the
defragmenter
every weekend for optimal performance.
Post by Christopher Faylor
2) Normal read/write behavior would not result in a file that has a
sparse block. I think it is a rare program which writes beyond EOF. So
this would normally be a non-issue.
Correct. I am only talking about why it is bad idea to blindly convert all
files to sparse files. This can be done with either GNU tar or GNU cp.
The above fragmentation behavior is going to happen and does happen when
the file in question is a database file, since databases tend to contain
lots of blank space intended for adding new records.
Post by Christopher Faylor
3) What no one seems to be mentioning is that we are trying to emulate
UNIX behavior here. If the above is an issue for Windows then it could
also be an issue for UNIX.
It sounds like we are really on the same page, but discussing different
issues.
CYGWIN should definitely support creating sparse files in the classical Unix
method of seeking beyond the end of the file. From what I've seen in this
discussion
it already does, and that is not an issue. What I'm arguing is that files
should not
be blindly converted into sparse files with GNU tar -S, GNU
cp --sparse=always, etc.

If for example, you convert a database file into a sparse file, it is not
uncommon for the
fragmentation to reduce database access times by an order of magnitude or
more.

Bill
Max Bowsher
2003-05-18 23:17:54 UTC
Permalink
Post by Martin Buchholz
As a result, a non-empty but small sparse file takes up a minimum of
16*clustersize bytes on the disk. My measurements suggest an overhead
of 32kb per file with a cluster size of 4kb.
I just thought I'd throw a few more numbers into the debate:

I patched Cygwin to respond to CYGWIN=sparse / CYGWIN=nosparse
Then, I did a cvs co winsup:

"Size on disc" of checked out dir, as shown in Windows properties box:
Sparse: 40.7MB
Not sparse: 43.6MB
OK, so sparse seems to win? But that makes no sense - backed up by noting
that for various individual sparse files, "Size on disc" is reporting a size
which is not an integer number of clusters.

Now, Properties of disc, look at "Used space":
Difference in creating sparse checkout: ~ 200MB !!!
Difference in creating normal checkout: ~ 40MB

Personally, I'm inclined to trust the overall disc stats more.

I think this evidence suggests that sparse files should NOT be on by default
in Cygwin.



Max.
Martin Buchholz
2003-05-18 23:56:15 UTC
Permalink
Post by Martin Buchholz
As a result, a non-empty but small sparse file takes up a minimum of
16*clustersize bytes on the disk. My measurements suggest an overhead
of 32kb per file with a cluster size of 4kb.
Max> I just thought I'd throw a few more numbers into the debate:

Max> I patched Cygwin to respond to CYGWIN=sparse / CYGWIN=nosparse
Max> Then, I did a cvs co winsup:

Max> "Size on disc" of checked out dir, as shown in Windows properties box:
Max> Sparse: 40.7MB
Max> Not sparse: 43.6MB
Max> OK, so sparse seems to win? But that makes no sense - backed up by noting
Max> that for various individual sparse files, "Size on disc" is reporting a size
Max> which is not an integer number of clusters.

Max> Now, Properties of disc, look at "Used space":
Max> Difference in creating sparse checkout: ~ 200MB !!!
Max> Difference in creating normal checkout: ~ 40MB

This 5-fold expansion is just like what I saw.

Max> Personally, I'm inclined to trust the overall disc stats more.

The thing that matters is, what happens when the disk gets to 100%?
This did happen to me. I do not recall getting any "disk full" error
messages, but the disk was unhappy nevertheless. Processes writing to
the disk would tend to hang.

Max> I think this evidence suggests that sparse files should NOT be on by default
Max> in Cygwin.

Yup.

Martin
Rolf Campbell
2003-05-20 16:11:13 UTC
Permalink
Post by Max Bowsher
Post by Martin Buchholz
As a result, a non-empty but small sparse file takes up a minimum of
16*clustersize bytes on the disk. My measurements suggest an overhead
of 32kb per file with a cluster size of 4kb.
I patched Cygwin to respond to CYGWIN=sparse / CYGWIN=nosparse
Sparse: 40.7MB
Not sparse: 43.6MB
OK, so sparse seems to win? But that makes no sense - backed up by noting
that for various individual sparse files, "Size on disc" is reporting a size
which is not an integer number of clusters.
Difference in creating sparse checkout: ~ 200MB !!!
Difference in creating normal checkout: ~ 40MB
Personally, I'm inclined to trust the overall disc stats more.
I think this evidence suggests that sparse files should NOT be on by default
in Cygwin.
I just checked out a corporate build system, which had average file size
much bigger. It decreases the available disk space by 300Megs with
non-sparce files, and 390Megs with sparce files.
Max Bowsher
2003-05-20 17:07:29 UTC
Permalink
Post by Rolf Campbell
Post by Max Bowsher
Post by Martin Buchholz
As a result, a non-empty but small sparse file takes up a minimum of
16*clustersize bytes on the disk. My measurements suggest an overhead
of 32kb per file with a cluster size of 4kb.
I patched Cygwin to respond to CYGWIN=sparse / CYGWIN=nosparse
Sparse: 40.7MB
Not sparse: 43.6MB
OK, so sparse seems to win? But that makes no sense - backed up by noting
that for various individual sparse files, "Size on disc" is reporting a size
which is not an integer number of clusters.
Difference in creating sparse checkout: ~ 200MB !!!
Difference in creating normal checkout: ~ 40MB
Personally, I'm inclined to trust the overall disc stats more.
I think this evidence suggests that sparse files should NOT be on by default
in Cygwin.
I just checked out a corporate build system, which had average file size
much bigger. It decreases the available disk space by 300Megs with
non-sparce files, and 390Megs with sparce files.
Well, that's still 90MB unneccessary disk usage. Horribly inelegant, even if
not directly inconveniencing.


Max.
John Vincent
2003-05-19 18:44:23 UTC
Permalink
Hi,

I looked up sparse files on MSDN and found the following link:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/sparse_file_operations.asp

The most interesting thing is that a sparse file is only sparse if the zeros
in the file are written with a special operation. I strongly suspect that
the patch to support sparse files introduced in cygwin is incorrect (or at
least incomplete)

I've quoted the contents of the entry below, I hope this is helpful:

Sparse File Operations

To determine whether a file system supports sparse files, call the
GetVolumeInformation function and examine the FILE_SUPPORTS_SPARSE_FILES bit
flag.


Most applications are not aware of sparse files and will not create sparse
files. The fact that an application is reading a sparse file is transparent
to the application. An application that is aware of sparse-files should
determine whether its data set is suitable to be kept in a sparse file.
After that determination is made, the application must explicitly declare a
file as sparse, using the FSCTL_SET_SPARSE control code.

After an application has set a file to be sparse, the application can use
the FSCTL_SET_ZERO_DATA control code to set a region of the file to zero. In
addition, the application can use the FSCTL_QUERY_ALLOCATED_RANGES control
code to speed searches for nonzero data in the sparse file.

When you perform a write operation (with a function or operation other than
FSCTL_SET_ZERO_DATA) whose data consists of nothing but zeros, zeros will be
written to the disk for the entire length of the write. To zero out a range
of the file and maintain sparseness, use FSCTL_SET_ZERO_DATA.

A sparseness-aware application may also set an existing file to be sparse.
If an application sets an existing file to be sparse, it should then scan
the file for regions which contain zeros, and use FSCTL_SET_ZERO_DATA to
reset those regions, thereby possibly deallocating some physical disk
storage. An application upgraded to sparse file awareness should perform
this conversion.

When you perform a read operation from a zeroed-out portion of a sparse
file, the operating system may not read from the hard drive. Instead, the
system recognizes that the portion of the file to be read contains zeros,
and it returns a buffer full of zeros without actually reading from the
disk.

As with any other file, the system can write data to or read data from any
position in a sparse file. Nonzero data being written to a previously zeroed
portion of the file may result in allocation of disk space. Zeros being
written over nonzero data (only with FSCTL_SET_ZERO_DATA) may result in a
deallocation of disk space.

Note It is up to the application to maintain sparseness by writing zeros
with FSCTL_SET_ZERO_DATA.

Defragmenting tools that handle compressed files on NTFS file systems will
correctly handle sparse files on NTFS volumes.

_________________________________________________________________
Stay in touch with absent friends - get MSN Messenger
http://www.msn.co.uk/messenger
Max Bowsher
2003-05-19 18:54:19 UTC
Permalink
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/sparse_file_operations.asp
Post by John Vincent
The most interesting thing is that a sparse file is only sparse if the zeros
in the file are written with a special operation. I strongly suspect that
the patch to support sparse files introduced in cygwin is incorrect (or at
least incomplete)
Areas that are simply seeked over, and never written to, should be sparse as
well.

Anyway:

Based on the posted numbers, global use of sparse files is a bad idea. Can
we conditionalize sparse files on a $CYGWIN option? (Or something else, I
don't mind, but the important thing is that it should not be on by default.)



Max.
Pierre A. Humblet
2003-05-19 20:08:49 UTC
Permalink
Post by John Vincent
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/sparse_file_operations.asp
Post by John Vincent
The most interesting thing is that a sparse file is only sparse if the
zeros
Post by John Vincent
in the file are written with a special operation. I strongly suspect that
the patch to support sparse files introduced in cygwin is incorrect (or at
least incomplete)
Areas that are simply seeked over, and never written to, should be sparse as
well.
Not only they "should be", they are.
Post by John Vincent
Based on the posted numbers, global use of sparse files is a bad idea. Can
we conditionalize sparse files on a $CYGWIN option? (Or something else, I
don't mind, but the important thing is that it should not be on by default.)
Why not make in on by default *when it can help*, i.e. when the write() leaves
holes?

Pierre
Max Bowsher
2003-05-19 19:40:27 UTC
Permalink
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/sparse_file_operations.asp
Post by Pierre A. Humblet
Post by Max Bowsher
Post by John Vincent
The most interesting thing is that a sparse file is only sparse if the
zeros
Post by John Vincent
in the file are written with a special operation. I strongly suspect that
the patch to support sparse files introduced in cygwin is incorrect (or at
least incomplete)
Areas that are simply seeked over, and never written to, should be sparse as
well.
Not only they "should be", they are.
Post by Max Bowsher
Based on the posted numbers, global use of sparse files is a bad idea. Can
we conditionalize sparse files on a $CYGWIN option? (Or something else, I
don't mind, but the important thing is that it should not be on by default.)
Why not make in on by default *when it can help*, i.e. when the write()
leaves holes?
Does anyone know of any programs which write non-sparse files, but not
contiguously? If there are none, then this approach should be fine.

In any case, the $CYGWIN approach could be a valid temporary measure until
someone writes the heuristic code.


Max.
Pierre A. Humblet
2003-05-19 20:22:10 UTC
Permalink
Post by Max Bowsher
Post by Pierre A. Humblet
Post by Max Bowsher
Based on the posted numbers, global use of sparse files is a bad idea.
Can
Post by Pierre A. Humblet
Post by Max Bowsher
we conditionalize sparse files on a $CYGWIN option? (Or something else, I
don't mind, but the important thing is that it should not be on by
default.)
Post by Pierre A. Humblet
Why not make in on by default *when it can help*, i.e. when the write()
leaves holes?
Does anyone know of any programs which write non-sparse files, but not
contiguously? If there are none, then this approach should be fine.
There must be some... Cygwin would check that the holes are bigger than a
minimum threshold. Thus such files would never be really short and the
penalty for guessing wrong would never be large.
Post by Max Bowsher
In any case, the $CYGWIN approach could be a valid temporary measure until
someone writes the heuristic code.
The code change I have in mind is very simple (because it's similar to what is
already done against a Win95 bug), so we might as well skip the intermediate step.
However a patch against current cvs might conflict with bigger changes that
Chris Faylor has already made in his sandbox.

Pierre
John Vincent
2003-05-19 19:26:22 UTC
Permalink
You say that any areas that are seeked over should be sparse as well. That
is true on many Unix/Linux file systems. I've not seen anything to suggest
it's true on NTFS though, have you?


From: "Max Bowsher" <maxb at ukf dot net>
To: <cygwin at cygwin dot com>
Date: Mon, 19 May 2003 19:54:19 +0100
Subject: Re: SPARSE files considered harmful - please revert
References: <Law10-***@hotmail.com>

--------------------------------------------------------------------------------
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/sparse_file_operations.asp
Post by John Vincent
The most interesting thing is that a sparse file is only sparse if the
zeros
Post by John Vincent
in the file are written with a special operation. I strongly suspect that
the patch to support sparse files introduced in cygwin is incorrect (or at
least incomplete)
Areas that are simply seeked over, and never written to, should be sparse as
well.

Anyway:

Based on the posted numbers, global use of sparse files is a bad idea. Can
we conditionalize sparse files on a $CYGWIN option? (Or something else, I
don't mind, but the important thing is that it should not be on by default.)

_________________________________________________________________
It's fast, it's easy and it's free. Get MSN Messenger today!
http://www.msn.co.uk/messenger
Max Bowsher
2003-05-19 19:37:04 UTC
Permalink
Post by John Vincent
You say that any areas that are seeked over should be sparse as well. That
is true on many Unix/Linux file systems. I've not seen anything to suggest
it's true on NTFS though, have you?
Just speculation.

But, the point is, given the numbers suggesting that sparse small files take
up way too many clusters, can sparse files in Cygwin be turned off by
default, please?



Max.
Ljubomir Josifovski
2003-05-21 11:03:34 UTC
Permalink
For the ignorant folk like me - does...
Post by Martin Buchholz
This patch is a bad idea.
* fhandler_disk_file.cc: Include winioctl.h for DeviceIoControl.
(fhandler_disk_file::open): Set newly created and truncated files as
sparse on platforms that support it.
As someone on the mailing list asked, "If making every file sparse is
such a good idea, why isn't it the default?".
My experience has been that for me, sparse files take up much more
disk space than non-sparse files, and are also signicantly slower.
...mean that by default, every file created by a cygwin app takes
at least 16 (NTFS?) clusters? Was this always the case, or is it
a recent thing?

(my cygwin1.dll is

948k 2003/03/18 C:\cygwin\bin\cygwin1.dll
Cygwin DLL version info:
DLL version: 1.3.22
DLL epoch: 19
DLL bad signal mask: 19005
DLL old termios: 5
DLL malloc env: 28
API major: 0
API minor: 78
Shared data: 3
DLL identifier: cygwin1
Mount registry: 2
Cygnus registry name: Cygnus Solutions
Cygwin registry name: Cygwin
Program options name: Program Options
Cygwin mount registry name: mounts v2
Cygdrive flags: cygdrive flags
Cygdrive prefix: cygdrive prefix
Cygdrive default prefix:
Build date: Tue Mar 18 09:20:11 EST 2003
CVS tag: dontuse-21
Shared id: cygwin1S3
)


thanks,
--
Ljubomir Josifovski
Loading...