Joshua Gertzen
2005-01-31 01:53:49 UTC
Hi,
DISCLAIMER: I'm sorry if this gets a little long winded but theres a lot
of detail to describe.
For the past few weeks I have been spending some weekends digging
intothe slow performance of jEdit's File System Browser over SMB
networkshares. I've been using jEdit for 3yrs and have just lived with
thisissue, but now that my company is considering standardizing on jEdit
asit's code editor, it has become a larger issue because few are
willingto accept it due to this issue. For example browsing a network
folderwith 1000 files using jEdit takes close to 17seconds over a
802.11bconnection, but using Windows Explorer takes less than a second.
Thegood news is that after one to many cups of coffee, I believe I
have asolution to the problem. Here's the details of what I found and
how I discovered the solution.
After checking out the source from CVS I hunted down the FileVFS
classand it's inner class LocalDirectoryEntry. As a test, I then
hard-coded in values for all but the 'name', 'path' and 'type', thus
eliminating the remaining calls on the File object. To my surprise,
this dropped the speed from 17 seconds to around 2 seconds. I then
restored the calls for each attribute one-by-one and I found that no
single attribute was slow, each one contributed 3-4 seconds to the load
time. So clearly there seemed to be an issue with the java.io.File
object and how it accessed attributes. A little google searching turned
up JSR-203 (originally targeted for JDK1.5, now targeted for JDK1.6)
that will add support for bulk-retrieval of file attributes. Further
review of the java.io.File class indicated that as suspected, it only
retrieves one attribute at a time from the underlying file system. So
at that point it was clearthat Java was to fault and for a while I was
concerned that nothingcould be done.
In disbelief that such a flaw existed, I decided to give theJFileChooser
demo that comes with the JDK a try and see if it had thesame slowness
when using the detail view and accessing the same networkdirectory.
Once again I was surprised because it seemed thatJFileChooser listed the
detail view of the directory in only 1 second.Further, I tried using the
jEdit File Browser from my Linux desktop tobrowse the same network drive
and I found that it also only took 1second. So apparently this flaw is
only an issue under Windows andapparently the Sun engineers did
something to get JFileChooser's accessto attributes under windows
running faster then what java.io.File provided.
So into the JFileChooser source I went and after many hours of
pouringover all the details and running a number of tests, it looks like
Sunspecifically addressed this slowness issue in there code by doing
thefollowing:
1. JFileChooser gets it's file listing from the the FileSystemView
classin the 'javax.swing.filechooser' package, not from java.io.File.
Youcall FileSystemView.getFileSystemView() to get an instance of the
classand then you call fsv.getFiles() to get your file listing.
The'getFiles()' method returns an array of 'File' objects, which in
thecase of Windows, are actually instances of
the'sun.awt.shell.Win32ShellFolder2' class that subclasses
'java.io.File'.Looking through the source code for 'Win32ShellFolder2'
reveals that theclass performs native bulk retrieval of certain
attributes and that itoverrides the 'isHidden()' & 'isDirectory()'
method of the 'File'class. Oddly though, the class does not override
the 'length()','lastModified()', 'canRead()' or 'canWrite()' methods
which means thespeed of those methods does not change. Why this is the
case isexplained in point 2.
2. JFileChooser only accesses the attributes for the files that
arelisted in the visible area of the JTable. Currently jEdit's
browser,lists all the files in a directory, creates a DirectoryEntry
instancesfor each 'File' and copies each attribute from the File to
theDirectoryEntry. In contrast, JFileChooser lists all the files in
adirectory but does not access the attributes for a given file until
theTableModel.getValueAt() method is called while rendering the
visiblearea of the JTable. This allows the FileBrowser to display very
fastsince it only access attributes for the 30 or so files visible in
thebrowser at any time. However, there are two attributes that
JFileChooser mustaccess for each file before displaying any files. The
isHidden() andisDirectory() properties are necessary for the model to
render anycontent, which is why those attributes are accessed in a
native bulk waythrough the 'Win32ShellFolder2' class.
So with this gained insight I proceeded to make the necessary changes
tothe jEdit 4.3 pre 2 code base that I have checked out.Making the first
change to jEdit was straight forward, I modified
theFileVFS._listDirectory() method to use FileSystemView.getFiles()
insteadof File.listFiles() and the result was the load time dropped from
17seconds to 14 seconds.
Implementing the second change was a bit more involved. Since I
neededto delay access the to a file's attributes as long as possible, I
had tomodify VFS.DirectoryEntry so that it provided access to it's
fields(name, path, symlinkPath, deletePath, type, length, hidden,
canRead &canWrite) through methods. That way I could lazy load the
attributesfrom the File class when a given method was called. This was
long andtedious since every reference to a given field had to be changed
to theappropriate method call. I then had to
modifyVFSDirectoryEntryTable.resizeColumnsAppropriately() so that it did
notcalculate the width of the attribute columns based on the width of
thevalues. It still calculates the width of the name column since
it'sfast to read the file name, but the attribute columns are hard set
to aspecific widths based on the maximum likely value of the attribute.
Forinstance the last modified column would not have a value larger
than"12/12/12 12:12 PM" so I calculate the width based on that
staticvalue. This solution may not be perfect since the Browser is also
usedfor other types of file views which may have different attributes
that arenot as static in their width, but I'm hoping a solution to this
issuecan be found. That was pretty much it for the changes and with
thesecond change in place the load time dropped from 14 seconds to 1 second.
I'm planning on building a special 4.2 final build with these changes
inplace since my company needs this fix soon and 4.3 is still in
earlydevelopment.
Any thoughts? Is there a chance this could be changed for 4.3?
Thanks,
-Josh Gertzen
DISCLAIMER: I'm sorry if this gets a little long winded but theres a lot
of detail to describe.
For the past few weeks I have been spending some weekends digging
intothe slow performance of jEdit's File System Browser over SMB
networkshares. I've been using jEdit for 3yrs and have just lived with
thisissue, but now that my company is considering standardizing on jEdit
asit's code editor, it has become a larger issue because few are
willingto accept it due to this issue. For example browsing a network
folderwith 1000 files using jEdit takes close to 17seconds over a
802.11bconnection, but using Windows Explorer takes less than a second.
Thegood news is that after one to many cups of coffee, I believe I
have asolution to the problem. Here's the details of what I found and
how I discovered the solution.
After checking out the source from CVS I hunted down the FileVFS
classand it's inner class LocalDirectoryEntry. As a test, I then
hard-coded in values for all but the 'name', 'path' and 'type', thus
eliminating the remaining calls on the File object. To my surprise,
this dropped the speed from 17 seconds to around 2 seconds. I then
restored the calls for each attribute one-by-one and I found that no
single attribute was slow, each one contributed 3-4 seconds to the load
time. So clearly there seemed to be an issue with the java.io.File
object and how it accessed attributes. A little google searching turned
up JSR-203 (originally targeted for JDK1.5, now targeted for JDK1.6)
that will add support for bulk-retrieval of file attributes. Further
review of the java.io.File class indicated that as suspected, it only
retrieves one attribute at a time from the underlying file system. So
at that point it was clearthat Java was to fault and for a while I was
concerned that nothingcould be done.
In disbelief that such a flaw existed, I decided to give theJFileChooser
demo that comes with the JDK a try and see if it had thesame slowness
when using the detail view and accessing the same networkdirectory.
Once again I was surprised because it seemed thatJFileChooser listed the
detail view of the directory in only 1 second.Further, I tried using the
jEdit File Browser from my Linux desktop tobrowse the same network drive
and I found that it also only took 1second. So apparently this flaw is
only an issue under Windows andapparently the Sun engineers did
something to get JFileChooser's accessto attributes under windows
running faster then what java.io.File provided.
So into the JFileChooser source I went and after many hours of
pouringover all the details and running a number of tests, it looks like
Sunspecifically addressed this slowness issue in there code by doing
thefollowing:
1. JFileChooser gets it's file listing from the the FileSystemView
classin the 'javax.swing.filechooser' package, not from java.io.File.
Youcall FileSystemView.getFileSystemView() to get an instance of the
classand then you call fsv.getFiles() to get your file listing.
The'getFiles()' method returns an array of 'File' objects, which in
thecase of Windows, are actually instances of
the'sun.awt.shell.Win32ShellFolder2' class that subclasses
'java.io.File'.Looking through the source code for 'Win32ShellFolder2'
reveals that theclass performs native bulk retrieval of certain
attributes and that itoverrides the 'isHidden()' & 'isDirectory()'
method of the 'File'class. Oddly though, the class does not override
the 'length()','lastModified()', 'canRead()' or 'canWrite()' methods
which means thespeed of those methods does not change. Why this is the
case isexplained in point 2.
2. JFileChooser only accesses the attributes for the files that
arelisted in the visible area of the JTable. Currently jEdit's
browser,lists all the files in a directory, creates a DirectoryEntry
instancesfor each 'File' and copies each attribute from the File to
theDirectoryEntry. In contrast, JFileChooser lists all the files in
adirectory but does not access the attributes for a given file until
theTableModel.getValueAt() method is called while rendering the
visiblearea of the JTable. This allows the FileBrowser to display very
fastsince it only access attributes for the 30 or so files visible in
thebrowser at any time. However, there are two attributes that
JFileChooser mustaccess for each file before displaying any files. The
isHidden() andisDirectory() properties are necessary for the model to
render anycontent, which is why those attributes are accessed in a
native bulk waythrough the 'Win32ShellFolder2' class.
So with this gained insight I proceeded to make the necessary changes
tothe jEdit 4.3 pre 2 code base that I have checked out.Making the first
change to jEdit was straight forward, I modified
theFileVFS._listDirectory() method to use FileSystemView.getFiles()
insteadof File.listFiles() and the result was the load time dropped from
17seconds to 14 seconds.
Implementing the second change was a bit more involved. Since I
neededto delay access the to a file's attributes as long as possible, I
had tomodify VFS.DirectoryEntry so that it provided access to it's
fields(name, path, symlinkPath, deletePath, type, length, hidden,
canRead &canWrite) through methods. That way I could lazy load the
attributesfrom the File class when a given method was called. This was
long andtedious since every reference to a given field had to be changed
to theappropriate method call. I then had to
modifyVFSDirectoryEntryTable.resizeColumnsAppropriately() so that it did
notcalculate the width of the attribute columns based on the width of
thevalues. It still calculates the width of the name column since
it'sfast to read the file name, but the attribute columns are hard set
to aspecific widths based on the maximum likely value of the attribute.
Forinstance the last modified column would not have a value larger
than"12/12/12 12:12 PM" so I calculate the width based on that
staticvalue. This solution may not be perfect since the Browser is also
usedfor other types of file views which may have different attributes
that arenot as static in their width, but I'm hoping a solution to this
issuecan be found. That was pretty much it for the changes and with
thesecond change in place the load time dropped from 14 seconds to 1 second.
I'm planning on building a special 4.2 final build with these changes
inplace since my company needs this fix soon and 4.3 is still in
earlydevelopment.
Any thoughts? Is there a chance this could be changed for 4.3?
Thanks,
-Josh Gertzen