Tar Unable to Continue Traversing Directory Tree

In order to perform some action on a file, you need to know where it is, and that's the information provided by file paths. Gradle builds on the standard Java File class, which represents the location of a single file, and provides new APIs for dealing with collections of paths. This section shows you how to use the Gradle APIs to specify file paths for use in tasks and file operations.

But first, an important note on using hard-coded file paths in your builds.

On hard-coded file paths

Many examples in this chapter use hard-coded paths as string literals. This makes them easy to understand, but it's not good practice for real builds. The problem is that paths often change and the more places you need to change them, the more likely you are to miss one and break the build.

Where possible, you should use tasks, task properties, and project properties — in that order of preference — to configure file paths. For example, if you were to create a task that packages the compiled classes of a Java application, you should aim for something like this:

Example 20. How to minimize the number of hard-coded paths in your build

build.gradle

                    def archivesDirPath = layout.buildDirectory.dir('archives')  tasks.register('packageClasses', Zip) {     archiveAppendix = "classes"     destinationDirectory = archivesDirPath      from compileJava }                  

build.gradle.kts

                    val archivesDirPath = layout.buildDirectory.dir("archives")  tasks.register<Zip>("packageClasses") {     archiveAppendix.set("classes")     destinationDirectory.set(archivesDirPath)      from(tasks.compileJava) }                  

See how we're using the compileJava task as the source of the files to package and we've created a project property archivesDirPath to store the location where we put archives, on the basis we're likely to use it elsewhere in the build.

Using a task directly as an argument like this relies on it having defined outputs, so it won't always be possible. In addition, this example could be improved further by relying on the Java plugin's convention for destinationDirectory rather than overriding it, but it does demonstrate the use of project properties.

Single files and directories

Gradle provides the Project.file(java.lang.Object) method for specifying the location of a single file or directory. Relative paths are resolved relative to the project directory, while absolute paths remain unchanged.

Never use new File(relative path) because this creates a path relative to the current working directory (CWD). Gradle can make no guarantees about the location of the CWD, which means builds that rely on it may break at any time.

Here are some examples of using the file() method with different types of argument:

Example 21. Locating files

build.gradle

                    // Using a relative path File configFile = file('src/config.xml')  // Using an absolute path configFile = file(configFile.absolutePath)  // Using a File object with a relative path configFile = file(new File('src/config.xml'))  // Using a java.nio.file.Path object with a relative path configFile = file(Paths.get('src', 'config.xml'))  // Using an absolute java.nio.file.Path object configFile = file(Paths.get(System.getProperty('user.home')).resolve('global-config.xml'))                  

build.gradle.kts

                    // Using a relative path var configFile = file("src/config.xml")  // Using an absolute path configFile = file(configFile.absolutePath)  // Using a File object with a relative path configFile = file(File("src/config.xml"))  // Using a java.nio.file.Path object with a relative path configFile = file(Paths.get("src", "config.xml"))  // Using an absolute java.nio.file.Path object configFile = file(Paths.get(System.getProperty("user.home")).resolve("global-config.xml"))                  

As you can see, you can pass strings, File instances and Path instances to the file() method, all of which result in an absolute File object. You can find other options for argument types in the reference guide, linked in the previous paragraph.

What happens in the case of multi-project builds? The file() method will always turn relative paths into paths that are relative to the current project directory, which may be a child project. If you want to use a path that's relative to the root project directory, then you need to use the special Project.getRootDir() property to construct an absolute path, like so:

Example 22. Creating a path relative to a parent project

build.gradle

                    File configFile = file("$rootDir/shared/config.xml")                  

build.gradle.kts

                    val configFile = file("$rootDir/shared/config.xml")                  

Let's say you're working on a multi-project build in a dev/projects/AcmeHealth directory. You use the above example in the build of the library you're fixing — at AcmeHealth/subprojects/AcmePatientRecordLib/build.gradle. The file path will resolve to the absolute version of dev/projects/AcmeHealth/shared/config.xml.

The file() method can be used to configure any task that has a property of type File. Many tasks, though, work on multiple files, so we look at how to specify sets of files next.

File collections

A file collection is simply a set of file paths that's represented by the FileCollection interface. Any file paths. It's important to understand that the file paths don't have to be related in any way, so they don't have to be in the same directory or even have a shared parent directory. You will also find that many parts of the Gradle API use FileCollection, such as the copying API discussed later in this chapter and dependency configurations.

The recommended way to specify a collection of files is to use the ProjectLayout.files(java.lang.Object...) method, which returns a FileCollection instance. This method is very flexible and allows you to pass multiple strings, File instances, collections of strings, collections of Files, and more. You can even pass in tasks as arguments if they have defined outputs. Learn about all the supported argument types in the reference guide.

Although the files() method accepts File instances, never use new File(relative path) with it because this creates a path relative to the current working directory (CWD). Gradle can make no guarantees about the location of the CWD, which means builds that rely on it may break at any time.

As with the Project.file(java.lang.Object) method covered in the previous section, all relative paths are evaluated relative to the current project directory. The following example demonstrates some of the variety of argument types you can use — strings, File instances, a list and a Path:

Example 23. Creating a file collection

build.gradle

                    FileCollection collection = layout.files('src/file1.txt',                                   new File('src/file2.txt'),                                   ['src/file3.csv', 'src/file4.csv'],                                   Paths.get('src', 'file5.txt'))                  

build.gradle.kts

                    val collection: FileCollection = layout.files(     "src/file1.txt",     File("src/file2.txt"),     listOf("src/file3.csv", "src/file4.csv"),     Paths.get("src", "file5.txt") )                  

File collections have some important attributes in Gradle. They can be:

• created lazily

• iterated over

• filtered

• combined

Lazy creation of a file collection is useful when you need to evaluate the files that make up a collection at the time a build runs. In the following example, we query the file system to find out what files exist in a particular directory and then make those into a file collection:

Example 24. Implementing a file collection

build.gradle

                    tasks.register('list') {     doLast {         File srcDir          // Create a file collection using a closure         collection = layout.files { srcDir.listFiles() }          srcDir = file('src')         println "Contents of $srcDir.name"         collection.collect { relativePath(it) }.sort().each { println it }          srcDir = file('src2')         println "Contents of $srcDir.name"         collection.collect { relativePath(it) }.sort().each { println it }     } }                  

build.gradle.kts

                    tasks.register("list") {     doLast {         var srcDir: File? = null          val collection = layout.files({             srcDir?.listFiles()         })          srcDir = file("src")         println("Contents of ${srcDir.name}")         collection.map { relativePath(it) }.sorted().forEach { println(it) }          srcDir = file("src2")         println("Contents of ${srcDir.name}")         collection.map { relativePath(it) }.sorted().forEach { println(it) }     } }                  

Output of gradle -q list

> gradle -q list Contents of src src/dir1 src/file1.txt Contents of src2 src2/dir1 src2/dir2

The key to lazy creation is passing a closure (in Groovy) or a Provider (in Kotlin) to the files() method. Your closure/provider simply needs to return a value of a type accepted by files(), such as List<File>, String, FileCollection, etc.

Iterating over a file collection can be done through the each() method (in Groovy) of forEach method (in Kotlin) on the collection or using the collection in a for loop. In both approaches, the file collection is treated as a set of File instances, i.e. your iteration variable will be of type File.

The following example demonstrates such iteration as well as how you can convert file collections to other types using the as operator or supported properties:

Example 25. Using a file collection

build.gradle

                                          // Iterate over the files in the collection         collection.each { File file ->             println file.name         }          // Convert the collection to various types         Set set = collection.files         Set set2 = collection as Set         List list = collection as List         String path = collection.asPath         File file = collection.singleFile          // Add and subtract collections         def union = collection + layout.files('src/file2.txt')         def difference = collection - layout.files('src/file2.txt')                  

build.gradle.kts

                                          // Iterate over the files in the collection         collection.forEach { file: File ->             println(file.name)         }          // Convert the collection to various types         val set: Set<File> = collection.files         val list: List<File> = collection.toList()         val path: String = collection.asPath         val file: File = collection.singleFile          // Add and subtract collections         val union = collection + layout.files("src/file2.txt")         val difference = collection - layout.files("src/file2.txt")                  

You can also see at the end of the example how to combine file collections using the + and - operators to merge and subtract them. An important feature of the resulting file collections is that they are live. In other words, when you combine file collections in this way, the result always reflects what's currently in the source file collections, even if they change during the build.

For example, imagine collection in the above example gains an extra file or two after union is created. As long as you use union after those files are added to collection, union will also contain those additional files. The same goes for the different file collection.

Live collections are also important when it comes to filtering. If you want to use a subset of a file collection, you can take advantage of the FileCollection.filter(org.gradle.api.specs.Spec) method to determine which files to "keep". In the following example, we create a new collection that consists of only the files that end with .txt in the source collection:

Example 26. Filtering a file collection

build.gradle

                                          FileCollection textFiles = collection.filter { File f ->             f.name.endsWith(".txt")         }                  

build.gradle.kts

                                          val textFiles: FileCollection = collection.filter { f: File ->             f.name.endsWith(".txt")         }                  

Output of gradle -q filterTextFiles

> gradle -q filterTextFiles src/file1.txt src/file2.txt src/file5.txt

If collection changes at any time, either by adding or removing files from itself, then textFiles will immediately reflect the change because it is also a live collection. Note that the closure you pass to filter() takes a File as an argument and should return a boolean.

File trees

A file tree is a file collection that retains the directory structure of the files it contains and has the type FileTree. This means that all the paths in a file tree must have a shared parent directory. The following diagram highlights the distinction between file trees and file collections in the common case of copying files:

Figure 3. The differences in how file trees and file collections behave when copying files

Although FileTree extends FileCollection (an is-a relationship), their behaviors do differ. In other words, you can use a file tree wherever a file collection is required, but remember: a file collection is a flat list/set of files, while a file tree is a file and directory hierarchy. To convert a file tree to a flat collection, use the FileTree.getFiles() property.

The simplest way to create a file tree is to pass a file or directory path to the Project.fileTree(java.lang.Object) method. This will create a tree of all the files and directories in that base directory (but not the base directory itself). The following example demonstrates how to use the basic method and, in addition, how to filter the files and directories using Ant-style patterns:

Example 27. Creating a file tree

build.gradle

                    // Create a file tree with a base directory ConfigurableFileTree tree = fileTree(dir: 'src/main')  // Add include and exclude patterns to the tree tree.include '**/*.java' tree.exclude '**/Abstract*'  // Create a tree using closure tree = fileTree('src') {     include '**/*.java' }  // Create a tree using a map tree = fileTree(dir: 'src', include: '**/*.java') tree = fileTree(dir: 'src', includes: ['**/*.java', '**/*.xml']) tree = fileTree(dir: 'src', include: '**/*.java', exclude: '**/*test*/**')                  

build.gradle.kts

                    // Create a file tree with a base directory var tree: ConfigurableFileTree = fileTree("src/main")  // Add include and exclude patterns to the tree tree.include("**/*.java") tree.exclude("**/Abstract*")  // Create a tree using closure tree = fileTree("src") {     include("**/*.java") }  // Create a tree using a map tree = fileTree("dir" to "src", "include" to "**/*.java") tree = fileTree("dir" to "src", "includes" to listOf("**/*.java", "**/*.xml")) tree = fileTree("dir" to "src", "include" to "**/*.java", "exclude" to "**/*test*/**")                  

You can see more examples of supported patterns in the API docs for PatternFilterable. Also, see the API documentation for fileTree() to see what types you can pass as the base directory.

By default, fileTree() returns a FileTree instance that applies some default exclude patterns for convenience — the same defaults as Ant in fact. For the complete default exclude list, see the Ant manual.

If those default excludes prove problematic, you can workaround the issue by changing the default excludes in the settings script:

Example 28. Changing default excludes in the settings script

settings.gradle

                    import org.apache.tools.ant.DirectoryScanner  DirectoryScanner.removeDefaultExclude('**/.git') DirectoryScanner.removeDefaultExclude('**/.git/**')                  

settings.gradle.kts

                    import org.apache.tools.ant.DirectoryScanner  DirectoryScanner.removeDefaultExclude("**/.git") DirectoryScanner.removeDefaultExclude("**/.git/**")                  

Currently, Gradle's default excludes are configured via Ant's DirectoryScanner class.

Gradle does not support changing default excludes during the execution phase.

You can do many of the same things with file trees that you can with file collections:

• iterate over them (depth first)

• filter them (using FileTree.matching(org.gradle.api.Action) and Ant-style patterns)

• merge them

You can also traverse file trees using the FileTree.visit(org.gradle.api.Action) method. All of these techniques are demonstrated in the following example:

Example 29. Using a file tree

build.gradle

                    // Iterate over the contents of a tree tree.each {File file ->     println file }  // Filter a tree FileTree filtered = tree.matching {     include 'org/gradle/api/**' }  // Add trees together FileTree sum = tree + fileTree(dir: 'src/test')  // Visit the elements of the tree tree.visit {element ->     println "$element.relativePath => $element.file" }                  

build.gradle.kts

                    // Iterate over the contents of a tree tree.forEach{ file: File ->     println(file) }  // Filter a tree val filtered: FileTree = tree.matching {     include("org/gradle/api/**") }  // Add trees together val sum: FileTree = tree + fileTree("src/test")  // Visit the elements of the tree tree.visit {     println("${this.relativePath} => ${this.file}") }                  

We've discussed how to create your own file trees and file collections, but it's also worth bearing in mind that many Gradle plugins provide their own instances of file trees, such as Java's source sets. These can be used and manipulated in exactly the same way as the file trees you create yourself.

Another specific type of file tree that users commonly need is the archive, i.e. ZIP files, TAR files, etc. We look at those next.

Using archives as file trees

An archive is a directory and file hierarchy packed into a single file. In other words, it's a special case of a file tree, and that's exactly how Gradle treats archives. Instead of using the fileTree() method, which only works on normal file systems, you use the Project.zipTree(java.lang.Object) and Project.tarTree(java.lang.Object) methods to wrap archive files of the corresponding type (note that JAR, WAR and EAR files are ZIPs). Both methods return FileTree instances that you can then use in the same way as normal file trees. For example, you can extract some or all of the files of an archive by copying its contents to some directory on the file system. Or you can merge one archive into another.

Here are some simple examples of creating archive-based file trees:

Example 30. Using an archive as a file tree

build.gradle

                    // Create a ZIP file tree using path FileTree zip = zipTree('someFile.zip')  // Create a TAR file tree using path FileTree tar = tarTree('someFile.tar')  //tar tree attempts to guess the compression based on the file extension //however if you must specify the compression explicitly you can: FileTree someTar = tarTree(resources.gzip('someTar.ext'))                  

build.gradle.kts

                    // Create a ZIP file tree using path val zip: FileTree = zipTree("someFile.zip")  // Create a TAR file tree using path val tar: FileTree = tarTree("someFile.tar")  // tar tree attempts to guess the compression based on the file extension // however if you must specify the compression explicitly you can: val someTar: FileTree = tarTree(resources.gzip("someTar.ext"))                  

You can see a practical example of extracting an archive file in among the common scenarios we cover.

Understanding implicit conversion to file collections

Many objects in Gradle have properties which accept a set of input files. For example, the JavaCompile task has a source property that defines the source files to compile. You can set the value of this property using any of the types supported by the files() method, as mentioned in the api docs. This means you can, for example, set the property to a File, String, collection, FileCollection or even a closure or Provider.

This is a feature of specific tasks! That means implicit conversion will not happen for just any task that has a FileCollection or FileTree property. If you want to know whether implicit conversion happens in a particular situation, you will need to read the relevant documentation, such as the corresponding task's API docs. Alternatively, you can remove all doubt by explicitly using ProjectLayout.files(java.lang.Object...) in your build.

Here are some examples of the different types of arguments that the source property can take:

Example 31. Specifying a set of files

build.gradle

                    tasks.register('compile', JavaCompile) {      // Use a File object to specify the source directory     source = file('src/main/java')      // Use a String path to specify the source directory     source = 'src/main/java'      // Use a collection to specify multiple source directories     source = ['src/main/java', '../shared/java']      // Use a FileCollection (or FileTree in this case) to specify the source files     source = fileTree(dir: 'src/main/java').matching { include 'org/gradle/api/**' }      // Using a closure to specify the source files.     source = {         // Use the contents of each zip file in the src dir         file('src').listFiles().findAll {it.name.endsWith('.zip')}.collect { zipTree(it) }     } }                  

build.gradle.kts

                    tasks.register<JavaCompile>("compile") {     // Use a File object to specify the source directory     source = fileTree(file("src/main/java"))      // Use a String path to specify the source directory     source = fileTree("src/main/java")      // Use a collection to specify multiple source directories     source = fileTree(listOf("src/main/java", "../shared/java"))      // Use a FileCollection (or FileTree in this case) to specify the source files     source = fileTree("src/main/java").matching { include("org/gradle/api/**") }      // Using a closure to specify the source files.     setSource({         // Use the contents of each zip file in the src dir         file("src").listFiles().filter { it.name.endsWith(".zip") }.map { zipTree(it) }     }) }                  

One other thing to note is that properties like source have corresponding methods in core Gradle tasks. Those methods follow the convention of appending to collections of values rather than replacing them. Again, this method accepts any of the types supported by the files() method, as shown here:

Example 32. Appending a set of files

build.gradle

                    compile {     // Add some source directories use String paths     source 'src/main/java', 'src/main/groovy'      // Add a source directory using a File object     source file('../shared/java')      // Add some source directories using a closure     source { file('src/test/').listFiles() } }                  

build.gradle.kts

                    tasks.named<JavaCompile>("compile") {     // Add some source directories use String paths     source("src/main/java", "src/main/groovy")      // Add a source directory using a File object     source(file("../shared/java"))      // Add some source directories using a closure     setSource({ file("src/test/").listFiles() }) }                  

As this is a common convention, we recommend that you follow it in your own custom tasks. Specifically, if you plan to add a method to configure a collection-based property, make sure the method appends rather than replaces values.

smithhisee1948.blogspot.com

Source: https://docs.gradle.org/current/userguide/working_with_files.html

Share this post