Directory Management

Section 5.6 Directory Management

Organizing your source files, external assets (like images) and other support files, can become a challenge as your project grows. PreTeXt supports managing external and generated files (including images), but this requires some setup and understanding of the correct way to specify assets in your source. This section explains both those steps.

If you are starting a new project, using the PreTeXt-CLI (with the command pretext new book, for example), then most of the setup portion is done for you.

Subsection 5.6.1 External and Generated Files

Early in your writing project, you will decide you want to add images, embedded YouTube videos, interactive demonstrations, or other enhancements that are more than just words on the page. Some of these objects will be created outside of your PreTeXt project, such as a photograph. But some images are described within your PreTeXt project, such as a diagram authored using the LaTeX TikZ package, and PreTeXt will help generate different versions of the diagram in different file formats for use in output formats that are not LaTeX. For an embedded YouTube video we provide tools that will automatically get you a thumbnail preview image from a YouTube server, which will then appear in your non-interactive PDF version. We describe the photograph as external, since it comes from “somewhere else,” independent of anything you authored in your source. In contrast, an SVG image of your TikZ diagram for HTML output, and a preview image of a YouTube video for use in a print version, are described as generated since they are dependent on what you have put in your source, and PreTeXt automates almost all of their creation for you.

As these files are added to your project, you want to organize them in a specific (but flexible) way. First, make a directory (folder) for your external files. You can use any name you like, including external. Within reason you can place it wherever you want. Natural choices are as a peer of a source directory that holds your PreTeXt files, or as a subdirectory of your project’s top-level directory that may hold all your PreTeXt files. You can also organize this directory with subdirectories of your own choice, if that helps you stay organized.

Similarly, you need a separate directory for your generated files. As above, it can be named anything, including generated, and you can place it almost anywhere (close by). But now, it must have a precise directory structure, described below in List 5.6.2, according to what sort of generation produced the files.

After you have read this section, see the discussion of generated and external files in Subsection 27.2.4 for some good examples of why this flexibility is useful.

Subsection 5.6.2 Managing Directories

The name, and location of the external and generated directories (both of them, always) are then specified in a publication file (see Subsection 44.7.1 for the precise specification). The values of these attributes are relative pathnames to the directories, relative to the location of the main PreTeXt file for your project. Let us take a look at an example.

Example 5.6.1. Sample Directory Management.

Suppose you are M. Jones, the author of the AOTA project, and so you might have the following directory and file structure. There is some sort of overall path from the root of your entire filesystem to your project, so on Linux this path could be

/home/mjones/books/

and on other operating systems it will be slightly different, but those differences are not relevant here. Inside of books there would be an aota directory for the AOTA project, structured as follows. We are most interested in the ext and gen directories.

aota
    source
        aota.ptx
        alligators.ptx
        dogs.ptx
    publication
        print.ptx
        online.ptx
        epub.ptx
    ext
        photos
            slow-alligator.jpeg
            fast-whippet.png
        movies
            alligator-chases-whippet.mp4
        data
            deer-weights.csv
    gen
        latex-image
            marsupial-life-cycle.svg
        youtube
            sloth.jpg

In your publication files you would then have the entry, as a sub-element of the source element,

<directories external="../ext" generated="../gen"/>

Notice that we have deliberately named our directories ext and gen as part of this illustration, so that they are not identical to the attribute names on the <directories> element. The two attribute values are relative to the location of the main PreTeXt file, in this case aota.ptx. The two periods, .., mean to go up a level, here to the aota directory, then the slash indicates a step down to either gen or ext. Note that latex-image and youtube within gen must be specified exactly that way. Now every PreTeXt production tool can deduce where your files are.

How do you specify in your PreTeXt source which file to use where? Suppose you want your two (external) photographs to be used in a <sidebyside> element. You would author

<sidebyside widths="50% 30%">
     <image source="photos/slow-alligator.jpeg"/>
     <image source="photos/fast-whippet.jpeg"/>
<sidebyside/>

Notice that the values of your @source attributes do not include ext or ../ext. This gives you the freedom to move your ext directory or rename it, and then you only change the publication file and not your source. The decision to have the subdirectories photos and movies are not easily changed, since they will be in your source, but they may have value for the organization of your project.

The chase video is of your own creation, you have included it as part of your project’s files, and will be hosting it on your web server, where the rest of your HTML version is made available for viewing. You might author

<figure xml:id="fig-chase">
     <caption>Alligator chases whippet</caption>
     <video source="movies/alligator-chases-whippet.mp4"/>
<figure/>

The situation with the generated files is presently a bit more complicated, but will eventually be transparent to an author. The marsupial life cycle diagram has been authored in your source using the language from the LaTeX TikZ package, which is delimited as a <latex-image> element. When building a PDF from LaTeX output this is a no-brainer—PreTeXt just puts your TikZ code into the LaTeX file. But how about HTML output? The pretext/pretext script will manufacture a standalone version of this image and then convert it to an SVG, and this version will work very well in the online version of your project. What do you author?

<figure xml:id="fig-life-cycle">
     <caption>Marsupial Life Cycle</caption>
     <latex-image xml:id="marsupial-life-cycle">
         -- some TiKZ code here --
     </latex-image>
<figure/>

With the right options to the pretext/pretext script, the file marsupial-life-cycle.svg will be generated. Presently, you must explicitly tell the script to place the output in the directory

/home/mjones/books/gen/latex-image

but we plan to make this the default behavior. Or you could generate the file anywhere, and move it to the latex-image subdirectory of gen. Notice how much of the process of creating a companion SVG image (which excels in a web browser) is automated, and how the @xml:id attribute and the directories in the publication file are used to keep everything coordinated. The latex-image directory is not negotiable, it is the home for every image authored using a PreTeXt <latex-image> element (see List 5.6.2).

The slow sloth video was created by somebody else and posted to YouTube as a super slo-mo 240fps video. You authored it within a <figure> simply by using the ID of the video on YouTube.

<figure xml:id="fig-sloth">
     <caption>Slow Sloth</caption>
     <video xml:id="sloth" youtube="DJWEqYcxUl8"/>
<figure/>

With the right options to the pretext/pretext script, the file sloth.jpg will be generated by requesting a thumbnail preview from YouTube’s servers. Note how this content is generated from your source (precisely, the YouTube ID) through an automated process. Presently, you must explicitly tell the script to place the output in the directory

/home/mjones/books/gen/youtube

but we plan to make this the default behavior. The youtube directory is not negotiable, it is the home for every server-generated preview image for an embedded YouTube video (see List 5.6.2). (Note that any <video> may have an author-provided image as its preview, and then this would be considered an external file.)

As you can tell from the above, different output formats have different expectations for additional files and when producing an output, there may be different expectations for where exactly where to find files. A good example is the EPUB format, which despite being basically an HTML format, is still very rigorous about names, formats, and locations for image files. With the above procedures you provide just enough information for PreTeXt to handle the remainder of the complexity for you, without you becoming an expert with the EPUB standard. So the pretext/pretext script can automatically produce PDF, HTML, EPUB, Kindle, and braille, each in a single step.

We have skirted one finer point of all this. How does an author know if a file is external or generated and what are the supported subdirectories of the generated directory? A PreTeXt @source attribute will almost always point to a file that belongs in the external directory. An exception is the @preview attribute which is used to specify static images to use in outputs like PDF or print to represent more dynamic objects employed in more-capable electronic formats (videos, audio, interactives). (Please alert us to other exceptions!) The list below describes the subdirectories of the generated directory. The files that belong in these directories are all generated by the pretext/pretext script using aspects of your authored source.

List 5.6.2. Subdirectories of the Generated Directory

asymptote: Images described by Asymptote code in an <asymptote> element.
dynamic_subs: XML files needed for dynamic fill-in-the-blank exercises.
latex-image: Images described by LaTeX code in a <latex-image> element.
mermaid: Images described by Mermaid code in a <mermaid> element.
prefigure: Images described by PreFigure code in a <prefigure> element.
preview: Static images of dynamic content, created with automatic screenshots via a headless web server.
problems: MyOpenMath problems in static PreTeXt forms.
qrcodes: QR codes for PDF files that point to live versions of some interactive elements.
sageplot: Images described by Sage code in a <sageplot> element.
trace: Program trace data for <codelens> elements
youtube: Static images for YouTube videos, automatically downloaded from YouTube servers.

Just one more obscure situation to address, requiring a peek under the hood. Occasionally there are external files used in ways that PreTeXt cannot “see” them and provide the management necessary. One example is a file of data points used to generate a plot using the <latex-image> element. Suppose we have a file

/home/mjones/books/ext/data/deer-weights.csv

as shown in Example 5.6.1. We will use this inside of a <latex-image> element employing PGF code with a table {}; construction. As part of building the LaTeX output, this file will be copied to a temporary directory as

external/data/deer-weights.csv

Note that your ext has been converted to external as files are copied—that is not a typo. PreTeXt is not going to examine all your code and try to guess where you have requested a file. So you need to author

\addplot table {external/data/deer-weights.csv};

using the filename created after the copy. Then the LaTeX compilation will locate the file successfully. The external directory after the file copy is not negotiable, that is what is always used.

Subsection 5.6.3 Backward-Compatibility and Migration

Directory management was added in the middle of 2021. For projects existing at that time, it is an “opt-in” feature. In other words, existing projects can still be built by moving additional files in the same ways as always, so long as you do not have a <directories> element in your publication file.

However, it is absolutely necessary to opt-in to directory management (via the publication file) if you want to build EPUB or Kindle formats. You will also then be able to build PDF and HTML output with a single invocation of the pretext/pretext script, without any extra effort to place additional files in the right places.

New projects should definitely adopt directory management as their project develops. New features and tools will assume this. We now describe how existing projects can migrate.

Previously there was not a firm distinction between external and generated files, though they were specified differently in PreTeXt source. Many authors placed both types of files into a single directory, and since the default for locating generated files was a directory called images, this was often the single directory employed. Thus, external files would be specified with a @source attribute such as

<image source="images/bobcat.jpg"/>

A solution, if a bit clunky, is to make a new directory named ext and copy images and its contents to become a subdirectory of ext. Then place a relative path to ext in the @external attribute in your publication file. This means you will not have to edit your PreTeXt source at all. At some later time, you could remove all references to images in your source and move the files themselves directly into ext.

For generated files it will be a bit more involved. You need a new directory, such as gen, and you need to add it to your publication file with the right relative path. Then you will need to create as many of the relevant directories from List 5.6.2 as necessary, as subdirectories of gen. Populate these subdirectories with the relevant files, which may have been located in images before you started migrating. There was a string parameter, directory.images, that could be used to specify an alternative directory to the default, images. That parameter is now announced as deprecated, but will still be effective until your project opts-in.

As another short-term option for Mac or Linux users, it is possible to take previously existing resource directories and make symbolic links from a previous image directory, such as my_images, to the names provided here. In that case, assuming the directory structure provided here, a command like ln -s my_images external (issued in the directory containing my_images) would create the correct symbolic link.

Prev Top Next