# Flystorage Documentation


# Visibility


Flystorage generalised over ACLs and permissions using **visibility**. The default provided
`Visibility` enum specifies _public_ and _private_ visibility, which is respected by all
implemented adapters.

The API does accepts a `string` input for visibility, allowing you to create more specialised
visibility handling strategies if you need it.

## What is public and private?

Public and private visibility is different depending on which adapter you use. For the local
filesystem, public files are readable by webserver whereas private files are not. For AWS S3,
public files can be read using public URLs whereas private files require a temporary URL to
be downloaded by a browser or HTTP client.


# Architecture


Flystorage is designed to abstract away the differences of underlying storage solutions as much
as possible. Most operations are entirely consistent across implementations. Some more specialised
operations, such as URL generation, moving files, and directory handling is _not_ normalised. A
pragmatic approach is taken for smoothing over behavioural differences.

## Normalised Behaviour

In general, the behaviour for operations on files are normalised across the board. For directories
case-by-case decisions are made. The most prominent exceptions are:

1. Passing a directory path to the `moveFile` method may result in a directory being moved on the
   local filesystem.
2. Explicitly creation a directory (which is not needed) may result in a no-op.

## Software Design Rationale

Flysystem implements the adapter pattern to enable an consistent experience across implementations.
The outer `FileStorage` class provides the top-level interface to which you are expected to
integrate against. The inner `StorageAdapter` implementations are designed to smooth other
provider-specific differences, but ultimately do not directly concern themselves with end-user
usability.

## Encapsulation

As much as possible, Flystorage encapsulates storage concerns. Implementation  differences are
normalised, but it also aims to hide any technical configuration from the application code.

Configuration, such as the root directory, are contained in the adapters as configuration. This
makes usage more portable than if they were to be exposed to the application code. You're advised
to leverage this as much as possible. The full path of a file can be cut into two pieces:

```text
[/path/to/root/directory]/[path/to/file.txt]
^ configuration           ^ known to app code
```

Hiding these details from application code gives you full mobility. You can move locations
and even storage solutions without changing application code.


# Setup


Setting up Flystorage

Flystorage is distributed through NPM. Install it using your favourite package manager.

```bash
npm install --save @flystorage/file-storage
```

Flystorage user the _adapter pattern_ to encapsulate storage. It's important not to interact
with an adapter directly. Instead, always interact with the main FileStorage class.

To get started install any of the pre-built adapters or build one yourself.


# Implement your own custom adapter


The adapter based design enables you to create your own adapter. Implement the `StorageAdapter` interface
and you're all set!

Use this template to get started quickly:

```typescript
import {Readable} from 'stream';
import {
    StorageAdapter,
    ChecksumOptions,
    CreateDirectoryOptions,
    FileContents, MimeTypeOptions,
    PublicUrlOptions,
    StatEntry, TemporaryUrlOptions,
    WriteOptions,
    PathPrefixer,
    CopyFileOptions,
    MoveFileOptions,
} from '@flystorage/file-storage';

export type AdapterFileStorageOptions = {
    prefix?: string,
}

export class AdapterFileStorage implements StorageAdapter {
    private readonly prefixer: PathPrefixer;

    constructor(
        private readonly options: AdapterFileStorageOptions = {},
    ) {
        this.prefixer = new PathPrefixer(options.prefix || '');
    }

    async write(path: string, contents: Readable, options: WriteOptions): Promise<void> {
        throw new Error('Not implemented');
    }

    async read(path: string): Promise<FileContents> {
        throw new Error('Not implemented');
    }
    async deleteFile(path: string): Promise<void> {
        throw new Error('Not implemented');
    }
    async createDirectory(path: string, options: CreateDirectoryOptions): Promise<void> {
        throw new Error('Not implemented');
    }
    async stat(path: string): Promise<StatEntry> {
        throw new Error('Not implemented');
    }
    list(path: string, options: {deep: boolean}): AsyncGenerator<StatEntry> {
        throw new Error('Not implemented');
    }
    async changeVisibility(path: string, visibility: string): Promise<void> {
        throw new Error('Not implemented');
    }
    async visibility(path: string): Promise<string> {
        throw new Error('Not implemented');
    }
    async deleteDirectory(path: string): Promise<void> {
        throw new Error('Not implemented');
    }
    async fileExists(path: string): Promise<boolean> {
        throw new Error('Not implemented');
    }
    async directoryExists(path: string): Promise<boolean> {
        throw new Error('Not implemented');
    }
    async publicUrl(path: string, options: PublicUrlOptions): Promise<string> {
        throw new Error('Not implemented');
    }
    async temporaryUrl(path: string, options: TemporaryUrlOptions): Promise<string> {
        throw new Error('Not implemented');
    }
    async checksum(path: string, options: ChecksumOptions): Promise<string> {
        throw new Error('Not implemented');
    }
    async mimeType(path: string, options: MimeTypeOptions): Promise<string> {
        throw new Error('Not implemented');
    }
    async lastModified(path: string): Promise<number> {
        throw new Error('Not implemented');
    }
    async fileSize(path: string): Promise<number> {
        throw new Error('Not implemented');
    }
    async copyFile(from: string, to: string, options: CopyFileOptions): Promise<void> {
        throw new Error('Not implemented');
    }
    async moveFile(from: string, to: string, options: MoveFileOptions): Promise<void> {
        throw new Error('Not implemented');
    }
}
```


# Azure Storage Blob


Flystorage adapter for Azure Storage Blob

This package contains the Flystorage adapter for Azure Storage Blob.

## Installation

Install all the required packages

```bash
npm install --save @flystorage/file-storage @flystorage/azure-storage-blob @azure/storage-blob
```

## Usage

```typescript
import {FileStorage} from '@flystorage/file-storage';
import {AzureStorageBlobStorageAdapter} from '@flystorage/azure-storage-blob';

const blobService = BlobServiceClient.fromConnectionString(process.env.AZURE_DSN!);
const container = blobService.getContainerClient('flysystem');
const adapter = new AzureStorageBlobStorageAdapter(container);
const storage = new FileStorage(adapter);
```


# Amazon S3


Flystorage adapter for AWS S3

This package contains the Flystorage adapter for AWS S3 using the V3 SDK.

## Installation

```bash
npm install --save @flystorage/file-storage @flystorage/aws-s3 @aws-sdk/client-s3
```

## Setup

```typescript
import {FileStorage} from '@flystorage/file-storage';
import {AwsS3StorageAdapter} from '@flystorage/aws-s3';
import {S3Client} from '@aws-sdk/client-s3';

const client = new S3Client();
const adapter = new AwsS3StorageAdapter(client, {
    bucket: '{your-bucket-name}',
    prefix: '{optional-path-prefix}',
});
const storage = new FileStorage(adapter);
```


# In-Memory


Flystorage adapter that uses only memory

This package contains the Flystorage adapter that uses only memory.

## Installation

Install all the required packages

```bash
npm install --save @flystorage/file-storage @flystorage/in-memory
```

## Setup

```typescript
import {FileStorage} from '@flystorage/file-storage';
import {InMemoryStorageAdapter} from '@flystorage/in-memory';

const adapter = new InMemoryStorageAdapter();
const storage = new FileStorage(adapter);
```


# Local FS


Flystorage adapter for the local filesystem

This package contains the Flystorage adapter for the local filesystem.

## Installation

```bash
npm install --save @flystorage/file-storage @flystorage/local-fs
```

## Setup

```typescript
import {FileStorage} from '@flystorage/file-storage';
import {LocalStorageAdapter} from '@flystorage/local-fs';

const rootDirectory = resolve(process.cwd(), 'my-files');
const adapter = new LocalStorageAdapter(rootDirectory);
const storage = new FileStorage(adapter);
```


# Google Cloud Storage


Flystorage adapter for Google Cloud Storage

This package contains the Flystorage adapter for Google Cloud Storage.

## Installation

```bash
npm install --save @flystorage/file-storage @flystorage/google-cloud-storage @google-cloud/storage
```

## Setup

```typescript
import {FileStorage} from '@flystorage/file-storage';
import {GoogleCloudStorageStorageAdapter} from '@flystorage/google-cloud-storage';
import {Storage} from '@google-cloud/storage';

const client = new Storage();
const bucket = googleStorage.bucket('{bucket-name}}', {
    userProject: '{user-project}}',
});
const adapter = new GoogleCloudStorageStorageAdapter(bucket, {
    prefix: '{optional-path-prefix}',
});
const storage = new FileStorage(adapter);
```

## Visibility

Setting and retrieving visibility is only meaningful for legacy buckets. To use this functionality
with Flystorage, pass the legacy visibility handling to the constructor:

```typescript
import {GoogleCloudStorageStorageAdapter, LegacyVisibilityHandling} from '@flystorage/google-cloud-storage';

const adapter = new GoogleCloudStorageStorageAdapter(bucket, {
    prefix: '{optional-path-prefix}',
}, new LegacyVisibilityHandling(
    'allUsers', // acl entity, optional
    'publicRead', // acl for Visibility.PUBLIC, optional,
    'projectPrivate', // acl for Visibility.PRIVATE, optional,
));
```


# Chaos (for testing)


Flystorage adapter for chaos engineering

This package contains an adapter decorator that causes instrumented failures. This is useful for
when you want to make sure your application code can deal with failures gracefully.

Use the chaos adapter decorator to wrap your actual adapter and instruct it to throw errors. Use
it to validate retry mechanisms, logging, and metric collection to gain better insights into the
functioning of your application.

## Installation

```bash
npm install --save @flystorage/file-storage @flystorage/chaos
```

## Setup

```typescript
import {FileStorage} from '@flystorage/file-storage';
import {ChaosStorageAdapterDecorator, TriggeredErrors} from '@flystorage/chaos';

const strategy = new TriggeredErrors();
const adapter = new ChaosStorageAdapterDecorator(
    createActualAdapter(),
    strategy,
);
const storage = new FileStorage(adapter);
```

## Usage

```typescript

import {TriggeredErrors} from '@flystorage/chaos';

const strategy = new TriggeredErrors();

// error on all write calls
strategy.on('write', () => new Error());

// error on first 2 stat calls
strategy.on('stat', () => new Error(), {times: 2});

// error after first 2 deleteFile calls
strategy.on('deleteFile', () => new Error(), {after: 2});

// error on 2nd and 3rd call to any method
strategy.on('*', () => new Error(), {after: 1, times: 2});
```


# FileStorage API


## Write Files

Writing file using the `write` method:

```typescript
try {
    await storage.write('path/to/file.txt', contents);
} catch (err) {
    if (err instanceof UnableToWriteFile) {
        // handle error
    }
}
```

The `write` method accepts a `string`, `Uint8Array` (or `Buffer`), or any `Readable` stream.
When writing a file, any of the parent directories are automatically created if the underlying
storage implementation requires directories to exist.

You can specify visibility of a file when writing it:

```typescript
import {VISIBILITY} from '@flystorage/file-storage';

await storage.write('path/to/file.txt', contents, {
    visibility: Visibility.PUBLIC,
});
```

---

## Read Files

Read a file using the `read`, `readToString`, `readToBuffer`, or `readToUint8Array` methods:

```typescript
try {
    /**
     * @type {Readable}
     */
    const contents = await storage.read('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToReadFile) {
        // handle error
    }
}
```

---

## Delete Files

Delete a file using the `delete` method. Deleting files deletes files _if_ they exist. To simplify
common scenarios, there is no error when you try to delete a non-existing file.

```typescript
try {
    await storage.deleteFile('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToDeleteFile) {
        // handle error
    }
}
```

---

## Create Directories

The `createDirectory` method allows you to explicitly create directories. Some implementations do
not support the creation of actual or virtual directories, in those cases this method is a no-op.

```typescript
try {
    await storage.createDirectory('path/to/directory');
} catch (err) {
    if (err instanceof UnableToCreateDirectory) {
        // handle error
    }
}
```

In cases the filesystem requires nested directories, all encapsulated directories are implicitly
created.

---

## Delete Directories

The `deleteDirectory` method allows you to delete a directory. Any contents of the directory is
implicitly deleted as well. Only use this method if you want to delete everything contained in the
directory.

```typescript
try {
    await storage.deleteDirectory('path/to/directory');
} catch (err) {
    if (err instanceof UnableToDeleteDirectory) {
        // handle error
    }
}
```

For storage implementation that do not support directories, these implementations emulate  the
deletion of anything that matches the directory prefix. This produces consistent behaviour
across adapters.

---

## File or directory info

Use the `stat` or `statFile` methods to retrieve _stat_ information for a file or directory.

```typescript
try {
    const stat = await storage.stat('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToGetStat) {
        // handle error
    }
}
```

The `stat` method has normalised behaviour across implementation for files. Some implementations
do not support directories, in which case you'll get an error. The `statFile` method ensures the
returned stat entry is a `FileInfo` and fails if the returned stat entry is a `DirectoryInfo`.

---

## Moving Files

Files can be moved using the `moveFile` method. The _visibility_ is a file is retained, unless the
`retainVisibility` option is set to `false`. When set to `false` the default visibility is used.
You can also explicitly set the `visibility` option to set it, which prevents a call to resolve it
from the existing file. This may be beneficial for performance.


```typescript
try {
    await storage.moveFile('from/here.txt', 'to/there.txt', {
        visibility: Visibility.PRIVATE,
    });
} catch (err) {
    if (err instanceof UnableToMoveFile) {
        // handle error
    }
}
```

Moving files works reliably across implementations. Some implementation support moving of directories
this behaviour is _not_ normalised across implementations.

---

## Copying Files

Files can be copied using the `copyFile` method. The _visibility_ is a file is retained, unless the
`retainVisibility` option is set to `false`. When set to `false` the default visibility is used.
You can also explicitly set the `visibility` option to set it, which prevents a call to resolve it
from the existing file. This may be beneficial for performance.


```typescript
try {
    await storage.copyFile('from/here.txt', 'to/there.txt', {
        visibility: Visibility.PRIVATE,
    });
} catch (err) {
    if (err instanceof UnableToCopyFile) {
        // handle error
    }
}
```

---

## Change visibility

The _visibility_ of files can be changed using the `changeVisibility` method. To understand visibility,
read up on the [visibility documentation](/visibility/).

```typescript
try {
    await storage.changeVisibility('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToCopyFile) {
        // handle error
    }
}
```

Some implementations, like [Azure Storage Blob](/adapter/aws-storage-blob/), do not support visibility
and may throw an error.

---

## Determine visibility

The _visibility_ of files can be resolved using the `visibility` method. To understand visibility,
read up on the [visibility documentation](/visibility/).

```typescript
try {
    const visibility = await storage.visibility('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToGetVisibility) {
        // handle error
    }
}
```

Some implementations, like [Azure Storage Blob](/adapter/aws-storage-blob/), do not support visibility
and may throw an error.

---

## File Existence

You can check if a file exists using the `fileExists` method.


```typescript
try {
    const exists = await storage.fileExists('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToCheckFileExistence) {
        // handle error
    }
}
```

---

## Directory Existence

You can check if a directory exists using the `directoryExists` method.

```typescript
try {
    const exists = await storage.directoryExists('path/to/directory');
} catch (err) {
    if (err instanceof UnableToCheckDirectoryExistence) {
        // handle error
    }
}
```

When the underlying implementation does not support _actual_ directories, the behaviour is emulated
by using the least expensive way to list files that match the directory prefix. 

---

## List Directory Contents

The contents is a directory can be listed using the `list` method. Recursive or deep listings can be
fetched by setting the `deep` option to true.

```typescript
try {
    const listing = storage.list('path/to/directory', {deep: true});
} catch (err) {
    if (err instanceof UnableToCheckDirectoryExistence) {
        // handle error
    }
}
```

### Looping over listings

The returned directory listing is an `AsyncIterable` and can be used to loop over the contents.

```typescript
for await (const entry of listing) {
    if (entry.type === 'file' || entry.isFile) {
        // handle the file
    }
    
    if (entry.type === 'directory' || entry.isDirectory) {
        // handle a directory
    }
}
```

You can check the type of the entry using the `type` property (`file` or `directory`) or by using the
`isFile` and/or `isDirectory` properties.

### Collect a listing as an array

Listings come with convenience method to make handling them easier. The `toArray` method collects
the entries and returns them as an array.

```typescript
const listingAsArray = await listing.toArray();
```

---

```typescript
try {
    const exists = await storage.directoryExists('path/to/directory');
} catch (err) {
    if (err instanceof UnableToCheckDirectoryExistence) {
        // handle error
    }
}
```

When the underlying implementation does not support _actual_ directories, the behaviour is emulated
by using the least expensive way to list files that match the directory prefix. 

---

## Public URLs

The `publicUrl` method resolves the public URL of a file.

```typescript
try {
    const url = await storage.publicUrl('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToGetPublicUrl) {
        // handle error
    }
}
```

Implementation that do not support public URLs may throw an error. Some adapter may require you to inject
a public url resolving strategy to enable them to produce public URLs.

---

## Temporary URLs

The `temporaryUrl` method resolves a temporary URL to a file. The second argument accepts either a
`Date` or a `number` (millisecond precise timestamp) to indicate when the link should expire.

```typescript
try {
    const expiresAt = Date.now() + 24 * 60 * 1000;
    const url = await storage.temporaryUrl('path/to/file.txt', expiresAt);
} catch (err) {
    if (err instanceof UnableToGetTemporaryUrl) {
        // handle error
    }
}
```

Implementation that do not support temporary URLs may throw an error. Some adapter may require you to inject
a temporary url resolving strategy to enable them to produce temporary URLs.

---

## Last Modified

The `lastModified` method resolves the last modification time of a file.

```typescript
try {
    const timestamp = await storage.lastModified('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToGetLastModified) {
        // handle error
    }
}
```

---

## Mime-type

The `mimeType` method resolves the mime-type of a file.

```typescript
try {
    const mimetype = await storage.mimeType('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToGetMimeType) {
        // handle error
    }
}
```

---

## File Size

The `fileSize` method resolves the size of a file.

```typescript
try {
    const size = await storage.fileSize('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToGetFileSize) {
        // handle error
    }
}
```

---

## Checksums

The `checksum` method resolves the hash/checksum/etag of a file. Optionally, an `algo` option can be
passed if a specific type of checksum should be retrieved.

```typescript
try {
    const checksum = await storage.checksum('path/to/file.txt', optionalOptions);
} catch (err) {
    if (err instanceof UnableToGetChecksum) {
        // handle error
    }
}
```

When the underlying storage solution does not expose pre-computed checksums, a checksum is calculated
in-memory. This involves streaming the entire file, which may be a costly operation.

When checksums are calculated, the resulting hash is encoded. By default `hex` is used, you can specify
any suitable `BinaryToTextEncoding` value using the `encoding` option.

---

```typescript
try {
    await storage.deleteFile('path/to/file.txt');
} catch (err) {
    if (err instanceof UnableToDeleteFile) {
        // handle error
    }
}
```


# Express / Multer Storage


The `@flystorage/multer-storage` package provides an storage implementation for multer,
which enables file uploads from express to any of the supported storage solutions.

## Installation

```bash
npm install --save @flystorage/file-storage @flystorage/multer-storage
```

## Usage

```typescript
import {FileStorage} from '@flystorage/file-storage';
import {FlystorageMulterStorageEngine} from '@flystorage/multer-storage';
import multer from 'multer';
import express from 'express';

const adapter = createYourAdapter();
const fileStorage = new FileStorage(adapter);

const storage = new FlystorageMulterStorageEngine(
    uploadStorage,
    async (action, _req: express.Request, file: Express.Multer.File) => {
        if (action === 'handle') {
            return file.originalname;
        } else {
            return file.destination;
        }
    }
);

const uploader = multer({storage});
const app = express();

app.post('/upload/document', uploader.single('document'), , (req, res, next) => {
  // req.file is the `document` file
  // req.body will hold the text fields, if there were any
});

app.listen(3000);
```

## Express examples

For more Express examples, read the [multer docs](https://www.npmjs.com/package/multer).


# Mime-type of stream


The `@flystorage/stream-mime-type` package provides a utility to determine the mime-type of
a stream in a non-destructive way.

Underneath, the package uses the `file-type` and `mime-types` packages to resolve the mime-type
of the stream. It first tries to determine the mime-type based on the file contents and will fall
back to resolving it based on the file extension.

## Installation

```bash
npm install --save @flystorage/stream-mime-type
```

## Usage

Resolve a mime-type:

```typescript
import {resolveMimeType} from '@flystorage/stream-mime-type';

const originalStream = fs.createReadStream(pathToFile);

const [mimetype, stream] = resolveMimeType(filename, originalStream);
```

Sampling the start of a stream:

```typescript
import {streamHead} from '@flystorage/stream-mime-type';

const originalStream = fs.createReadStream(pathToFile);

const [sample, stream] = streamHead(originalStream, sizeInBytes);
```