Media Storage¶

Piccolo Admin has excellent support for managing media files (images, audio, video, and more). The files can be stored locally on the server, or in S3 compatible storage.

User Interface¶

You’re able to view many file types within the user interface - images, videos, audio, documents, and more.

Viewing images:

Viewing video:

Viewing audio:

Viewing PDFs:

Supported columns¶

We store the files in a folder on the server, or in a S3 bucket, and store unique references to those files in the database.

Note

We don’t store the files directly in the database, because database storage is typically much more expensive than block / object storage.

An example file reference is my-file-abc-123.jpeg. Since it’s a string, we can only store it in a database column which stores strings.

Varchar¶

Varchar is a good choice for storing file references. For example:

from piccolo.table import Table
from piccolo.column.column_types import Varchar


class Movie(Table):
    poster = Varchar()

Text¶

Text can also be used. For example:

from piccolo.table import Table
from piccolo.column.column_types import Text


class Movie(Table):
    poster = Text()

Array¶

We also support Array, but only when the base_column is either Varchar or Text. For example:

from piccolo.table import Table
from piccolo.column.column_types import Array, Varchar


class Movie(Table):
    screenshots = Array(base_column=Varchar())

This allows us to store multiple file references in a single column.

MediaStorage¶

For each column we want to use for media storage, we associate it with a MediaStorage instance.

Out of the box we have two subclasses - LocalMediaStorage and S3MediaStorage. You can also create your own subclass of MediaStorage to implement your own storage backend.

LocalMediaStorage¶

This stores media in a folder on the server.

Example¶

In order to associate a column with LocalMediaStorage, we do the following:

import os

from piccolo.columns import Array, Varchar
from piccolo_admin.endpoints import  (
    TableConfig,
    create_admin
)
from piccolo_api.media.local import LocalMediaStorage


class Movie(Table):
    title = Varchar()
    poster = Varchar()
    screenshots = Array(base_column=Varchar())


MEDIA_ROOT = '/srv/piccolo-admin/'


MOVIE_POSTER_MEDIA = LocalMediaStorage(
    column=Movie.poster,
    media_path=os.path.join(MEDIA_ROOT, 'movie_poster'),
    allowed_extensions=['jpg', 'jpeg', 'png']
)


MOVIE_SCREENSHOTS_MEDIA = LocalMediaStorage(
    column=Movie.screenshots,
    media_path=os.path.join(MEDIA_ROOT, 'movie_screenshots'),
    allowed_extensions=['jpg', 'jpeg', 'png']
)


movie_config = TableConfig(
    table_class=Movie,
    media_storage=[MOVIE_POSTER_MEDIA, MOVIE_SCREENSHOTS_MEDIA],
)


APP = create_admin([movie_config])

Some things to be aware of:

By specifiying allowed_extensions, we make sure that only images can be uploaded.
We store the files for posters and screenshots in separate folders - this is important when cleaning up files.

Source¶

class piccolo_api.media.local.LocalMediaStorage(column: t.Union[Text, Varchar, Array], media_path: str, executor: t.Optional[Executor] = None, allowed_extensions: t.Optional[t.Sequence[str]] = ALLOWED_EXTENSIONS, allowed_characters: t.Optional[t.Sequence[str]] = ALLOWED_CHARACTERS, file_permissions: t.Optional[int] = 0o600)[source]¶

Stores media files on a local path. This is good for simple applications, where you’re happy with the media files being stored on a single server.

Parameters:

column – The Piccolo Column which the storage is for.
media_path – This is the local folder where the media files will be stored. It should be an absolute path. For example, '/srv/piccolo-media/poster/'.
executor – An executor, which file save operations are run in, to avoid blocking the event loop. If not specified, we use a sensibly configured ThreadPoolExecutor.
allowed_extensions – Which file extensions are allowed. If None, then all extensions are allowed (not recommended unless the users are trusted).
allowed_characters – Which characters are allowed in the file name. By default, it’s very strict. If set to None then all characters are allowed.
file_permissions – If set to a value other than None, then all uploaded files are given these file permissions.

S3MediaStorage¶

This allows us to store files in a private S3 bucket. When we need to access a file, a signed URL is generated, so the file can be viewed securely.

Example¶

In order to associate a column with S3MediaStorage, we do the following:

import os

from piccolo.columns import Array, Varchar
from piccolo_admin.endpoints import  (
    TableConfig,
    create_admin
)
from piccolo_api.media.s3 import S3MediaStorage


class Movie(Table):
    title = Varchar()
    poster = Varchar()
    screenshots = Array(base_column=Varchar())


# Note - don't store credentials in source code if possible.
# It's safer to read them from environment variables.
# A tool like `python-dotenv` can help with this.
S3_CONNECTION_KWARGS = {
    "aws_access_key_id": os.environ.get("AWS_ACCESS_KEY_ID"),
    "aws_secret_access_key": os.environ.get("AWS_SECRET_ACCESS_KEY"),
}


MOVIE_POSTER_MEDIA = S3MediaStorage(
    column=Movie.poster,
    bucket_name="bucket123"
    folder_name="movie_poster"
    connection_kwargs=S3_CONNECTION_KWARGS,
    allowed_extensions=['jpg', 'jpeg', 'png']
)


MOVIE_SCREENSHOTS_MEDIA = S3MediaStorage(
    column=Movie.screenshots,
    bucket_name="bucket123"
    folder_name="movie_screenshots"
    connection_kwargs=S3_CONNECTION_KWARGS,
    allowed_extensions=['jpg', 'jpeg', 'png']
)


movie_config = TableConfig(
    table_class=Movie,
    media_storage=[MOVIE_POSTER_MEDIA, MOVIE_SCREENSHOTS_MEDIA],
)


APP = create_admin([movie_config])

Some things to be aware of:

By specifiying allowed_extensions, we make sure that only images can be uploaded.
We store the files for posters and screenshots in separate folders within the bucket - this is important when cleaning up files. You could even store them in separate buckets if you prefer.

Source¶

class piccolo_api.media.s3.S3MediaStorage(column: t.Union[Text, Varchar, Array], bucket_name: str, folder_name: t.Optional[str] = None, connection_kwargs: t.Optional[t.Dict[str, t.Any]] = None, sign_urls: bool = True, signed_url_expiry: int = 3600, upload_metadata: t.Optional[t.Dict[str, t.Any]] = None, executor: t.Optional[Executor] = None, allowed_extensions: t.Optional[t.Sequence[str]] = ALLOWED_EXTENSIONS, allowed_characters: t.Optional[t.Sequence[str]] = ALLOWED_CHARACTERS)[source]¶

Stores media files in S3 compatible storage. This is a good option when you have lots of files to store, and don’t want them stored locally on a server. Many cloud providers provide S3 compatible storage, besides from Amazon Web Services.

Parameters:

column – The Piccolo Column which the storage is for.
bucket_name – Which S3 bucket the files are stored in.
folder_name – The files will be stored in this folder within the bucket. S3 buckets don’t really have folders, but if folder is 'movie_screenshots', then we store the file at 'movie_screenshots/my-file-abc-123.jpeg', to simulate it being in a folder.

connection_kwargs –

These kwargs are passed directly to the boto3 client. For example:

S3MediaStorage(
    ...,
    connection_kwargs={
        'aws_access_key_id': 'abc123',
        'aws_secret_access_key': 'xyz789',
        'endpoint_url': 's3.cloudprovider.com',
        'region_name': 'uk'
    }
)

sign_urls – Whether to sign the URLs - by default this is True, as it’s highly recommended that your store your files in a private bucket.
signed_url_expiry – Files are accessed via signed URLs, which are only valid for this number of seconds.
upload_metadata –
You can provide additional metadata to the uploaded files. To see all available options see S3Transfer.ALLOWED_UPLOAD_ARGS. Below we show examples of common use cases.

To set the ACL:
```
S3MediaStorage(
    ...,
    upload_metadata={'ACL': 'my_acl'}
)
```
To set the content disposition (how the file behaves when opened - is it downloaded, or shown in the browser):
```
S3MediaStorage(
    ...,
    # Shows the file within the browser:
    upload_metadata={'ContentDisposition': 'inline'}
)
```
To attach user defined metadata to the file:
```
S3MediaStorage(
    ...,
    upload_metadata={'Metadata': {'myfield': 'abc123'}}
)
```
To specify how long browsers should cache the file for:
```
S3MediaStorage(
    ...,
    # Cache the file for 24 hours:
    upload_metadata={'CacheControl': 'max-age=86400'}
)
```
Note: We automatically add the ContentType field based on the file type.
executor – An executor, which file save operations are run in, to avoid blocking the event loop. If not specified, we use a sensibly configured ThreadPoolExecutor.
allowed_extensions – Which file extensions are allowed. If None, then all extensions are allowed (not recommended unless the users are trusted).
allowed_characters – Which characters are allowed in the file name. By default, it’s very strict. If set to None then all characters are allowed.

Integrating it with your wider app¶

If you’re using Piccolo Admin as part of a larger application, you can easily gain access to the stored files, and use them within your own code.

For example:

# We can fetch a file from storage
file = await MOVIE_POSTER_MEDIA.get_file('some-file-key.jpeg')

# We can delete files from storage
await MOVIE_POSTER_MEDIA.delete_file('some-file-key.jpeg')

To see all of the methods available, look at MediaStorage.

Cleaning up files¶

If we delete a row from the database which references a stored file, the file isn’t automatically deleted. This is common practice, as it gives a bit more safety against accidentally deleting files.

We can periodically delete any files which are no longer referenced in the database.

await MOVIE_POSTER_MEDIA.delete_unused_files()

Warning

It’s very important that each column stores files in its own folder or S3 bucket. If multiple columns share the same folder, when we run delete_unused_files we may delete files needed by another column.