New file management

[susanu]

Hi guys,

In backpack v5 i implemented a file management to make uploads easier.
The way it works is like this:

• Encapsulate everything related to that file in a new class called File
• Create a new Castable to cast the model attribute to this new class
• Attach a new trait that deals with the uploaded files and does all the jobs, upload, replace, delete.
• Create a new field type called file_upload which is just a wrapper for the upload field but supports this File class.
• Create a new column type called file which displays a link to the file.

This system does not update, upload or delete the files if the model throws an error when saving.

IMPORTANT: All of my files are stored in a s3 disk and the paths look like this: {model_class}/{id}/{?attribute}/{filename}

Path examples (please note that path include the model id which helps me find everything related to that model in 1 folder):

• banner/1/2/mobile/file.png
• banner/1/2/desktop/file.png
• banner/1/2/external.jpg
• product/1/main.png

I propose something similar for the next version of backpack, i know you implemented uploaders but controlling everything from one place would be a nice addition.
Let me know if you like any of the ideas presented here so we can start building.

Source files (please keep in mind that these are oriented towards my use cases and need to be adjusted):

• Model configuration

use \App\General\Traits\HasFileUploads;
protected $casts = [
    'file' => AsFile::class,
    // 'file' => AsFile::class.':cloud-new',
];
// If you want custom base path not the one generated based on class name
public function getBasePath(): string
{
    return 'custom/path' . implode('/', str_split($this->getKey()));
}

• CRUD field configuration is as simple as this.

CRUD::addField([
    'name' => 'file',
    'type' => 'file_upload',
]);

• HasFileUploads.php

<?php

namespace App\General\Traits;

use App\General\Casts\AsFile;
use App\Storage\File;
use Illuminate\Http\Request;
use Illuminate\Http\UploadedFile;
use Illuminate\Support\Str;
use LogicException;

trait HasFileUploads
{
    /**
     * Set a given attribute on the model.
     *
     * @param  string  $key
     * @param  mixed  $value
     * @return $this
     */
    abstract public function setAttribute($key, $value);

    /**
     * Get an attribute from the model.
     *
     * @param  string  $key
     * @return mixed
     */
    abstract public function getAttribute($key);

    /**
     * Get the model's original attribute values.
     *
     * @param  string|null  $key
     * @param  mixed  $default
     * @return mixed|array
     */
    abstract public function getOriginal($key = null, $default = null);

    /**
     * Get the value of the model's primary key.
     *
     * @return mixed
     */
    abstract public function getKey();

    /**
     * Get the primary key for the model.
     *
     * @return string
     */
    abstract public function getKeyName();

    /** @var UploadedFile[] */
    protected array $deferredUploads = [];

    /** @var File[] */
    protected array $deferredFiles = [];

    /** @var File[] */
    protected array $deferredDeletedFiles = [];

    /**
     * The model attributes that store the file names of uploaded files
     *
     * @return array
     */
    protected function getFileAttributes(): array
    {
        $attributes = [];
        foreach($this->casts as $attribute => $castType) {
            if(str_starts_with($castType, AsFile::class)) {
                $attributes[] = $attribute;
            }
        }

        return $attributes;
    }

    /**
     * Trait initializer, used internally by Laravel
     */
    public static function bootHasFileUploads()
    {
        static::saving(function ($model) {
            foreach($model->getFileAttributes() as $fileAttribute) {
                $file = $model->getAttribute($fileAttribute);

                if(is_null($file)) {
                    $model->deleteOldFile($fileAttribute);
                    continue;
                }

                if($file instanceof UploadedFile) {
                    $model->uploadFile($file, $fileAttribute);
                }
            }
        });

        static::saved(function ($model) {
            $model->processDeferredUploads();
            $model->processDeferredFiles();
        });

        static::replicating(function ($model) {
            $query = $model->newModelQuery();

            foreach($model->getAttributes() as $attribute => $value) {
                $query->where($attribute, $value);
            }

            $sourceModel = $query->get()->first();

            if(!$sourceModel) {
                return;
            }

            foreach($model->getFileAttributes() as $fileAttribute) {
                if($file = $sourceModel->getFileByAttribute($fileAttribute)) {
                    $model->deferFileUpload($file, $fileAttribute);
                }
            }
        });

        static::deleting(function($model) {
            foreach($model->getFileAttributes() as $fileAttribute) {
                if($file = $model->getFileByAttribute($fileAttribute)) {
                    $model->deferFileDelete($file);
                }
            }
        });

        static::deleted(function($model) {
            $model->processDeferredDeletedFiles();
        });
    }

    /**
     * @return string
     */
    public function getBasePath(): string
    {
        $className = $this::class;
        if ($pos = strrpos($this::class, '\\')) {
            $className = substr($this::class, $pos + 1);
        }

        return Str::snake($className) . '/' . implode('/', str_split($this->getKey()));
    }

    /**
     * @param string $attribute
     * @return string
     */
    public function getFileBasePath(string $attribute): string
    {
        return $this->getBasePath();
    }

    /**
     * @param string $attribute
     * @param string|null $value
     * @return string
     */
    public function getFilePath(string $attribute, string $value = null): string
    {
        if(!$value) {
            $value = $this->getAttribute($attribute);
        }

        return "{$this->getFileBasePath($attribute)}/{$value}";
    }

    /**
     * Build a file object based on a given attribute.
     *
     * @param string $attribute
     * @return File|null
     */
    public function getFileByAttribute(string $attribute): ?File
    {
        $value = $this->getAttribute($attribute);

        if($value instanceof UploadedFile) {
            $value = $value->getClientOriginalName();
        }

        if($value instanceof File) {
            $value = $value->filename();
        }

        if($value) {
            return new File($this->getFilePath($attribute, $value));
        }

        return null;
    }

    /**
     * Build a file object based on a given attribute.
     *
     * @param string $attribute
     * @return File|null
     */
    public function getFileByOriginalAttribute(string $attribute): ?File
    {
        if($value = $this->getOriginal($attribute)) {
            if($value instanceof File) {
                return $value;
            }

            return new File($this->getFilePath($attribute, $value));
        }

        return null;
    }

    public function processDeferredUploads(): void
    {
        foreach ($this->deferredUploads as $attribute => $file) {
            $this->uploadNow($file, $attribute);
        }
        $this->deferredUploads = [];
    }

    public function processDeferredFiles(): void
    {
        foreach ($this->deferredFiles as $attribute => $originalFile) {
            if($newFile = $this->getFileByAttribute($attribute)) {
                $newFile->put($originalFile->readStream());
            }
        }

        $this->deferredFiles = [];
    }

    public function processDeferredDeletedFiles(): void
    {
        foreach ($this->deferredDeletedFiles as $file) {
            $file->delete();
        }

        $this->deferredDeletedFiles = [];
    }

    /**
     * @param Request $request
     * @param string $requestAttribute
     * @param string|null $modelAttribute
     * @return string|false
     */
    public function uploadFromRequest(Request $request, string $requestAttribute, ?string $modelAttribute = null)
    {
        return $this->uploadFile($request->file($requestAttribute), $modelAttribute ?? $requestAttribute);
    }

    /**
     * @param UploadedFile|null $file
     * @param string $attribute
     * @return string|false
     */
    public function uploadFile(?UploadedFile $file, string $attribute)
    {
        if (!($file && $file->isValid())) {
            return false;
        }

        $this->deleteOldFile($attribute);

        $this->setAttribute($attribute, $this->getFileName($file, $attribute));

        // We can only upload the file if the model has a primary key
        // Otherwise the upload is deferred until after the model is saved
        return $this->getKey() ? $this->uploadNow($file, $attribute) : $this->uploadDeferred($file, $attribute);
    }

    protected function getFileName(UploadedFile $file, string $attribute): string
    {
        return $attribute . '-' . $file->hashName();
        //return $file->hashName();
    }

    protected function deleteOldFile(string $attribute): void
    {
        if ($oldFile = $this->getFileByOriginalAttribute($attribute)) {
            $oldFile->delete();
        }
    }

    /**
     * @param UploadedFile $file
     * @param string $attribute
     * @return string|false
     */
    protected function uploadNow(UploadedFile $file, string $attribute)
    {
        $newFile = $this->getFileByAttribute($attribute);
        if (!$newFile) {
            throw new LogicException('Expected to receive a file object and got null');
        }

        return $newFile->upload($file);
    }

    protected function uploadDeferred(UploadedFile $file, string $attribute): string
    {
        $this->deferredUploads[$attribute] = $file;
        return $file->path();
    }

    public function deferFileUpload(File $file, string $attribute)
    {
        $this->deferredFiles[$attribute] = $file;
    }

    public function deferFileDelete(File $file)
    {
        $this->deferredDeletedFiles[] = $file;
    }
}

• File.php

<?php

namespace App\Storage;

use Carbon\Carbon;
use Illuminate\Contracts\Filesystem\Filesystem;
use Illuminate\Http\UploadedFile;
use Illuminate\Support\Facades\Storage;

class File
{
    private string $path;
    private Filesystem $filesystem;

    public function __construct(string $path, ?Filesystem $filesystem = null)
    {
        $this->path = $path;
        $this->filesystem = $filesystem ?? Storage::disk('cloud');
    }

    public function withPath(string $path): self
    {
        return new static($path, $this->filesystem);
    }

    public function withFilesystem(Filesystem $filesystem): self
    {
        return new static($this->path(), $filesystem);
    }

    public function getFilesystem(): Filesystem
    {
        return $this->filesystem;
    }

    /**
     * Get a resource to read the file.
     *
     * @return resource|null
     */
    public function readStream()
    {
        return $this->filesystem->readStream($this->path);
    }

    /**
     * Write the contents to the file.
     *
     * @param string|resource $contents
     * @param mixed $options
     * @return bool
     */
    public function put($contents, $options = []): bool
    {
        return $this->filesystem->put($this->path, $contents, $options);
    }

    public function exists(): bool
    {
        return $this->filesystem->exists($this->path);
    }

    public function delete(): bool
    {
        return $this->filesystem->delete($this->path);
    }

    public function path(): string
    {
        return $this->path;
    }

    public function prefix(): string
    {
        return dirname($this->path);
    }

    public function filename(): string
    {
        return basename($this->path);
    }

    public function extension(): string
    {
        return pathinfo($this->path(), PATHINFO_EXTENSION);
    }

    public function url(): string
    {
        return $this->filesystem->temporaryUrl($this->path(), Carbon::now()->addDay());
    }

    public function type(): ?string
    {
        return Helper::getFileType($this->path());
    }

    public function size(): int
    {
        return $this->filesystem->size($this->path());
    }

    public function downloadUrl(): string
    {
        return $this->filesystem->temporaryUrl($this->path(), Carbon::now()->addDay(), [
            'ResponseContentDisposition' => 'attachment',
        ]);
    }

    public function upload(UploadedFile $file): bool|string
    {
        return $this->filesystem->putFileAs($this->prefix(), $file, $this->filename());
    }

    public function __toString()
    {
        return $this->path();
    }
}

• AsFile.php

<?php

namespace App\General\Casts;

use App\Storage\File;
use Illuminate\Contracts\Database\Eloquent\Castable;
use Illuminate\Contracts\Database\Eloquent\CastsAttributes;
use Illuminate\Support\Facades\Storage;

class AsFile implements Castable
{
    /**
     * @inheritDoc
     */
    public static function castUsing(array $arguments)
    {
        return new class implements CastsAttributes {
            public function get($model, $key, $value, $attributes): ?File
            {
                if ($value) {
                    return new File($model->getFilePath($key, $value), Storage::disk($arguments[0] ?? 'cloud'));
                }

                return null;
            }

            public function set($model, $key, $value, $attributes): ?string
            {
                if (is_string($value) && filter_var($value, FILTER_VALIDATE_URL)) {
                    if ($urlParsed = parse_url($value)) {
                        return $urlParsed['path'];
                    }
                }

                if ($value instanceof File) {
                    return $value->filename();
                }

                return $value;
            }
        };
    }
}

• file_upload field

@php
    if(isset($field['value']) && $field['value'] instanceof \App\Storage\File) {
        $field['prefix'] = $field['value']->prefix() . '/';
        $field['disk'] = 'cloud';
        $field['temporary'] = 1440;
        $field['value'] = $field['value']->filename();
    }

    $field['type'] = 'upload';
    $field = $crud->makeSureFieldHasNecessaryAttributes($field);
@endphp

@include($crud->getFirstFieldView($field['type'], $field['view_namespace'] ?? false), $field)

• file column

@php
    $originalColumn = $column;

    $column['value'] = $column['value'] ?? data_get($entry, $column['name']);
    $file = $column['value'];

    if($file instanceof \App\Storage\File) {
        $column['value'] = !empty($column['icon']) ? ('<i class="' . $column['icon'] . '"></i>') : $file->filename();
        $column['wrapper'] = [
            'href' => fn() => $file->url(),
            'target' => '_blank',
            'class' => $column['class'] ?? '',
            'title' => $column['label']
        ];
        $column['limit'] = '64';
        $column['escaped'] = false;
    }
@endphp

@includeWhen(!empty($originalColumn['wrapper']), 'crud::columns.inc.wrapper_start', ['column' => $originalColumn])
    @include('crud::columns.text', ['column' => $column])
@includeWhen(!empty($originalColumn['wrapper']), 'crud::columns.inc.wrapper_end', ['column' => $originalColumn])

[pxpm]

This looks alot like uploaders https://backpackforlaravel.com/docs/6.x/crud-uploaders#about have you checked them ?

[susanu]

Hi @pxpm,

I know about uploaders, i also mentioned this in the post, please review the post one more time.
My implementation takes a slightly different approach.

• The path can be computed based on current entry (after the model was saved)
• Everything is controlled from one place
• The path is not saved into the database, just the filename
• The file is uploaded/removed after the model was successfully saved

Cheers!

[pxpm]

Hey @susanu sorry I probably missed that bit. I've read it better now.
I think our aim with uploaders was exactly what you said was missing, controlling everything from the same place, aka the controller where you already define your field/column. No need to add traits to your model or anything like that, we register the events on the model.

The path can be computed based on current entry (after the model was saved)

This was not contemplated in uploaders, probably we can add a FilePathGenerator similar to how we have the FileNameGenerator and this would be a non issue as long as we also register something like your defered solution.

The path is not saved into the database, just the filename

This is very dependent on the setup. Indeed, if you are only going to generate your path after the model is saved, it makes sense to not store it in the database, otherwise you would need two queries. But mostly I agree with this.

I like the idea of the casts you presented. We can use them to define the only things that makes sense to define in the model (the path and the disk), and we use it in our uploaders retrieve event. We miss the ability to define the path and disk in the uploader config, but it means you can generate paths after the model is saved and are able to link them back when working with the model.

This may be a good idea, depending on how we frame it. Thanks a lot for the example code 👍