Skip to content

Commit 0c3619e

Browse files
nagmat84kamil4ildyria
authored
Use filestreams (#1351)
* Started to refactor GdImageHandler * Added exceptions * Refactored interface * Refactored GD handler * Streams for all real image handlers * Adopted image handlers to file classes. * Fixed generic/automatic image handler. * A lot of work * Remove `checksum` from `Extractor` * Cleaned up MOVFormat * Refactored handling of Google Motion Pictures and finalized AddStandaloneStrategy. * Made collection of stream statistics optional on save. * Removed ImageHandler as a service. * Fixed size variant factory and interface * Use exception-free report in ImageHandler * Repaired exception handling in AddStandaloneStrategy * Fixed exception handling in RotateStrategy * Fixed exceptions and checksum in AddPhotoPartnerStrategy * Somehow unrelated refactoring of SizeVariantNamingStrategy * Follow up of unrelated fix of naming strategy. * Fixed AddVideoPartnerStrategy * New naming strategies for size variants * Switched to SizeVariantSharedPrefixRandomNamingStrategy * fix-to-squash: Fixes in AddStandaloneStrategy * Fixed unrelated error in test which suddenly fails * fix-to-squash: Repaired type errors und undefined variables in ImgagickHandler * Refactored tests * More tests * More tests * Improved tests * Remove dead code * More tests * Removed dead code * Update dependencies due to bug in Laravel * Fixed a lot of tests * More tests * Fixed test * Uses helper method `public_path` * Removed storage facade from test. * Cleaned up hard-coded paths * Make tests more pretty * Added test for rotated photo and broken motion photo * Added "Location" EXIF tag to sample * Refactored Photo Rotation Tests * Refactored upload-method * More tests for photo rotation. * Added Archive Tests * More tests for archive * Fixed SonarCloud issues * Fixed typo * Added test for import via download. * Bugfix for falsely skipped image optimization. * Added @ildyria's suggestion for a naming strategy * Moved repeated test code into traits * Refactored test for adding photos in preperation for testing GD handler. * Fixed GD handler * Added test for GD handler * Fix in GD image handler * Added different file types for test * Added rotation tests for GD handler * Added samples for all kind of auto-orientation. * Removed dead code * Removed absolute path from FlysystemFile. * Bug fix for naming strategies wrt. raw variants, part I. * Added negative test for unsupported raw. * Bug fix for naming strategies wrt. raw variants, part II. * Added positive tests for raw upload. * Quick fix to track down error which only occurs with Github workflows * Exchanged PDF with XCF as raw test case, because Githib workflows forbid PDF due to security concerns. * Revert "Quick fix to track down error which only occurs with Github workflows" This reverts commit 00e2c09. * Exchanged XCF with TIFF as raw test case, because Githib craches for XCF. * Test for import of supported/unsupported raw file via URL * Make SonarCloud happy about string constants. * Removed dead code * Split command tests and make command tests runnable (renamed to ...Test). * Fixed trivial bugs in commands * Make Sonar Cloud happy. * Make Sonar Cloud even more hapyy. * First positive test for command `lychee:generate_thumbs`. * Added test to re-create video thumbnail. * Fixed error when re-creating a subset of missing size variants * Test set creation time from file creation time. * More tests (for ghostbuster) * More tests (for ghostbuster), this time for real. * Nuked unused size variants * Fixed `makefile` as pointed out by @qwerty287 * Added stub for GD to make PHPStan happy. * Apply suggestions from code review Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Re-phrased comment as suggested by code review. * Remove stupid addition (sorry) * Update app/Assets/SizeVariantGroupedWithRandomSuffixNamingStrategy.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Update app/Assets/SizeVariantGroupedWithRandomSuffixNamingStrategy.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Update app/Actions/Photo/Strategies/AddStandaloneStrategy.php Co-authored-by: Matthias Nagel <matthias.h.nagel@posteo.de> * Update app/Actions/Photo/Strategies/RotateStrategy.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Update app/Assets/SizeVariantGroupedWithRandomSuffixNamingStrategy.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Update app/Image/GdHandler.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Apply suggestions from code review Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Fix comment. * Apply suggestions from code review Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Simplified code for compression quality * Fixed condition for creation of size variants * Updated comment on Quicktime container * Merge saveFrame and extractFrame into one and removed optimization. * Apply suggestions from code review Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Fixed subleties * Fixed file permissions on uploading * Fix efficiency regression for GD handler * Fixed base test class to avoid crash with chmod * Test for all size variants in simple upload * Avoid failing tests due to incorrect file owner * Efficiency boost for GD handler * Added "dry-run" to console command "fix-permissions" * better messages * fix phpstan with useless default * Update app/Console/Commands/FixPermissions.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Update app/Actions/Diagnostics/Checks/BasicPermissionCheck.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Update app/Image/GdHandler.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * fix commands (because yes, I tested them, they didn't work) * nice message if nothing to fix is required * fix phpstan * fix tests * Moved command from composer to migration * Added disclaimer to end of command `fix-permissions` * Make file permissions more configurable * Added config option to `.env` file * Update app/Actions/Diagnostics/Checks/BasicPermissionCheck.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Update .env.example Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Update app/Console/Commands/FixPermissions.php Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> * Made permission check more robust (no silent failure); no hard-coded disk name * Repaired broken umask * Fixed `neither ... nor` vs. `not ... or` * Fixed "neither ... nor" vs. "not ... or" * Added comment about bug for world-writeable directories * Fixed unused parameter. * bump to version 4.5.2 Co-authored-by: Kamil Iskra <kamil.01482@iskra.name> Co-authored-by: ildyria <beviguier@gmail.com> Co-authored-by: Benoît Viguier <ildyria@users.noreply.github.com>
1 parent 03b5062 commit 0c3619e

File tree

96 files changed

+5377
-2481
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

96 files changed

+5377
-2481
lines changed

.env.example

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,30 @@ DB_LOG_SQL=false
3535
# Don't use a timezone offset (like +01:00) or a timezone abbreviation (like CEST)
3636
# TIMEZONE=Europe/Paris
3737

38+
# Visibility of directories and (media) files in LYCHEE_UPLOADS
39+
# Possible values are:
40+
#
41+
# - private: world group has neither read nor write access
42+
# - public: world group has read access but no write access (the default)
43+
# - world: world group has read and write access
44+
#
45+
# The default should suffice for most installations.
46+
# For improved security, change this setting to "private".
47+
# Some rare setups may require directories and files to be world writeable.
48+
# In this case, use "world" here.
49+
# USE WITH PRECAUTIONS: world writeable files and folders may be a SECURITY RISK.
50+
# Note at the time of writing, the Flysystem package v1 has a bug which
51+
# prevents the "world" preset from working.
52+
# This is a temporary problem and will be solved after
53+
# https://github.com/thephpleague/flysystem/pull/1523 will have been merged
54+
# upstream or after Lychee will have migrated to Laravel 9.
55+
# Until then, if you need to use world-writable directories, please edit
56+
# `config/filesystems.php` and re-define the values for
57+
# `disks.images.permissions.(file|dir).public` such that they equal
58+
# `disks.images.permissions.(file|dir).world` and use the preset `public`
59+
# instead.
60+
# LYCHEE_IMAGE_VISIBILITY=public
61+
3862
# folders in which the files will be stored
3963
# LYCHEE_DIST="/var/www/html/Lychee-Laravel/public/dist/"
4064
# LYCHEE_UPLOADS="/var/www/html/Lychee-Laravel/public/uploads/"

.gitignore

Lines changed: 3 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -7,21 +7,9 @@ public/Lychee-front/node_modules/
77
public/Lychee-front/bower_components/
88
public/Lychee-front/package-lock.json
99

10-
public/uploads/big/*
11-
public/uploads/import/*
12-
public/uploads/medium/*
13-
public/uploads/raw/*
14-
public/uploads/small/*
15-
public/uploads/thumb/*
16-
public/uploads/tracks/*
17-
18-
!public/uploads/big/index.html
19-
!public/uploads/import/index.html
20-
!public/uploads/medium/index.html
21-
!public/uploads/raw/index.html
22-
!public/uploads/small/index.html
23-
!public/uploads/thumb/index.html
24-
!public/uploads/tracks/index.html
10+
public/dist/user.css
11+
12+
public/uploads/**
2513

2614
/storage/*.key
2715
/storage/clockwork/
@@ -36,7 +24,6 @@ yarn-error.log
3624
.env
3725
.env.*
3826

39-
public/dist/user.css
4027
aliases
4128

4229
Lychee/*

.phpstorm.meta.php

Lines changed: 95 additions & 94 deletions
Large diffs are not rendered by default.

app/Actions/Diagnostics/Checks/BasicPermissionCheck.php

Lines changed: 204 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,27 +3,105 @@
33
namespace App\Actions\Diagnostics\Checks;
44

55
use App\Contracts\DiagnosticCheckInterface;
6+
use App\Contracts\SizeVariantNamingStrategy;
7+
use App\Exceptions\Handler;
8+
use App\Exceptions\Internal\InvalidConfigOption;
69
use App\Facades\Helpers;
10+
use App\Models\SymLink;
11+
use Illuminate\Contracts\Container\BindingResolutionException;
12+
use Illuminate\Contracts\Filesystem\Filesystem;
713
use Illuminate\Support\Facades\Storage;
14+
use League\Flysystem\Adapter\Local as LocalFlysystem;
15+
use Psr\Container\ContainerExceptionInterface;
16+
use Psr\Container\NotFoundExceptionInterface;
17+
use function Safe\sprintf;
818

919
class BasicPermissionCheck implements DiagnosticCheckInterface
1020
{
21+
public const MAX_ISSUE_REPORTS_PER_TYPE = 5;
22+
23+
/**
24+
* @var int[] IDs of all (POSIX) groups to which the process belongs
25+
*/
26+
protected array $groupIDs;
27+
28+
/**
29+
* @var string Comma-separated list of names of (POSIX) groups to which the process belongs
30+
*/
31+
protected string $groupNames;
32+
33+
protected int $numOwnerIssues;
34+
35+
protected int $numPermissionIssues;
36+
37+
protected int $numAccessIssues;
38+
39+
/**
40+
* @param string[] $errors
41+
*
42+
* @return void
43+
*/
1144
public function check(array &$errors): void
1245
{
1346
$this->folders($errors);
1447
$this->userCSS($errors);
1548
}
1649

50+
/**
51+
* @param string[] $errors
52+
*
53+
* @return void
54+
*/
1755
public function folders(array &$errors): void
1856
{
19-
$paths = ['big', 'medium', 'small', 'thumb', 'import', ''];
57+
if (!extension_loaded('posix')) {
58+
return;
59+
}
2060

21-
foreach ($paths as $path) {
22-
$p = Storage::path($path);
23-
if (!Helpers::hasPermissions($p)) {
24-
$errors[] = "Error: '" . $p . "' is missing or has insufficient read/write privileges";
61+
clearstatcache(true);
62+
$this->numOwnerIssues = 0;
63+
$this->numPermissionIssues = 0;
64+
$this->numAccessIssues = 0;
65+
$groupIDsOrFalse = posix_getgroups();
66+
if ($groupIDsOrFalse === false) {
67+
$errors[] = 'Error: Could not determine groups of process';
68+
69+
return;
70+
}
71+
$this->groupIDs = $groupIDsOrFalse;
72+
$this->groupIDs[] = posix_getegid();
73+
$this->groupIDs[] = posix_getgid();
74+
$this->groupIDs = array_unique($this->groupIDs);
75+
$this->groupNames = implode(', ', array_map(
76+
function (int $gid): string {
77+
$groupNameOrFalse = posix_getgrgid($gid);
78+
79+
return $groupNameOrFalse === false ? '<unknown>' : $groupNameOrFalse['name'];
80+
},
81+
$this->groupIDs
82+
));
83+
84+
/** @var Filesystem[] $disks */
85+
$disks = [
86+
SizeVariantNamingStrategy::getImageDisk(),
87+
Storage::disk(SymLink::DISK_NAME),
88+
];
89+
90+
foreach ($disks as $disk) {
91+
if ($disk->getDriver()->getAdapter() instanceof LocalFlysystem) {
92+
$this->checkDirectoryPermissionsRecursively($disk->path(''), $errors);
2593
}
2694
}
95+
96+
if ($this->numOwnerIssues > self::MAX_ISSUE_REPORTS_PER_TYPE) {
97+
$errors[] = sprintf('Warning: %d more directories with wrong owner', $this->numOwnerIssues - self::MAX_ISSUE_REPORTS_PER_TYPE);
98+
}
99+
if ($this->numPermissionIssues > self::MAX_ISSUE_REPORTS_PER_TYPE) {
100+
$errors[] = sprintf('Warning: %d more directories with wrong permissions', $this->numPermissionIssues - self::MAX_ISSUE_REPORTS_PER_TYPE);
101+
}
102+
if ($this->numAccessIssues > self::MAX_ISSUE_REPORTS_PER_TYPE) {
103+
$errors[] = sprintf('Warning: %d more inaccessible directories', $this->numAccessIssues - self::MAX_ISSUE_REPORTS_PER_TYPE);
104+
}
27105
}
28106

29107
public function userCSS(array &$errors): void
@@ -37,4 +115,125 @@ public function userCSS(array &$errors): void
37115
}
38116
}
39117
}
118+
119+
/**
120+
* Check permissions of (local) image directories.
121+
*
122+
* For efficiency reasons only the directory permissions are checked,
123+
* not the permissions of every single file.
124+
*
125+
* @param string $path the path of the directory or file to check
126+
* @param string[] $errors the list of errors to append to
127+
* @noinspection PhpComposerExtensionStubsInspection
128+
*/
129+
private function checkDirectoryPermissionsRecursively(string $path, array &$errors): void
130+
{
131+
try {
132+
if (!is_dir($path)) {
133+
return;
134+
}
135+
136+
$actualPerm = fileperms($path);
137+
if ($actualPerm === false) {
138+
$errors[] = sprintf('Warning: Unable to determine permissions for %s' . PHP_EOL, $path);
139+
140+
return;
141+
}
142+
143+
// `fileperms` also returns the higher bits of the inode mode.
144+
// Hence, we must AND it with 07777 to only get what we are
145+
// interested in
146+
$actualPerm &= 07777;
147+
$owningGroupIdOrFalse = filegroup($path);
148+
$owningGroupNameOrFalse = $owningGroupIdOrFalse === false ? false : posix_getgrgid($owningGroupIdOrFalse);
149+
$owningGroupName = $owningGroupNameOrFalse === false ? '<unknown>' : $owningGroupNameOrFalse['name'];
150+
$expectedPerm = self::getConfiguredDirectoryPerm();
151+
152+
if (!in_array($owningGroupIdOrFalse, $this->groupIDs, true)) {
153+
$this->numOwnerIssues++;
154+
if ($this->numOwnerIssues <= self::MAX_ISSUE_REPORTS_PER_TYPE) {
155+
$errors[] = sprintf('Warning: %s is owned by group %s, but should be owned by one out of %s', $path, $owningGroupName, $this->groupNames);
156+
}
157+
}
158+
159+
if ($expectedPerm !== $actualPerm) {
160+
$this->numPermissionIssues++;
161+
if ($this->numPermissionIssues <= self::MAX_ISSUE_REPORTS_PER_TYPE) {
162+
$errors[] = sprintf(
163+
'Warning: %s has permissions %04o, but should have %04o',
164+
$path,
165+
$actualPerm,
166+
$expectedPerm
167+
);
168+
}
169+
}
170+
171+
if (!is_writable($path) || !is_readable($path)) {
172+
$this->numAccessIssues++;
173+
if ($this->numAccessIssues <= self::MAX_ISSUE_REPORTS_PER_TYPE) {
174+
$problem = match (true) {
175+
(!is_writable($path) && !is_readable($path)) => 'neither readable nor writable',
176+
!is_writable($path) => 'not writable',
177+
!is_readable($path) => 'not readable',
178+
default => ''
179+
};
180+
$errors[] = sprintf('Error: %s is %s by %s', $path, $problem, $this->groupNames);
181+
}
182+
}
183+
184+
$dir = new \DirectoryIterator($path);
185+
foreach ($dir as $dirEntry) {
186+
if ($dirEntry->isDir() && !$dirEntry->isDot()) {
187+
$this->checkDirectoryPermissionsRecursively($dirEntry->getPathname(), $errors);
188+
}
189+
}
190+
} catch (\Exception $e) {
191+
$errors[] = 'Error: ' . $e->getMessage();
192+
Handler::reportSafely($e);
193+
}
194+
}
195+
196+
/**
197+
* @throws InvalidConfigOption
198+
*/
199+
public static function getConfiguredDirectoryPerm(): int
200+
{
201+
return self::getConfiguredPerm('dir');
202+
}
203+
204+
/**
205+
* @throws InvalidConfigOption
206+
*/
207+
public static function getConfiguredFilePerm(): int
208+
{
209+
return self::getConfiguredPerm('file');
210+
}
211+
212+
/**
213+
* @param string $type either 'dir' or 'file'
214+
*
215+
* @return int
216+
*
217+
* @phpstan-param 'dir'|'file' $type
218+
*
219+
* @throws InvalidConfigOption
220+
*/
221+
private static function getConfiguredPerm(string $type): int
222+
{
223+
try {
224+
$visibility = (string) config(sprintf('filesystems.disks.%s.visibility', SizeVariantNamingStrategy::IMAGE_DISK_NAME));
225+
if ($visibility === '') {
226+
throw new InvalidConfigOption('File/directory visibility not configured');
227+
}
228+
229+
$perm = (int) config(sprintf('filesystems.disks.%s.permissions.%s.%s', SizeVariantNamingStrategy::IMAGE_DISK_NAME, $type, $visibility));
230+
if ($perm === 0) {
231+
throw new InvalidConfigOption('Configured file/directory permission is invalid');
232+
}
233+
234+
return $perm;
235+
} catch (ContainerExceptionInterface|BindingResolutionException|NotFoundExceptionInterface $e) {
236+
throw new InvalidConfigOption('Could not read configuration for file/directory permission', $e);
237+
}
238+
}
40239
}

app/Actions/Diagnostics/Errors.php

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ public function __construct(DiagnosticsChecksFactory $diagnosticsChecksFactory)
2020
*/
2121
public function get(): array
2222
{
23-
// Declare
23+
/** @var string[] $errors */
2424
$errors = [];
2525

2626
// @codeCoverageIgnoreStart

app/Actions/Import/Extensions/Checks.php

Lines changed: 0 additions & 32 deletions
This file was deleted.

app/Actions/Import/FromUrl.php

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,9 @@
22

33
namespace App\Actions\Import;
44

5-
use App\Actions\Import\Extensions\Checks;
65
use App\Actions\Photo\Create;
76
use App\Actions\Photo\Strategies\ImportMode;
87
use App\Exceptions\Handler;
9-
use App\Exceptions\InsufficientFilesystemPermissions;
108
use App\Exceptions\MassImportException;
119
use App\Image\DownloadedFile;
1210
use App\Image\MediaFile;
@@ -21,19 +19,6 @@
2119

2220
class FromUrl
2321
{
24-
use Checks;
25-
26-
/**
27-
* @throws InsufficientFilesystemPermissions
28-
*/
29-
public function __construct()
30-
{
31-
// TODO: Why do we explicitly perform this check here? We don't check the other import classes. We could just let the import fail.
32-
// Moreover, we do not even use the `import` folder which is checked by this method.
33-
// There is similar odd test in {@link \App\Actions\Photo\Create::add()} which uses another "check" trait.
34-
$this->checkPermissions();
35-
}
36-
3722
/**
3823
* Imports photos from a list of URLs.
3924
*

0 commit comments

Comments
 (0)