File uploading fails with current master #130

New Issue

matthijskooijman · 2019-07-25T23:14:46+01:00

matthijskooijman commented

2019-07-25 23:14:46 +01:00

It seems you recently made a change to involve some processing queue in the file uploads, but now I believe this means that the queue moves away the image file from its original /tmp location, which causes subsequent code to fail.

In terms of the code:

                $queuedFileName = $queueStorage->uploadToAnalysisQueue($photoFile, $queueUid);
                $uploadedTempFile = new File($photoFile);

I believe the file is now at $queuedFileName, but the File instance still uses $photoFile. Changing that to $queuedFileName seems obvious and lets the upload POST request actually return 200, but still shows an "upload failed" error message and a trace in the laravel log.

Here's the (what I think is the relevant part of) the backtrace of the original error:

[2019-07-25 22:58:34] production.ERROR: The file "/tmp/phpPiLGDS" does not exist {"userId":1,"email":"matthijs@stdin.nl","exception":"[object] (Symfony\\Compon
ent\\HttpFoundation\\File\\Exception\\FileNotFoundException(code: 0): The file \"/tmp/phpPiLGDS\" does not exist at /var/www/blue-twilight/vendor/symfony/http-foundation/File/File.php:43)                                                                                                                                   
[stacktrace]                                                                                                                                                  
#0 /var/www/blue-twilight/app/Http/Controllers/Admin/PhotoController.php(351): Symfony\\Component\\HttpFoundation\\File\\File->__construct(Object(Illuminate\\H
ttp\\UploadedFile))                                                                                                                                           
#1 [internal function]: App\\Http\\Controllers\\Admin\\PhotoController->store(Object(Illuminate\\Http\\Request))

When "fixing" the error as suggested above, it changes to:

[2019-07-25 23:02:14] production.ERROR: fopen(/var/www/blue-twilight/storage/app/albums/Test/admin-preview/a1dkomf7dacg8f4pfxlz.jpeg): failed to open stream: No such file or directory {"userId":1,"email":"matthijs@stdin.nl","exception":"[object] (ErrorException(code: 0): fopen(/var/www/blue-twilight/storage/app/album
s/Test/admin-preview/a1dkomf7dacg8f4pfxlz.jpeg): failed to open stream: No such file or directory at /var/www/blue-twilight/app/AlbumSources/LocalFilesystemSource.php:52)
[stacktrace]                                                                                                                                                  
#0 [internal function]: Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError(2, 'fopen(/var/www/...', '/var/www/blue-t...', 52, Array)            
#1 /var/www/blue-twilight/app/AlbumSources/LocalFilesystemSource.php(52): fopen('/var/www/blue-t...', 'r+')                                                   
#2 /var/www/blue-twilight/app/Http/Controllers/Gallery/PhotoController.php(74): App\\AlbumSources\\LocalFilesystemSource->fetchPhotoContent(Object(App\\Photo),
 'admin-preview')                                                                                                                                             
#3 [internal function]: App\\Http\\Controllers\\Gallery\\PhotoController->download(Object(Illuminate\\Http\\Request), 'Test', 'a1dkomf7dacg8f4...')

I have not investigated further, I guess I'll stick with a released version instead.

It seems you recently made a change to involve some processing queue in the file uploads, but now I believe this means that the queue moves away the image file from its original `/tmp` location, which causes subsequent code to fail. In terms of [the code](https://apps.andysh.uk/aheathershaw/blue-twilight/src/branch/master/app/Http/Controllers/Admin/PhotoController.php#L340-L341): $queuedFileName = $queueStorage->uploadToAnalysisQueue($photoFile, $queueUid); $uploadedTempFile = new File($photoFile); I believe the file is now at `$queuedFileName`, but the `File` instance still uses `$photoFile`. Changing that to `$queuedFileName` seems obvious and lets the upload POST request actually return 200, but still shows an "upload failed" error message and a trace in the laravel log. Here's the (what I think is the relevant part of) the backtrace of the original error: ``` [2019-07-25 22:58:34] production.ERROR: The file "/tmp/phpPiLGDS" does not exist {"userId":1,"email":"matthijs@stdin.nl","exception":"[object] (Symfony\\Compon ent\\HttpFoundation\\File\\Exception\\FileNotFoundException(code: 0): The file \"/tmp/phpPiLGDS\" does not exist at /var/www/blue-twilight/vendor/symfony/http-foundation/File/File.php:43) [stacktrace] #0 /var/www/blue-twilight/app/Http/Controllers/Admin/PhotoController.php(351): Symfony\\Component\\HttpFoundation\\File\\File->__construct(Object(Illuminate\\H ttp\\UploadedFile)) #1 [internal function]: App\\Http\\Controllers\\Admin\\PhotoController->store(Object(Illuminate\\Http\\Request)) ``` When "fixing" the error as suggested above, it changes to: ``` [2019-07-25 23:02:14] production.ERROR: fopen(/var/www/blue-twilight/storage/app/albums/Test/admin-preview/a1dkomf7dacg8f4pfxlz.jpeg): failed to open stream: No such file or directory {"userId":1,"email":"matthijs@stdin.nl","exception":"[object] (ErrorException(code: 0): fopen(/var/www/blue-twilight/storage/app/album s/Test/admin-preview/a1dkomf7dacg8f4pfxlz.jpeg): failed to open stream: No such file or directory at /var/www/blue-twilight/app/AlbumSources/LocalFilesystemSource.php:52) [stacktrace] #0 [internal function]: Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError(2, 'fopen(/var/www/...', '/var/www/blue-t...', 52, Array) #1 /var/www/blue-twilight/app/AlbumSources/LocalFilesystemSource.php(52): fopen('/var/www/blue-t...', 'r+') #2 /var/www/blue-twilight/app/Http/Controllers/Gallery/PhotoController.php(74): App\\AlbumSources\\LocalFilesystemSource->fetchPhotoContent(Object(App\\Photo), 'admin-preview') #3 [internal function]: App\\Http\\Controllers\\Gallery\\PhotoController->download(Object(Illuminate\\Http\\Request), 'Test', 'a1dkomf7dacg8f4...') ``` I have not investigated further, I guess I'll stick with a released version instead.

aheathershaw commented

2019-07-25 23:34:37 +01:00

Yeah this change is part of the upcoming 2.2.0 beta.

It allows for the analysis of uploaded photos to be shifted off to a separate server, using a Storage location as the temporary queue (which can then be accessed by both servers) instead of the previously hard-coded local file system.

The File instance still uses the $photoFile not the $queuedFilename as this is the temp file still on disk from the upload. In the case of a cloud storage queue (eg. Amazon S3) the $queuedFilename would be the path on a remotely-stored service where we couldn’t (as easily) get the file size and MIME type.

What I think is happening is that with the Amazon S3 driver, the image is COPIED from the uploaded temp folder to S3, so we still have the temp file to play with.

However the local storage driver (the default that should mirror current behaviour) MOVES the temp file into the analysis queue location, so it no longer exists to play with afterward.

This is the offending line.

This needs changing to be a copy, not a move. There doesn’t seem to be a Symfony or base PHP copy method on the File object so this will probably need to use the global “copy” function.

Yeah this change is part of the upcoming 2.2.0 beta. It allows for the analysis of uploaded photos to be shifted off to a separate server, using a Storage location as the temporary queue (which can then be accessed by both servers) instead of the previously hard-coded local file system. The File instance still uses the $photoFile not the $queuedFilename as this is the temp file still on disk from the upload. In the case of a cloud storage queue (eg. Amazon S3) the $queuedFilename would be the path on a remotely-stored service where we couldn’t (as easily) get the file size and MIME type. What I think is happening is that with the Amazon S3 driver, the image is COPIED from the uploaded temp folder to S3, so we still have the temp file to play with. However the local storage driver (the default that should mirror current behaviour) MOVES the temp file into the analysis queue location, so it no longer exists to play with afterward. This is [the offending line](https://apps.andysh.uk/aheathershaw/blue-twilight/src/branch/master/app/AlbumSources/LocalFilesystemSource.php#L180). This needs changing to be a copy, not a move. There doesn’t seem to be a Symfony or base PHP copy method on the File object so this will probably need to use the global “copy” function.

aheathershaw added the

bug

label 2019-07-25 23:35:50 +01:00

aheathershaw added this to the Version 2.2.0-beta.1 milestone 2019-07-25 23:35:52 +01:00

aheathershaw self-assigned this 2019-07-25 23:35:54 +01:00

matthijskooijman commented

2019-07-26 00:18:36 +01:00

This is the offending line.

Yeah, I found that. Your analysis seems correct, and if you've been testing with the S3 backend, that also explains why it didn't show up before :-)

> This is the offending line. Yeah, I found that. Your analysis seems correct, and if you've been testing with the S3 backend, that also explains why it didn't show up before :-)

aheathershaw referenced this issue from a commit

2019-07-27 13:50:33 +01:00

Fixes #130: Local file system queue copies the file instead of moving it, so the temp file is still available to analyse

aheathershaw closed this issue

2019-07-27 13:50:33 +01:00

Sign in to join this conversation.