Backblaze B2 storage driver #135

New Issue

aheathershaw · 2019-09-03T19:01:52+01:00

aheathershaw commented

2019-09-03 19:01:52 +01:00

Allow photos to be stored on the Backblaze B2 cloud.

https://www.backblaze.com/b2/docs/

Allow photos to be stored on the Backblaze B2 cloud. https://www.backblaze.com/b2/docs/

aheathershaw self-assigned this 2019-09-03 19:01:52 +01:00

aheathershaw added the

feature

label 2019-09-03 19:01:52 +01:00

aheathershaw added the

wip

label 2019-09-09 13:07:47 +01:00

aheathershaw commented

2019-09-09 18:24:47 +01:00

We should support both public and private buckets.

If set to public, generate a public URL for Backblaze.

If set to private, generate a 5-minute auth token using b2_get_download_authorization and use b2_download_file_by_name.

If set to auto-detect (the default), check if the bucket is public or private on the first call to the B2 API, and set the public/private flag accordingly.

We should support both public and private buckets. If set to public, generate a public URL for Backblaze. If set to private, generate a 5-minute auth token using [b2_get_download_authorization](https://www.backblaze.com/b2/docs/b2_get_download_authorization.html) and use [b2_download_file_by_name](https://www.backblaze.com/b2/docs/b2_download_file_by_name.html). If set to auto-detect (the default), check if the bucket is public or private on the first call to the B2 API, and set the public/private flag accordingly.

aheathershaw added this to the Version 2.2.0-beta.2 milestone 2019-09-09 18:30:27 +01:00

aheathershaw referenced this issue from a commit

2019-09-09 20:35:41 +01:00

Backblaze #135 - added the storage UI and initial connectivity to B2

aheathershaw referenced this issue from a commit

2019-09-09 21:52:28 +01:00

Backblaze #135 - introduced the config setting to choose whether to generate private or public URLs, or to auto-detect. Photos are now displayed from B2.

aheathershaw commented

2019-09-09 22:02:00 +01:00

B2’s delete method requires a file ID, whereas we only store the filename - the path is generated on-demand.

To avoid an additional call to B2 to delete a file (to get the file ID) we should cache the path to the file and it’s ID within the Blue Twilight DB - the result of the upload operation.

We can then just issue the delete calls with the cached ID when needed. Obviously the cached records needs deleting when the file is deleted.

For private buckets we may be able to generate URLs to get_download_file_by_id so visitors can’t guess at the original image URL.

B2’s delete method requires a file ID, whereas we only store the filename - the path is generated on-demand. To avoid an additional call to B2 to delete a file (to get the file ID) we should cache the path to the file and it’s ID within the Blue Twilight DB - the result of the upload operation. We can then just issue the delete calls with the cached ID when needed. Obviously the cached records needs deleting when the file is deleted. For private buckets we may be able to generate URLs to get_download_file_by_id so visitors can’t guess at the original image URL.

aheathershaw commented

2019-09-09 22:02:35 +01:00

Also need to test/consider what happens with the modification methods - rotate, flip and replace image - for example.

aheathershaw referenced this issue from a commit

2019-09-10 15:11:57 +01:00

Backblaze #135 - album storage driver is now cached to maintain state within the same request, prevents multiple calls to B2. Images can now be deleted and (I think) edited.

aheathershaw referenced this issue from a commit

2019-09-10 16:24:31 +01:00

Backblaze #135 - tried to implement b2_download_file_by_id for private buckets, but this doesn't work correctly, logged with Backblaze

aheathershaw commented

2019-09-10 22:52:56 +01:00

We should follow the logic outlined here so we can submit Blue Twilight as an integration to Backblaze.

https://www.backblaze.com/b2/docs/integration_checklist.html

We should follow the logic outlined here so we can submit Blue Twilight as an integration to Backblaze. https://www.backblaze.com/b2/docs/integration_checklist.html

aheathershaw referenced this issue from a commit

2019-09-11 14:59:30 +01:00

Backblaze #135 - implemented a retry and backoff period for 500/503 errors

aheathershaw referenced this issue from a commit

2019-09-14 10:04:15 +01:00

Backblaze #135 - implemented the re-use of the upload token/URL. Fetching file contents now works by using the b2_download_file_by_id method with an auth header.

aheathershaw commented

2019-09-14 10:09:56 +01:00

This is all working apart from file versions.

When deleting a file, we need to make sure we delete all versions.

There doesn't seem to be a way to retrieve all file versions - perhaps before we upload a changed file, we issue a delete request first.

This is all working apart from file versions. When deleting a file, we need to make sure we delete all versions. There doesn't seem to be a way to retrieve all file versions - perhaps before we upload a changed file, we issue a delete request first.

aheathershaw referenced this issue from a commit

2019-09-14 15:35:16 +01:00

Backblaze #135 - B2 storage source now removes the current file version before uploading a new one

aheathershaw removed the

wip

label 2019-10-19 21:04:31 +01:00

aheathershaw closed this issue

2019-10-19 21:04:35 +01:00

Sign in to join this conversation.