Add three new flags to bpy.data.file_path_foreach so render farms and
other scripts can find all relevant files for the Cycles texture cache.
- EXPAND_TOKENS: visit each UDIM tile path.
- EXPAND_SEQUENCES: visit each image and volume sequence file path.
- EXPAND_CACHES: visit cache files, like texture cache files
associated with images.
The previously reserved metadata field is now provided to callers,
with is_expanded and is_cache booleans to indicate the file kind.
Also is_readonly indicates if the path can be edited.
Ref #158964
Pull Request: https://projects.blender.org/blender/blender/pulls/159159
When assets have been downloaded from a remote asset library, these are
now checked against the library's listing. If their hash or file size
doesn't match, this is shown in the UI.
This uses the Disk File Hash Service (DFHS) from various threads for the
same asset library (and thus the same storage path for the DFHS SQLite
database). This proved troublesome, as SQLite connections are not meant
to be shared across threads. To avoid this sharing, DFHS instances are
now created on a per-thread basis, and each thread is responsible for
releasing its DFHS instance. The mass-release at loading another blend
file or quitting Blender is still in place.
Pull Request: https://projects.blender.org/blender/blender/pulls/158963
Fix a bug in the file locking mechanism that's used to ensure only a
single Blender instance syncs a remote asset library's listing.
The file locking in Windows actually locks a specific byte offset. This
offset MUST be the same in the unlock call. This was failing, because
downloading a file tends to write to it. Adding `the_file.seek(0)` to
both lock & unlock functions now ensures the same offset is (un)locked,
regardless of what the caller does with the files.
Pull Request: https://projects.blender.org/blender/blender/pulls/159063
The HTTP downloader first downloads to a temporary file, only renaming
it to the final name once the download is completed. If this rename
fails, the error message now includes the old/new path of the rename
operation. Previously only the cause was shown (like "permission
denied") but not.
Pull Request: https://projects.blender.org/blender/blender/pulls/159042
Show a progress for all current asset downloads combined in the
statusbar, with a button to cancel all pending downloads. The progress
is reported by the asset downloader.
A few bits were needed for this:
- Store a mapping of library URLs + potentially relative asset URL to
the absolute URL used by the downloader and its progress reporting.
- Store the `size_in_bytes` for asset files as non-optional field. It's
required as part of the asset listing schema, so absence should/will
result in an error.
- Keep track of in-progress and finished files in the `ProgressTracker`.
- New `asset_library_status_ping_asset_file_progress()` Python API
function for the downloader to use.
- The `asset_library_status_ping_asset_file_done()` Python API function
now requires the absolute asset URL as parameter.
- Progress UI changes.
- Timer to redraw progress UIs.
Co-authored-by: Sybren A. Stüvel <sybren@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/158666
The statusbar would keep showing the "Downloading Asset(s)" button, even
after all asset downloads finished.
After a recent change (I guess 80b3fc1080 ?), the status of downloaders
would only indicate that they are done downloading _after_ the
individual pings for finished asset file downloads were sent. That means
our idea of querying the status on this "finished asset file download"
ping to recognize when the downloaders are done wouldn't work anymore.
Instead, use the new `download queue empty` callback (added in
15abc945a2) of the downloader to send a dedicated ping to Blender. This
makes the previously needed function to query the downloader status from
C++ unnecessary, so it's removed here.
Semantically, the callback should also be called when all asset
downloads were cancelled. This wasn't the case and is fixed too here to
ensure the status bar button disappears in that case too.
Co-authored-by: Sybren A. Stüvel <sybren@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/158773
Add two Blender versions to each asset in the online asset listing:
`min` and `until`. The asset is shown in Blender if `min <=
BLENDER_VERSION < until`.
This includes changes to the schema, the listing generator, and
Blender itself to do the filtering.
This is intended for two things:
- Ensuring older Blenders don't see assets that are made with a newer
version of Blender.
- Making it possible to replace assets, where older Blender versions
still see the older version of the asset, and newer versions of
Blender only see the newer asset.
Example listing JSON:
```json
{
"name": "Mrežnica",
"id_type": "WORLD",
"files": ["worlds.blend"],
"bl_versions": {
"min": "5.2"
},
"thumbnail": {
"url": "worlds_thumbnails/World/mreznica.webp",
"hash": "SHA256:9785a7e242dbd8ff973736c51fb09a6a1efe401dfff62d38f38d9fb48b7f6e46"
},
"meta": {
"catalog_id": "efd4ea51-c968-44a6-9781-09c20796e8a6",
"tags": ["River", "Croatia", "HDRi"],
"author": "Sybren A. Stüvel",
"description": "HDRi of river Mrežnica in Croatia"
}
},
```
The listing generator writes the version of Blender that was used to
save the blend file.
Once an asset is downloaded, it will always be shown, regardless of
the version of Blender used. That may be changed in a future PR, but
such filtering is out of scope for this one.
Note: after landing this, all remote asset listings need to be
regenerated. Without a `bl_versions.max` field, the asset will not be
shown in Blender.
Pull Request: https://projects.blender.org/blender/blender/pulls/158768
This makes it possible for asset authors to set a "preferred" import
method (link, append, packing) for individual assets.
This is important for base meshes in the essentials asset library:
Currently they would pack by default, meaning they are not editable.
Users would have to go through additional steps to make the base meshes
editable. Such "base" assets should just work well without additional
steps, especially ones from the essentials library.
Adds an _Import_ panel to the asset browser sidebar. It will only be
shown if the active asset is either in the current file, or preferred
import method was set by the asset author.
In the _Import Settings_ popover of the asset browser, the
_Follow Preferences_ import method was renamed to
_Follow Asset or Preferences_.
Co-authored-by: Sybren A. Stüvel <sybren@blender.org>
Pull Request: https://projects.blender.org/blender/blender/pulls/158486
When downloading a remote asset library's listing, make a backup of
the already-downloaded files. If the download gets aborted for some
reason, the backup can be put back, and the local listing is at least
correct (if possibly outdated).
Pull Request: https://projects.blender.org/blender/blender/pulls/158659
Fix a bug where the relative path to a file was used instead of the
absolute path. This now makes it possible to run the listing generator
on other directories than the current working directory.
Pull Request: https://projects.blender.org/blender/blender/pulls/158771
Rename `scripts/modules/_bpy_internal/assets/remote_library_listing/`
to `.../remote_library/`. The Python module started out as code for
the remote listing, but now has all the asset-related code (so also
downloading the assets themselves).
No functional changes.
Pull Request: https://projects.blender.org/blender/blender/pulls/158651
While downloading, the statusbar (or any other UI showing progress, like
the topbar when the statusbar is collapsed) will show a "Downloading
Asset(s)" button with a cancel icon. Clicking it will cancel all
asset downloads.
This lays some foundations for progress reporting of assets.
Pull Request: https://projects.blender.org/blender/blender/pulls/158556
Add a callback function to the asset downloader that gets called
whenever that downloader's queue is emptied. The implementation ensures
that the status of the downloader is set before this callback is called,
so that when it's used for pinging Blender's C++ code, that code sees
the correct status.
Pull Request: https://projects.blender.org/blender/blender/pulls/158638
The asset listing generator now sorts the assets, by file, then ID type,
then asset name. Sorting by file places related assets together, and
minimises the repeats of the same file across multiple listing pages.
The other sorting is just there to ensure stability of the generated
listing, so that diffs between generator runs are clean.
Pull Request: https://projects.blender.org/blender/blender/pulls/158637
Replace the 'on done' callback function (which would be called both on
error and success) with a DownloadReporter protocol implementation, so
that richer information is known for passing to the C++ code.
This commit just refactors the code to be able to do so.
Pull Request: https://projects.blender.org/blender/blender/pulls/158605
Download progress is now reported as `DownloadProgress` object, which
contains:
- The number of bytes streamed so far,
- the total number of bytes expected to be streamed (if the HTTP
response included a `content-length` header, otherwise `None`).
- the number of bytes written to disk so far.
This also adds support for servers that did not send a `content-length`
header. In that case it's not possible for the downloader to compute a
percentage, but the user (in our case, the asset system) may know the
expected on-disk size of the asset, and with the 'number of bytes
written to disk so far' number, can still calculate this.
Related: !157644
Co-authored by: Jakub Ružička <yac@noreply.localhost>
Pull Request: https://projects.blender.org/blender/blender/pulls/158340
When the same request is queued multiple times, it's only downloaded
once. All registered 'on done' callbacks are still called once the
download has completed.
Earlier the 'on done' callbacks were deduplicated, but the downloads
were not. For webserver that support conditional downloading this was
not a practical issue (they'd respond `304 Not Modified` on all but the
first request), but the approach in this commit is much cleaner.
Pull Request: https://projects.blender.org/blender/blender/pulls/158601
Add a Python function to cancel all remote asset listing downloads.
Calling this function will leave the on-disk listings in an unusable
state. This isn't user friendly, so in the future it would be nice to
store a backup of the listing's files, and restore those on
cancellation. That is for a future PR, though, as the current limitation
already exists when quitting Blender while the listing is being
downloaded.
Pull Request: https://projects.blender.org/blender/blender/pulls/158495
Take the 'description' fields from the OpenAPI schema definition file,
and use those to generate docstrings for the dataclasses.
This adds another dependency to the generator's virtualenv:
docformatter. The generator already runs autopep8 via Blender's `make
format` command, but that doesn't rewrap long docstrings. That's what
docformatter now does.
No functional changes, just added developer comfort.
Pull Request: https://projects.blender.org/blender/blender/pulls/158598
This commit removes code related to making bug reports to the
blender/blender-addons repository.
The Blender addons repository was archived ~10 months ago, so users
can't make bug reports to it.
And the buttons to make bug reports to the Blender addons repository
was removed in Blender 4.2 with the switch to the Blender extensions
platform.
As such, all the code related to handling bug reports for addons is
unused and non-functioning. So it can be removed.
Pull Request: https://projects.blender.org/blender/blender/pulls/158429
Add a Python function to cancel all pending asset downloads. This is
to support the soon-to-be-implemented downloader progress view in
Blender's status bar, which will have a single 'cancel' button for all
asset downloads combined.
Pull Request: https://projects.blender.org/blender/blender/pulls/158482
Fix an issue running the background downloader when the Python entry
point was a text datablock in the text editor.
The issue is that Python's multiprocessing module expects
`sys.modules['__main__'].__file__` to be a valid file IF it exists.
It's now checked to see if it actually exists on disk as a file, and
erased if it's not. This fixes the issue, because that triggers a
special case in the multiprocessing module.
Pull Request: https://projects.blender.org/blender/blender/pulls/158284
Currently the stub generation mostly serves to expose gaps/errors in
Python's documentation, although it should be reasonably complete too.
While it is possible to use the generated stubs with mypy,
most scripts generate many errors, mainly because of `bpy.props.*`
annotations which aren't recognized by mypy.
- Generate mypy-compatible `.pyi` stubs from RST (`sphinx_stub_gen.py`).
- Validate stubs via syntax + import check (`sphinx_stub_validate.py`).
- Add `python_stubs` make target.
- Include dunder methods (`__*__` methods) in RST doc-strings,
these used for the stub files, but also available in generated docs.
- Use foldable `.. details::` so as not to add unnecessary noise
to the HTML output.
- Use literal default reprs for Matrix defaults so signatures can be
evaluated.
Ref !156260
When HTTP stream compression was used, it would break conditional HTTP
downloading, because the streamed (so compressed) number of bytes of the
request was compared with the on-disk (so decompressed) size of the
downloaded file.
This is now fixed, and some other size-related fields have been renamed
to make it explicit that they are about the on-disk size.
Since this changes how the HTTP request metadata is stored on disk,
exising entries never match the on-disk file, and so every conditional
download will just re-download the file. This can't be helped, because
it's unknown whether the original request used stream compression or
not.
Pull Request: https://projects.blender.org/blender/blender/pulls/158334
Introduce download cancellation functions, for the HTTP downloader in
general, and for asset downloading.
Thumbnails and the downloading of asset library listings cannot be
cancelled. These are expected to be relatively small anyway.
The design:
- The HTTP downloader returns the exact URL that's being downloaded.
This value is returned by the various abstraction layers in between.
This ensures that the correct URL is known, regardless of the
adjustments made by these abstraction layers (turning relative URLs
into an absolute one, adding the expected file hash, etc.)
- The downloading URL is used to identify which download should be
cancelled.
- When the same URL is queued for downloading multiple times, all of
them are cancelled by a single call to the cancel function.
- Cancelling a URL that is not known to the downloader (because it was
never requested, or because it's already fully downloaded) is a
valid operation. It's simply a no-op.
Relevant for the calls from C++:
- `download_asset_file()` now returns the URL that was queued.
- `cancel_download()` takes the URL of the asset library (to identify
the downloader instance) and the URL returned by the function above.
- Cancellation will be reported as 'download error' with
`http_dl.DownloadCancelled` as exception type.
Pull Request: https://projects.blender.org/blender/blender/pulls/158264
Pull Request: https://projects.blender.org/blender/blender/pulls/158264
In the messaging protocol between Blender and the HTTP downloader's
background process, rename 'CANCEL' to 'SHUTDOWN', as that message is
intended to shut down the entire background process.
This makes space for using 'CANCEL' for cancellation of individual
downloads.
Also I snuck in two tiny non-functional changes that are too small to
warrant their own commit.
No functional changes.
https://projects.blender.org/blender/blender/pulls/158264
The online listing schema uses `IDP_BOOLEAN` as type for boolean
properties, but the Blender custom property deserializer expects it to
be `IDP_BOOL`.
Online brush assets would not show in the asset shelf because of this
issue.
Pull Request: https://projects.blender.org/blender/blender/pulls/157793
Introduced with c946d9355b
The 'Smooth' and 'Erase' options for stroke mode was migrated to a
different operator property. This commit ensures that any custom keymaps
with these commonly used paint operators are now functional upon
reimporting to 5.2
Pull Request: https://projects.blender.org/blender/blender/pulls/157207
Searching by "Key-Binding" did not return results for key bindings
that use double-click events.
This happened because the search parser does not recognize
event value prefixes such as "dbl", "press", "release",
or "drag". Tokens like "dbl-A" are treated as a single string
and therefore never match the underlying keymap item value
(e.g. KM_DBL_CLICK).
Added parsing for common event value prefixes and map them
to the corresponding keymap item values during search processing.
This allows queries such as: dbl-A, drag-leftmouse,
press-A to correctly match key bindings using those event values.
Also support:
- Searching with translated modifier / event names.
- Partial prefix matching is also supported (e.g. "db-A" -> "dbl-A").
Ref !155291
The online asset library listing generator crashed with
FileNotFoundError when a data-block had a preview attached but the
preview's pixel data was empty (image_size == (0, 0)). In that case
_save_thumbnail() silently returns without writing the .webp file, but
_find_assets() then unconditionally hashed the non-existent path.
Guard the URLWithHash construction on the thumbnail file actually
existing on disk. Assets without a usable thumbnail are now reported
with thumbnail=None instead of aborting the entire run. This also
covers the case where a thumbnail was deleted externally between
generator runs (should_write_thumbnails=False path).
Pull Request: https://projects.blender.org/blender/blender/pulls/157466
The Asset Catalog definition file is colon-separated, so colons should
not appear in any of the fields themselves. Rather than adding elaborate
escaping rules, just don't write those characters.
This is just implemented for online asset listings, as that's considered
3rd-party data, and thus needs more sanitising before writing to a local
file.
Pull Request: https://projects.blender.org/blender/blender/pulls/157439
The remote library listing already included custom properties. However,
the format they used didn't match what the C++ indexer expected at all.
This means the custom properties would essentially get lost. Sometimes
they contain important information though, for example, the type of node
tree so the compositor asset shelf can display only compositing assets.
The C++ indexer expected the same format as for the local asset index.
This avoids duplicating the reading code. This commit makes the listing
match that.
Another issue is, that a top level custom property should always be a
group property (this is the one iterated within when doing
`bpy.types.AssetData.items(), like the listing generator does).
Otherwise adding more properties isn't supported, and property lookups
assumed this. We can handle that by letting the indexer create the top
level group if needed, that way it can be implicit in the remote library
listing.
Pull Request: https://projects.blender.org/blender/blender/pulls/156305
Expose grease pencil `fill_id` and `hide_stroke` attributes to high
level python API. This allows to easily get or set those attributes
using the stroke object itself in scripts or interactive console.
#### Usage:
For the example, let's store a drawing (first frame on first layer of
an active GP object):
`drawing = bpy.context.object.data.layers[0].frames[0].drawing`
Using low level API:
`drawing.attributes['fill_id'].data[0].value = 1`
`drawing.attributes['hide_stroke'].data[0].value = True`
Now also possible directly on stroke object with high level API:
`drawing.strokes[0].fill_id = 1`
`drawing.strokes[0].hide_stroke = True`
Pull Request: https://projects.blender.org/blender/blender/pulls/156246
