feat(Blenvy:Blender): fixes & enhancements to auto export, particularly assets

* injected BlueprintAssets are now reusing existing asset scan boilerplate (wip)
 * added back per blueprint assets
 * experimenting with how to export all vs local only assets
 * renamed (finally) export main scenes
 * found issue with scene serialization (hellooo collections !), working on fix

tools/blenvy/TODO.md

@@ -185,7 +185,37 @@ Blender side:
 
 - [x] BLENVY_OT_item_select is missing handling for the other types (outside of object & collection)
     - [x] fix selection logic 
+- [x] update testing blend files
+- [x] disable 'export_hierarchy_full_collections' for all cases: not reliable and redudant
+- [ ] fix systematic material exports despite no changes
+- [ ] investigate lack of detection of changes of adding/changing components
+    - [ ] change scene serialization to account for collections ...sigh
+- [ ] also remove ____dummy____.bin when export format is gltf
 
+- [ ] fix/cleanup asset information injection (also needed for hot reload)
+    - [ ] add back per blueprint assets
+    - [ ] reuse the already existing asset_scan + export thing
+    - thoughts: 
+       - the "list of all assets" is actually the "fake"/generated one: nobody would write a list of assets for sub assets,
+    you would just add the assets to your blueprint
+       - in Bevy at spawning we have
+               blueprint => assets
+            for hot reload we need
+               asset => blueprint instances so we can despawn/respawn etc blueprint instances when one of their assets has changed 
+
+               problem of untyped vs typed
+               perhaps have a mapping of untyped => typed id
+               map asset id => [entity ids]
+
+- [ ] add option to 'split out' meshes from blueprints ? 
+    - [ ] ie considering meshletts etc , it would make sense to keep blueprints seperate from purely mesh gltfs
+- [ ] persist exported materials path in blueprints so that it can be read from library file users
+    - [ ] just like "export_path" write it into each blueprint's collection
+    - [ ] scan for used materials per blueprint !
+    - [ ] for scenes, scan for used materials of all non instance objects (TODO: what about overrides ?)
+
+- [ ] add a way of visualizing per blueprint instances ?
+- [ ] display export path of blueprints (mostly external) ?
 - [ ] hidden objects/collections only semi respected at export
     - this is because blueprints are external ?
     - [ ] verify based on gltf settings
@@ -199,20 +229,8 @@ Blender side:
     - [ ] disabled components 
     - [ ] blueprint instances as children of blueprint instances
     - [ ] blueprint instances as children of empties
-- [x] update testing blend files
-- [ ] disable 'export_hierarchy_full_collections' for all cases: not reliable and redudant
 
 
-- [ ] add option to 'split out' meshes from blueprints ? 
-    - [ ] ie considering meshletts etc , it would make sense to keep blueprints seperate from purely mesh gltfs
-- [ ] persist exported materials path in blueprints so that it can be read from library file users
-    - [ ] just like "export_path" write it into each blueprint's collection
-    - [ ] scan for used materials per blueprint !
-    - [ ] for scenes, scan for used materials of all non instance objects (TODO: what about overrides ?)
-
-- [ ] add a way of visualizing per blueprint instances ?
-- [ ] display export path of blueprints (mostly external) ?
-
 Bevy Side:
 - [x] deprecate BlueprintName & BlueprintPath & use BlueprintInfo instead
 - [x] make blueprint instances invisible until spawning is done to avoid "spawn flash"?
@@ -237,17 +255,26 @@ Bevy Side:
     - [x] blueprint level/ collection level components are now visible in instances in Blender
     - [x] they do not seem to be transfered to the (instance) entity above:
         could they be on the "empty node" ? 
+- [ ] add back & cleanup animation frame triggers
 
 - [ ] simplify testing example:
     - [x] remove use of rapier physics (or even the whole common boilerplate ?)
     - [ ] remove/replace bevy editor pls with some native ui to display hierarchies
     - [ ] a full fledged demo (including physics & co)
     - [ ] other examples without interactions or physics 
+
 - [ ] add hot reloading
     - [x] basics
     - [x] make it enabled/disabled based on general flag
-    - [ ] make 
-    - [ ] cleanup internals
+    - [x] account for changes impact both parent & children (ie "world" and "blueprint3") for example, which leads to a crash as there is double despawn /respawn so we need to filter things out 
+    - [x] if there are many assets/blueprints that have changed at the same time, it causes issues similar to the above, so apply a similar fix
+        - [x] also ignore any entities currently spawning (better to loose some information, than cause a crash)
+    - [ ] something is off with blueprint level components
+    - [ ] add the root blueprint itself to the assets either on the blender side or on the bevy side programatically
+    - [x] for sub blueprint tracking: do not propagate/ deal with parent blueprints if they are not themselves Spawning (ie filter out by "BlueprintSpawning")
+    - [ ] invalidate despawned entity & parent entities AABB 
+    - [x] cleanup internals
+
 
 - [x] review & change general component insertion & spawning ordering & logic
     - GltfComponentsSet::Injection => GltfBlueprintsSet::Spawn => GltfBlueprintsSet::AfterSpawn

tools/blenvy/add_ons/auto_export/blueprints/export_blueprints.py

@@ -5,41 +5,28 @@ from blenvy.assets.generate_asset_file import write_ron_assets_file
 from ..constants import TEMPSCENE_PREFIX
 from ..common.generate_temporary_scene_and_export import generate_temporary_scene_and_export, copy_hollowed_collection_into, clear_hollow_scene
 from ..common.export_gltf import generate_gltf_export_settings
-
-def assets_to_fake_ron(list_like):
-    result = []
-    for item in list_like:
-        result.append(f"(name: \"{item['name']}\", path: \"{item['path']}\")")
-
-    return f"(assets: {result})".replace("'", '')
-
-    return f"({result})".replace("'", '')
-        
+from ..utils import upsert_blueprint_assets
 
 def export_blueprints(blueprints, settings, blueprints_data):
     blueprints_path_full = getattr(settings, "blueprints_path_full")
     gltf_export_settings = generate_gltf_export_settings(settings)
-    
+    export_materials_library = getattr(settings.auto_export, "export_materials_library")
+
     try:
         # save current active collection
         active_collection =  bpy.context.view_layer.active_layer_collection
-        export_materials_library = getattr(settings.auto_export, "export_materials_library")
 
         for blueprint in blueprints:
             print("exporting collection", blueprint.name)
             gltf_output_path = os.path.join(blueprints_path_full, blueprint.name) # TODO: reuse the export_path custom property ?
             gltf_export_settings = { **gltf_export_settings, 'use_active_scene': True, 'use_active_collection': True, 'use_active_collection_with_nested':True}
             
+            collection = bpy.data.collections[blueprint.name]
             # if we are using the material library option, do not export materials, use placeholder instead
             if export_materials_library:
-                gltf_export_settings['export_materials'] = 'PLACEHOLDER'
-
-            collection = bpy.data.collections[blueprint.name]
-
-            all_assets = []
-            auto_assets = []
-            collection["BlueprintAssets"] = assets_to_fake_ron([]) #assets_to_fake_ron([{"name": asset.name, "path": asset.path} for asset in collection.user_assets] + auto_assets) #all_assets + [{"name": asset.name, "path": asset.path} for asset in collection.user_assets] + auto_assets)
+                gltf_export_settings['export_materials'] = 'PLACEHOLDER'  
 
+            upsert_blueprint_assets(blueprint, blueprints_data=blueprints_data, settings=settings)
 
             # do the actual export
             generate_temporary_scene_and_export(

tools/blenvy/add_ons/auto_export/common/auto_export.py

@@ -10,7 +10,7 @@ from ..blueprints.get_blueprints_to_export import get_blueprints_to_export
 from ..levels.get_levels_to_export import get_levels_to_export
 from .export_gltf import get_standard_exporter_settings
 
-from ..levels.export_main_scenes import export_main_scene
+from ..levels.export_levels import export_main_scene
 from ..blueprints.export_blueprints import export_blueprints
 from .export_materials import cleanup_materials, export_materials
 from ..levels.bevy_scene_components import remove_scene_components, upsert_scene_components
@@ -56,6 +56,7 @@ def auto_export(changes_per_scene, changed_export_parameters, settings):
         #inject/ update light shadow information
         for light in bpy.data.lights:
             enabled = 'true' if light.use_shadow else 'false'
+            # TODO: directly set relevant components instead ?
             light['BlenderLightShadows'] = f"(enabled: {enabled}, buffer_bias: {light.shadow_buffer_bias})"
 
         # export
@@ -69,7 +70,8 @@ def auto_export(changes_per_scene, changed_export_parameters, settings):
 
             # since materials export adds components we need to call this before blueprints are exported
             # export materials & inject materials components into relevant objects
-            if export_materials_library:
+            # FIXME: improve change detection, perhaps even add "material changes"
+            if export_materials_library and (changed_export_parameters or len(changes_per_scene.keys()) > 0 ):
                 export_materials(blueprints_data.blueprint_names, settings.library_scenes, settings)
 
             # update the list of tracked exports

tools/blenvy/add_ons/auto_export/common/serialize_scene.py

@@ -235,7 +235,9 @@ def custom_properties_hash(obj):
     custom_properties = {}
     for property_name in obj.keys():
         if property_name not in '_RNA_UI' and property_name != 'components_meta':
+            print("custom properties stuff for", obj, property_name)
             custom_properties[property_name] = obj[property_name]
+    print("custom props for hashing", custom_properties, str(h1_hash(str(custom_properties))) )
     return str(h1_hash(str(custom_properties)))
 
 def camera_hash(obj):
@@ -316,7 +318,7 @@ def modifiers_hash(object, settings):
 
 def serialize_scene(settings): 
     cache = {"materials":{}}
-    print("serializing scene")
+    print("serializing scenes")
     data = {}
 
 
@@ -325,6 +327,7 @@ def serialize_scene(settings):
 
     # TODO: only go through scenes actually in our list
     for scene in bpy.data.scenes:
+        print("scene", scene.name)
         # ignore temporary scenes
         if scene.name.startswith(TEMPSCENE_PREFIX):
             continue

tools/blenvy/add_ons/auto_export/levels/export_levels.py

@@ -9,16 +9,8 @@ from ..constants import TEMPSCENE_PREFIX
 from ..common.generate_temporary_scene_and_export import generate_temporary_scene_and_export, copy_hollowed_collection_into, clear_hollow_scene
 from ..common.export_gltf import (generate_gltf_export_settings, export_gltf)
 from .is_object_dynamic import is_object_dynamic, is_object_static
+from ..utils import upsert_scene_assets
 
-def assets_to_fake_ron(list_like):
-    result = []
-    for item in list_like:
-        result.append(f"(name: \"{item['name']}\", path: \"{item['path']}\")")
-
-    return f"(assets: {result})".replace("'", '')
-
-    return f"({result})".replace("'", '')
-        
 
 def export_main_scene(scene, settings, blueprints_data): 
     gltf_export_settings = generate_gltf_export_settings(settings)
@@ -41,53 +33,7 @@ def export_main_scene(scene, settings, blueprints_data):
         gltf_output_path = os.path.join(levels_path_full, scene.name)
 
         inject_blueprints_list_into_main_scene(scene, blueprints_data, settings)
-        """print("main scene", scene)
-        for asset in scene.user_assets:
-            print("  user asset", asset.name, asset.path)
-        for asset in scene.generated_assets:
-            print("  generated asset", asset)"""
-        """for blueprint in blueprints_data.blueprints_per_scenes[scene.name]:
-            print("BLUEPRINT", blueprint)"""
-        blueprint_instances_in_scene = blueprints_data.blueprint_instances_per_main_scene.get(scene.name, {}).keys()
-        blueprints_in_scene = [blueprints_data.blueprints_per_name[blueprint_name] for blueprint_name in blueprint_instances_in_scene]
-        #yala = [blueprint.collection.user_assets for blueprint in blueprints_in_scene]
-        #print("dsfsdf", yala)
-        auto_assets = []
-
-        all_assets = []
-        export_gltf_extension = getattr(settings, "export_gltf_extension", ".glb")
-
-        blueprints_path =  getattr(settings, "blueprints_path")
-        for blueprint in blueprints_in_scene:
-            if blueprint.local:
-                blueprint_exported_path = os.path.join(blueprints_path, f"{blueprint.name}{export_gltf_extension}")
-            else:
-                # get the injected path of the external blueprints
-                blueprint_exported_path = blueprint.collection['export_path'] if 'export_path' in blueprint.collection else None
-                # add their material path
-                materials_exported_path = blueprint.collection['materials_path'] if 'materials_path' in blueprint.collection else None
-                auto_assets.append({"name": blueprint.name+"_material", "path": materials_exported_path})#, "generated": True, "internal":blueprint.local, "parent": None})
-
-
-            if blueprint_exported_path is not None: # and not does_asset_exist(assets_list, blueprint_exported_path):
-                auto_assets.append({"name": blueprint.name, "path": blueprint_exported_path})#, "generated": True, "internal":blueprint.local, "parent": None})
-
-            # now also add the assets of the blueprints # TODO: wait no , these should not be a part of the (scene) local assets
-            for asset in blueprint.collection.user_assets:
-                #print("adding assets of blueprint", asset.name)
-                all_assets.append({"name": asset.name, "path": asset.path})
-
-        """for asset in auto_assets:
-            print("  generated asset", asset.name, asset.path)"""
-        
-        materials_path =  getattr(settings, "materials_path")
-        current_project_name = Path(bpy.context.blend_data.filepath).stem
-        materials_library_name = f"{current_project_name}_materials"
-        materials_exported_path = os.path.join(materials_path, f"{materials_library_name}{export_gltf_extension}")
-        material_assets = [{"name": materials_library_name, "path": materials_exported_path}] # we also add the material library as an asset
-        print("material_assets", material_assets, "extension", export_gltf_extension)
-        scene["BlueprintAssets"] = assets_to_fake_ron(all_assets + [{"name": asset.name, "path": asset.path} for asset in scene.user_assets] + auto_assets + material_assets)
-        #scene["BlueprintAssets"] = assets_to_fake_ron([{'name':'foo', 'path':'bar'}])
+        upsert_scene_assets(scene, blueprints_data=blueprints_data, settings=settings)
 
         if export_separate_dynamic_and_static_objects:
             #print("SPLIT STATIC AND DYNAMIC")

tools/blenvy/add_ons/auto_export/utils.py

@@ -0,0 +1,84 @@
+import bpy
+import os
+from pathlib import Path
+from blenvy.assets.assets_scan import get_blueprint_asset_tree, get_main_scene_assets_tree2
+
+def assets_to_fake_ron(list_like):
+    result = []
+    for item in list_like:
+        result.append(f"(name: \"{item['name']}\", path: \"{item['path']}\")")
+
+    return f"(assets: {result})".replace("'", '')
+
+    return f"({result})".replace("'", '')
+
+
+# TODO : move to assets
+def upsert_scene_assets(scene, blueprints_data, settings):
+    """print("main scene", scene)
+    for asset in scene.user_assets:
+        print("  user asset", asset.name, asset.path)
+    for asset in scene.generated_assets:
+        print("  generated asset", asset)"""
+    """for blueprint in blueprints_data.blueprints_per_scenes[scene.name]:
+        print("BLUEPRINT", blueprint)"""
+    blueprint_instances_in_scene = blueprints_data.blueprint_instances_per_main_scene.get(scene.name, {}).keys()
+    blueprints_in_scene = [blueprints_data.blueprints_per_name[blueprint_name] for blueprint_name in blueprint_instances_in_scene]
+    #yala = [blueprint.collection.user_assets for blueprint in blueprints_in_scene]
+    #print("dsfsdf", yala)
+    level_assets = []
+    all_assets = []
+    export_gltf_extension = getattr(settings, "export_gltf_extension", ".glb")
+
+    blueprints_path =  getattr(settings, "blueprints_path")
+    for blueprint in blueprints_in_scene:
+        if blueprint.local:
+            blueprint_exported_path = os.path.join(blueprints_path, f"{blueprint.name}{export_gltf_extension}")
+        else:
+            # get the injected path of the external blueprints
+            blueprint_exported_path = blueprint.collection['export_path'] if 'export_path' in blueprint.collection else None
+            # add their material path
+            materials_exported_path = blueprint.collection['materials_path'] if 'materials_path' in blueprint.collection else None
+            level_assets.append({"name": blueprint.name+"_material", "path": materials_exported_path})#, "generated": True, "internal":blueprint.local, "parent": None})
+
+
+        if blueprint_exported_path is not None: # and not does_asset_exist(assets_list, blueprint_exported_path):
+            level_assets.append({"name": blueprint.name, "path": blueprint_exported_path})#, "generated": True, "internal":blueprint.local, "parent": None})
+
+        # now also add the assets of the blueprints # TODO: wait no , these should not be a part of the (scene) local assets
+        for asset in blueprint.collection.user_assets:
+            #print("adding assets of blueprint", asset.name)
+            all_assets.append({"name": asset.name, "path": asset.path})
+
+    """for asset in level_assets:
+        print("  generated asset", asset.name, asset.path)"""
+    
+    materials_path =  getattr(settings, "materials_path")
+    current_project_name = Path(bpy.context.blend_data.filepath).stem
+    materials_library_name = f"{current_project_name}_materials"
+    materials_exported_path = os.path.join(materials_path, f"{materials_library_name}{export_gltf_extension}")
+    material_assets = [{"name": materials_library_name, "path": materials_exported_path}] # we also add the material library as an asset
+    print("material_assets", material_assets, "extension", export_gltf_extension)
+
+
+    all_assets_raw = get_main_scene_assets_tree2(main_scene=scene, blueprints_data=blueprints_data, settings=settings)
+    local_assets =  [{"name": asset["name"], "path": asset["path"]} for asset in all_assets_raw if asset['parent'] is None and asset["path"] != "" ] 
+    all_assets = [{"name": asset["name"], "path": asset["path"]} for asset in all_assets_raw if asset["path"] != "" ] 
+    print("all_assets_raw", all_assets_raw)
+    print("all_assets", all_assets)
+    print("local assets", local_assets + material_assets)
+    scene["BlueprintAssets"] = assets_to_fake_ron(local_assets + material_assets)
+
+
+    #scene["BlueprintAssets"] = assets_to_fake_ron(all_assets + [{"name": asset.name, "path": asset.path} for asset in scene.user_assets] + level_assets + material_assets)
+    #scene["BlueprintAssets"] = assets_to_fake_ron([{'name':'foo', 'path':'bar'}])
+
+def upsert_blueprint_assets(blueprint, blueprints_data, settings):   
+    all_assets_raw = get_blueprint_asset_tree(blueprint=blueprint, blueprints_data=blueprints_data, settings=settings)
+   
+    all_assets = []
+    auto_assets = []
+    local_assets =  [{"name": asset["name"], "path": asset["path"]} for asset in all_assets_raw if asset['parent'] is None and asset["path"] != "" ]
+    print("all_assets_raw", all_assets_raw)
+    print("local assets", local_assets)
+    blueprint.collection["BlueprintAssets"] = assets_to_fake_ron(local_assets)

tools/blenvy/assets/assets_scan.py

@@ -121,6 +121,30 @@ def get_main_scene_assets_tree(main_scene, blueprints_data, settings):
             added_asset.path = asset["path"]
     return assets_list
 
+# same as the above, withouth the clutter below : TODO: unify
+def get_main_scene_assets_tree2(main_scene, blueprints_data, settings):
+    blueprints_path =  getattr(settings, "blueprints_path")
+    export_gltf_extension = getattr(settings, "export_gltf_extension", ".glb")
+    blueprint_instance_names_for_scene = blueprints_data.blueprint_instances_per_main_scene.get(main_scene.name, None)
+
+    assets_list = get_user_assets_as_list(main_scene)
+    if blueprint_instance_names_for_scene:
+        for blueprint_name in blueprint_instance_names_for_scene:
+            blueprint = blueprints_data.blueprints_per_name.get(blueprint_name, None)
+            if blueprint is not None: 
+                blueprint_exported_path = None
+                if blueprint.local:
+                    blueprint_exported_path = os.path.join(blueprints_path, f"{blueprint.name}{export_gltf_extension}")
+                else:
+                    # get the injected path of the external blueprints
+                    blueprint_exported_path = blueprint.collection['export_path'] if 'export_path' in blueprint.collection else None
+                if blueprint_exported_path is not None and not does_asset_exist(assets_list, blueprint_exported_path):
+                    assets_list.append({"name": blueprint.name, "path": blueprint_exported_path, "type": "MODEL", "generated": True, "internal":blueprint.local, "parent": None})
+                
+                assets_list += get_blueprint_assets_tree(blueprint, blueprints_data, parent=blueprint.name, settings=settings)
+   
+    return assets_list
+
 def get_blueprint_asset_tree(blueprint, blueprints_data, settings):
     blueprints_path =  getattr(settings, "blueprints_path")
     export_gltf_extension = getattr(settings, "export_gltf_extension", ".glb")