In a previous article we shared our way of organizing the files. In this one, we’ll introduce you to the use of a simple python library called Lucidity, very useful to start handling your project naming conventions. The aim is to quickly check if your file paths are correct, or build paths from a set of variables, without coding your own parser.
“Lucidity is a framework for templating filesystem structure.” It’s a library made to handle the paths of your project through templates easy to write. It’s based on regex, but provides an abstraction layer to create your pattern in an easier way. You are going to describe, with simple placeholders, what you expect in your paths.
Table of Contents
This is a tutorial as an introduction to the library lucidity. It’s answering a naming convention and parsing problems you might encounter. That said, we must admit we do not use it in our pipeline. We use something else that we might present in an incoming article depending on the future of that other library (not ours). Meanwhile lucidity looks to be a good option you can use out of the box. And it was interesting for me to test and it’s also helpful in small cases. For example, you can use this library within your Digital Content Creation Tool in which you can do naming sanity checks (if you have rules for naming your 3D object within blender, maya, etc.) It’s not limited to a filesystem structure.
Also, this article contains code with long lines (yes, it’s not PEP8 compliant but paths are long and I don’t like breaking paths). Unfortunately some browsers might not display the scrolls in some cases (for example chrome on mac if you don’t have a mouse and you’re using your trackpad hides them). So just scroll to see the rest. 🙂
Installing lucidity
Lucidity is available on github : github.com/4degrees/lucidity or 4degrees’s gitlab. The documentation is available here : lucidity.readthedocs.io.
Lucidity can be easily installed using pip, by just typing:
pip install lucidity
then start your python and check you can import the library:
import lucidity
Let’s start parsing
Those examples are based on what was described in the article an introduction to organizing project files. You obviously will have to adapt them to your own file naming conventions. The first path I want to describe is for my Library. A path to store a simple asset task file (shading for example). So following the conventions described in my previous article, this is the path I’m looking for:
/myProject/LIB/chars/main/Tomoe/shad/LIB_chars_main_Tomoe_shad_high_v05.blend
if we try to abstract that folder path to isolate the variables, we can describe it as:
/project/library/type/family/assetName/department/
and then for the filename, some variables are the same as the folder and others are new:
library_type_family_assetName_department_tags_version.extension
So we have the following variables ton handle in our complete path:
- project
- library
- type
- family
- assetName
- department
- tags
- version
- extension
In lucidity variables are called placeholders, and they are described between braces ({}). So we create a new lucitity template and we give it a name (in this case task_file)
template = lucidity.Template('task_file', '/{project}/{library}/{type}/{family}/{asset_name}/{department}/{library}_{type}_{family}_{asset_name}_{department}_{tags}_v{version}.{extension}')
That’s it! Your template is ready to be used. But I won’t abandon you here.
Let’s check if all the placeholders are listed correctly. We can check the keys of the template:
print(template.keys()) # set(['asset_name', 'extension', 'family', 'tags', 'library', 'project', 'version', 'type'])
Everything looks good. So let’s check if the example path is legit:
path = "/myProject/LIB/chars/main/Tomoe/shad/LIB_chars_main_Tomoe_shad_high_v05.blend" data = template.parse(path)
So far no error was raised by python. Apparently my path is correct, and I can now ask for the value of all the parsed keys:
print(data)
# {'asset_name': 'Tomoe',
# 'department': 'shad',
# 'extension': 'blend',
# 'family': 'main',
# 'library': 'LIB',
# 'project': 'myProject',
# 'tags': 'high',
# 'type': 'chars',
# 'version': '05'}
My path is compliant to what was described in the template and I can get the information in a nice dictionary. With that, I can already go through all the files of my Library, get their full path and checks if it matches my template. If not, the file might be named wrongly.
For example, we could have found the following file, with the “v” missing next to the version number:
path = "/myProject/LIB/chars/main/Tomoe/shad/LIB_chars_main_Tomoe_shad_high_05.blend"
Let’s look what happens now if we run the template on that path:
data = template.parse(path) # lucidity.error.ParseError: Path '/myProject/LIB/chars/main/Tomoe/shad/LIB_chars_main_Tomoe_shad_high_05.blend' did not match template patterns.
Python raises an error message! We know the file path is not compliant to our organization.
Being strict on parsing
Let’s now consider the following path:
path = "/myProject/LIB/chars/main/Tomoe/shad/LIB_chars_main_Agent327_shad_high_v05.blend"
As you can see, I’ve changed the asset_name on the filename (Agent327 instead of Tomoe). As my first value of asset_name is different from the second one, I could expect to raise an Error. Let’s try it:
data = template.parse(path)
print(data)
# {'project': 'myProject',
# 'extension': 'blend',
# 'family': 'main',
# 'tags': 'high',
# 'library': 'LIB',
# 'asset_name': 'Agent327',
# 'version': '05',
# 'department': 'shad',
# 'type': 'chars'}
My asset_name is “Agent327”. It’s because the default behavior of lucidity is permissive and it will take the last occurrence of the placeholder value. The documentation says it’s the “RELAXED” mode. I don’t know why it was set up like this, but I’ll rather have something stricter and less permissive. With the current behavior this path is correct when it should not.
Except if you really want it, you should not mix your assets in the library. In my case I do not want any Agent327 files in the Tomoe asset folder. So you can use a parameter when the template has been set like this:
template = lucidity.Template('task_file', '/{project}/{library}/{type}/{family}/{asset_name}/{department}/{library}_{type}_{family}_{asset_name}_{department}_{tags}_v{version}.{extension}')
template.duplicate_placeholder_mode = template.STRICT
if you need it at a specific time, or:
template = lucidity.Template('task_file', '/{project}/{library}/{type}/{family}/{asset_name}/{department}/{library}_{type}_{family}_{asset_name}_{department}_{tags}_v{version}.{extension}', duplicate_placeholder_mode = lucidity.template.STRICT)
if you want it activated by default all the time, and then:
path = "/myProject/LIB/chars/main/Tomoe/shad/LIB_chars_main_Agent327_shad_high_v05.blend" data = template.parse(path) # lucidity.error.ParseError: Different extracted values for placeholder 'asset_name' detected. Values were 'Tomoe' and 'Agent327'
Now an error is raised, and it is beautifully displayed for you to know why. So if I were you, I will always setup lucidity like this right after creating the templates.
Now if I want to check all the files of the library, using python glob I can do something like (nb: it’s a dirty example):
import lucidity
import glob
template = lucidity.Template('task_file', '/{project}/{library}/{type}/{family}/{asset_name}/{department}/{library}_{type}_{family}_{asset_name}_{department}_{tags}_v{version}.{extension}')
for f in glob.glob("/myProject/LIB/*/*/*/*/*"):
try:
path = template.parse(f)
except:
print("WRONG NAMING: %s" % f)
10 simple lines of code allows me to check potentially thousands of file names within seconds!
Formatting data
Now, instead of checking the path against my template, I want to build a path from data. I want to build it from a dictionary I fill. It’s called formatting and it’s pretty easy:
data = {'asset_name': 'Agent327',
'department': 'modeling',
'extension': 'ma',
'family': 'main',
'library': 'LIB',
'project': 'myProject',
'tags': 'low',
'type': 'chars',
'version': '05'}
path = template.format(data)
print(path)
# '/myProject/LIB/chars/main/Agent327/modeling/LIB_chars_main_Agent327_modeling_low_v05.ma'
So now you can get and create any path you want following your template and using custom data. That data can come from a CSV of assets to make, or from a simple wizard asking the user in a nice GUI to fill the placeholders.
And once you have that path, you can build all the folder hierarchy without any human interaction, doing a simple:
import os folder_path = os.path.dirname(path) os.makedirs(forlder_path)
You can now virtually build any task file in your library. Of course, errors are handled well here too, for example if you forget a key:
data = {'asset_name': 'Agent327',
'department': 'modeling',
'extension': 'ma',
'library': 'LIB',
'project': 'myProject',
'tags': 'low',
'type': 'chars',
'version': '05'}
path = template.format(data)
# lucidity.error.FormatError: Could not format data {...} due to missing key 'family'
This is super handy!
Regex: More Power for the Parsing
With regex you can go even go further allowing you to make powerful templates. For example, the version might always be a digit as currently it might be anything. You can specify the placeholder regex in the template, adding the rule after a semicolon (“:”) following the placeholder name :
template = lucidity.Template('name', 'file_v{version:\d+}.ext')
path = "file_v01.ext"
data = template.parse(path)
print(data)
# {'version': '01'}
path = "file_vONE.ext"
data = template.parse(path)
# lucidity.error.ParseError: Path 'file_vOne.ext' did not match template pattern.
So you can go pretty wild with powerful regex. Like for example saying that the department should be one of these choices : chars OR sets OR props
template = lucidity.Template('name', 'file_{department:chars|sets|props}.ext')
path = "file_chars.ext"
data = template.parse(path)
print(data)
# {'department': 'chars'}
path = "file_cats.ext"
data = template.parse(path)
# lucidity.error.ParseError: Path 'file_cats.ext' did not match template pattern.
BUT, unfortunatly, this is not working the way we can expect the other way arround, using a dictionary to format a path:
template = lucidity.Template('name', 'file_{department:chars|sets|props}.ext')
data = {'department': 'VFX'}
path = template.format(data)
print(path)
# file_VFX.ext
As you see, the regex was not used. If you want to be sure, the provided path is right, you can just parse it against your template:
template = lucidity.Template('name', 'file_{department:chars|sets|props}.ext')
data = {'department': 'VFX'}
path = template.format(data)
datacheck = template.parse(path)
# lucidity.error.ParseError: Path 'file_VFX.ext' did not match template pattern.
According to the documentation it’s normal: “And of course, any custom expression text is omitted when formatting data”. Why??? This is a weird and sad choice as the library is providing you a wrong path at first… It would be more interesting and powerful to check the regex while creating the path.
Multiple Templates
Now that my Library template is working as I want, I would like to also have my FILM template. So if I provide any path coming from my library or film directory, lucidity could check if the path is right by itself, without forcing me to prepare switch cases.
In our case, the film template should look like:
/{project}/{film}/{sequence}/{shot}/{department}/{film}_{sequence}_{shot}_{department}_{tags}_v{version}.{extension}
This is called multiple templates and you must store them in a list. You assemble them like this:
templates = [
lucidity.Template('lib_task_file', '/{project}/{library}/{type}/{family}/{asset_name}/{department}/{library}_{type}_{family}_{asset_name}_{department}_{tags}_v{version}.{extension}'),
lucidity.Template('film_task_file','/{project}/{film}/{sequence}/{shot}/{department}/{film}_{sequence}_{shot}_{department}_{tags}_v{version}.{extension}')
]
And now to check any path calling lucidity.parse() providing the path and the templates:
path = "/myProject/MYFILM/Seq01/Shot0010/Animation/MYFILM_Seq01_Shot0010_Animation_Blocking_v07.blend"
data = lucidity.parse(path, templates)
print(data)
# ({'department': 'Animation',
# 'extension': 'blend',
# 'film': 'MYFILM',
# 'project': 'myProject',
# 'sequence': 'Seq01',
# 'shot': 'Shot0010',
# 'tags': 'Blocking',
# 'version': '07'},
# Template(name='lib_task_file', pattern='/{project}/{film}/{sequence}/{shot}/{department}/{film}_{sequence}_{shot}_{department}_{tags}_v{version}.{extension}'))
As you can see now, the information provided within a tupple is a little bit more complex. You get the keys as the first element, a template object as second element including the template name and the recognized pattern.
And of course, you can go the other way, formatting data instead of parsing a path:
data = {'department': 'Lighting',
'extension': 'blend',
'film': 'MYFILM',
'project': 'myProject',
'sequence': 'Seq13',
'shot': 'Shot0042',
'tags': 'low-samples',
'version': '03'},
path = lucidity.format(data, templates)
print(path)
# /myProject/MYFILM/Seq01/Shot0010/Animation/MYFILM_Seq01_Shot0010_Animation_Blocking_v07.blend
Be aware that the order of your patterns is important as lucidity will check your path pattern by pattern and stop when the first pattern matches. So the next patterns are not considered.
Other features
There is other features documented in lucitity. Some help you to customize the patterns
Nested structures
This is a way of creating data from a path or creating a path from data, but in a more complex way than a simple key/value dictionnary. For example you could setup your template as :
import lucidity
template = lucidity.Template('shot_file', '{shot.film}_{shot.sequence}_{shot.name}_{task.name}_{task.tags}_v{file.version}.{file.extension}')
path = "DIL_s01_p001_animation_main_v05.blend"
data = template.parse(path)
print(data)
# {'file': {'extension': 'blend', 'version': '05'},
# 'shot': {'film': 'DIL', 'name': 'p001', 'sequence': 's01'},
# 'task': {'name': 'animation', 'tags': 'main'}}
Nested structures could be useful to handle complex naming conventions.
Anchors
The anchor is the direction your pattern is parsed. By default, it’s parsed from left to right or start to end.
template = lucidity.Template('project', '/production/{project}')
print(template.parse('/production/Dilili'))
# {'project': 'dilili'}
print(template.parse('/production/Dilili/DIL/S01/file.ext'))
# {'project': 'dilili'}
print(template.parse('/media/production/Dilili/')
# ParseError: Input '/media/production/Dilili/' did not match template pattern.
The first case is matching our template just as described, no surprise. The second case, even if the path is longer, the regex is matching and provides a result. Because the default behavior looks for the pattern at the beginning of the string, and it’s matching here. But in the third case, as the regex is anchored to the left, or start of the string, the beginning of the provided string does not match.
So you can specify you are anchoring the right side (or from the end). It could be useful if you just want to parse the file name and not the full path:
filename_template = lucidity.Template(
'version',
'v{version}.{ext}',
anchor=lucidity.Template.ANCHOR_END
)
print(filename_template.parse('/path/to/MYFILM_Seq01_Shot0010_Animation_Blocking_v07.blend'))
# {'ext': 'blend', 'version': '07'}
You can see the regex is matching something even if I didn’t specify the rest of the path. It wouldn’t have worked without specifying the Anchor (anchor=lucidity.Template.ANCHOR_END)to the end (or right) of the string . You can also have very strict patterns using ANCHOR_BOTH where the exact pattern should match on the whole string.
Or you can set it to None, and the pattern could match anywhere in the string :
template = lucidity.Template(
'animation_tags',
'_Animation_{tags}_',
anchor=None
)
print(template.parse('/path/to/MYFILM_Seq01_Shot0010_Animation_Blocking_v07.blend'))
# {'tags': 'Blocking'}
Use that function wisely.
Patterns discovery
As projects organization might change with time, you can save a templates.py in your project folders, describing the specific templates of the project. Then you can look for the templates.py files automatically with the following command :
import lucidity templates = lucidity.discover_templates(paths=['/myProject'])
That loads any template.py file it found in myProject, as long as there is a register function within the python file:
# templates.py
from lucidity import Template
def register():
'''Register templates for my project'''
return [
Template('lib_task_file', '/{project}/{library}/{type}/{family}/{asset_name}/{department}/{library}_{type}_{family}_{asset_name}_{department}_{tags}_v{version}.{extension}'),
Template('lib_task_file','/{project}/{film}/{sequence}/{shot}/{department}/{film}_{sequence}_{shot}_{department}_{tags}_v{version}.{extension}')
]
This way of using the loader could be useful when having scripts that are projected agnostic, script that adapt to any project templates.
Note that by default the research will be recursive! It means, pointing at your project root folder will look through all the subdirectories. This might take a long time and be a bad idea. So you might want to specify your script to be not recursive:
templates = lucidity.discover_templates(paths=['/myProject'], recursive=False)
Templates of templates!
Let’s imagine we have a lot of templates to handle. Some share a lot of same placeholders or paths. It’s annoying and not easy to read. But Lucidity allows you to create a template, and reuse-it in another template, it’s called template reference.
For example, you have two differently organized task files in your film library. One is versioned as described sooner, the other one is not, because it’s a reference file. But they share the same path and I want to reuse a maximum of elements for readability and easy updates. In that case I can create a template for the folder path, and then two templates for the task files, which share the same folder path.
Template reference are indicated using the arroba sign (@).
# the folder path
shot_path = lucidity.Template('shot_path','/{project}/{film}/{sequence}/{shot}/{department}/')
# Then the two files, one is versioned and the other is not
shot_task_versioned = lucidity.Template('shot_task_versioned', '{@shot_path}/taskname_v{version}.{extension}')
shot_task_reference = lucidity.Template('shot_task_reference', '{@shot_path}/taskname_REF.{extension}')
You can run that code and everything will go fine untill you try to parse it or call it using the template.key() method. Lucidity does not know yet what to do with the @shot_path. You need a resolver, a dictionnary of templates needed. You include the shared template, and then you provide that resolver to the templates needing template reference.
resolver = {}
resolver[shot_path.name] = shot_path
You resolver is ready. Now you can add it to your two other templates
shot_task_versioned.template_resolver = resolver shot_task_reference.template_resolver = resolver
Note that you can also pass the template_resolver argument when instantiating a new template as :
shot_task_reference = lucidity.Template('shot_task_reference',
'{@shot_path}/taskname_REF.{extension}',
template_resolver=resolver)
Of course, your resolver can contain more templates. So a template can reference more than one reference template. Or you can do templatesceptions, references of references. But don’t make it too complex then.
And you can now run all the usual methodes like .keys(), .parse() or .format(). It will be easier to update in case of complex templates. And it’s more readable.
Errors handling
The list of errors is listed in the documentation and you might play with them in advance python scripts. It might be enough, but sometimes is not precise. For example, when an element of the path is wrong, the error doesn’t really tell you where or why.
Conclusion
Lucidity could be a nice start to handle your project organization. There are limits, as I mentioned for example in the regex. It’s very annoying it’s not used when formating. You also might have a lot of templates to setup depending on your organization.
I also miss a real option for optional fields. You can use regex, but it’s harder to get a bulletproof optional field. And when you prepare your data you don’t know if a field is optional or not, and you don’t know the choices and options (in version a digit or any string?). So it’s harder to make dynamic wizards for the users.
But that said, it’s simple and fast to deploy and to kick-start your first tools. A pretty good option to start rather than coding your own parser!
Thanks for Kevin, Dorian and Manu for the feedback on this article.
