tf.data.Datasetを使っていた時に AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason:

Deep Learning

2022.09.15

CIFAR10でtf.dataを使った時に表題のwarning。

model.fitは問題なく出来るので無視しようと思ったが、気になったので調べてみた。

(train_features, _), (_, _) = tf.keras.datasets.cifar10.load_data()

dataset = tf.data.Dataset.from_tensor_slices(train_features)

 W tensorflow/core/grappler/optimizers/data/auto_shard.cc:776] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Fou
nd an unshardable source dataset: name: "TensorSliceDataset/_1"

AUTO sharding policyでこけてるようなので、手動でFILEに設定して解決。

options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.FILE
dataset = dataset.with_options(options)

参考

Distributed Input | TensorFlow Core

Using tf.data.Dataset.list_files prints "unshardable source dataset" warning · Issue #55474 · tensorflow/tensorflow

System information Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes O...