This notebook series was updated from the previous one sds-2-x-dl to work with Spark 3.0.1 and Python3+. The notebook series was updated on 2021-01-18. See changes from previous version in table below as well as current flaws that needs revision.
Thanks to Christian von Koch and William Anzén for their contributions towards making these materials Spark 3.0.1 and Python 3+ compliant.
Table of changes and current flaws
| Notebook | Changes | Current flaws |
|---|---|---|
| 049 | cmd6 & cmd15: Picture missing | |
| 051 | cmd9: Changed syntax with dataframes in Pandas. Changed to df.loc for it to work properly. Also changed all .asmatrix() to .values since asmatrix() will be depreciated in future versions | |
| 053 | cmd4: Python WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/opdeflibrary.py:263: colocatewith (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. <tf.Variable 'y:0' shape=() dtype=int32ref> | |
| 054 | cmd15: Python WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/opdeflibrary.py:263: colocatewith (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. <tf.Variable 'y:0' shape=() dtype=int32ref> | |
| 055 | cmd25: Changed parameter in Dropout layer to rate=1-keepprob (see comments in code) since keepprob will be depreciated in future versions. | |
| 057 | cmd9: Changed parameter in Dropout layer to rate=1-keepprob (see comments in code) since keepprob will be depreciated in future versions. | cmd4: Python WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/opdeflibrary.py:263: colocatewith (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. <tf.Variable 'y:0' shape=() dtype=int32ref>, WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/ops/mathops.py:3066: toint32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. |
| 058 | cmd7 & cmd9: Updated path to cifar-10-batches-py | cmd17: Python WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/opdeflibrary.py:263: colocatewith (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. <tf.Variable 'y:0' shape=() dtype=int32ref>, WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/ops/mathops.py:3066: toint32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. |
| 060 | cmd2: Python WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/opdeflibrary.py:263: colocatewith (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. <tf.Variable 'y:0' shape=() dtype=int32ref>, WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/ops/mathops.py:3066: toint32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. | |
| 062 | cmd21: Changed parameter in Dropout layer to rate=1-keepprob (see comments in code) since keepprob will be depreciated in future versions. | |
| 063 | cmd16-cmd18: Does not work to mount on dbfs directly with save_weights() in keras. Workaround: Save first locally on tmp and then move files to dbfs. See https://github.com/h5py/h5py/issues/709 | cmd16: Python WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/opdeflibrary.py:263: colocatewith (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. <tf.Variable 'y:0' shape=() dtype=int32ref>, WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/ops/mathops.py:3066: toint32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. |