957 KiB
957 KiB
In [1]:
!apt-get update
!apt-get install graphviz -y
!pip install tensorflow
!pip install numpy
!pip install pandas
!pip install keras
!pip install scikit-learn
!pip install matplotlib
!pip install joblib
!pip install pyarrow
!pip install fastparquet
!pip install scipy
!pip install seaborn
!pip install tqdm
!pip install pydot
!pip install tensorflow-io
!pip install tensorflow-addonsGet:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease [1581 B]
Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 Packages [1192 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
Get:9 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [45.2 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2738 kB]
Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [3323 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [3446 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1514 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [53.3 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [33.8 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [81.4 kB]
Get:19 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2454 kB]
Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1225 kB]
Fetched 36.5 MB in 2s (18.2 MB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
fontconfig fonts-liberation libann0 libcairo2 libcdt5 libcgraph6 libdatrie1
libfribidi0 libgraphite2-3 libgts-0.7-5 libgts-bin libgvc6 libgvpr2
libharfbuzz0b libice6 liblab-gamut1 libltdl7 libpango-1.0-0
libpangocairo-1.0-0 libpangoft2-1.0-0 libpathplan4 libpixman-1-0 libsm6
libthai-data libthai0 libxaw7 libxcb-render0 libxmu6 libxrender1 libxt6
x11-common
Suggested packages:
gsfonts graphviz-doc
The following NEW packages will be installed:
fontconfig fonts-liberation graphviz libann0 libcairo2 libcdt5 libcgraph6
libdatrie1 libfribidi0 libgraphite2-3 libgts-0.7-5 libgts-bin libgvc6
libgvpr2 libharfbuzz0b libice6 liblab-gamut1 libltdl7 libpango-1.0-0
libpangocairo-1.0-0 libpangoft2-1.0-0 libpathplan4 libpixman-1-0 libsm6
libthai-data libthai0 libxaw7 libxcb-render0 libxmu6 libxrender1 libxt6
x11-common
0 upgraded, 32 newly installed, 0 to remove and 121 not upgraded.
Need to get 7298 kB of archives.
After this operation, 18.3 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libfribidi0 amd64 1.0.8-2ubuntu3.1 [26.1 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 fontconfig amd64 2.13.1-4.2ubuntu5 [177 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 fonts-liberation all 1:1.07.4-11 [822 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libann0 amd64 1.1.2+doc-7build1 [26.0 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libcdt5 amd64 2.42.2-6ubuntu0.1 [21.1 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libcgraph6 amd64 2.42.2-6ubuntu0.1 [45.4 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libgts-0.7-5 amd64 0.7.6+darcs121130-5 [164 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpixman-1-0 amd64 0.40.0-1ubuntu0.22.04.1 [264 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxcb-render0 amd64 1.14-3ubuntu3 [16.4 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxrender1 amd64 1:0.9.10-1build4 [19.7 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libcairo2 amd64 1.16.0-5ubuntu2 [628 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libltdl7 amd64 2.4.6-15build2 [39.6 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libgraphite2-3 amd64 1.3.14-1build2 [71.3 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libharfbuzz0b amd64 2.7.4-1ubuntu3.1 [352 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libthai-data all 0.1.29-1build1 [162 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 libdatrie1 amd64 0.2.13-2 [19.9 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libthai0 amd64 0.1.29-1build1 [19.2 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpango-1.0-0 amd64 1.50.6+ds-2ubuntu1 [230 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpangoft2-1.0-0 amd64 1.50.6+ds-2ubuntu1 [54.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpangocairo-1.0-0 amd64 1.50.6+ds-2ubuntu1 [39.8 kB]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libpathplan4 amd64 2.42.2-6ubuntu0.1 [23.4 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libgvc6 amd64 2.42.2-6ubuntu0.1 [724 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libgvpr2 amd64 2.42.2-6ubuntu0.1 [192 kB]
Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 liblab-gamut1 amd64 2.42.2-6ubuntu0.1 [1965 kB]
Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 x11-common all 1:7.7+23ubuntu2 [23.4 kB]
Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libice6 amd64 2:1.0.10-1build2 [42.6 kB]
Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 libsm6 amd64 2:1.2.3-1build2 [16.7 kB]
Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxt6 amd64 1:1.2.1-1 [177 kB]
Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxmu6 amd64 2:1.1.3-3 [49.6 kB]
Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxaw7 amd64 2:1.0.14-1 [191 kB]
Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 graphviz amd64 2.42.2-6ubuntu0.1 [653 kB]
Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libgts-bin amd64 0.7.6+darcs121130-5 [44.3 kB]
Fetched 7298 kB in 2s (4771 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libfribidi0:amd64.
(Reading database ... 20752 files and directories currently installed.)
Preparing to unpack .../00-libfribidi0_1.0.8-2ubuntu3.1_amd64.deb ...
Unpacking libfribidi0:amd64 (1.0.8-2ubuntu3.1) ...
Selecting previously unselected package fontconfig.
Preparing to unpack .../01-fontconfig_2.13.1-4.2ubuntu5_amd64.deb ...
Unpacking fontconfig (2.13.1-4.2ubuntu5) ...
Selecting previously unselected package fonts-liberation.
Preparing to unpack .../02-fonts-liberation_1%3a1.07.4-11_all.deb ...
Unpacking fonts-liberation (1:1.07.4-11) ...
Selecting previously unselected package libann0.
Preparing to unpack .../03-libann0_1.1.2+doc-7build1_amd64.deb ...
Unpacking libann0 (1.1.2+doc-7build1) ...
Selecting previously unselected package libcdt5:amd64.
Preparing to unpack .../04-libcdt5_2.42.2-6ubuntu0.1_amd64.deb ...
Unpacking libcdt5:amd64 (2.42.2-6ubuntu0.1) ...
Selecting previously unselected package libcgraph6:amd64.
Preparing to unpack .../05-libcgraph6_2.42.2-6ubuntu0.1_amd64.deb ...
Unpacking libcgraph6:amd64 (2.42.2-6ubuntu0.1) ...
Selecting previously unselected package libgts-0.7-5:amd64.
Preparing to unpack .../06-libgts-0.7-5_0.7.6+darcs121130-5_amd64.deb ...
Unpacking libgts-0.7-5:amd64 (0.7.6+darcs121130-5) ...
Selecting previously unselected package libpixman-1-0:amd64.
Preparing to unpack .../07-libpixman-1-0_0.40.0-1ubuntu0.22.04.1_amd64.deb ...
Unpacking libpixman-1-0:amd64 (0.40.0-1ubuntu0.22.04.1) ...
Selecting previously unselected package libxcb-render0:amd64.
Preparing to unpack .../08-libxcb-render0_1.14-3ubuntu3_amd64.deb ...
Unpacking libxcb-render0:amd64 (1.14-3ubuntu3) ...
Selecting previously unselected package libxrender1:amd64.
Preparing to unpack .../09-libxrender1_1%3a0.9.10-1build4_amd64.deb ...
Unpacking libxrender1:amd64 (1:0.9.10-1build4) ...
Selecting previously unselected package libcairo2:amd64.
Preparing to unpack .../10-libcairo2_1.16.0-5ubuntu2_amd64.deb ...
Unpacking libcairo2:amd64 (1.16.0-5ubuntu2) ...
Selecting previously unselected package libltdl7:amd64.
Preparing to unpack .../11-libltdl7_2.4.6-15build2_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.6-15build2) ...
Selecting previously unselected package libgraphite2-3:amd64.
Preparing to unpack .../12-libgraphite2-3_1.3.14-1build2_amd64.deb ...
Unpacking libgraphite2-3:amd64 (1.3.14-1build2) ...
Selecting previously unselected package libharfbuzz0b:amd64.
Preparing to unpack .../13-libharfbuzz0b_2.7.4-1ubuntu3.1_amd64.deb ...
Unpacking libharfbuzz0b:amd64 (2.7.4-1ubuntu3.1) ...
Selecting previously unselected package libthai-data.
Preparing to unpack .../14-libthai-data_0.1.29-1build1_all.deb ...
Unpacking libthai-data (0.1.29-1build1) ...
Selecting previously unselected package libdatrie1:amd64.
Preparing to unpack .../15-libdatrie1_0.2.13-2_amd64.deb ...
Unpacking libdatrie1:amd64 (0.2.13-2) ...
Selecting previously unselected package libthai0:amd64.
Preparing to unpack .../16-libthai0_0.1.29-1build1_amd64.deb ...
Unpacking libthai0:amd64 (0.1.29-1build1) ...
Selecting previously unselected package libpango-1.0-0:amd64.
Preparing to unpack .../17-libpango-1.0-0_1.50.6+ds-2ubuntu1_amd64.deb ...
Unpacking libpango-1.0-0:amd64 (1.50.6+ds-2ubuntu1) ...
Selecting previously unselected package libpangoft2-1.0-0:amd64.
Preparing to unpack .../18-libpangoft2-1.0-0_1.50.6+ds-2ubuntu1_amd64.deb ...
Unpacking libpangoft2-1.0-0:amd64 (1.50.6+ds-2ubuntu1) ...
Selecting previously unselected package libpangocairo-1.0-0:amd64.
Preparing to unpack .../19-libpangocairo-1.0-0_1.50.6+ds-2ubuntu1_amd64.deb ...
Unpacking libpangocairo-1.0-0:amd64 (1.50.6+ds-2ubuntu1) ...
Selecting previously unselected package libpathplan4:amd64.
Preparing to unpack .../20-libpathplan4_2.42.2-6ubuntu0.1_amd64.deb ...
Unpacking libpathplan4:amd64 (2.42.2-6ubuntu0.1) ...
Selecting previously unselected package libgvc6.
Preparing to unpack .../21-libgvc6_2.42.2-6ubuntu0.1_amd64.deb ...
Unpacking libgvc6 (2.42.2-6ubuntu0.1) ...
Selecting previously unselected package libgvpr2:amd64.
Preparing to unpack .../22-libgvpr2_2.42.2-6ubuntu0.1_amd64.deb ...
Unpacking libgvpr2:amd64 (2.42.2-6ubuntu0.1) ...
Selecting previously unselected package liblab-gamut1:amd64.
Preparing to unpack .../23-liblab-gamut1_2.42.2-6ubuntu0.1_amd64.deb ...
Unpacking liblab-gamut1:amd64 (2.42.2-6ubuntu0.1) ...
Selecting previously unselected package x11-common.
Preparing to unpack .../24-x11-common_1%3a7.7+23ubuntu2_all.deb ...
Unpacking x11-common (1:7.7+23ubuntu2) ...
Selecting previously unselected package libice6:amd64.
Preparing to unpack .../25-libice6_2%3a1.0.10-1build2_amd64.deb ...
Unpacking libice6:amd64 (2:1.0.10-1build2) ...
Selecting previously unselected package libsm6:amd64.
Preparing to unpack .../26-libsm6_2%3a1.2.3-1build2_amd64.deb ...
Unpacking libsm6:amd64 (2:1.2.3-1build2) ...
Selecting previously unselected package libxt6:amd64.
Preparing to unpack .../27-libxt6_1%3a1.2.1-1_amd64.deb ...
Unpacking libxt6:amd64 (1:1.2.1-1) ...
Selecting previously unselected package libxmu6:amd64.
Preparing to unpack .../28-libxmu6_2%3a1.1.3-3_amd64.deb ...
Unpacking libxmu6:amd64 (2:1.1.3-3) ...
Selecting previously unselected package libxaw7:amd64.
Preparing to unpack .../29-libxaw7_2%3a1.0.14-1_amd64.deb ...
Unpacking libxaw7:amd64 (2:1.0.14-1) ...
Selecting previously unselected package graphviz.
Preparing to unpack .../30-graphviz_2.42.2-6ubuntu0.1_amd64.deb ...
Unpacking graphviz (2.42.2-6ubuntu0.1) ...
Selecting previously unselected package libgts-bin.
Preparing to unpack .../31-libgts-bin_0.7.6+darcs121130-5_amd64.deb ...
Unpacking libgts-bin (0.7.6+darcs121130-5) ...
Setting up libgraphite2-3:amd64 (1.3.14-1build2) ...
Setting up libpixman-1-0:amd64 (0.40.0-1ubuntu0.22.04.1) ...
Setting up fontconfig (2.13.1-4.2ubuntu5) ...
Regenerating fonts cache... done.
Setting up libxrender1:amd64 (1:0.9.10-1build4) ...
Setting up libdatrie1:amd64 (0.2.13-2) ...
Setting up libxcb-render0:amd64 (1.14-3ubuntu3) ...
Setting up liblab-gamut1:amd64 (2.42.2-6ubuntu0.1) ...
Setting up x11-common (1:7.7+23ubuntu2) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up libcairo2:amd64 (1.16.0-5ubuntu2) ...
Setting up libgts-0.7-5:amd64 (0.7.6+darcs121130-5) ...
Setting up libpathplan4:amd64 (2.42.2-6ubuntu0.1) ...
Setting up libann0 (1.1.2+doc-7build1) ...
Setting up libfribidi0:amd64 (1.0.8-2ubuntu3.1) ...
Setting up libltdl7:amd64 (2.4.6-15build2) ...
Setting up fonts-liberation (1:1.07.4-11) ...
Setting up libharfbuzz0b:amd64 (2.7.4-1ubuntu3.1) ...
Setting up libthai-data (0.1.29-1build1) ...
Setting up libcdt5:amd64 (2.42.2-6ubuntu0.1) ...
Setting up libcgraph6:amd64 (2.42.2-6ubuntu0.1) ...
Setting up libgts-bin (0.7.6+darcs121130-5) ...
Setting up libice6:amd64 (2:1.0.10-1build2) ...
Setting up libthai0:amd64 (0.1.29-1build1) ...
Setting up libgvpr2:amd64 (2.42.2-6ubuntu0.1) ...
Setting up libsm6:amd64 (2:1.2.3-1build2) ...
Setting up libpango-1.0-0:amd64 (1.50.6+ds-2ubuntu1) ...
Setting up libxt6:amd64 (1:1.2.1-1) ...
Setting up libpangoft2-1.0-0:amd64 (1.50.6+ds-2ubuntu1) ...
Setting up libpangocairo-1.0-0:amd64 (1.50.6+ds-2ubuntu1) ...
Setting up libxmu6:amd64 (2:1.1.3-3) ...
Setting up libxaw7:amd64 (2:1.0.14-1) ...
Setting up libgvc6 (2.42.2-6ubuntu0.1) ...
Setting up graphviz (2.42.2-6ubuntu0.1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.3) ...
Requirement already satisfied: tensorflow in /usr/local/lib/python3.11/dist-packages (2.14.0)
Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (2.0.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (23.5.26)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (0.5.4)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (3.9.0)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (16.0.6)
Requirement already satisfied: ml-dtypes==0.2.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: numpy>=1.23.5 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (1.26.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (3.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from tensorflow) (23.1)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (4.24.3)
Requirement already satisfied: setuptools in /usr/local/lib/python3.11/dist-packages (from tensorflow) (68.2.2)
Requirement already satisfied: six>=1.12.0 in /usr/lib/python3/dist-packages (from tensorflow) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (2.3.0)
Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (4.8.0)
Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (1.14.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (0.34.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (1.58.0)
Requirement already satisfied: tensorboard<2.15,>=2.14 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (2.14.0)
Requirement already satisfied: tensorflow-estimator<2.15,>=2.14.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (2.14.0)
Requirement already satisfied: keras<2.15,>=2.14.0 in /usr/local/lib/python3.11/dist-packages (from tensorflow) (2.14.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.11/dist-packages (from astunparse>=1.6.0->tensorflow) (0.41.2)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.15,>=2.14->tensorflow) (2.23.1)
Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.15,>=2.14->tensorflow) (1.0.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.15,>=2.14->tensorflow) (3.4.4)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.15,>=2.14->tensorflow) (2.31.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.15,>=2.14->tensorflow) (0.7.1)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from tensorboard<2.15,>=2.14->tensorflow) (2.3.7)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow) (5.3.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow) (4.9)
Requirement already satisfied: urllib3>=2.0.5 in /usr/local/lib/python3.11/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow) (2.0.5)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow) (1.3.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.15,>=2.14->tensorflow) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.15,>=2.14->tensorflow) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests<3,>=2.21.0->tensorboard<2.15,>=2.14->tensorflow) (2023.7.22)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.11/dist-packages (from werkzeug>=1.0.1->tensorboard<2.15,>=2.14->tensorflow) (2.1.3)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.11/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow) (0.5.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/lib/python3/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow) (3.2.0)
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Requirement already satisfied: numpy in /usr/local/lib/python3.11/dist-packages (1.26.0)
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting pandas
Obtaining dependency information for pandas from https://files.pythonhosted.org/packages/cd/5f/4dba1d39bb9c38d574a9a22548c540177f78ea47b32f99c0ff2ec499fac5/pandas-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading pandas-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (89 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m89.9/89.9 kB[0m [31m2.1 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: numpy>=1.23.2 in /usr/local/lib/python3.11/dist-packages (from pandas) (1.26.0)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from pandas) (2.8.2)
Collecting pytz>=2020.1 (from pandas)
Obtaining dependency information for pytz>=2020.1 from https://files.pythonhosted.org/packages/11/c3/005fcca25ce078d2cc29fd559379817424e94885510568bc1bc53d7d5846/pytz-2024.2-py2.py3-none-any.whl.metadata
Downloading pytz-2024.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.7 (from pandas)
Obtaining dependency information for tzdata>=2022.7 from https://files.pythonhosted.org/packages/a6/ab/7e5f53c3b9d14972843a647d8d7a853969a58aecc7559cb3267302c94774/tzdata-2024.2-py2.py3-none-any.whl.metadata
Downloading tzdata-2024.2-py2.py3-none-any.whl.metadata (1.4 kB)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
Downloading pandas-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m13.1/13.1 MB[0m [31m74.0 MB/s[0m eta [36m0:00:00[0m:00:01[0m:01[0m
[?25hDownloading pytz-2024.2-py2.py3-none-any.whl (508 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m508.0/508.0 kB[0m [31m106.3 MB/s[0m eta [36m0:00:00[0m
[?25hDownloading tzdata-2024.2-py2.py3-none-any.whl (346 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m346.6/346.6 kB[0m [31m103.8 MB/s[0m eta [36m0:00:00[0m
[?25hInstalling collected packages: pytz, tzdata, pandas
Successfully installed pandas-2.2.3 pytz-2024.2 tzdata-2024.2
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Requirement already satisfied: keras in /usr/local/lib/python3.11/dist-packages (2.14.0)
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting scikit-learn
Obtaining dependency information for scikit-learn from https://files.pythonhosted.org/packages/49/21/3723de321531c9745e40f1badafd821e029d346155b6c79704e0b7197552/scikit_learn-1.5.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading scikit_learn-1.5.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)
Requirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.11/dist-packages (from scikit-learn) (1.26.0)
Collecting scipy>=1.6.0 (from scikit-learn)
Obtaining dependency information for scipy>=1.6.0 from https://files.pythonhosted.org/packages/93/6b/701776d4bd6bdd9b629c387b5140f006185bd8ddea16788a44434376b98f/scipy-1.14.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading scipy-1.14.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m60.8/60.8 kB[0m [31m2.3 MB/s[0m eta [36m0:00:00[0m
[?25hCollecting joblib>=1.2.0 (from scikit-learn)
Obtaining dependency information for joblib>=1.2.0 from https://files.pythonhosted.org/packages/91/29/df4b9b42f2be0b623cbd5e2140cafcaa2bef0759a00b7b70104dcfe2fb51/joblib-1.4.2-py3-none-any.whl.metadata
Downloading joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)
Collecting threadpoolctl>=3.1.0 (from scikit-learn)
Obtaining dependency information for threadpoolctl>=3.1.0 from https://files.pythonhosted.org/packages/4b/2c/ffbf7a134b9ab11a67b0cf0726453cedd9c5043a4fe7a35d1cefa9a1bcfb/threadpoolctl-3.5.0-py3-none-any.whl.metadata
Downloading threadpoolctl-3.5.0-py3-none-any.whl.metadata (13 kB)
Downloading scikit_learn-1.5.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.3 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m13.3/13.3 MB[0m [31m78.0 MB/s[0m eta [36m0:00:00[0m:00:01[0m00:01[0m
[?25hDownloading joblib-1.4.2-py3-none-any.whl (301 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m301.8/301.8 kB[0m [31m104.1 MB/s[0m eta [36m0:00:00[0m
[?25hDownloading scipy-1.14.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.2 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m41.2/41.2 MB[0m [31m55.0 MB/s[0m eta [36m0:00:00[0m00:01[0m:00:01[0m
[?25hDownloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB)
Installing collected packages: threadpoolctl, scipy, joblib, scikit-learn
Successfully installed joblib-1.4.2 scikit-learn-1.5.2 scipy-1.14.1 threadpoolctl-3.5.0
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Requirement already satisfied: matplotlib in /usr/local/lib/python3.11/dist-packages (3.8.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.11/dist-packages (from matplotlib) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib) (4.42.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib) (1.4.5)
Requirement already satisfied: numpy<2,>=1.21 in /usr/local/lib/python3.11/dist-packages (from matplotlib) (1.26.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib) (23.1)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib) (10.0.1)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3/dist-packages (from matplotlib) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.11/dist-packages (from matplotlib) (2.8.2)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Requirement already satisfied: joblib in /usr/local/lib/python3.11/dist-packages (1.4.2)
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting pyarrow
Obtaining dependency information for pyarrow from https://files.pythonhosted.org/packages/5e/b5/9e14e9f7590e0eaa435ecea84dabb137284a4dbba7b3c337b58b65b76d95/pyarrow-18.1.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata
Downloading pyarrow-18.1.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Downloading pyarrow-18.1.0-cp311-cp311-manylinux_2_28_x86_64.whl (40.1 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m40.1/40.1 MB[0m [31m58.8 MB/s[0m eta [36m0:00:00[0m:00:01[0m00:01[0m
[?25hInstalling collected packages: pyarrow
Successfully installed pyarrow-18.1.0
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting fastparquet
Obtaining dependency information for fastparquet from https://files.pythonhosted.org/packages/8d/e8/e1ede861bea68394a755d8be1aa2e2d60a3b9f6b551bfd56aeca74987e2e/fastparquet-2024.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading fastparquet-2024.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Requirement already satisfied: pandas>=1.5.0 in /usr/local/lib/python3.11/dist-packages (from fastparquet) (2.2.3)
Requirement already satisfied: numpy in /usr/local/lib/python3.11/dist-packages (from fastparquet) (1.26.0)
Collecting cramjam>=2.3 (from fastparquet)
Obtaining dependency information for cramjam>=2.3 from https://files.pythonhosted.org/packages/79/1d/180f2ca168625073f0df80b16c795926deed91b7e89dbfc045263ba7444b/cramjam-2.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading cramjam-2.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
Collecting fsspec (from fastparquet)
Obtaining dependency information for fsspec from https://files.pythonhosted.org/packages/c6/b2/454d6e7f0158951d8a78c2e1eb4f69ae81beb8dca5fee9809c6c99e9d0d0/fsspec-2024.10.0-py3-none-any.whl.metadata
Downloading fsspec-2024.10.0-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from fastparquet) (23.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.5.0->fastparquet) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.5.0->fastparquet) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.5.0->fastparquet) (2024.2)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas>=1.5.0->fastparquet) (1.16.0)
Downloading fastparquet-2024.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.8 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m1.8/1.8 MB[0m [31m16.2 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hDownloading cramjam-2.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m2.4/2.4 MB[0m [31m66.7 MB/s[0m eta [36m0:00:00[0m:00:01[0m
[?25hDownloading fsspec-2024.10.0-py3-none-any.whl (179 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m179.6/179.6 kB[0m [31m37.0 MB/s[0m eta [36m0:00:00[0m
[?25hInstalling collected packages: fsspec, cramjam, fastparquet
Successfully installed cramjam-2.9.0 fastparquet-2024.11.0 fsspec-2024.10.0
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Requirement already satisfied: scipy in /usr/local/lib/python3.11/dist-packages (1.14.1)
Requirement already satisfied: numpy<2.3,>=1.23.5 in /usr/local/lib/python3.11/dist-packages (from scipy) (1.26.0)
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting seaborn
Obtaining dependency information for seaborn from https://files.pythonhosted.org/packages/83/11/00d3c3dfc25ad54e731d91449895a79e4bf2384dc3ac01809010ba88f6d5/seaborn-0.13.2-py3-none-any.whl.metadata
Downloading seaborn-0.13.2-py3-none-any.whl.metadata (5.4 kB)
Requirement already satisfied: numpy!=1.24.0,>=1.20 in /usr/local/lib/python3.11/dist-packages (from seaborn) (1.26.0)
Requirement already satisfied: pandas>=1.2 in /usr/local/lib/python3.11/dist-packages (from seaborn) (2.2.3)
Requirement already satisfied: matplotlib!=3.6.1,>=3.4 in /usr/local/lib/python3.11/dist-packages (from seaborn) (3.8.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.11/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (4.42.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (23.1)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.11/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (10.0.1)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/lib/python3/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.11/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.2->seaborn) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas>=1.2->seaborn) (2024.2)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.7->matplotlib!=3.6.1,>=3.4->seaborn) (1.16.0)
Downloading seaborn-0.13.2-py3-none-any.whl (294 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m294.9/294.9 kB[0m [31m4.7 MB/s[0m eta [36m0:00:00[0m [36m0:00:01[0m
[?25hInstalling collected packages: seaborn
Successfully installed seaborn-0.13.2
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting tqdm
Obtaining dependency information for tqdm from https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl.metadata
Downloading tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m57.7/57.7 kB[0m [31m2.3 MB/s[0m eta [36m0:00:00[0m
[?25hDownloading tqdm-4.67.1-py3-none-any.whl (78 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m78.5/78.5 kB[0m [31m13.0 MB/s[0m eta [36m0:00:00[0m
[?25hInstalling collected packages: tqdm
Successfully installed tqdm-4.67.1
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting pydot
Obtaining dependency information for pydot from https://files.pythonhosted.org/packages/3e/1b/ef569ac44598b6b24bc0f80d5ac4f811af59d3f0d0d23b0216e014c0ec33/pydot-3.0.3-py3-none-any.whl.metadata
Downloading pydot-3.0.3-py3-none-any.whl.metadata (10 kB)
Collecting pyparsing>=3.0.9 (from pydot)
Obtaining dependency information for pyparsing>=3.0.9 from https://files.pythonhosted.org/packages/be/ec/2eb3cd785efd67806c46c13a17339708ddc346cbb684eade7a6e6f79536a/pyparsing-3.2.0-py3-none-any.whl.metadata
Downloading pyparsing-3.2.0-py3-none-any.whl.metadata (5.0 kB)
Downloading pydot-3.0.3-py3-none-any.whl (35 kB)
Downloading pyparsing-3.2.0-py3-none-any.whl (106 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m106.9/106.9 kB[0m [31m3.4 MB/s[0m eta [36m0:00:00[0m
[?25hInstalling collected packages: pyparsing, pydot
Attempting uninstall: pyparsing
Found existing installation: pyparsing 2.4.7
Uninstalling pyparsing-2.4.7:
Successfully uninstalled pyparsing-2.4.7
Successfully installed pydot-3.0.3 pyparsing-3.2.0
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting tensorflow-io
Obtaining dependency information for tensorflow-io from https://files.pythonhosted.org/packages/f0/5e/f47443a14a00816fab54caf74599e2fcb34c05d6059e91f82126f8f4c68d/tensorflow_io-0.37.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading tensorflow_io-0.37.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (14 kB)
Collecting tensorflow-io-gcs-filesystem==0.37.1 (from tensorflow-io)
Obtaining dependency information for tensorflow-io-gcs-filesystem==0.37.1 from https://files.pythonhosted.org/packages/66/7f/e36ae148c2f03d61ca1bff24bc13a0fef6d6825c966abef73fc6f880a23b/tensorflow_io_gcs_filesystem-0.37.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading tensorflow_io_gcs_filesystem-0.37.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (14 kB)
Downloading tensorflow_io-0.37.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (49.6 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m49.6/49.6 MB[0m [31m22.1 MB/s[0m eta [36m0:00:00[0m00:01[0m00:01[0m
[?25hDownloading tensorflow_io_gcs_filesystem-0.37.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.1 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m5.1/5.1 MB[0m [31m61.0 MB/s[0m eta [36m0:00:00[0m00:01[0m00:01[0m
[?25hInstalling collected packages: tensorflow-io-gcs-filesystem, tensorflow-io
Attempting uninstall: tensorflow-io-gcs-filesystem
Found existing installation: tensorflow-io-gcs-filesystem 0.34.0
Uninstalling tensorflow-io-gcs-filesystem-0.34.0:
Successfully uninstalled tensorflow-io-gcs-filesystem-0.34.0
Successfully installed tensorflow-io-0.37.1 tensorflow-io-gcs-filesystem-0.37.1
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
Collecting tensorflow-addons
Obtaining dependency information for tensorflow-addons from https://files.pythonhosted.org/packages/24/94/80165946ec4986505cbfac29b5ae79544bfe2200d9d7883e1ad7c7342a55/tensorflow_addons-0.23.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading tensorflow_addons-0.23.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.8 kB)
Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from tensorflow-addons) (23.1)
Collecting typeguard<3.0.0,>=2.7 (from tensorflow-addons)
Obtaining dependency information for typeguard<3.0.0,>=2.7 from https://files.pythonhosted.org/packages/9a/bb/d43e5c75054e53efce310e79d63df0ac3f25e34c926be5dffb7d283fb2a8/typeguard-2.13.3-py3-none-any.whl.metadata
Downloading typeguard-2.13.3-py3-none-any.whl.metadata (3.6 kB)
Downloading tensorflow_addons-0.23.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (611 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m611.8/611.8 kB[0m [31m8.6 MB/s[0m eta [36m0:00:00[0m00:01[0m0:01[0m
[?25hDownloading typeguard-2.13.3-py3-none-any.whl (17 kB)
Installing collected packages: typeguard, tensorflow-addons
Successfully installed tensorflow-addons-0.23.0 typeguard-2.13.3
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m24.3.1[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpython3 -m pip install --upgrade pip[0m
In [2]:
import tensorflow as tf
import keras
print(f"Keras version: {keras.__version__}")
print(f"TensorFlow version: {tf.__version__}")
print(f"TensorFlow version: {tf.__version__}")
print(f"CUDA available: {tf.test.is_built_with_cuda()}")
print(f"GPU devices: {tf.config.list_physical_devices('GPU')}")
# GPU configuration
import tensorflow as tf
import os
# Limita la crescita della memoria GPU
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
try:
# Imposta la crescita di memoria dinamica
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
# Opzionalmente, limita la memoria GPU massima (uncomment se necessario)
# tf.config.experimental.set_virtual_device_configuration(
# gpus[0],
# [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024*4)] # 4GB
# )
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
print(e)
# Imposta le opzioni di logging
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # Riduce i messaggi di log
# Configura la modalità mista di precisione
tf.keras.mixed_precision.set_global_policy('float32')
# Imposta il seed per la riproducibilità
##tf.random.set_seed(42)2024-12-06 10:36:10.368632: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-12-06 10:36:10.368679: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-12-06 10:36:10.368726: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2024-12-06 10:36:10.377750: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Keras version: 2.14.0 TensorFlow version: 2.14.0 TensorFlow version: 2.14.0 CUDA available: True GPU devices: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')] 1 Physical GPUs, 1 Logical GPUs
2024-12-06 10:36:13.233242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1886] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 43404 MB memory: -> device: 0, name: NVIDIA L40, pci bus id: 0000:81:00.0, compute capability: 8.9
In [3]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
import tensorflow_addons as tfa
from datetime import datetime
import os
import joblib
import re
from typing import List
random_state_value = None
execute_name = datetime.now().strftime("%Y-%m-%d_%H-%M")
base_project_dir = './'
data_dir = '../../sources/'
models_project_dir = base_project_dir
os.makedirs(base_project_dir, exist_ok=True)
os.makedirs(models_project_dir, exist_ok=True)/usr/local/lib/python3.11/dist-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning: TensorFlow Addons (TFA) has ended development and introduction of new features. TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024. Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP). For more information see: https://github.com/tensorflow/addons/issues/2807 warnings.warn(
In [4]:
def clean_column_name(name: str) -> str:
"""
Rimuove caratteri speciali e spazi, converte in snake_case e abbrevia.
Parameters
----------
name : str
Nome della colonna da pulire
Returns
-------
str
Nome della colonna pulito
"""
# Rimuove caratteri speciali
name = re.sub(r'[^a-zA-Z0-9\s]', '', name)
# Converte in snake_case
name = name.lower().replace(' ', '_')
# Abbreviazioni comuni
abbreviations = {
'production': 'prod',
'percentage': 'pct',
'hectare': 'ha',
'tonnes': 't',
'litres': 'l',
'minimum': 'min',
'maximum': 'max',
'average': 'avg'
}
for full, abbr in abbreviations.items():
name = name.replace(full, abbr)
return name
def clean_column_names(df: pd.DataFrame) -> List[str]:
"""
Pulisce tutti i nomi delle colonne in un DataFrame.
Parameters
----------
df : pd.DataFrame
DataFrame con le colonne da pulire
Returns
-------
list
Lista dei nuovi nomi delle colonne puliti
"""
new_columns = []
for col in df.columns:
# Usa regex per separare le varietà
varieties = re.findall(r'([a-z]+)_([a-z_]+)', col)
if varieties:
new_columns.append(f"{varieties[0][0]}_{varieties[0][1]}")
else:
new_columns.append(col)
return new_columnsIn [5]:
def save_plot(plt, title, output_dir=f'{base_project_dir}/{execute_name}_plots'):
os.makedirs(output_dir, exist_ok=True)
filename = "".join(x for x in title if x.isalnum() or x in [' ', '-', '_']).rstrip()
filename = filename.replace(' ', '_').lower()
filepath = os.path.join(output_dir, f"{filename}.png")
plt.savefig(filepath, bbox_inches='tight', dpi=300)
print(f"Plot salvato come: {filepath}")
def encode_techniques(df, mapping_path=f'{data_dir}technique_mapping.joblib'):
if not os.path.exists(mapping_path):
raise FileNotFoundError(f"Mapping not found at {mapping_path}. Run create_technique_mapping first.")
technique_mapping = joblib.load(mapping_path)
# Trova tutte le colonne delle tecniche
tech_columns = [col for col in df.columns if col.endswith('_tech')]
# Applica il mapping a tutte le colonne delle tecniche
for col in tech_columns:
df[col] = df[col].str.lower().map(technique_mapping).fillna(0).astype(int)
return df
def decode_single_technique(technique_value, mapping_path=f'{data_dir}technique_mapping.joblib'):
if not os.path.exists(mapping_path):
raise FileNotFoundError(f"Mapping not found at {mapping_path}")
technique_mapping = joblib.load(mapping_path)
reverse_mapping = {v: k for k, v in technique_mapping.items()}
reverse_mapping[0] = ''
return reverse_mapping.get(technique_value, '')
def prepare_comparison_data(simulated_data, olive_varieties):
# Pulisci i nomi delle colonne
df = simulated_data.copy()
df.columns = clean_column_names(df)
df = encode_techniques(df)
all_varieties = olive_varieties['Varietà di Olive'].unique()
varieties = [clean_column_name(variety) for variety in all_varieties]
comparison_data = []
for variety in varieties:
olive_prod_col = next((col for col in df.columns if col.startswith(f'{variety}_') and col.endswith('_olive_prod')), None)
oil_prod_col = next((col for col in df.columns if col.startswith(f'{variety}_') and col.endswith('_avg_oil_prod')), None)
tech_col = next((col for col in df.columns if col.startswith(f'{variety}_') and col.endswith('_tech')), None)
water_need_col = next((col for col in df.columns if col.startswith(f'{variety}_') and col.endswith('_water_need')), None)
if olive_prod_col and oil_prod_col and tech_col and water_need_col:
variety_data = df[[olive_prod_col, oil_prod_col, tech_col, water_need_col]]
variety_data = variety_data[variety_data[tech_col] != 0] # Esclude le righe dove la tecnica è 0
if not variety_data.empty:
avg_olive_prod = pd.to_numeric(variety_data[olive_prod_col], errors='coerce').mean()
avg_oil_prod = pd.to_numeric(variety_data[oil_prod_col], errors='coerce').mean()
avg_water_need = pd.to_numeric(variety_data[water_need_col], errors='coerce').mean()
efficiency = avg_oil_prod / avg_olive_prod if avg_olive_prod > 0 else 0
water_efficiency = avg_oil_prod / avg_water_need if avg_water_need > 0 else 0
comparison_data.append({
'Variety': variety,
'Avg Olive Production (kg/ha)': avg_olive_prod,
'Avg Oil Production (L/ha)': avg_oil_prod,
'Avg Water Need (m³/ha)': avg_water_need,
'Oil Efficiency (L/kg)': efficiency,
'Water Efficiency (L oil/m³ water)': water_efficiency
})
return pd.DataFrame(comparison_data)
def plot_variety_comparison(comparison_data, metric):
plt.figure(figsize=(12, 6))
bars = plt.bar(comparison_data['Variety'], comparison_data[metric])
plt.title(f'Comparison of {metric} across Olive Varieties')
plt.xlabel('Variety')
plt.ylabel(metric)
plt.xticks(rotation=45, ha='right')
for bar in bars:
height = bar.get_height()
plt.text(bar.get_x() + bar.get_width() / 2., height,
f'{height:.2f}',
ha='center', va='bottom')
plt.tight_layout()
plt.show()
save_plot(plt, f'variety_comparison_{metric.lower().replace(" ", "_").replace("/", "_").replace("(", "").replace(")", "")}')
plt.close()
def plot_efficiency_vs_production(comparison_data):
plt.figure(figsize=(10, 6))
plt.scatter(comparison_data['Avg Olive Production (kg/ha)'],
comparison_data['Oil Efficiency (L/kg)'],
s=100)
for i, row in comparison_data.iterrows():
plt.annotate(row['Variety'],
(row['Avg Olive Production (kg/ha)'], row['Oil Efficiency (L/kg)']),
xytext=(5, 5), textcoords='offset points')
plt.title('Oil Efficiency vs Olive Production by Variety')
plt.xlabel('Average Olive Production (kg/ha)')
plt.ylabel('Oil Efficiency (L oil / kg olives)')
plt.tight_layout()
save_plot(plt, 'efficiency_vs_production')
plt.close()
def plot_water_efficiency_vs_production(comparison_data):
plt.figure(figsize=(10, 6))
plt.scatter(comparison_data['Avg Olive Production (kg/ha)'],
comparison_data['Water Efficiency (L oil/m³ water)'],
s=100)
for i, row in comparison_data.iterrows():
plt.annotate(row['Variety'],
(row['Avg Olive Production (kg/ha)'], row['Water Efficiency (L oil/m³ water)']),
xytext=(5, 5), textcoords='offset points')
plt.title('Water Efficiency vs Olive Production by Variety')
plt.xlabel('Average Olive Production (kg/ha)')
plt.ylabel('Water Efficiency (L oil / m³ water)')
plt.tight_layout()
plt.show()
save_plot(plt, 'water_efficiency_vs_production')
plt.close()
def plot_water_need_vs_oil_production(comparison_data):
plt.figure(figsize=(10, 6))
plt.scatter(comparison_data['Avg Water Need (m³/ha)'],
comparison_data['Avg Oil Production (L/ha)'],
s=100)
for i, row in comparison_data.iterrows():
plt.annotate(row['Variety'],
(row['Avg Water Need (m³/ha)'], row['Avg Oil Production (L/ha)']),
xytext=(5, 5), textcoords='offset points')
plt.title('Oil Production vs Water Need by Variety')
plt.xlabel('Average Water Need (m³/ha)')
plt.ylabel('Average Oil Production (L/ha)')
plt.tight_layout()
plt.show()
save_plot(plt, 'water_need_vs_oil_production')
plt.close()
def analyze_by_technique(simulated_data, olive_varieties):
# Pulisci i nomi delle colonne
df = simulated_data.copy()
df.columns = clean_column_names(df)
df = encode_techniques(df)
all_varieties = olive_varieties['Varietà di Olive'].unique()
varieties = [clean_column_name(variety) for variety in all_varieties]
technique_data = []
for variety in varieties:
olive_prod_col = next((col for col in df.columns if col.startswith(f'{variety}_') and col.endswith('_olive_prod')), None)
oil_prod_col = next((col for col in df.columns if col.startswith(f'{variety}_') and col.endswith('_avg_oil_prod')), None)
tech_col = next((col for col in df.columns if col.startswith(f'{variety}_') and col.endswith('_tech')), None)
water_need_col = next((col for col in df.columns if col.startswith(f'{variety}_') and col.endswith('_water_need')), None)
if olive_prod_col and oil_prod_col and tech_col and water_need_col:
variety_data = df[[olive_prod_col, oil_prod_col, tech_col, water_need_col]]
variety_data = variety_data[variety_data[tech_col] != 0]
if not variety_data.empty:
for tech in variety_data[tech_col].unique():
tech_data = variety_data[variety_data[tech_col] == tech]
avg_olive_prod = pd.to_numeric(tech_data[olive_prod_col], errors='coerce').mean()
avg_oil_prod = pd.to_numeric(tech_data[oil_prod_col], errors='coerce').mean()
avg_water_need = pd.to_numeric(tech_data[water_need_col], errors='coerce').mean()
efficiency = avg_oil_prod / avg_olive_prod if avg_olive_prod > 0 else 0
water_efficiency = avg_oil_prod / avg_water_need if avg_water_need > 0 else 0
technique_data.append({
'Variety': variety,
'Technique': tech,
'Technique String': decode_single_technique(tech),
'Avg Olive Production (kg/ha)': avg_olive_prod,
'Avg Oil Production (L/ha)': avg_oil_prod,
'Avg Water Need (m³/ha)': avg_water_need,
'Oil Efficiency (L/kg)': efficiency,
'Water Efficiency (L oil/m³ water)': water_efficiency
})
return pd.DataFrame(technique_data)In [6]:
def calculate_real_error(model, test_data, test_targets, scaler_y):
# Fare predizioni
predictions = model.predict(test_data)
# Denormalizzare predizioni e target
predictions_real = scaler_y.inverse_transform(predictions)
targets_real = scaler_y.inverse_transform(test_targets)
# Calcolare errore percentuale per ogni target
percentage_errors = []
absolute_errors = []
for i in range(predictions_real.shape[1]):
mae = np.mean(np.abs(predictions_real[:, i] - targets_real[:, i]))
mape = np.mean(np.abs((predictions_real[:, i] - targets_real[:, i]) / targets_real[:, i])) * 100
percentage_errors.append(mape)
absolute_errors.append(mae)
# Stampa risultati per ogni target
target_names = ['olive_prod', 'min_oil_prod', 'max_oil_prod', 'avg_oil_prod', 'total_water_need']
print("\nErrori per target:")
print("-" * 50)
for i, target in enumerate(target_names):
print(f"{target}:")
print(f"MAE assoluto: {absolute_errors[i]:.2f}")
print(f"Errore percentuale medio: {percentage_errors[i]:.2f}%")
print(f"Precisione: {100 - percentage_errors[i]:.2f}%")
print("-" * 50)
return percentage_errors, absolute_errorsIn [7]:
simulated_data = pd.read_parquet(f"{data_dir}olive_training_dataset.parquet")
olive_varieties = pd.read_parquet(f"{data_dir}olive_varieties.parquet")
# Esecuzione dell'analisi
comparison_data = prepare_comparison_data(simulated_data, olive_varieties)
# Genera i grafici
plot_variety_comparison(comparison_data, 'Avg Olive Production (kg/ha)')
plot_variety_comparison(comparison_data, 'Avg Oil Production (L/ha)')
plot_variety_comparison(comparison_data, 'Avg Water Need (m³/ha)')
plot_variety_comparison(comparison_data, 'Oil Efficiency (L/kg)')
plot_variety_comparison(comparison_data, 'Water Efficiency (L oil/m³ water)')
plot_efficiency_vs_production(comparison_data)
plot_water_efficiency_vs_production(comparison_data)
plot_water_need_vs_oil_production(comparison_data)
# Analisi per tecnica
technique_data = analyze_by_technique(simulated_data, olive_varieties)
print(technique_data)
# Stampa un sommario statistico
print("Comparison by Variety:")
print(comparison_data.set_index('Variety'))
print("\nBest Varieties by Water Efficiency:")
print(comparison_data.sort_values('Water Efficiency (L oil/m³ water)', ascending=False).head())
Plot salvato come: .//2024-12-06_10-36_plots/variety_comparison_avg_olive_production_kg_ha.png
Plot salvato come: .//2024-12-06_10-36_plots/variety_comparison_avg_oil_production_l_ha.png
Plot salvato come: .//2024-12-06_10-36_plots/variety_comparison_avg_water_need_m³_ha.png
Plot salvato come: .//2024-12-06_10-36_plots/variety_comparison_oil_efficiency_l_kg.png
Plot salvato come: .//2024-12-06_10-36_plots/variety_comparison_water_efficiency_l_oil_m³_water.png Plot salvato come: .//2024-12-06_10-36_plots/efficiency_vs_production.png
Plot salvato come: .//2024-12-06_10-36_plots/water_efficiency_vs_production.png
Plot salvato come: .//2024-12-06_10-36_plots/water_need_vs_oil_production.png
Variety Technique Technique String \
0 nocellara_delletna 3 tradizionale
1 nocellara_delletna 1 intensiva
2 nocellara_delletna 2 superintensiva
3 leccino 1 intensiva
4 leccino 2 superintensiva
5 leccino 3 tradizionale
6 frantoio 2 superintensiva
7 frantoio 3 tradizionale
8 frantoio 1 intensiva
9 coratina 1 intensiva
10 coratina 2 superintensiva
11 coratina 3 tradizionale
12 taggiasca 3 tradizionale
13 taggiasca 2 superintensiva
14 taggiasca 1 intensiva
15 pendolino 1 intensiva
16 pendolino 2 superintensiva
17 pendolino 3 tradizionale
18 moraiolo 2 superintensiva
19 moraiolo 1 intensiva
20 moraiolo 3 tradizionale
Avg Olive Production (kg/ha) Avg Oil Production (L/ha) \
0 9564.638687 2088.362004
1 13699.079622 2991.183032
2 17826.710664 3892.059753
3 16432.379678 3229.053194
4 20528.499013 4033.942398
5 10937.982122 2149.449585
6 24621.040119 6047.876212
7 13740.739760 3375.103688
8 20550.900635 5047.942655
9 16429.706879 4215.265516
10 19164.700743 4916.649709
11 12318.510310 3160.037128
12 6839.506230 1381.247995
13 16433.741502 3319.210170
14 10968.603159 2215.371493
15 13705.431414 2468.678455
16 19183.689269 3455.879324
17 10960.549241 1974.357984
18 17793.971752 3885.415851
19 13144.222436 2870.020002
20 8765.195655 1913.745255
Avg Water Need (m³/ha) Oil Efficiency (L/kg) \
0 32997.227891 0.218342
1 33079.012125 0.218349
2 33118.708645 0.218327
3 25013.303736 0.196506
4 24989.459147 0.196504
5 24981.219100 0.196512
6 28874.473543 0.245639
7 29003.452741 0.245628
8 28921.261327 0.245631
9 38270.638622 0.256564
10 38264.650562 0.256547
11 38253.676395 0.256528
12 26219.134374 0.201951
13 26253.317778 0.201975
14 26284.027794 0.201974
15 26154.359691 0.180124
16 26153.199618 0.180147
17 26152.823801 0.180133
18 32561.911109 0.218356
19 32577.899255 0.218348
20 32594.860153 0.218335
Water Efficiency (L oil/m³ water)
0 0.063289
1 0.090425
2 0.117518
3 0.129093
4 0.161426
5 0.086043
6 0.209454
7 0.116369
8 0.174541
9 0.110144
10 0.128491
11 0.082607
12 0.052681
13 0.126430
14 0.084286
15 0.094389
16 0.132140
17 0.075493
18 0.119324
19 0.088097
20 0.058713
Comparison by Variety:
Avg Olive Production (kg/ha) Avg Oil Production (L/ha) \
Variety
nocellara_delletna 13696.683690 2990.507461
leccino 15971.162702 3138.439782
frantoio 19648.631813 4826.360700
coratina 15974.164423 4098.136472
taggiasca 11412.636779 2305.011278
pendolino 14617.432649 2633.129635
moraiolo 13232.961913 2889.399172
Avg Water Need (m³/ha) Oil Efficiency (L/kg) \
Variety
nocellara_delletna 33064.983905 0.218338
leccino 24994.676451 0.196507
frantoio 28932.932409 0.245633
coratina 38262.995517 0.256548
taggiasca 26252.184893 0.201970
pendolino 26153.461822 0.180136
moraiolo 32578.228327 0.218349
Water Efficiency (L oil/m³ water)
Variety
nocellara_delletna 0.090443
leccino 0.125564
frantoio 0.166812
coratina 0.107104
taggiasca 0.087803
pendolino 0.100680
moraiolo 0.088691
Best Varieties by Water Efficiency:
Variety Avg Olive Production (kg/ha) \
2 frantoio 19648.631813
1 leccino 15971.162702
3 coratina 15974.164423
5 pendolino 14617.432649
0 nocellara_delletna 13696.683690
Avg Oil Production (L/ha) Avg Water Need (m³/ha) Oil Efficiency (L/kg) \
2 4826.360700 28932.932409 0.245633
1 3138.439782 24994.676451 0.196507
3 4098.136472 38262.995517 0.256548
5 2633.129635 26153.461822 0.180136
0 2990.507461 33064.983905 0.218338
Water Efficiency (L oil/m³ water)
2 0.166812
1 0.125564
3 0.107104
5 0.100680
0 0.090443
In [8]:
def prepare_transformer_data(df, olive_varieties_df):
# Crea una copia del DataFrame per evitare modifiche all'originale
df = df.copy()
# Ordina per zona e anno
df = df.sort_values(['zone', 'year'])
# Definisci le feature
temporal_features = ['temp_mean', 'precip_sum', 'solar_energy_sum']
static_features = ['ha'] # Feature statiche base
target_features = ['olive_prod', 'min_oil_prod', 'max_oil_prod', 'avg_oil_prod', 'total_water_need']
# Ottieni le varietà pulite
all_varieties = olive_varieties_df['Varietà di Olive'].unique()
varieties = [clean_column_name(variety) for variety in all_varieties]
# Crea la struttura delle feature per ogni varietà
variety_features = [
'tech', 'pct', 'prod_t_ha', 'oil_prod_t_ha', 'oil_prod_l_ha',
'min_yield_pct', 'max_yield_pct', 'min_oil_prod_l_ha', 'max_oil_prod_l_ha',
'avg_oil_prod_l_ha', 'l_per_t', 'min_l_per_t', 'max_l_per_t', 'avg_l_per_t'
]
# Prepara dizionari per le nuove colonne
new_columns = {}
# Prepara le feature per ogni varietà
for variety in varieties:
# Feature esistenti
for feature in variety_features:
col_name = f"{variety}_{feature}"
if col_name in df.columns:
if feature != 'tech': # Non includere la colonna tech direttamente
static_features.append(col_name)
# Feature binarie per le tecniche di coltivazione
for technique in ['tradizionale', 'intensiva', 'superintensiva']:
col_name = f"{variety}_{technique}"
new_columns[col_name] = df[f"{variety}_tech"].notna() & (
df[f"{variety}_tech"].str.lower() == technique
).fillna(False)
static_features.append(col_name)
# Aggiungi tutte le nuove colonne in una volta sola
new_df = pd.concat([df] + [pd.Series(v, name=k) for k, v in new_columns.items()], axis=1)
# Ordiniamo per zona e anno per mantenere la continuità temporale
df_sorted = new_df.sort_values(['zone', 'year'])
# Definiamo la dimensione della finestra temporale
window_size = 41
# Liste per raccogliere i dati
temporal_sequences = []
static_features_list = []
targets_list = []
# Iteriamo per ogni zona
for zone in df_sorted['zone'].unique():
zone_data = df_sorted[df_sorted['zone'] == zone].reset_index(drop=True)
if len(zone_data) >= window_size: # Verifichiamo che ci siano abbastanza dati
# Creiamo sequenze temporali scorrevoli
for i in range(len(zone_data) - window_size + 1):
# Sequenza temporale
temporal_window = zone_data.iloc[i:i + window_size][temporal_features].values
# Verifichiamo che non ci siano valori NaN
if not np.isnan(temporal_window).any():
temporal_sequences.append(temporal_window)
# Feature statiche (prendiamo quelle dell'ultimo timestep della finestra)
static_features_list.append(zone_data.iloc[i + window_size - 1][static_features].values)
# Target (prendiamo quelli dell'ultimo timestep della finestra)
targets_list.append(zone_data.iloc[i + window_size - 1][target_features].values)
# Convertiamo in array numpy
X_temporal = np.array(temporal_sequences)
X_static = np.array(static_features_list)
y = np.array(targets_list)
print(f"Dataset completo - Temporal: {X_temporal.shape}, Static: {X_static.shape}, Target: {y.shape}")
# Split dei dati (usando indici casuali per una migliore distribuzione)
indices = np.random.permutation(len(X_temporal))
#train_idx = int(len(indices) * 0.7) # 70% training
#val_idx = int(len(indices) * 0.85) # 15% validation
# Il resto rimane 15% test
train_idx = int(len(indices) * 0.65) # 65% training
val_idx = int(len(indices) * 0.85) # 20% validation
# Il resto rimane 15% test
#train_idx = int(len(indices) * 0.60) # 60% training
#val_idx = int(len(indices) * 0.85) # 25% validation
# Il resto rimane 15% test
train_indices = indices[:train_idx]
val_indices = indices[train_idx:val_idx]
test_indices = indices[val_idx:]
# Split dei dati
X_temporal_train = X_temporal[train_indices]
X_temporal_val = X_temporal[val_indices]
X_temporal_test = X_temporal[test_indices]
X_static_train = X_static[train_indices]
X_static_val = X_static[val_indices]
X_static_test = X_static[test_indices]
y_train = y[train_indices]
y_val = y[val_indices]
y_test = y[test_indices]
# Standardizzazione
scaler_temporal = StandardScaler()
scaler_static = StandardScaler()
scaler_y = StandardScaler()
# Standardizzazione dei dati temporali
X_temporal_train = scaler_temporal.fit_transform(X_temporal_train.reshape(-1, len(temporal_features))).reshape(X_temporal_train.shape)
X_temporal_val = scaler_temporal.transform(X_temporal_val.reshape(-1, len(temporal_features))).reshape(X_temporal_val.shape)
X_temporal_test = scaler_temporal.transform(X_temporal_test.reshape(-1, len(temporal_features))).reshape(X_temporal_test.shape)
# Standardizzazione dei dati statici
X_static_train = scaler_static.fit_transform(X_static_train)
X_static_val = scaler_static.transform(X_static_val)
X_static_test = scaler_static.transform(X_static_test)
# Standardizzazione dei target
y_train = scaler_y.fit_transform(y_train)
y_val = scaler_y.transform(y_val)
y_test = scaler_y.transform(y_test)
print("\nShape dopo lo split e standardizzazione:")
print(f"Train - Temporal: {X_temporal_train.shape}, Static: {X_static_train.shape}, Target: {y_train.shape}")
print(f"Val - Temporal: {X_temporal_val.shape}, Static: {X_static_val.shape}, Target: {y_val.shape}")
print(f"Test - Temporal: {X_temporal_test.shape}, Static: {X_static_test.shape}, Target: {y_test.shape}")
# Prepara i dizionari di input
train_data = {'temporal': X_temporal_train, 'static': X_static_train}
val_data = {'temporal': X_temporal_val, 'static': X_static_val}
test_data = {'temporal': X_temporal_test, 'static': X_static_test}
joblib.dump(scaler_temporal, os.path.join(base_project_dir, f'{execute_name}_scaler_temporal.joblib'))
joblib.dump(scaler_static, os.path.join(base_project_dir, f'{execute_name}_scaler_static.joblib'))
joblib.dump(scaler_y, os.path.join(base_project_dir, f'{execute_name}_scaler_y.joblib'))
return (train_data, y_train), (val_data, y_val), (test_data, y_test), (scaler_temporal, scaler_static, scaler_y)In [10]:
simulated_data = pd.read_parquet(f"{data_dir}olive_training_dataset.parquet")
olive_varieties = pd.read_parquet(f"{data_dir}olive_varieties.parquet")
(train_data, train_targets), (val_data, val_targets), (test_data, test_targets), scalers = prepare_transformer_data(simulated_data, olive_varieties)
scaler_temporal, scaler_static, scaler_y = scalers
print("Temporal data shape:", train_data['temporal'].shape)
print("Static data shape:", train_data['static'].shape)
print("Target shape:", train_targets.shape)Dataset completo - Temporal: (3920000, 41, 3), Static: (3920000, 113), Target: (3920000, 5) Shape dopo lo split e standardizzazione: Train - Temporal: (2548000, 41, 3), Static: (2548000, 113), Target: (2548000, 5) Val - Temporal: (784000, 41, 3), Static: (784000, 113), Target: (784000, 5) Test - Temporal: (588000, 41, 3), Static: (588000, 113), Target: (588000, 5) Temporal data shape: (2548000, 41, 3) Static data shape: (2548000, 113) Target shape: (2548000, 5)
In [11]:
@keras.saving.register_keras_serializable()
class DataAugmentation(tf.keras.layers.Layer):
"""Custom layer per l'augmentation dei dati"""
def __init__(self, noise_stddev=0.03, **kwargs):
super().__init__(**kwargs)
self.noise_stddev = noise_stddev
def call(self, inputs, training=None):
if training:
return inputs + tf.random.normal(
shape=tf.shape(inputs),
mean=0.0,
stddev=self.noise_stddev
)
return inputs
def get_config(self):
config = super().get_config()
config.update({"noise_stddev": self.noise_stddev})
return config
@keras.saving.register_keras_serializable()
class PositionalEncoding(tf.keras.layers.Layer):
"""Custom layer per l'encoding posizionale"""
def __init__(self, d_model, **kwargs):
super().__init__(**kwargs)
self.d_model = d_model
def build(self, input_shape):
_, seq_length, _ = input_shape
# Crea la matrice di encoding posizionale
position = tf.range(seq_length, dtype=tf.float32)[:, tf.newaxis]
div_term = tf.exp(
tf.range(0, self.d_model, 2, dtype=tf.float32) *
(-tf.math.log(10000.0) / self.d_model)
)
# Calcola sin e cos
pos_encoding = tf.zeros((1, seq_length, self.d_model))
pos_encoding_even = tf.sin(position * div_term)
pos_encoding_odd = tf.cos(position * div_term)
# Assegna i valori alle posizioni pari e dispari
pos_encoding = tf.concat(
[tf.expand_dims(pos_encoding_even, -1),
tf.expand_dims(pos_encoding_odd, -1)],
axis=-1
)
pos_encoding = tf.reshape(pos_encoding, (1, seq_length, -1))
pos_encoding = pos_encoding[:, :, :self.d_model]
# Salva l'encoding come peso non trainabile
self.pos_encoding = self.add_weight(
shape=(1, seq_length, self.d_model),
initializer=tf.keras.initializers.Constant(pos_encoding),
trainable=False,
name='positional_encoding'
)
super().build(input_shape)
def call(self, inputs):
# Broadcast l'encoding posizionale sul batch
batch_size = tf.shape(inputs)[0]
pos_encoding_tiled = tf.tile(self.pos_encoding, [batch_size, 1, 1])
return inputs + pos_encoding_tiled
def get_config(self):
config = super().get_config()
config.update({"d_model": self.d_model})
return config
@keras.saving.register_keras_serializable()
class WarmUpLearningRateSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
"""Custom learning rate schedule with linear warmup and exponential decay."""
def __init__(self, initial_learning_rate=1e-3, warmup_steps=500, decay_steps=5000):
super().__init__()
self.initial_learning_rate = initial_learning_rate
self.warmup_steps = warmup_steps
self.decay_steps = decay_steps
def __call__(self, step):
warmup_pct = tf.cast(step, tf.float32) / self.warmup_steps
warmup_lr = self.initial_learning_rate * warmup_pct
decay_factor = tf.pow(0.1, tf.cast(step, tf.float32) / self.decay_steps)
decayed_lr = self.initial_learning_rate * decay_factor
return tf.where(step < self.warmup_steps, warmup_lr, decayed_lr)
def get_config(self):
return {
'initial_learning_rate': self.initial_learning_rate,
'warmup_steps': self.warmup_steps,
'decay_steps': self.decay_steps
}
def create_olive_oil_transformer(temporal_shape, static_shape, num_outputs,
d_model=128, num_heads=8, ff_dim=256,
num_transformer_blocks=4, mlp_units=None,
dropout=0.2):
"""
Crea un transformer per la predizione della produzione di olio d'oliva.
"""
# Input layers
if mlp_units is None:
mlp_units = [256, 128, 64]
temporal_input = tf.keras.layers.Input(shape=temporal_shape, name='temporal')
static_input = tf.keras.layers.Input(shape=static_shape, name='static')
# === TEMPORAL PATH ===
x = tf.keras.layers.LayerNormalization(epsilon=1e-6)(temporal_input)
x = DataAugmentation()(x)
# Temporal projection
x = tf.keras.layers.Dense(
d_model // 2,
activation='swish',
kernel_regularizer=tf.keras.regularizers.l2(1e-5)
)(x)
x = tf.keras.layers.Dropout(dropout)(x)
x = tf.keras.layers.Dense(
d_model,
activation='swish',
kernel_regularizer=tf.keras.regularizers.l2(1e-5)
)(x)
# Positional encoding
x = PositionalEncoding(d_model)(x)
# Transformer blocks
skip_connection = x
for _ in range(num_transformer_blocks):
# Self-attention
attention_output = tf.keras.layers.MultiHeadAttention(
num_heads=num_heads,
key_dim=d_model // num_heads,
value_dim=d_model // num_heads
)(x, x)
attention_output = tf.keras.layers.Dropout(dropout)(attention_output)
# Residual connection con pesi addestrabili
residual_weights = tf.keras.layers.Dense(d_model, activation='sigmoid')(x)
x = tfa.layers.StochasticDepth(survival_probability=0.3)([x, residual_weights * attention_output])
x = tf.keras.layers.LayerNormalization(epsilon=1e-6)(x)
# Feed-forward network
ffn = tf.keras.layers.Dense(ff_dim, activation="swish")(x)
ffn = tf.keras.layers.Dropout(dropout)(ffn)
ffn = tf.keras.layers.Dense(d_model)(ffn)
ffn = tf.keras.layers.Dropout(dropout)(ffn)
# Second residual connection
x = tfa.layers.StochasticDepth()([x, ffn])
x = tf.keras.layers.LayerNormalization(epsilon=1e-6)(x)
# Add final skip connection
x = tfa.layers.StochasticDepth(survival_probability=0.5)([x, skip_connection])
# Temporal pooling
attention_pooled = tf.keras.layers.MultiHeadAttention(
num_heads=num_heads,
key_dim=d_model // 4
)(x, x)
attention_pooled = tf.keras.layers.GlobalAveragePooling1D()(attention_pooled)
# Additional pooling operations
avg_pooled = tf.keras.layers.GlobalAveragePooling1D()(x)
max_pooled = tf.keras.layers.GlobalMaxPooling1D()(x)
# Combine pooling results
temporal_features = tf.keras.layers.Concatenate()(
[attention_pooled, avg_pooled, max_pooled]
)
# === STATIC PATH ===
static_features = tf.keras.layers.LayerNormalization(epsilon=1e-6)(static_input)
for units in [256, 128, 64]:
static_features = tf.keras.layers.Dense(
units,
activation='swish',
kernel_regularizer=tf.keras.regularizers.l2(1e-5)
)(static_features)
static_features = tf.keras.layers.Dropout(dropout)(static_features)
# === FEATURE FUSION ===
combined = tf.keras.layers.Concatenate()([temporal_features, static_features])
# === MLP HEAD ===
x = combined
for units in mlp_units:
x = tf.keras.layers.BatchNormalization()(x)
x = tf.keras.layers.Dense(
units,
activation="swish",
kernel_regularizer=tf.keras.regularizers.l2(1e-5)
)(x)
x = tf.keras.layers.Dropout(dropout)(x)
# Output layer
outputs = tf.keras.layers.Dense(
num_outputs,
activation='linear',
kernel_regularizer=tf.keras.regularizers.l2(1e-5)
)(x)
# Create model
model = tf.keras.Model(
inputs={'temporal': temporal_input, 'static': static_input},
outputs=outputs,
name='OilTransformer'
)
return model
def create_transformer_callbacks(target_names, val_data, val_targets):
"""
Crea i callbacks per il training del modello.
Parameters:
-----------
target_names : list
Lista dei nomi dei target per il monitoraggio specifico
val_data : dict
Dati di validazione
val_targets : array
Target di validazione
Returns:
--------
list
Lista dei callbacks configurati
"""
# Custom Metric per target specifici
class TargetSpecificMetric(tf.keras.callbacks.Callback):
def __init__(self, validation_data, target_names):
super().__init__()
self.validation_data = validation_data
self.target_names = target_names
def on_epoch_end(self, epoch, logs={}):
x_val, y_val = self.validation_data
y_pred = self.model.predict(x_val, verbose=0)
for i, name in enumerate(self.target_names):
mae = np.mean(np.abs(y_val[:, i] - y_pred[:, i]))
logs[f'val_{name}_mae'] = mae
callbacks = [
# Early Stopping
tf.keras.callbacks.EarlyStopping(
monitor='val_loss',
patience=20,
restore_best_weights=True,
min_delta=0.0005,
mode='min'
),
# Model Checkpoint
tf.keras.callbacks.ModelCheckpoint(
filepath=f'{execute_name}_best_oil_model.h5',
monitor='val_loss',
save_best_only=True,
mode='min',
save_weights_only=True
),
# Metric per target specifici
TargetSpecificMetric(
validation_data=(val_data, val_targets),
target_names=target_names
),
# Reduce LR on Plateau
tf.keras.callbacks.ReduceLROnPlateau(
monitor='val_loss',
factor=0.5,
patience=10,
min_lr=1e-6,
verbose=1
),
# TensorBoard logging
tf.keras.callbacks.TensorBoard(
log_dir=f'./logs_{execute_name}',
histogram_freq=1,
write_graph=True,
update_freq='epoch'
)
]
return callbacks
def compile_model(model, learning_rate=1e-3):
"""
Compila il modello con le impostazioni standard.
"""
lr_schedule = WarmUpLearningRateSchedule(
initial_learning_rate=learning_rate,
warmup_steps=500,
decay_steps=5000
)
model.compile(
optimizer=tf.keras.optimizers.AdamW(
learning_rate=lr_schedule,
weight_decay=0.01
),
loss=tf.keras.losses.Huber(),
metrics=['mae']
)
return model
def setup_transformer_training(train_data, train_targets, val_data, val_targets):
"""
Configura e prepara il transformer con dimensioni dinamiche basate sui dati.
"""
# Estrai le shape dai dati
temporal_shape = (train_data['temporal'].shape[1], train_data['temporal'].shape[2])
static_shape = (train_data['static'].shape[1],)
num_outputs = train_targets.shape[1]
print(f"Shape rilevate:")
print(f"- Temporal shape: {temporal_shape}")
print(f"- Static shape: {static_shape}")
print(f"- Numero di output: {num_outputs}")
# Target names basati sul numero di output
target_names = ['olive_prod', 'min_oil_prod', 'max_oil_prod', 'avg_oil_prod', 'total_water_need']
# Assicurati che il numero di target names corrisponda al numero di output
assert len(target_names) == num_outputs, \
f"Il numero di target names ({len(target_names)}) non corrisponde al numero di output ({num_outputs})"
# Crea il modello con le dimensioni rilevate
model = create_olive_oil_transformer(
temporal_shape=temporal_shape,
static_shape=static_shape,
num_outputs=num_outputs
)
# Compila il modello
model = compile_model(model)
# Crea i callbacks
callbacks = create_transformer_callbacks(target_names, val_data, val_targets)
return model, callbacks, target_names
def train_transformer(train_data, train_targets, val_data, val_targets, epochs=150, batch_size=64, save_name='final_model'):
"""
Funzione principale per l'addestramento del transformer con ottimizzazioni.
"""
# Conversione dei dati in tf.data.Dataset per una gestione più efficiente della memoria
train_dataset = tf.data.Dataset.from_tensor_slices((train_data, train_targets))\
.cache()\
.shuffle(buffer_size=1024)\
.batch(batch_size)\
.prefetch(tf.data.AUTOTUNE)
val_dataset = tf.data.Dataset.from_tensor_slices((val_data, val_targets))\
.cache()\
.batch(batch_size)\
.prefetch(tf.data.AUTOTUNE)
# Setup del modello
strategy = tf.distribute.MirroredStrategy() if len(tf.config.list_physical_devices('GPU')) > 1 else tf.distribute.get_strategy()
with strategy.scope():
model, callbacks, target_names = setup_transformer_training(
train_data, train_targets, val_data, val_targets
)
# Mostra il summary del modello
model.summary()
try:
keras.utils.plot_model(model, f"{execute_name}_{save_name}.png", show_shapes=True)
except Exception as e:
print(f"Warning: Could not create model plot: {e}")
# Training con gestione degli errori
try:
history = model.fit(
train_dataset,
validation_data=val_dataset,
epochs=epochs,
callbacks=callbacks,
verbose=1,
workers=4,
use_multiprocessing=True
)
except tf.errors.ResourceExhaustedError:
print("Memoria GPU esaurita, riprovo con batch size più piccolo...")
# Riprova con batch size più piccolo
batch_size = batch_size // 2
train_dataset = train_dataset.unbatch().batch(batch_size)
val_dataset = val_dataset.unbatch().batch(batch_size)
history = model.fit(
train_dataset,
validation_data=val_dataset,
epochs=epochs,
callbacks=callbacks,
verbose=1
)
# Salva il modello finale
try:
save_path = f'{execute_name}_{save_name}.keras'
model.save(save_path, save_format='keras')
os.makedirs(f'{execute_name}/weights', exist_ok=True)
model.save_weights(f'{execute_name}/weights')
print(f"\nModello salvato in: {save_path}")
except Exception as e:
print(f"Warning: Could not save model: {e}")
return model, historyIn [12]:
model, history = train_transformer(train_data, train_targets, val_data, val_targets, 150, 512)Shape rilevate: - Temporal shape: (41, 3) - Static shape: (113,) - Numero di output: 5
2024-12-06 11:43:09.026945: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
Model: "OilTransformer"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
temporal (InputLayer) [(None, 41, 3)] 0 []
layer_normalization (Layer (None, 41, 3) 6 ['temporal[0][0]']
Normalization)
data_augmentation (DataAug (None, 41, 3) 0 ['layer_normalization[0][0]']
mentation)
dense (Dense) (None, 41, 64) 256 ['data_augmentation[0][0]']
dropout (Dropout) (None, 41, 64) 0 ['dense[0][0]']
dense_1 (Dense) (None, 41, 128) 8320 ['dropout[0][0]']
positional_encoding (Posit (None, 41, 128) 5248 ['dense_1[0][0]']
ionalEncoding)
multi_head_attention (Mult (None, 41, 128) 66048 ['positional_encoding[0][0]',
iHeadAttention) 'positional_encoding[0][0]']
dense_2 (Dense) (None, 41, 128) 16512 ['positional_encoding[0][0]']
dropout_1 (Dropout) (None, 41, 128) 0 ['multi_head_attention[0][0]']
tf.math.multiply (TFOpLamb (None, 41, 128) 0 ['dense_2[0][0]',
da) 'dropout_1[0][0]']
stochastic_depth (Stochast (None, 41, 128) 0 ['positional_encoding[0][0]',
icDepth) 'tf.math.multiply[0][0]']
layer_normalization_1 (Lay (None, 41, 128) 256 ['stochastic_depth[0][0]']
erNormalization)
dense_3 (Dense) (None, 41, 256) 33024 ['layer_normalization_1[0][0]'
]
dropout_2 (Dropout) (None, 41, 256) 0 ['dense_3[0][0]']
dense_4 (Dense) (None, 41, 128) 32896 ['dropout_2[0][0]']
dropout_3 (Dropout) (None, 41, 128) 0 ['dense_4[0][0]']
stochastic_depth_1 (Stocha (None, 41, 128) 0 ['layer_normalization_1[0][0]'
sticDepth) , 'dropout_3[0][0]']
layer_normalization_2 (Lay (None, 41, 128) 256 ['stochastic_depth_1[0][0]']
erNormalization)
multi_head_attention_1 (Mu (None, 41, 128) 66048 ['layer_normalization_2[0][0]'
ltiHeadAttention) , 'layer_normalization_2[0][0]
']
dense_5 (Dense) (None, 41, 128) 16512 ['layer_normalization_2[0][0]'
]
dropout_4 (Dropout) (None, 41, 128) 0 ['multi_head_attention_1[0][0]
']
tf.math.multiply_1 (TFOpLa (None, 41, 128) 0 ['dense_5[0][0]',
mbda) 'dropout_4[0][0]']
stochastic_depth_2 (Stocha (None, 41, 128) 0 ['layer_normalization_2[0][0]'
sticDepth) , 'tf.math.multiply_1[0][0]']
layer_normalization_3 (Lay (None, 41, 128) 256 ['stochastic_depth_2[0][0]']
erNormalization)
dense_6 (Dense) (None, 41, 256) 33024 ['layer_normalization_3[0][0]'
]
dropout_5 (Dropout) (None, 41, 256) 0 ['dense_6[0][0]']
dense_7 (Dense) (None, 41, 128) 32896 ['dropout_5[0][0]']
dropout_6 (Dropout) (None, 41, 128) 0 ['dense_7[0][0]']
stochastic_depth_3 (Stocha (None, 41, 128) 0 ['layer_normalization_3[0][0]'
sticDepth) , 'dropout_6[0][0]']
layer_normalization_4 (Lay (None, 41, 128) 256 ['stochastic_depth_3[0][0]']
erNormalization)
multi_head_attention_2 (Mu (None, 41, 128) 66048 ['layer_normalization_4[0][0]'
ltiHeadAttention) , 'layer_normalization_4[0][0]
']
dense_8 (Dense) (None, 41, 128) 16512 ['layer_normalization_4[0][0]'
]
dropout_7 (Dropout) (None, 41, 128) 0 ['multi_head_attention_2[0][0]
']
tf.math.multiply_2 (TFOpLa (None, 41, 128) 0 ['dense_8[0][0]',
mbda) 'dropout_7[0][0]']
stochastic_depth_4 (Stocha (None, 41, 128) 0 ['layer_normalization_4[0][0]'
sticDepth) , 'tf.math.multiply_2[0][0]']
layer_normalization_5 (Lay (None, 41, 128) 256 ['stochastic_depth_4[0][0]']
erNormalization)
dense_9 (Dense) (None, 41, 256) 33024 ['layer_normalization_5[0][0]'
]
dropout_8 (Dropout) (None, 41, 256) 0 ['dense_9[0][0]']
dense_10 (Dense) (None, 41, 128) 32896 ['dropout_8[0][0]']
dropout_9 (Dropout) (None, 41, 128) 0 ['dense_10[0][0]']
stochastic_depth_5 (Stocha (None, 41, 128) 0 ['layer_normalization_5[0][0]'
sticDepth) , 'dropout_9[0][0]']
layer_normalization_6 (Lay (None, 41, 128) 256 ['stochastic_depth_5[0][0]']
erNormalization)
multi_head_attention_3 (Mu (None, 41, 128) 66048 ['layer_normalization_6[0][0]'
ltiHeadAttention) , 'layer_normalization_6[0][0]
']
dense_11 (Dense) (None, 41, 128) 16512 ['layer_normalization_6[0][0]'
]
dropout_10 (Dropout) (None, 41, 128) 0 ['multi_head_attention_3[0][0]
']
tf.math.multiply_3 (TFOpLa (None, 41, 128) 0 ['dense_11[0][0]',
mbda) 'dropout_10[0][0]']
stochastic_depth_6 (Stocha (None, 41, 128) 0 ['layer_normalization_6[0][0]'
sticDepth) , 'tf.math.multiply_3[0][0]']
layer_normalization_7 (Lay (None, 41, 128) 256 ['stochastic_depth_6[0][0]']
erNormalization)
dense_12 (Dense) (None, 41, 256) 33024 ['layer_normalization_7[0][0]'
]
dropout_11 (Dropout) (None, 41, 256) 0 ['dense_12[0][0]']
dense_13 (Dense) (None, 41, 128) 32896 ['dropout_11[0][0]']
static (InputLayer) [(None, 113)] 0 []
dropout_12 (Dropout) (None, 41, 128) 0 ['dense_13[0][0]']
layer_normalization_9 (Lay (None, 113) 226 ['static[0][0]']
erNormalization)
stochastic_depth_7 (Stocha (None, 41, 128) 0 ['layer_normalization_7[0][0]'
sticDepth) , 'dropout_12[0][0]']
dense_14 (Dense) (None, 256) 29184 ['layer_normalization_9[0][0]'
]
layer_normalization_8 (Lay (None, 41, 128) 256 ['stochastic_depth_7[0][0]']
erNormalization)
dropout_13 (Dropout) (None, 256) 0 ['dense_14[0][0]']
stochastic_depth_8 (Stocha (None, 41, 128) 0 ['layer_normalization_8[0][0]'
sticDepth) , 'positional_encoding[0][0]']
dense_15 (Dense) (None, 128) 32896 ['dropout_13[0][0]']
multi_head_attention_4 (Mu (None, 41, 128) 131968 ['stochastic_depth_8[0][0]',
ltiHeadAttention) 'stochastic_depth_8[0][0]']
dropout_14 (Dropout) (None, 128) 0 ['dense_15[0][0]']
global_average_pooling1d ( (None, 128) 0 ['multi_head_attention_4[0][0]
GlobalAveragePooling1D) ']
global_average_pooling1d_1 (None, 128) 0 ['stochastic_depth_8[0][0]']
(GlobalAveragePooling1D)
global_max_pooling1d (Glob (None, 128) 0 ['stochastic_depth_8[0][0]']
alMaxPooling1D)
dense_16 (Dense) (None, 64) 8256 ['dropout_14[0][0]']
concatenate (Concatenate) (None, 384) 0 ['global_average_pooling1d[0][
0]',
'global_average_pooling1d_1[0
][0]',
'global_max_pooling1d[0][0]']
dropout_15 (Dropout) (None, 64) 0 ['dense_16[0][0]']
concatenate_1 (Concatenate (None, 448) 0 ['concatenate[0][0]',
) 'dropout_15[0][0]']
batch_normalization (Batch (None, 448) 1792 ['concatenate_1[0][0]']
Normalization)
dense_17 (Dense) (None, 256) 114944 ['batch_normalization[0][0]']
dropout_16 (Dropout) (None, 256) 0 ['dense_17[0][0]']
batch_normalization_1 (Bat (None, 256) 1024 ['dropout_16[0][0]']
chNormalization)
dense_18 (Dense) (None, 128) 32896 ['batch_normalization_1[0][0]'
]
dropout_17 (Dropout) (None, 128) 0 ['dense_18[0][0]']
batch_normalization_2 (Bat (None, 128) 512 ['dropout_17[0][0]']
chNormalization)
dense_19 (Dense) (None, 64) 8256 ['batch_normalization_2[0][0]'
]
dropout_18 (Dropout) (None, 64) 0 ['dense_19[0][0]']
dense_20 (Dense) (None, 5) 325 ['dropout_18[0][0]']
==================================================================================================
Total params: 972077 (3.71 MB)
Trainable params: 965165 (3.68 MB)
Non-trainable params: 6912 (27.00 KB)
__________________________________________________________________________________________________
Epoch 1/150
2024-12-06 11:43:25.651745: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7d7e70d1ce40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2024-12-06 11:43:25.651778: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA L40, Compute Capability 8.9 2024-12-06 11:43:25.659099: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable. 2024-12-06 11:43:25.722749: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:442] Loaded cuDNN version 8905 2024-12-06 11:43:25.861911: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
9954/9954 [==============================] - 481s 46ms/step - loss: 0.0460 - mae: 0.1872 - val_loss: 0.0145 - val_mae: 0.0865 - val_olive_prod_mae: 0.0964 - val_min_oil_prod_mae: 0.0935 - val_max_oil_prod_mae: 0.0936 - val_avg_oil_prod_mae: 0.0894 - val_total_water_need_mae: 0.0598 - lr: 1.0219e-05 Epoch 2/150 9954/9954 [==============================] - 473s 47ms/step - loss: 0.0273 - mae: 0.1505 - val_loss: 0.0143 - val_mae: 0.0863 - val_olive_prod_mae: 0.0963 - val_min_oil_prod_mae: 0.0931 - val_max_oil_prod_mae: 0.0929 - val_avg_oil_prod_mae: 0.0889 - val_total_water_need_mae: 0.0603 - lr: 1.0438e-07 Epoch 3/150 9954/9954 [==============================] - 477s 48ms/step - loss: 0.0273 - mae: 0.1506 - val_loss: 0.0143 - val_mae: 0.0861 - val_olive_prod_mae: 0.0964 - val_min_oil_prod_mae: 0.0929 - val_max_oil_prod_mae: 0.0927 - val_avg_oil_prod_mae: 0.0886 - val_total_water_need_mae: 0.0602 - lr: 1.0661e-09 Epoch 4/150 9954/9954 [==============================] - 508s 51ms/step - loss: 0.0272 - mae: 0.1505 - val_loss: 0.0143 - val_mae: 0.0867 - val_olive_prod_mae: 0.0967 - val_min_oil_prod_mae: 0.0932 - val_max_oil_prod_mae: 0.0930 - val_avg_oil_prod_mae: 0.0889 - val_total_water_need_mae: 0.0616 - lr: 1.0889e-11 Epoch 5/150 9954/9954 [==============================] - 431s 43ms/step - loss: 0.0273 - mae: 0.1507 - val_loss: 0.0143 - val_mae: 0.0865 - val_olive_prod_mae: 0.0965 - val_min_oil_prod_mae: 0.0931 - val_max_oil_prod_mae: 0.0929 - val_avg_oil_prod_mae: 0.0889 - val_total_water_need_mae: 0.0612 - lr: 1.1122e-13 Epoch 6/150 9954/9954 [==============================] - 438s 44ms/step - loss: 0.0273 - mae: 0.1506 - val_loss: 0.0143 - val_mae: 0.0863 - val_olive_prod_mae: 0.0965 - val_min_oil_prod_mae: 0.0931 - val_max_oil_prod_mae: 0.0929 - val_avg_oil_prod_mae: 0.0889 - val_total_water_need_mae: 0.0598 - lr: 1.1361e-15 Epoch 7/150 9954/9954 [==============================] - 413s 41ms/step - loss: 0.0273 - mae: 0.1506 - val_loss: 0.0143 - val_mae: 0.0868 - val_olive_prod_mae: 0.0967 - val_min_oil_prod_mae: 0.0932 - val_max_oil_prod_mae: 0.0930 - val_avg_oil_prod_mae: 0.0890 - val_total_water_need_mae: 0.0620 - lr: 1.1604e-17 Epoch 8/150 9954/9954 [==============================] - 433s 43ms/step - loss: 0.0272 - mae: 0.1505 - val_loss: 0.0143 - val_mae: 0.0865 - val_olive_prod_mae: 0.0966 - val_min_oil_prod_mae: 0.0931 - val_max_oil_prod_mae: 0.0929 - val_avg_oil_prod_mae: 0.0888 - val_total_water_need_mae: 0.0611 - lr: 1.1852e-19 Epoch 9/150 9954/9954 [==============================] - 413s 41ms/step - loss: 0.0273 - mae: 0.1507 - val_loss: 0.0143 - val_mae: 0.0865 - val_olive_prod_mae: 0.0967 - val_min_oil_prod_mae: 0.0933 - val_max_oil_prod_mae: 0.0930 - val_avg_oil_prod_mae: 0.0890 - val_total_water_need_mae: 0.0608 - lr: 1.2106e-21 Epoch 10/150 9954/9954 [==============================] - 430s 43ms/step - loss: 0.0273 - mae: 0.1508 - val_loss: 0.0143 - val_mae: 0.0864 - val_olive_prod_mae: 0.0965 - val_min_oil_prod_mae: 0.0931 - val_max_oil_prod_mae: 0.0929 - val_avg_oil_prod_mae: 0.0889 - val_total_water_need_mae: 0.0607 - lr: 1.2365e-23 Epoch 11/150 9954/9954 [==============================] - 438s 44ms/step - loss: 0.0273 - mae: 0.1507 - val_loss: 0.0143 - val_mae: 0.0863 - val_olive_prod_mae: 0.0965 - val_min_oil_prod_mae: 0.0930 - val_max_oil_prod_mae: 0.0928 - val_avg_oil_prod_mae: 0.0887 - val_total_water_need_mae: 0.0604 - lr: 1.2630e-25 Epoch 12/150 9954/9954 [==============================] - 430s 43ms/step - loss: 0.0273 - mae: 0.1505 - val_loss: 0.0144 - val_mae: 0.0866 - val_olive_prod_mae: 0.0968 - val_min_oil_prod_mae: 0.0933 - val_max_oil_prod_mae: 0.0932 - val_avg_oil_prod_mae: 0.0891 - val_total_water_need_mae: 0.0606 - lr: 1.2900e-27 Epoch 13/150 9954/9954 [==============================] - 425s 43ms/step - loss: 0.0273 - mae: 0.1507 - val_loss: 0.0144 - val_mae: 0.0868 - val_olive_prod_mae: 0.0966 - val_min_oil_prod_mae: 0.0932 - val_max_oil_prod_mae: 0.0930 - val_avg_oil_prod_mae: 0.0890 - val_total_water_need_mae: 0.0619 - lr: 1.3177e-29 Epoch 14/150 9954/9954 [==============================] - 409s 41ms/step - loss: 0.0272 - mae: 0.1504 - val_loss: 0.0144 - val_mae: 0.0865 - val_olive_prod_mae: 0.0967 - val_min_oil_prod_mae: 0.0933 - val_max_oil_prod_mae: 0.0932 - val_avg_oil_prod_mae: 0.0891 - val_total_water_need_mae: 0.0605 - lr: 1.3459e-31 Epoch 15/150 9954/9954 [==============================] - 439s 44ms/step - loss: 0.0273 - mae: 0.1509 - val_loss: 0.0143 - val_mae: 0.0863 - val_olive_prod_mae: 0.0964 - val_min_oil_prod_mae: 0.0929 - val_max_oil_prod_mae: 0.0926 - val_avg_oil_prod_mae: 0.0886 - val_total_water_need_mae: 0.0609 - lr: 1.3747e-33 Epoch 16/150 9954/9954 [==============================] - 421s 42ms/step - loss: 0.0273 - mae: 0.1508 - val_loss: 0.0143 - val_mae: 0.0862 - val_olive_prod_mae: 0.0963 - val_min_oil_prod_mae: 0.0930 - val_max_oil_prod_mae: 0.0928 - val_avg_oil_prod_mae: 0.0887 - val_total_water_need_mae: 0.0604 - lr: 1.4041e-35 Epoch 17/150 9954/9954 [==============================] - 429s 43ms/step - loss: 0.0272 - mae: 0.1505 - val_loss: 0.0143 - val_mae: 0.0863 - val_olive_prod_mae: 0.0966 - val_min_oil_prod_mae: 0.0931 - val_max_oil_prod_mae: 0.0929 - val_avg_oil_prod_mae: 0.0888 - val_total_water_need_mae: 0.0600 - lr: 1.4342e-37 Epoch 18/150 9954/9954 [==============================] - 414s 41ms/step - loss: 0.0272 - mae: 0.1505 - val_loss: 0.0144 - val_mae: 0.0865 - val_olive_prod_mae: 0.0967 - val_min_oil_prod_mae: 0.0933 - val_max_oil_prod_mae: 0.0931 - val_avg_oil_prod_mae: 0.0890 - val_total_water_need_mae: 0.0602 - lr: 0.0000e+00 Epoch 19/150 9954/9954 [==============================] - 441s 44ms/step - loss: 0.0272 - mae: 0.1506 - val_loss: 0.0143 - val_mae: 0.0864 - val_olive_prod_mae: 0.0965 - val_min_oil_prod_mae: 0.0930 - val_max_oil_prod_mae: 0.0928 - val_avg_oil_prod_mae: 0.0888 - val_total_water_need_mae: 0.0608 - lr: 0.0000e+00 Epoch 20/150 9954/9954 [==============================] - 440s 44ms/step - loss: 0.0272 - mae: 0.1505 - val_loss: 0.0143 - val_mae: 0.0862 - val_olive_prod_mae: 0.0963 - val_min_oil_prod_mae: 0.0930 - val_max_oil_prod_mae: 0.0929 - val_avg_oil_prod_mae: 0.0888 - val_total_water_need_mae: 0.0601 - lr: 0.0000e+00 Epoch 21/150 9954/9954 [==============================] - 448s 45ms/step - loss: 0.0273 - mae: 0.1508 - val_loss: 0.0143 - val_mae: 0.0862 - val_olive_prod_mae: 0.0964 - val_min_oil_prod_mae: 0.0935 - val_max_oil_prod_mae: 0.0936 - val_avg_oil_prod_mae: 0.0894 - val_total_water_need_mae: 0.0598 - lr: 0.0000e+00 Modello salvato in: 2024-12-06_10-36_final_model.keras
In [13]:
percentage_errors, absolute_errors = calculate_real_error(model, val_data, val_targets, scaler_y)24500/24500 [==============================] - 102s 4ms/step Errori per target: -------------------------------------------------- olive_prod: MAE assoluto: 1585.45 Errore percentuale medio: 6.91% Precisione: 93.09% -------------------------------------------------- min_oil_prod: MAE assoluto: 319.12 Errore percentuale medio: 6.61% Precisione: 93.39% -------------------------------------------------- max_oil_prod: MAE assoluto: 387.31 Errore percentuale medio: 6.74% Precisione: 93.26% -------------------------------------------------- avg_oil_prod: MAE assoluto: 337.11 Errore percentuale medio: 6.46% Precisione: 93.54% -------------------------------------------------- total_water_need: MAE assoluto: 1775.48 Errore percentuale medio: 4.24% Precisione: 95.76% --------------------------------------------------
In [14]:
def evaluate_model_performance(model, data, targets, set_name=""):
"""
Valuta le performance del modello su un set di dati specifico.
"""
predictions = model.predict(data, verbose=0)
target_names = ['olive_prod', 'min_oil_prod', 'max_oil_prod', 'avg_oil_prod', 'total_water_need']
metrics = {}
for i, name in enumerate(target_names):
mae = np.mean(np.abs(targets[:, i] - predictions[:, i]))
mse = np.mean(np.square(targets[:, i] - predictions[:, i]))
rmse = np.sqrt(mse)
mape = np.mean(np.abs((targets[:, i] - predictions[:, i]) / (targets[:, i] + 1e-7))) * 100
metrics[f"{name}_mae"] = mae
metrics[f"{name}_rmse"] = rmse
metrics[f"{name}_mape"] = mape
if set_name:
print(f"\nPerformance sul set {set_name}:")
for metric, value in metrics.items():
print(f"{metric}: {value:.4f}")
return metrics
def retrain_model(base_model, train_data, train_targets,
val_data, val_targets,
test_data, test_targets,
epochs=50, batch_size=128):
"""
Implementa il retraining del modello con i dati combinati.
"""
print("Valutazione performance iniziali del modello...")
initial_metrics = {
'train': evaluate_model_performance(base_model, train_data, train_targets, "training"),
'val': evaluate_model_performance(base_model, val_data, val_targets, "validazione"),
'test': evaluate_model_performance(base_model, test_data, test_targets, "test")
}
# Combina i dati per il retraining
combined_data = {
'temporal': np.concatenate([train_data['temporal'], val_data['temporal'], test_data['temporal']]),
'static': np.concatenate([train_data['static'], val_data['static'], test_data['static']])
}
combined_targets = np.concatenate([train_targets, val_targets, test_targets])
# Crea una nuova suddivisione per la validazione
indices = np.arange(len(combined_targets))
np.random.shuffle(indices)
split_idx = int(len(indices) * 0.9)
train_idx, val_idx = indices[:split_idx], indices[split_idx:]
# Prepara i dati per il retraining
retrain_data = {k: v[train_idx] for k, v in combined_data.items()}
retrain_targets = combined_targets[train_idx]
retrain_val_data = {k: v[val_idx] for k, v in combined_data.items()}
retrain_val_targets = combined_targets[val_idx]
# Configura callbacks
callbacks = [
tf.keras.callbacks.EarlyStopping(
monitor='val_loss',
patience=10,
restore_best_weights=True,
min_delta=0.0001
),
tf.keras.callbacks.ReduceLROnPlateau(
monitor='val_loss',
factor=0.2,
patience=5,
min_lr=1e-6,
verbose=1
),
tf.keras.callbacks.ModelCheckpoint(
filepath=f'{execute_name}_retrained_best_oil_model.h5',
monitor='val_loss',
save_best_only=True,
mode='min',
save_weights_only=True
)
]
# Imposta learning rate per il fine-tuning
optimizer = tf.keras.optimizers.AdamW(
learning_rate=tf.keras.optimizers.schedules.ExponentialDecay(
initial_learning_rate=1e-4,
decay_steps=1000,
decay_rate=0.9
),
weight_decay=0.01
)
# Ricompila il modello con il nuovo optimizer
base_model.compile(
optimizer=optimizer,
loss=tf.keras.losses.Huber(),
metrics=['mae']
)
print("\nAvvio retraining...")
history = base_model.fit(
retrain_data,
retrain_targets,
validation_data=(retrain_val_data, retrain_val_targets),
epochs=epochs,
batch_size=batch_size,
callbacks=callbacks,
verbose=1
)
print("\nValutazione performance finali...")
final_metrics = {
'train': evaluate_model_performance(base_model, train_data, train_targets, "training"),
'val': evaluate_model_performance(base_model, val_data, val_targets, "validazione"),
'test': evaluate_model_performance(base_model, test_data, test_targets, "test")
}
# Salva il modello finale
save_path = f'{execute_name}_retrained_model.keras'
os.makedirs(f'{execute_name}_retrained/weights', exist_ok=True)
base_model.save_weights(f'{execute_name}_retrained/weights')
base_model.save(save_path, save_format='keras')
print(f"\nModello riaddestrato salvato in: {save_path}")
# Report miglioramenti
print("\nMiglioramenti delle performance:")
for dataset in ['train', 'val', 'test']:
print(f"\nSet {dataset}:")
for metric in initial_metrics[dataset].keys():
initial = initial_metrics[dataset][metric]
final = final_metrics[dataset][metric]
improvement = ((initial - final) / initial) * 100
print(f"{metric}: {improvement:.2f}% di miglioramento")
return base_model, history, final_metrics
def start_retraining(model_path, train_data, train_targets,
val_data, val_targets,
test_data, test_targets,
epochs=50, batch_size=128):
"""
Avvia il processo di retraining in modo sicuro.
"""
try:
print("Caricamento del modello...")
base_model = tf.keras.models.load_model(model_path, compile=False)
print("Modello caricato con successo!")
return retrain_model(
base_model=base_model,
train_data=train_data,
train_targets=train_targets,
val_data=val_data,
val_targets=val_targets,
test_data=test_data,
test_targets=test_targets,
epochs=epochs,
batch_size=batch_size
)
except Exception as e:
print(f"Errore durante il retraining: {str(e)}")
raiseIn [15]:
model_path = f'{execute_name}_final_model.keras'
retrained_model, retrain_history, final_metrics = start_retraining(
model_path=model_path,
train_data=train_data,
train_targets=train_targets,
val_data=val_data,
val_targets=val_targets,
test_data=test_data,
test_targets=test_targets,
epochs=50,
batch_size=128
)Caricamento del modello... Modello caricato con successo! Valutazione performance iniziali del modello... Performance sul set training: olive_prod_mae: 0.0963 olive_prod_rmse: 0.1300 olive_prod_mape: 77.2491 min_oil_prod_mae: 0.0936 min_oil_prod_rmse: 0.1312 min_oil_prod_mape: 91.4612 max_oil_prod_mae: 0.0936 max_oil_prod_rmse: 0.1304 max_oil_prod_mape: 88.9396 avg_oil_prod_mae: 0.0895 avg_oil_prod_rmse: 0.1238 avg_oil_prod_mape: 89.5317 total_water_need_mae: 0.0598 total_water_need_rmse: 0.0808 total_water_need_mape: 44.4531 Performance sul set validazione: olive_prod_mae: 0.0964 olive_prod_rmse: 0.1301 olive_prod_mape: 133.2427 min_oil_prod_mae: 0.0935 min_oil_prod_rmse: 0.1310 min_oil_prod_mape: 120.7693 max_oil_prod_mae: 0.0936 max_oil_prod_rmse: 0.1304 max_oil_prod_mape: 86.2224 avg_oil_prod_mae: 0.0894 avg_oil_prod_rmse: 0.1237 avg_oil_prod_mape: 83.8138 total_water_need_mae: 0.0598 total_water_need_rmse: 0.0809 total_water_need_mape: 53.9347 Performance sul set test: olive_prod_mae: 0.0962 olive_prod_rmse: 0.1298 olive_prod_mape: 77.9806 min_oil_prod_mae: 0.0935 min_oil_prod_rmse: 0.1312 min_oil_prod_mape: 95.5886 max_oil_prod_mae: 0.0934 max_oil_prod_rmse: 0.1301 max_oil_prod_mape: 76.3217 avg_oil_prod_mae: 0.0893 avg_oil_prod_rmse: 0.1237 avg_oil_prod_mape: 111.2211 total_water_need_mae: 0.0596 total_water_need_rmse: 0.0806 total_water_need_mape: 38.1699 Avvio retraining... Epoch 1/50 27563/27563 [==============================] - 851s 30ms/step - loss: 0.0261 - mae: 0.1520 - val_loss: 0.0118 - val_mae: 0.0804 - lr: 5.4806e-06 Epoch 2/50 27563/27563 [==============================] - 852s 31ms/step - loss: 0.0245 - mae: 0.1478 - val_loss: 0.0117 - val_mae: 0.0803 - lr: 3.0034e-07 Epoch 3/50 27563/27563 [==============================] - 836s 30ms/step - loss: 0.0244 - mae: 0.1476 - val_loss: 0.0117 - val_mae: 0.0807 - lr: 1.6459e-08 Epoch 4/50 27563/27563 [==============================] - 863s 31ms/step - loss: 0.0244 - mae: 0.1476 - val_loss: 0.0118 - val_mae: 0.0807 - lr: 9.0196e-10 Epoch 5/50 27563/27563 [==============================] - 854s 31ms/step - loss: 0.0243 - mae: 0.1474 - val_loss: 0.0119 - val_mae: 0.0812 - lr: 4.9428e-11 Epoch 6/50 27563/27563 [==============================] - 869s 32ms/step - loss: 0.0244 - mae: 0.1475 - val_loss: 0.0118 - val_mae: 0.0807 - lr: 2.7087e-12 Epoch 7/50 27563/27563 [==============================] - 867s 31ms/step - loss: 0.0244 - mae: 0.1475 - val_loss: 0.0118 - val_mae: 0.0806 - lr: 1.4844e-13 Epoch 8/50 27563/27563 [==============================] - 899s 33ms/step - loss: 0.0244 - mae: 0.1475 - val_loss: 0.0117 - val_mae: 0.0803 - lr: 8.1345e-15 Epoch 9/50 27563/27563 [==============================] - 966s 35ms/step - loss: 0.0244 - mae: 0.1475 - val_loss: 0.0117 - val_mae: 0.0804 - lr: 4.4578e-16 Epoch 10/50 27563/27563 [==============================] - 930s 34ms/step - loss: 0.0244 - mae: 0.1474 - val_loss: 0.0118 - val_mae: 0.0807 - lr: 2.4429e-17 Epoch 11/50 27563/27563 [==============================] - 921s 33ms/step - loss: 0.0244 - mae: 0.1475 - val_loss: 0.0118 - val_mae: 0.0809 - lr: 1.3387e-18 Valutazione performance finali... Performance sul set training: olive_prod_mae: 0.0901 olive_prod_rmse: 0.1222 olive_prod_mape: 75.7735 min_oil_prod_mae: 0.0886 min_oil_prod_rmse: 0.1245 min_oil_prod_mape: 91.0646 max_oil_prod_mae: 0.0888 max_oil_prod_rmse: 0.1243 max_oil_prod_mape: 89.5375 avg_oil_prod_mae: 0.0845 avg_oil_prod_rmse: 0.1171 avg_oil_prod_mape: 86.3355 total_water_need_mae: 0.0495 total_water_need_rmse: 0.0678 total_water_need_mape: 41.0436 Performance sul set validazione: olive_prod_mae: 0.0901 olive_prod_rmse: 0.1222 olive_prod_mape: 138.3196 min_oil_prod_mae: 0.0885 min_oil_prod_rmse: 0.1243 min_oil_prod_mape: 126.9523 max_oil_prod_mae: 0.0888 max_oil_prod_rmse: 0.1243 max_oil_prod_mape: 82.7593 avg_oil_prod_mae: 0.0843 avg_oil_prod_rmse: 0.1169 avg_oil_prod_mape: 84.3605 total_water_need_mae: 0.0495 total_water_need_rmse: 0.0679 total_water_need_mape: 48.6941 Performance sul set test: olive_prod_mae: 0.0899 olive_prod_rmse: 0.1219 olive_prod_mape: 77.0356 min_oil_prod_mae: 0.0886 min_oil_prod_rmse: 0.1243 min_oil_prod_mape: 96.3498 max_oil_prod_mae: 0.0885 max_oil_prod_rmse: 0.1238 max_oil_prod_mape: 76.4509 avg_oil_prod_mae: 0.0843 avg_oil_prod_rmse: 0.1167 avg_oil_prod_mape: 87.8912 total_water_need_mae: 0.0494 total_water_need_rmse: 0.0677 total_water_need_mape: 30.6997 Modello riaddestrato salvato in: 2024-12-06_10-36_retrained_model.keras Miglioramenti delle performance: Set train: olive_prod_mae: 6.48% di miglioramento olive_prod_rmse: 6.00% di miglioramento olive_prod_mape: 1.91% di miglioramento min_oil_prod_mae: 5.29% di miglioramento min_oil_prod_rmse: 5.12% di miglioramento min_oil_prod_mape: 0.43% di miglioramento max_oil_prod_mae: 5.11% di miglioramento max_oil_prod_rmse: 4.70% di miglioramento max_oil_prod_mape: -0.67% di miglioramento avg_oil_prod_mae: 5.58% di miglioramento avg_oil_prod_rmse: 5.45% di miglioramento avg_oil_prod_mape: 3.57% di miglioramento total_water_need_mae: 17.16% di miglioramento total_water_need_rmse: 15.99% di miglioramento total_water_need_mape: 7.67% di miglioramento Set val: olive_prod_mae: 6.51% di miglioramento olive_prod_rmse: 6.04% di miglioramento olive_prod_mape: -3.81% di miglioramento min_oil_prod_mae: 5.33% di miglioramento min_oil_prod_rmse: 5.16% di miglioramento min_oil_prod_mape: -5.12% di miglioramento max_oil_prod_mae: 5.13% di miglioramento max_oil_prod_rmse: 4.70% di miglioramento max_oil_prod_mape: 4.02% di miglioramento avg_oil_prod_mae: 5.62% di miglioramento avg_oil_prod_rmse: 5.48% di miglioramento avg_oil_prod_mape: -0.65% di miglioramento total_water_need_mae: 17.23% di miglioramento total_water_need_rmse: 16.08% di miglioramento total_water_need_mape: 9.72% di miglioramento Set test: olive_prod_mae: 6.52% di miglioramento olive_prod_rmse: 6.09% di miglioramento olive_prod_mape: 1.21% di miglioramento min_oil_prod_mae: 5.32% di miglioramento min_oil_prod_rmse: 5.22% di miglioramento min_oil_prod_mape: -0.80% di miglioramento max_oil_prod_mae: 5.22% di miglioramento max_oil_prod_rmse: 4.83% di miglioramento max_oil_prod_mape: -0.17% di miglioramento avg_oil_prod_mae: 5.64% di miglioramento avg_oil_prod_rmse: 5.59% di miglioramento avg_oil_prod_mape: 20.98% di miglioramento total_water_need_mae: 17.22% di miglioramento total_water_need_rmse: 16.03% di miglioramento total_water_need_mape: 19.57% di miglioramento
In [16]:
percentage_errors, absolute_errors = calculate_real_error(retrained_model, val_data, val_targets, scaler_y)24500/24500 [==============================] - 137s 6ms/step Errori per target: -------------------------------------------------- olive_prod: MAE assoluto: 1482.22 Errore percentuale medio: 5.77% Precisione: 94.23% -------------------------------------------------- min_oil_prod: MAE assoluto: 302.12 Errore percentuale medio: 5.68% Precisione: 94.32% -------------------------------------------------- max_oil_prod: MAE assoluto: 367.45 Errore percentuale medio: 5.78% Precisione: 94.22% -------------------------------------------------- avg_oil_prod: MAE assoluto: 318.15 Errore percentuale medio: 5.49% Precisione: 94.51% -------------------------------------------------- total_water_need: MAE assoluto: 1469.51 Errore percentuale medio: 3.31% Precisione: 96.69% --------------------------------------------------
In [34]:
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
from typing import List, Dict, Tuple, Union
def analyze_feature_importance(model: tf.keras.Model,
test_data: dict,
feature_names: List[str]) -> Dict[str, float]:
"""
Analizza l'importanza delle feature usando perturbazione.
Args:
model: Modello TensorFlow addestrato
test_data: Dizionario con chiavi 'temporal' e 'static' contenenti i dati
feature_names: Lista dei nomi delle feature
Returns:
dict: Dizionario con l'importanza relativa di ogni feature
"""
# Estrai i dati temporali e statici
temporal_data = test_data['temporal']
static_data = test_data['static']
# Ottieni la predizione base
base_prediction = model.predict(test_data)
feature_importance = {}
# Per ogni feature temporale
for i, feature in enumerate(feature_names):
if feature in ['temp_mean', 'precip_sum', 'solar_energy_sum']:
# Crea copia perturbata dei dati
perturbed_data = {
'temporal': temporal_data.copy(),
'static': static_data.copy()
}
# Trova l'indice della feature temporale
temp_idx = ['temp_mean', 'precip_sum', 'solar_energy_sum'].index(feature)
# Crea rumore per la feature temporale
feature_values = temporal_data[..., temp_idx]
noise = np.random.normal(0, np.std(feature_values) * 0.1,
size=feature_values.shape)
# Applica il rumore alla feature temporale
perturbed_temporal = perturbed_data['temporal'].copy()
perturbed_temporal[..., temp_idx] = feature_values + noise
perturbed_data['temporal'] = perturbed_temporal
else: # Feature statiche
# Crea copia perturbata dei dati
perturbed_data = {
'temporal': temporal_data.copy(),
'static': static_data.copy()
}
# Trova l'indice della feature statica
static_idx = ['ha'].index(feature)
# Crea rumore per la feature statica
feature_values = static_data[..., static_idx]
noise = np.random.normal(0, np.std(feature_values) * 0.1,
size=feature_values.shape)
# Applica il rumore alla feature statica
perturbed_static = perturbed_data['static'].copy()
perturbed_static[..., static_idx] = feature_values + noise
perturbed_data['static'] = perturbed_static
# Calcola nuova predizione
perturbed_prediction = model.predict(perturbed_data)
# Calcola impatto della perturbazione
impact = np.mean(np.abs(perturbed_prediction - base_prediction))
feature_importance[feature] = float(impact)
# Normalizza le importanze
total_importance = sum(feature_importance.values())
feature_importance = {k: v/total_importance
for k, v in feature_importance.items()}
return feature_importance
class ProbabilityFunctions:
@staticmethod
def calculate_statistics(data: Union[np.ndarray, tf.Tensor]) -> Dict[str, float]:
"""
Calcola statistiche di base usando TensorFlow.
Args:
data: Tensor o array dei dati
Returns:
dict: Dizionario con le statistiche
"""
if not isinstance(data, tf.Tensor):
data = tf.convert_to_tensor(data, dtype=tf.float32)
mean = tf.reduce_mean(data)
# Calcola varianza manualmente
squared_deviations = tf.square(data - mean)
variance = tf.reduce_mean(squared_deviations)
std = tf.sqrt(variance)
# Ordina il tensor per il calcolo della mediana
sorted_data = tf.sort(data)
size = tf.size(data)
mid_index = size // 2
median = sorted_data[mid_index]
return {
'mean': mean.numpy(),
'variance': variance.numpy(),
'std': std.numpy(),
'min': tf.reduce_min(data).numpy(),
'max': tf.reduce_max(data).numpy(),
'median': median.numpy()
}
@staticmethod
def calculate_pmf(data: np.ndarray, bins: int = 50) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Calcola la Probability Mass Function (PMF) dei dati.
Args:
data: Array di dati
bins: Numero di bin per l'istogramma
Returns:
tuple: (bin_centers, pmf, bin_edges)
"""
# Calcola l'istogramma
hist, bin_edges = np.histogram(data, bins=bins, density=True)
# Calcola i centri dei bin
bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2
# Normalizza per ottenere la PMF
pmf = hist / np.sum(hist)
return bin_centers, pmf, bin_edges
@staticmethod
def calculate_cmf(pmf: np.ndarray) -> np.ndarray:
"""
Calcola la Cumulative Mass Function (CMF) dalla PMF.
Args:
pmf: Probability Mass Function
Returns:
array: Cumulative Mass Function
"""
return np.cumsum(pmf)
def plot_distributions(self, data: np.ndarray,
bins: int = 50,
title: str = "Distribuzione") -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
"""
Calcola e visualizza PMF e CMF delle distribuzioni.
Args:
data: Array di dati da analizzare
bins: Numero di bin per l'istogramma
title: Titolo del grafico
Returns:
tuple: (bin_centers, pmf, cmf)
"""
# Calcola PMF e CMF
bin_centers, pmf, bin_edges = self.calculate_pmf(data, bins)
cmf = self.calculate_cmf(pmf)
# Crea il plot
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 8))
# Plot PMF
width = np.diff(bin_edges)
ax1.bar(bin_centers, pmf, width=width, alpha=0.5, label='PMF')
ax1.set_title('Probability Mass Function')
ax1.set_ylabel('Probability')
ax1.grid(True, alpha=0.3)
ax1.legend()
# Plot CMF
ax2.plot(bin_centers, cmf, 'r-', label='CMF')
ax2.set_title('Cumulative Mass Function')
ax2.set_xlabel('Value')
ax2.set_ylabel('Cumulative Probability')
ax2.grid(True, alpha=0.3)
ax2.legend()
# Imposta il titolo generale
fig.suptitle(title, y=1.02)
plt.tight_layout()
plt.show()
return bin_centers, pmf, cmf
def analyze_model_predictions(model: tf.keras.Model,
test_data: np.ndarray,
test_targets: np.ndarray,
scaler_y) -> None:
"""
Analizza le distribuzioni di probabilità delle predizioni del modello.
Args:
model: Modello TensorFlow addestrato
test_data: Dati di test
test_targets: Target di test
scaler_y: Scaler usato per denormalizzare i target
"""
# Ottieni le predizioni
predictions = model.predict(test_data)
# Denormalizza predizioni e target
predictions_real = scaler_y.inverse_transform(predictions)
targets_real = scaler_y.inverse_transform(test_targets)
# Inizializza la classe per l'analisi delle probabilità
prob = ProbabilityFunctions()
# Analizza ogni target
target_names = ['olive_prod', 'min_oil_prod', 'max_oil_prod',
'avg_oil_prod', 'total_water_need']
for i, target in enumerate(target_names):
print(f"\nAnalisi per {target}")
print("-" * 50)
# Calcola errori
errors = predictions_real[:, i] - targets_real[:, i]
# Calcola statistiche degli errori
error_stats = prob.calculate_statistics(errors)
print("\nStatistiche degli Errori:")
for key, value in error_stats.items():
print(f"{key}: {value:.3f}")
# Visualizza le distribuzioni degli errori
bin_centers, pmf, cmf = prob.plot_distributions(
errors,
bins=50,
title=f"Distribuzione degli Errori - {target}"
)
# Calcola intervalli di confidenza
confidence_levels = [0.68, 0.95, 0.99] # 1σ, 2σ, 3σ
for level in confidence_levels:
lower_idx = np.searchsorted(cmf, (1 - level) / 2)
upper_idx = np.searchsorted(cmf, (1 + level) / 2)
print(f"\nIntervallo di Confidenza {level*100}%:")
print(f"Range: [{bin_centers[lower_idx]:.2f}, {bin_centers[upper_idx]:.2f}]")
def run_comprehensive_analysis(retrained_model, test_data, test_targets, scaler_y):
"""
Esegue un'analisi completa del modello includendo errori,
importanza delle feature e distribuzioni.
"""
print("=== ANALISI COMPLETA DEL MODELLO ===")
# 1. Analisi degli errori
print("\n1. ANALISI DEGLI ERRORI")
print("-" * 50)
analyze_model_predictions(retrained_model, test_data, test_targets, scaler_y)
# 2. Analisi dell'importanza delle feature
print("\n2. IMPORTANZA DELLE FEATURE")
print("-" * 50)
# Definisci i nomi delle feature
temporal_features = ['temp_mean', 'precip_sum', 'solar_energy_sum']
static_features = ['ha']
all_features = temporal_features + static_features
importance = analyze_feature_importance(retrained_model, test_data, all_features)
print("\nImportanza relativa delle feature:")
for feature, imp in sorted(importance.items(), key=lambda x: x[1], reverse=True):
print(f"{feature}: {imp:.4f}")
# 3. Analisi distribuzionale
print("\n3. ANALISI DISTRIBUZIONALE")
print("-" * 50)
prob = ProbabilityFunctions()
predictions = retrained_model.predict(test_data)
predictions_real = scaler_y.inverse_transform(predictions)
targets_real = scaler_y.inverse_transform(test_targets)
target_names = ['olive_prod', 'min_oil_prod', 'max_oil_prod',
'avg_oil_prod', 'total_water_need']
for i, target in enumerate(target_names):
print(f"\nAnalisi distribuzionale per {target}")
# Statistiche
stats_pred = prob.calculate_statistics(predictions_real[:, i])
stats_true = prob.calculate_statistics(targets_real[:, i])
print("\nStatistiche Predizioni:")
for key, value in stats_pred.items():
print(f"{key}: {value:.3f}")
print("\nStatistiche Target Reali:")
for key, value in stats_true.items():
print(f"{key}: {value:.3f}")
# Visualizza distribuzioni
prob.plot_distributions(predictions_real[:, i], bins=50,
title=f"Distribuzione Predizioni - {target}")
prob.plot_distributions(targets_real[:, i], bins=50,
title=f"Distribuzione Target Reali - {target}")
def analyze_model_predictions(model, test_data, test_targets, scaler_y):
"""
Analizza le distribuzioni di probabilità delle predizioni del modello.
Args:
model: Modello TensorFlow addestrato
test_data: Dati di test
test_targets: Target di test
scaler_y: Scaler usato per denormalizzare i target
"""
# Ottieni le predizioni
predictions = model.predict(test_data)
# Denormalizza predizioni e target
predictions_real = scaler_y.inverse_transform(predictions)
targets_real = scaler_y.inverse_transform(test_targets)
# Inizializza la classe per l'analisi delle probabilità
prob = ProbabilityFunctions()
# Analizza ogni target
target_names = ['olive_prod', 'min_oil_prod', 'max_oil_prod',
'avg_oil_prod', 'total_water_need']
for i, target in enumerate(target_names):
print(f"\nAnalisi per {target}")
print("-" * 50)
# Calcola errori
errors = predictions_real[:, i] - targets_real[:, i]
# Calcola statistiche degli errori
error_stats = prob.calculate_statistics(errors)
print("\nStatistiche degli Errori:")
for key, value in error_stats.items():
print(f"{key}: {value:.3f}")
# Visualizza le distribuzioni degli errori
bin_centers, pmf, cmf = prob.plot_distributions(
errors,
bins=50,
title=f"Distribuzione degli Errori - {target}"
)
# Calcola intervalli di confidenza
confidence_levels = [0.80,0.85, 0.90, 0.95, 0.99] # 1σ, 2σ, 3σ
for level in confidence_levels:
lower_idx = np.searchsorted(cmf, (1 - level) / 2)
upper_idx = np.searchsorted(cmf, (1 + level) / 2)
print(f"\nIntervallo di Confidenza {level*100}%:")
print(f"Range: [{bin_centers[lower_idx]:.2f}, {bin_centers[upper_idx]:.2f}]")
class ProbabilityFunctions:
@staticmethod
def calculate_statistics(data):
"""
Calcola statistiche di base usando TensorFlow.
Args:
data: Tensor dei dati
Returns:
dict: Dizionario con le statistiche
"""
if not isinstance(data, tf.Tensor):
data = tf.convert_to_tensor(data, dtype=tf.float32)
mean = tf.reduce_mean(data)
# Calculate variance manually
squared_deviations = tf.square(data - mean)
variance = tf.reduce_mean(squared_deviations)
std = tf.sqrt(variance)
# Sort the tensor for median calculation
sorted_data = tf.sort(data)
size = tf.size(data)
mid_index = size // 2
median = sorted_data[mid_index]
return {
'mean': mean.numpy(),
'variance': variance.numpy(),
'std': std.numpy(),
'min': tf.reduce_min(data).numpy(),
'max': tf.reduce_max(data).numpy(),
'median': median.numpy()
}
@staticmethod
def calculate_pmf(data, bins=50):
"""
Calcola la Probability Mass Function (PMF) dei dati.
Args:
data: Array di dati
bins: Numero di bin per l'istogramma
Returns:
tuple: (bin_centers, pmf, bin_edges)
"""
# Calcola l'istogramma
hist, bin_edges = np.histogram(data, bins=bins, density=True)
# Calcola i centri dei bin
bin_centers = (bin_edges[:-1] + bin_edges[1:]) / 2
# Normalizza per ottenere la PMF
pmf = hist / np.sum(hist)
return bin_centers, pmf, bin_edges
@staticmethod
def calculate_cmf(pmf):
"""
Calcola la Cumulative Mass Function (CMF) dalla PMF.
Args:
pmf: Probability Mass Function
Returns:
array: Cumulative Mass Function
"""
return np.cumsum(pmf)
def plot_distributions(self, data, bins=50, title="Distribuzione"):
"""
Calcola e visualizza PMF e CMF delle distribuzioni.
Args:
data: Array di dati da analizzare
bins: Numero di bin per l'istogramma
title: Titolo del grafico
Returns:
tuple: (bin_centers, pmf, cmf)
"""
# Calcola PMF e CMF
bin_centers, pmf, bin_edges = self.calculate_pmf(data, bins)
cmf = self.calculate_cmf(pmf)
# Crea il plot
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(10, 8))
# Plot PMF
width = np.diff(bin_edges)
ax1.bar(bin_centers, pmf, width=width, alpha=0.5, label='PMF')
ax1.set_title('Probability Mass Function')
ax1.set_ylabel('Probability')
ax1.grid(True, alpha=0.3)
ax1.legend()
# Plot CMF
ax2.plot(bin_centers, cmf, 'r-', label='CMF')
ax2.set_title('Cumulative Mass Function')
ax2.set_xlabel('Value')
ax2.set_ylabel('Cumulative Probability')
ax2.grid(True, alpha=0.3)
ax2.legend()
# Set overall title
fig.suptitle(title, y=1.02)
plt.tight_layout()
plt.show()
return bin_centers, pmf, cmfIn [ ]:
run_comprehensive_analysis(retrained_model, test_data, test_targets, scaler_y)=== ANALISI COMPLETA DEL MODELLO === 1. ANALISI DEGLI ERRORI -------------------------------------------------- 18375/18375 [==============================] - 78s 4ms/step Analisi per olive_prod -------------------------------------------------- Statistiche degli Errori: mean: -71.944 variance: 4009595.000 std: 2002.397 min: -18637.889 max: 12871.579 median: 48.672
Intervallo di Confidenza 68.0%: Range: [-1937.87, 1843.27] Intervallo di Confidenza 95.0%: Range: [-4458.63, 3733.83] Intervallo di Confidenza 99.0%: Range: [-6979.39, 5624.40] Analisi per min_oil_prod -------------------------------------------------- Statistiche degli Errori: mean: -32.785 variance: 179026.016 std: 423.115 min: -4439.664 max: 3453.714 median: -12.655
Intervallo di Confidenza 68.0%: Range: [-414.04, 375.30] Intervallo di Confidenza 95.0%: Range: [-1045.51, 848.90] Intervallo di Confidenza 99.0%: Range: [-1519.11, 1164.63] Analisi per max_oil_prod -------------------------------------------------- Statistiche degli Errori: mean: -34.971 variance: 261409.344 std: 511.282 min: -5732.709 max: 4274.197 median: -11.391
Intervallo di Confidenza 68.0%: Range: [-429.05, 371.50] Intervallo di Confidenza 95.0%: Range: [-1229.60, 971.92] Intervallo di Confidenza 99.0%: Range: [-1830.02, 1572.33] Analisi per avg_oil_prod -------------------------------------------------- Statistiche degli Errori: mean: -33.549 variance: 192810.531 std: 439.102 min: -4876.229 max: 3813.953 median: -13.710
Intervallo di Confidenza 68.0%: Range: [-444.24, 250.98] Intervallo di Confidenza 95.0%: Range: [-965.65, 772.39] Intervallo di Confidenza 99.0%: Range: [-1487.06, 1293.80] Analisi per total_water_need -------------------------------------------------- Statistiche degli Errori: mean: -216.226 variance: 3987062.750 std: 1996.763 min: -22812.350 max: 13374.520 median: -119.823
Intervallo di Confidenza 68.0%: Range: [-2185.83, 1432.85] Intervallo di Confidenza 95.0%: Range: [-4357.05, 3604.06] Intervallo di Confidenza 99.0%: Range: [-7252.00, 5051.54] 2. IMPORTANZA DELLE FEATURE -------------------------------------------------- 18375/18375 [==============================] - 79s 4ms/step 18375/18375 [==============================] - 80s 4ms/step 18375/18375 [==============================] - 99s 5ms/step 18375/18375 [==============================] - 96s 5ms/step 13976/18375 [=====================>........] - ETA: 21s